AI Biases (V): Introduction of risks in the AI system lifecycle (part 1)

In the previous installment of this series, after introducing the concept of bias and its taxonomy—as well as analyzing a number of related or adjacent concepts—we began to address the management of risks associated with bias, starting with its potential impacts.

However, when discussing risk management, the first step in any risk management strategy is to identify the risks. In the case of bias, this means identifying the sources of bias, which may be introduced at different stages of an AI system's lifecycle. Therefore, it is crucial to understand how bias enters at various phases of that lifecycle.

As stated by the National Institute of Standards and Technology (NIST), organizations that design and develop AI technologies use the AI lifecycle to monitor their processes and ensure that the technology performs as intended. However, they do not necessarily to identify or manage potential risks and harms.

Organizations developing AI technologies use the AI lifecycle to ensure functionality, but not necessarily to identify or manage risks and harms.

Furthermore, in current approaches to bias, classification tends to be by type (statistical, cognitive, etc.) or by use case and sector (hiring, healthcare, etc.). While this categorization is helpful, it may fail to offer the broader perspective needed to manage bias effectively as a context-specific phenomenon.

For this reason, the NIST report proposes an approach to managing and reducing the impact of harmful biases across all contexts, by considering critical points within the stages of an AI system’s lifecycle.

Phases of the AI system lifecycle

Identifying bias sources is the first step in any bias mitigation strategy. As noted, these sources may be introduced at various points throughout an AI system’s development. Therefore, we must first understand what those phases are—and, in turn, relate them to the different operators involved.

The development of an AI system comprises three core phases, as defined by NIST: pre-design, design and development, and testing and evaluation. Meanwhile, ISO 22989 outlines a more granular sequence: initiation, design and development, verification and validation, deployment/implementation, operation and monitoring, continuous validation, re-evaluation, and retirement.

■ ISO 22989 includes additional phases but does not imply that NIST overlooks their importance. Rather, they may be implicitly addressed or incorporated into a broader operational framework.

Phase 1. Initiation: Pre-design or definition scope

AI systems begin in the pre-design phase, which is foundational for establishing the conditions that will determine the system’s effectiveness and fairness. This phase includes several key milestones:

• The first step is to clearly define the problem the AI system is intended to solve and establish its objectives.
  
  If the system’s objectives are shaped by biased assumptions, the system will inevitably reflect those biases.
  
  —For example, if it is decided that a hiring system should prioritize candidates from so-called prestigious universities, this may exclude equally qualified candidates from other institutions.
  
  This kind of bias is known as institutional or systemic bias. To mitigate it, it is advisable to scrutinize such assumptions, applying measures that range from reviewing data sources to assessing potential impacts, which requires the involvement of experts in areas such as ethics and human rights.

Identifying bias sources is the first step in any bias mitigation strategy.

• At this stage, functional requirements (what the system should do) and non-functional requirements (how it should behave) are defined. This includes data collection for requirements analysis (determining what data is needed to address the problem) and preliminary data gathering (acquiring initial data to better understand the domain and challenges involved).
  
  If the data isn't sufficiently representative of the target population, the model may learn flawed patterns.
  
  —For example, training a healthcare system on data from a single region may limit its performance in areas with different characteristics.
  
  This situation can lead to abstraction traps, which occur when real-world complexity is oversimplified in the form of inputs and outputs for an AI system. Key abstraction traps include:
  
  
  • Formalism traps: the assumption that AI models fully capture real-world complexity.
  • Ripple effect traps: the failure to anticipate how small changes in the system might produce disproportionate downstream consequences.
  • Solutionism traps: the belief that all problems can be solved through technical means alone.

If the data are not sufficiently representative of the target population, the model may learn flawed patterns.

• A technical, economic, and operational feasibility analysis is conducted to determine whether the proposed AI system is viable.
• Appropriate tools and platforms are selected to support the system development.
• A detailed plan is prepared, including timelines, resource allocations, and milestone definitions, to ensure efficient and effective project management.

Phase 2. Design and development

This phase includes the following sub-processes:

• Design
• Data understanding and preparation
• Development

At this point, high-impact decisions are made—such as whether to build in-house or buy existing solutions, whether to rely on open-source or proprietary components, and so forth.

Given the role of design in determining outcomes, construction validity bias—where a chosen variable fails to accurately represent the concept it's meant to capture—is particularly relevant here. This is especially problematic when a AI system addresses complex problems.

—For instance, if socioeconomic status is narrowly equated with income, ignoring other relevant dimensions like education, wealth, occupation, or prestige, the system may operate on a deeply flawed conceptual model. It is therefore essential to include multiple measures of such complex phenomena and to consider culturally diverse interpretations.

Data understanding and preparation also occur here. The most common and widely discussed bias at this stage is representation bias.

This must be addressed by ensuring correct representation, and using techniques such as sampling where appropriate.

Other critical biases in this phase include measurement bias, historical bias, labeling bias, and selection bias.

During the development phase, models are built and trained on selected datasets.

At the end of the design phase—prior to deployment—a thorough bias mitigation assessment must be carried out to ensure that the system remains within predefined ethical and technical boundaries.

The primary type of bias encountered at this stage is algorithmic bias. This bias does not reside in the data itself but in algorithmic logic. For example, a candidate-selection algorithm might assign disproportionate weight to a certain feature—unrelated to actual performance—even when trained on balanced data.

There are multiple forms of algorithmic bias. Among those relevant at this stage are aggregation bias, omitted variables bias, and learning bias.

As NIST notes, a comprehensive bias mitigation review at the end of this phase should include:

• Identified bias sources.
• Implemented mitigation techniques.
• Performance evaluations before the model is released for implementation.

To address these risks, NIST recommends practices such as the “cultural effective challenge”—an internal practice aimed at fostering an environment where technologists actively challenge and interrogate the modeling and engineering steps, with the goal of rooting out statistical and decision-making biases. While we've situated this here, the practice should ideally be iterative across phases.

■ In our view, the implementation of a formal pause, possibly documented in a report—such as in Data Protection Impact Assessments (DPIAs) under the General Data Protection Regulation (GDPR) or in Algorithmic Impact Assessments (FRAIAs) under the Artificial Intelligence Act (AIA)—would be a valuable mechanism for ensuring such a pause actually occurs and has meaningful weight.

If, as a result, bias is identified in the algorithm and its potential impact deemed significant, deployment could—and perhaps should—be halted.

Phase 3. Testing and evaluation (verification and validation)

Testing and evaluation are continuous processes throughout the AI development lifecycle. At this stage:

• Post-deployment model performance is monitored, and ongoing maintenance is conducted.
  
  If evaluation metrics fail to account for fairness, the model may appear accurate while perpetuating or even amplifying pre-existing biases once in production.
  
  —For example, a loan recommendation system trained on historical data could continue to discriminate against certain groups if that data reflects past discriminatory practices.
• Model updates are carried out using new data, with any necessary improvements implemented.

If fairness is not considered in updates, they may introduce or reinforce existing biases.

—For instance, updating a product recommendation system with recent purchase data reflecting a temporary trend might skew the system toward those products and reduce diversity in recommendations.

The NIST report underscores the need for a "culturally effective challenge” to eliminate decision-making bias and continuously improve models.

Another issue that may arise here is evaluation bias, which occurs when evaluation procedures or metrics are misaligned with the real-world deployment context. This can lead to inaccurate conclusions about system performance and fairness.

It is therefore necessary to adopt measures such as revisiting and adjusting evaluation metrics, benchmarking model outcomes against real-world data, and involving all stakeholders to ensure previously identified issues are resolved satisfactorily.

■ In this article, we examined how bias can be introduced in the early stages of the AI system lifecycle. In the next article of this series, we will explore the remaining phases in greater depth, focusing on the risks and control mechanisms associated with implementation, operation, and eventual reassessment.