AI Biases (IV): Risk management and impact

As we have seen in the previous chapters of this series, biases are one of the main risks that AI systems can have.

If we take into account that AI-based technology already has and will have even more connections and broader impacts on society than traditional software, as they are sure to gain capillarity in almost all sectors and contexts, either directly or in instrumental processes, this implies, from the outset and in general, a multiplying effect of the possible risks arising from AI biases.

Having said this, it is essential to turn to the concept of risk and its measurement in order to advance in the relationship between biases and risks. As we know, the way to measure risk is to take into account the probability of a threat mateAIRlizing combined with the impact it would have if it were to occur. There are therefore two factors in the equation: probability and impact.

In AI systems, biases are one of the main risks.

1. Impacts produced by IA systems and their calculation

Focusing on the calculation of impact, practices based on criteAIR accepted as global standards should be taken into account for the specificities of AIR.

In this case ISO 31000:2009 Risk Management - Principles and Guidelines, from which the rest of the ISO standards are inspired and adapted to specific environments, in addition to the specific ISOs in IA, particularly ISO/IEC 23894:2023, guidance on risk management in IA.

Particularly relevant is the work developed by the Massachusetts Institute of Technology (MIT) with the AI Risk Repository, framework, which offers a dynamic database compiling more than 1000 AI risks, drawn from 56 frameworks.

The concept of impact in the field of AI systems is used on different fronts: for risk analysis in AI systems in Article 9 AIR, for FRAIA in Article 29 bis; and there are also examples such as the pioneering Law 1/2022, of April 13, on Transparency and Good Governance of the Valencian Community that, in addressing when it is considered an essential element for whether or not active publicity of AI systems should be made, refers to the fact that “they have an impact on administrative procedures or the provision of public services”.

The impact of AI technology on society is greater than that of traditional software, which amplifies the risks associated with its biases.

Based on the above, the impacts that AI may have on people should be analyzed, when it is used in accordance with its intended purpose and also, when it is put to a reasonably foreseeable misuse. These impacts can be classified in different ways, such as, for example:

a) Legal, affecting Fundamental Rights such as, for example:

• Discrimination: biases can perpetuate discrimination in different activities in both the public and private spheres.
• Inequality: biased algorithms can aggravate inequality instead of reducing it.
• Injustice: legal or significant effects on individuals through denial of subsidies, suspicions of non-compliance with existing regulations etc.

b) Social, such as:

• Loss of self-judgment, self-confidence.
• Perpetuation of biases: structural risks.
• Loss of confidence in one's own technology.

2. Lack of explainability and its impact on the management of biases

Explainability is the ability to understand and thus to be able to explain how an AI system makes its decisions. Therefore, if an AI system is not explainable, it is difficult to identify possible biases and manage them.

We refer to systems such as, for example:

• Those of deep neural networks or deep learning where these layers of neurons create a black box where it is difficult to understand how a specific decision has been reached. Think for example of an image recognition model in which the model classifies an animal, but we do not know what specific elements or characteristics it uses to do so. In cases linked to face recognition, this has led to discrimination cases.
• Self-learning systems, among which are for example reinforcement learning systems (in which the agent learns to make optimal decisions by interacting with the environment and receiving rewards) and where there are strategies used that can be difficult to understand and explain. Think for example of an autonomous car that in a situation where there is no visible obstacle decides to stop because its sensors have identified one.
• In data classification models, such as those that classify operations as fraudulent or not and in which the patterns used can be complex to understand. In these cases, if the system erroneously classifies as illegitimate the operations of certain groups (for example, taking into account their demographic location or some other element), it may be discriminatory.

The relationship between biases and risks must be addressed in terms of risk and its measurement.

Notwithstanding the above, there are several techniques to solve the lack of explainability, such as, for instance:

• Interpretable models: this consists of using AI models that are inherently easier to interpret, such as decision trees, linear regressions and simple neural networks and that allow understanding how decisions are made based on the inputs by applying clear rules and constraints to guide their operation and ensure that decisions are understandable and justifiable.
• Post-hoc methods: this involves applying techniques that explain the decisions of complex models after they have been made, such as LIME (Local Interpretable Model-agnostic Explanations) that generates local explanations for individual predictions or SHAP (SHapley Additive exPlanations) that assigns importance values to each input feature, based on game theory.
• Education and training: train teams in understanding and managing AI decisions. This includes training in the use of explainability tools and interpretation of results.
• Audits and evaluations: conduct regular audits and external evaluations to review and validate AI system decisions, ensuring that they are transparent and equitable.

In assessing AI's impact, we must account for both its intended uses and its reasonable foreseeable misuses.

3. Criteria for measuring impact

Standards are not oblivious to the potential impacts and certain legal provisions already provide criteAIR for this.

Article 22 of the General Data Protection Regulation (GDPR) prohibits automated decisions with significant legal effects (such as credit denials or criminal risk assessments) without relevant human intervention. Algorithmic biases affecting fundamental rights, such as equal access to public services could be subject to legal challenge.

The SCHUFA judgment of the Court of Justice of the EU expanded this concept, adopting a guaranteeing criterion, by considering that even the automatic generation of a predictive value (such as a credit score) constitutes an automated decision if it has a decisive influence on the final decision of a third party. This criterion makes it necessary to re-evaluate systems that combine automatic processing with superficial human review.

In this sense, the CJEU expands, through a broad interpretation of Article 22, the scope of the term automated decisions to include the following cases: (i) “semi-automated” decisions based on the automated decision, but with greater or lesser human participation; (ii) predictions of probabilities or profiling that are configured as determinant for the adoption of the decision by a third party.

Meanwhile, Article 14 of the EU AI Act requires that human intervention be meaningful, avoiding blind automation. This is not just a superficial review, but a real and substantive assessment of all relevant factors.

In assessing the impact, special consideration must be given to, on the one hand, the purpose of the use or purpose of the AI system. It is a good starting point to follow the AIR's “high risk” uses of AI.

As we know, the AIR has a risk approach and it is already clear that it takes into account biases as elements to be considered. As an example, recital 61 expressly states:

“In particular, in order to address the risk of potential biases, errors and opacities, it is appropAIRte to classify as high risk those IA systems intended to be used by or on behalf of a judicial authority to assist judicial authorities in investigating and interpreting facts and law and in applying the law to specific facts…”.

High-risk systems in the AIR (Article 6) will occur in two scenarios:

• Either when a security component or product is involved.
• Or when it is one of the high-risk IA systems referred to in Annex III. Also, if an AI system falls within one of the areas of Annex III, it may not be considered high risk if its influence on decision making is not substantial and one or more of the following conditions are met: the system performs a limited procedural task, enhances previous human activities, detects patterns of decision or deviation without replacing human assessment, or performs preparatory tasks.

The list of high-risk AI systems is based on criteria such as autonomy and complexity, social and personal impact and safety in critical infrastructures.

Annex III of the AIR lists the systems that are considered high-risk.

All of the above would also have to be grounded to the use in the public or private sector, where there may be elements that make the risk “weighted” and even recalibrated. In the case of the public sector, the proposal made by Cotino in relation to the criteria and variables to determine the impact, level of risk and legal relevance of public algorithmic systems, outlines the following criteria:

• That it produces legal effects on him or significantly affects him, a normative criterion that follows, for example, another norm such as art. 22 of the GDPR.
• That the system takes individualized decisions regarding individuals, involves the making of internal administrative decisions or for the development of policies and their collective impact.
• That they are high-risk systems, being critical of the list of systems catalogued as high-risk by the IA because the aforementioned author considers it incomplete due to the existence of public IA systems that have a particularly high impact but are not included in the aforementioned list.
• That they are mass uses, which implies that “the danger of a massive error or bias in countless future cases will have to be weighed, as well as the significant benefit of avoiding its replication in thousands or millions of decisions” and concludes that therefore for public high-risk AI systems that apply massively it is necessary to “recalibrate” these acceptable thresholds and the applicable safeguards. And he concludes - a view with which I agree - that for such systems “in general we need to be much less tolerant”.

When measuring risk, it is necessary to consider the probability of a threat materializing along with its impact.

In the case of the private sector, some criteria can be applied with corresponding peculiarities:

• That they produce significant legal effects: as in the public sphere, private IA systems that produce significant legal effects or considerably affect individuals should be evaluated with greater rigor, as is the case with decisions that may impact fundamental rights, as is the case for example with access to financial services, employment, or housing.
• That the system makes individualized decisions: AI systems that make individualized decisions about individuals, such as in hiring processes, credit evaluation, or personalization of services, must be carefully monitored to ensure transparency and fairness, so that automated decisions must be explainable and justifiable.
• High-risk systems: in the private sector, high-risk AI systems may refer, for example, to those used in critical sectors such as healthcare, finance, and transport.

MIT is developing a dynamic database with more than 1,000 AI risks called the AI Risk Repository.

In conclusion, biases in Artificial Intelligence constitute a multidimensional challenge that requires a rigorous approach, combining normative, ethical and technical analysis. Explainability not only facilitates the detection of biases, but is also an essential legal requirement.

The legal frameworks and standards mentioned above provide clear tools for managing risks, although it should be noted that their success will depend on effective implementation and interdisciplinary collaboration between legal, technical and ethical experts.