Responsible AI

What responsible AI means

Responsible AI is the practice of designing, building, deploying and using AI systems in a way that is fair, safe, transparent and accountable, with attention to the people affected by them and not only to the people using them. It is not one checkpoint at the end of a project but a set of questions asked at every stage: what data, for which purpose, tested on whom, explained how, overseen by whom, stopped when?

Many organizations publish their own principles. Microsoft's six are a good example, and are part of the AI-901 exam:

Other lists say much the same in different words. The OECD AI Principles (2019, updated 2024) and the UNESCO Recommendation on the Ethics of AI (2021) are international examples, and other cloud providers publish their own sets. The EU AI Act turns several of these ideas into legal obligations; the US and the UK rely more on existing laws (consumer protection, anti-discrimination, data protection), guidance and, in the US, a growing number of state laws. See AI regulation and governance. Privacy and security have their own pages: privacy and data protection and AI security.

Algorithmic bias

Algorithmic bias is a systematic error in the output of an algorithm that unfairly advantages or disadvantages particular groups of people. It usually comes from the data (history repeated: past decisions were biased, or some groups are under-represented; see data for AI), from the design (which features are used, what the model optimizes for) or from the way the output is used.

A common misunderstanding is that removing a sensitive attribute such as gender, age or nationality removes the bias. It does not when other features act as a proxy: a postal code can stand in for ethnic background, a gap in a CV for parenthood, a first name for gender. Three well-documented cases show how bias arises in practice.

The Dutch childcare benefits scandal (toeslagenaffaire)

In the Netherlands, the Tax and Customs Administration (Belastingdienst) wrongly treated many thousands of parents who received childcare benefits as fraudsters over several years. Families were ordered to repay large sums, often for minor administrative errors, and many were pushed into debt and lasting hardship. A parliamentary inquiry called it "unprecedented injustice", and the government resigned over the affair in January 2021.

The causes were broad — strict rules, an institutional focus on fraud, and failing legal protection — but data processing played a clear part. The Dutch Data Protection Authority, the Autoriteit Persoonsgegevens (AP), found that the Belastingdienst had processed applicants' (dual) nationality unlawfully and in a discriminatory way. Nationality was used, among other things, as an indicator in a risk classification model, so applicants with a non-Dutch nationality were more likely to be selected for manual review. In December 2021 the AP fined the Belastingdienst €2.75 million for this.

Lessons that apply far beyond tax offices: a sensitive attribute was used directly as a risk signal; being selected led to harsh consequences with little room for explanation or appeal; and the people affected could not see why they were selected. The affair is one of the reasons the Dutch government now publishes an algorithm register and gave the AP a coordinating role in supervising algorithms.

Amazon's recruiting tool

In October 2018, Reuters reported that Amazon had stopped using an experimental tool that rated job applicants. It had been trained on about ten years of CVs submitted to the company, mostly from men, and according to the report it learned to downgrade CVs that contained the word "women's" (as in "women's chess club captain") and graduates of all-women's colleges. Gender was never an explicit input; the model found proxies for it in the text. Amazon told Reuters the tool was never used by its recruiters to evaluate candidates.

COMPAS and conflicting definitions of fairness

COMPAS is a commercial tool used in parts of the US justice system to estimate the risk that a defendant will reoffend. In 2016, ProPublica analyzed its scores and reported that Black defendants who did not reoffend were much more likely to have been labeled high-risk than white defendants who did not reoffend (a higher false positive rate). The vendor, Northpointe, disputed the analysis: its scores were equally calibrated for both groups — a given score meant roughly the same chance of reoffending regardless of race.

Both could be right at the same time. Kleinberg, Mullainathan and Raghavan (2016) proved that when two groups have different base rates, a risk score cannot in general be both calibrated and have equal error rates across the groups. Fairness is not one number; choosing which definition matters most is a decision about values that must be made openly, by people who can answer for it.

Measuring fairness

The simplest check compares selection rates: the share of each group that receives the favorable outcome (invited, approved, hired). Dividing the lowest rate by the highest gives the disparate-impact ratio. A widely used rule of thumb, the four-fifths rule, treats a ratio below 0.8 as a sign of possible adverse impact that needs investigation. It comes from US employment law: the Uniform Guidelines on Employee Selection Procedures (29 CFR 1607.4(D)) generally regard a selection rate for any race, sex or ethnic group below four-fifths of the rate of the group with the highest rate as evidence of adverse impact. It is not proof of discrimination, and outside that US setting it is not a legal standard, but it is a useful first alarm for any group.

The program checks a made-up CV-screening model that decided which of 50 applicants to invite for an interview:

from collections import Counter

# Made-up results of a CV-screening model: (sex, invited to interview?)
decisions = (
    [("men", True)] * 18 + [("men", False)] * 12
    + [("women", True)] * 7 + [("women", False)] * 13
)

applicants = Counter(group for group, _ in decisions)
invited = Counter(group for group, was_invited in decisions if was_invited)

rates = {}
for group in sorted(applicants):
    rates[group] = invited[group] / applicants[group]
    print(f"{group:<12} {invited[group]:>2} of {applicants[group]:>2} invited "
          f"-> selection rate {rates[group]:.2f}")

overall = sum(invited.values()) / sum(applicants.values())
print(f"overall      selection rate {overall:.2f}")

ratio = min(rates.values()) / max(rates.values())
print(f"disparate-impact ratio: {ratio:.2f}")
if ratio < 0.8:
    print("below 0.8 (four-fifths rule): investigate before using this model")
else:
    print("at or above 0.8: no red flag by this heuristic alone")

men          18 of 30 invited -> selection rate 0.60
women         7 of 20 invited -> selection rate 0.35
overall      selection rate 0.50
disparate-impact ratio: 0.58
below 0.8 (four-fifths rule): investigate before using this model

The overall rate of 0.50 looks unremarkable; only the per-group view shows that women were invited at a bit more than half the rate of men. That does not yet prove the model is unfair — the groups might differ in relevant qualifications — but it means someone must find out why before the model is used. Note that you can only run this check if you know the group of each applicant; measuring fairness sometimes requires carefully protected sensitive data.

Other metrics compare error rates instead (are qualified applicants rejected equally often in each group?), and as COMPAS showed, they can disagree with each other. Libraries such as Fairlearn and AIF360 compute dozens of fairness metrics; they do not tell you which one to choose.

Explainability

Explainability is the degree to which people can understand why an AI system produced a particular output, in terms that are meaningful to them. A decision tree or a small linear model is explainable by itself: you can read the rules or the weights. A deep neural network or a large language model is a black box: billions of parameters, no readable reasons.

Explanation methods work from the outside:

• Feature importance: which inputs influence the model's predictions most overall (for example, "payment history matters most, postal code second" — the second is a warning sign for a proxy).
• LIME (Local Interpretable Model-agnostic Explanations; Ribeiro, Singh and Guestrin, 2016): explains one prediction by changing the input slightly many times and fitting a simple model to how the output changes around that point.
• SHAP (SHapley Additive exPlanations; Lundberg and Lee, 2017): divides one prediction among the input features using Shapley values from game theory, so you can see that, for this applicant, income pushed the score up and a short employment history pushed it down.

These methods are approximations and can themselves mislead, but they make it possible to spot a model that relies on the wrong things. In public services, black boxes are a particular problem: a government decision must be explainable and open to appeal, and a citizen cannot challenge reasons nobody can state. The GDPR gives people the right to meaningful information about the logic involved in automated decisions about them and, with limited exceptions, the right not to be subject to decisions based solely on automated processing that significantly affect them. In the UK such decisions are allowed with safeguards, including information about the decision and the right to contest it (UK GDPR Article 22C); in the US, Colorado will require a plain-language explanation of the role automated decision-making played within 30 days of an adverse decision, from 1 January 2027.

Transparency

AI transparency means being open about the use of AI: telling people when they interact with an AI system or receive AI-generated content, and documenting what a system does, what data it was built on, how well it performs and where its limits are. It is broader than explainability, which is about individual outputs.

Model cards and datasheets

A model card is a short, structured document published with a trained model that describes its intended use, how it was trained and evaluated, its performance (including per group of people), its limitations and ethical considerations. The idea comes from Mitchell et al., "Model Cards for Model Reporting" (2019); model providers and hubs now publish them as standard practice. The companion idea for data is datasheets for datasets (Gebru et al., 2018; published in Communications of the ACM in 2021): who collected the data, why, how it was labeled, who is represented and what it should not be used for.

Before you adopt a model, read its model card. The most useful sections are often the least glamorous: "out-of-scope uses" and "known limitations".

Human oversight and automation bias

Human oversight means that people monitor an AI system and can understand, question, override or stop its outputs and its operation, especially where decisions affect people's rights or safety. The EU AI Act requires it for high-risk systems; UK data protection law counts a human role only when it is meaningful human involvement (UK GDPR Article 22A); US federal agencies must ensure human oversight, intervention and accountability for high-impact AI (OMB memorandum M-25-21). It takes several forms: a person approves every decision (human in the loop), a person monitors and can intervene (human on the loop), or people decide when and how the system is used at all.

Oversight fails quietly through automation bias: the tendency of people to over-trust the output of an automated system, accepting its suggestions without enough scrutiny and overlooking evidence that contradicts them. A reviewer who handles a hundred AI-flagged cases a day, with targets to meet and no information about why each case was flagged, will mostly confirm the flag. The human is then formally in the loop but in practice a rubber stamp. In the toeslagenaffaire, applications selected by the risk model were reviewed by people; that human step did not prevent the harm to the families who were selected.

Meaningful oversight needs:

• Time and workload that allow real review;
• Information: the reasons for the output, its confidence, and the evidence, not just a label;
• Authority to disagree without having to justify it more than an agreement;
• Monitoring of the overseers: if reviewers agree with the system 99.9% of the time, check whether they are still reviewing.

Accountability

"The algorithm did it" is never an acceptable answer. Someone chose to build or buy the system, someone decided what data it uses, someone decided to act on its output. Accountability means those roles are named in advance and the people in them can be asked to explain and correct decisions. In practice:

• The organization that deploys an AI system is responsible for how it is used, even if the model comes from a supplier. The EU AI Act distinguishes providers (who build or market a system) from deployers (who use it), with duties for both; the AI laws of Colorado and Texas make a similar split between developers and deployers.
• Each AI system has an owner, just like any IT service, who answers for its behavior and can decide to switch it off (see AI governance). The data it was trained on needs an owner too; see owners and stewards.
• People affected have a way to complain and appeal, handled by humans.
• Decisions and the data behind them are logged, so they can be reconstructed later.

For individual employees the same principle holds: if you use AI output in your work, you are accountable for it; see using AI at work.

Environmental impact

Training and running AI models uses electricity, water for cooling and hardware that has to be manufactured and eventually discarded. According to the International Energy Agency, data centers used about 1.5% of the world's electricity in 2024, and AI is the main reason that share is expected to grow; the figures and their uncertainty are discussed under energy use. Responsible use means proportionality: choosing the smallest model that does the job, not using AI where a simple rule works, and counting energy as a cost when deciding whether a use case is worth it. Some providers publish energy and emissions data for their models and data centers; many do not, which makes comparison hard.