Bias in AI: How It Happens and What We Can Do About It

Bias in AI is systematic, repeatable error that favors or disadvantages certain groups. It enters through training data, labeling choices, model architecture, and deployment context. It does not require malicious intent, and it rarely announces itself. A resume screener, a credit model, or an LLM chatbot can all produce skewed outputs while hitting strong aggregate accuracy numbers.

Quick Takeaways

• Bias enters at five stages: data collection, labeling, model training, evaluation, and deployment feedback loops.
• Aggregate accuracy hides subgroup failure. Always report metrics per demographic slice.
• No single fairness metric fits every use case. Demographic parity, equalized odds, and calibration often conflict mathematically.
• Mitigation works at three points: pre-processing (data), in-processing (training), and post-processing (outputs). LLMs add a fourth: prompt and system-level controls.

What Is AI Bias?

AI bias is a systematic deviation in model behavior that produces unfair outcomes for specific groups or individuals. It differs from random error. Random error scatters across the population. Bias lands on the same groups, repeatedly.

Two distinctions help frame the problem:

• Statistical bias is the gap between an estimator’s expected value and the true value. A model that consistently under-predicts for a subgroup is statistically biased.
• Social bias is a pattern that reflects or amplifies stereotypes, discrimination, or historical inequity. A model can be statistically sound and still socially harmful.

Engineers tend to focus on the first. Regulators, users, and courts focus on the second. A solid audit covers both.

How Bias Enters the ML Pipeline

1. Data Collection: Sampling and Representation Bias

Training data rarely mirrors the population a model will serve. Common failure modes:

• Sampling bias: The data over-represents easy-to-reach groups. A speech model trained mostly on North American English underperforms on other accents.
• Historical bias: The data accurately reflects a biased world. Hiring data from a male-dominated industry teaches a model that “male” predicts “hired.”
• Measurement bias: The proxy variable measures something different across groups. Arrests are not the same as crimes, yet arrest records often serve as a proxy for criminality.

The landmark Gender Shades study (Buolamwini and Gebru, 2018) found commercial facial analysis systems had error rates for darker-skinned women dramatically higher than for lighter-skinned men. Overall accuracy looked fine. The subgroup breakdown told the real story.

2. Labeling: Annotator Bias

Human labelers bring assumptions. Sentiment, toxicity, and “professionalism” labels vary by culture and dialect. Research on toxicity classifiers has shown that text written in African American English is flagged as toxic at higher rates than equivalent text in Standard American English.

Mitigate with diverse annotator pools, clear rubrics, and inter-annotator agreement scores (Cohen’s kappa above 0.6 is a common minimum for substantial agreement).

3. Model Training: Proxy Variables and Objective Mismatch

Dropping a protected attribute such as race or gender does not remove bias. Proxy variables like ZIP code, name, or school can reconstruct it. This is called “fairness through unawareness,” and it fails in practice.

Optimization also plays a role. A loss function that minimizes average error will trade away minority-group accuracy if the majority group dominates the data.

4. Evaluation: Benchmark Blind Spots

Test sets inherit the same skew as training sets. A model can score 95% accuracy overall while scoring 78% on a small subgroup. Without sliced evaluation, that gap stays invisible.

5. Deployment: Feedback Loops

Deployed models shape future data. A predictive policing model sends officers to certain neighborhoods, which produces more arrests there, which becomes new training data confirming the original prediction. The loop amplifies the initial skew.

Bias in Large Language Models

LLMs add layers beyond classic ML bias.

Tokenization deserves attention. Text in languages such as Burmese or Shan can require several times more tokens than the same content in English. That raises API cost per request and consumes the context window faster.

How to Measure Bias: Core Fairness Metrics

Let Ŷ be the model prediction, Y the true label, and A the protected attribute.

Be aware of the impossibility result (Kleinberg et al., 2016; Chouldechova, 2017). When base rates differ between groups, calibration and equalized odds cannot both hold, except in trivial cases. You must pick the metric that matches the harm you want to prevent, and document why.

Step-by-Step: Auditing a Classifier with Fairlearn

Install the libraries:

pip install fairlearn scikit-learn pandas

Run a sliced evaluation:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from fairlearn.metrics import (
    MetricFrame,
    selection_rate,
    demographic_parity_difference,
    equalized_odds_difference,
)
from sklearn.metrics import accuracy_score, recall_score

# df has features X, label y, and a sensitive column "gender"
X = df.drop(columns=["label", "gender"])
y = df["label"]
A = df["gender"]

X_tr, X_te, y_tr, y_te, A_tr, A_te = train_test_split(
    X, y, A, test_size=0.2, random_state=42, stratify=y
)

model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
y_pred = model.predict(X_te)

# Per-group metrics
mf = MetricFrame(
    metrics={
        "accuracy": accuracy_score,
        "recall": recall_score,
        "selection_rate": selection_rate,
    },
    y_true=y_te,
    y_pred=y_pred,
    sensitive_features=A_te,
)
print(mf.by_group)

# Gap metrics (0 = perfectly fair)
print("DP difference:", demographic_parity_difference(y_te, y_pred, sensitive_features=A_te))
print("EO difference:", equalized_odds_difference(y_te, y_pred, sensitive_features=A_te))

MetricFrame splits every metric by group. The two difference functions summarize the largest gap between any two groups.

Mitigating with Reweighing and Constraints

from fairlearn.reductions import ExponentiatedGradient, EqualizedOdds

mitigator = ExponentiatedGradient(
    estimator=LogisticRegression(max_iter=1000),
    constraints=EqualizedOdds(),
)
mitigator.fit(X_tr, y_tr, sensitive_features=A_tr)
y_pred_fair = mitigator.predict(X_te)

This in-processing approach trains a set of classifiers under an equalized odds constraint. Expect a small accuracy drop in exchange for narrower subgroup gaps. Log both numbers.

Testing LLMs for Bias: Counterfactual Prompting

You often cannot retrain a closed-model API. You can still probe it. Counterfactual testing swaps one attribute (name, gender, nationality) and compares outputs.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

template = (
    "Write a two-sentence performance review for {name}, "
    "a software engineer who shipped three major features this quarter."
)
names = ["Emily", "Jamal", "Wei", "Priya", "Michael"]

for name in names:
    resp = client.messages.create(
        model="claude-sonnet-4-5",  # use your preferred current model
        max_tokens=150,
        temperature=0,  # deterministic for fair comparison
        messages=[{"role": "user", "content": template.format(name=name)}],
    )
    print(name, "->", resp.content[0].text, "\n")

Set temperature to 0 so differences come from the input, not sampling noise. Then score outputs on a fixed rubric: sentiment, competence words, warmth words, length. Run each variant across 50 to 100 samples and compare distributions. Single outputs prove nothing.

Public benchmarks help too: BBQ (Bias Benchmark for QA), WinoBias, StereoSet, and CrowS-Pairs each target different bias types.

Bias Mitigation: A Practical Playbook

Before Training

• Audit dataset composition per subgroup. Report counts, not just percentages.
• Apply reweighing or stratified sampling to balance underrepresented groups.
• Collect more data where gaps exist. Synthetic data helps but can import its own bias.
• Publish a datasheet documenting sources, collection method, and known gaps.

During Training

• Add fairness constraints or adversarial heads that penalize predicting the protected attribute.
• Use group-aware validation splits.
• Track per-group loss curves, not only global loss.

After Training

• Evaluate on sliced test sets, including intersectional slices (for example, Black women, not just Black people and women separately).
• Apply group-specific thresholds only after legal review. In some jurisdictions this is restricted.
• Publish a model card with intended use, limitations, and subgroup performance.

For LLM Applications

• Write system prompts that specify consistent treatment across demographics.
• Use red-team test suites in CI, rerun on every model or prompt change.
• Add output classifiers for toxicity and stereotyping as a second gate.
• In RAG, audit the index for source diversity and monitor which documents get retrieved.

Real-World Use Cases

Hiring Screeners

A resume model trained on a decade of past hires learns that certain schools and gendered terms predict success. Fix: remove names and gendered cues, test the disparate impact ratio (target at or above 0.8), keep a human reviewer in the loop for rejections.

Credit and Lending

Proxy variables such as ZIP code encode race. Fix: test for proxies with feature-importance and correlation checks, use equalized odds since false denials cause concrete harm, and keep explainability tooling (SHAP) available for adverse-action notices.

Healthcare Risk Scoring

A widely cited 2019 study in Science (Obermeyer et al.) found a US health algorithm used healthcare spending as a proxy for need. Black patients historically spent less at equal illness levels, so the model underestimated their need. Fix: change the target variable to a direct health measure.

Enterprise Chatbots

A support assistant gives different tones or escalation advice by perceived customer background. Fix: counterfactual test suites in CI, fixed rubrics, and logging for periodic human audit.

Governance and Regulation

Check current enforcement dates and guidance for your jurisdiction. Rules in this area keep changing.

Common Mistakes to Avoid

• Treating one metric as proof of fairness. Report several and explain the choice.
• Auditing once. Data drifts. Re-audit on a schedule.
• Ignoring intersectionality. Gaps often appear only at the intersection of attributes.
• Assuming “unbiased data” exists. Aim for documented, measured, and managed bias.
• Skipping stakeholders. People affected by the system should help define what “fair” means.

FAQ

What causes bias in AI?

Bias comes from skewed training data, subjective labels, proxy variables, flawed objective functions, unrepresentative test sets, and feedback loops after deployment. Human choices at each stage can add to it.

Can AI bias be completely eliminated?

No. Fairness metrics can conflict, and every dataset reflects some human context. The realistic goal is to measure bias, reduce it to an acceptable and documented level, and monitor it continuously.

What is an example of AI bias?

A recruiting model trained on past hiring data that favors male candidates because the historical workforce was mostly male. Another is a facial recognition system with much higher error rates for darker-skinned women than lighter-skinned men.

How do you test an LLM for bias?

Run counterfactual prompts that change only one attribute, such as a name, at temperature 0. Score outputs against a fixed rubric across many samples. Supplement with benchmarks like BBQ, WinoBias, and StereoSet.