Responsible Machine Learning

Contents

• Introduction and overview
• 0. Review: probability, classification metrics and convexity
• 1. Why responsible machine learning?
• 2. Fairness: legal notions, criteria and impossibility
• 3. Privacy: from anonymisation to differential privacy
• 4. Transparency: interpretable models, explanations and recourse
• 5. Accountability, human agency and context
• 6. Complements: fair learning, privacy accounting, federated learning, generative AI
• 7. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with a good M1 in mathematics, computer science or physics who can train a classifier and wants to know, with definitions and proofs, when its decisions are fair, private, explainable and accountable, and what the law asks of it.

Empirical first: a credit-like score that never sees the protected attribute still selects 50 % of one group and 28 % of the other (ch. 2). A patrol rule fed by its own discoveries, between two districts whose true incident rates are 0.3 and 0.2, sends after 20,000 days more than 90 % of patrols to the first in two runs out of three, against a proportional share of 60 % (ch. 1). This course explains such failures and the tools that measure and correct them.

Responsible machine learning sits where computer science, law and public policy meet. The course description organises it around the pillars of trustworthy AI: fairness, privacy, transparency and accountability, to which the Université Paris-Saclay programme adds human agency. Each pillar is treated twice on this page: as a mathematical object, and as a legal or policy requirement. The course is given by Nicolas Vayatis and Aurélien Bellet, as listed on master-mva.com.

Related MVA pages: Regulating AI for the law itself, Multimodal Explainable AI for explanation methods, and Turing Seminar: An Introduction to AGI Safety for the safety of general-purpose models.

What you will be able to do

• Locate where harm enters a machine-learning system, from historical bias in the data to feedback loops after deployment.
• Translate the legal notions of direct and indirect discrimination into statistical criteria, and compute them from confusion matrices.
• Prove the impossibility theorems of Chouldechova and of Kleinberg, Mullainathan and Raghavan, and explain the COMPAS controversy with them.
• Equalise error rates across groups by post-processing a score, and state the price in accuracy.
• Explain why removing names does not anonymise data, and define differential privacy as a bound on any membership test.
• Prove the Laplace and randomized-response mechanisms, calibrate the Gaussian mechanism, compose them, and train a model with DP-SGD.
• Compute exact Shapley values, derive them in closed form for linear models, and compute minimal counterfactual explanations with immutable features.
• Audit a deployed model with confidence intervals, and design a reject option that defers uncertain cases to a human.
• Read a deployment in healthcare, finance or criminal justice against the GDPR and the AI Act, and write the matching documentation.
• Compute the privacy budget of a training run, and compare central, local and federated privacy.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com examines the mutual relationship between law, computer science and public policy, a link it describes as more and more ambiguous with the spread of AI. Growth in computing capacity and data has made machine learning excellent at specialised tasks, from speech recognition and translation to decision support, fault detection and image classification, with applications in health, industry and the social sciences.

The description sets this against the costs of automation. In predictive justice, hiring and finance, algorithms can exacerbate structural inequalities, favouring those already in power and further disadvantaging minorities and vulnerable groups. Automation is not a “simple” substitution: it encodes norms into code that obscures the problem (Carr, 2017; Dekker and Woods, 2002; Zuboff, 2019).

It adds that recommendation systems carry harms such as political destabilisation, mental-health issues, misinformation and disinformation; that the technology is resource-intensive, with effects on the environment (Strubell et al., 2019); and that generative AI widens the gap between benefits and harms.

The course then debates how to balance the benefits and harms, by unpacking the pillars of AI from a computer-science and from a legal and policy perspective. Each session dives into one component, discusses what each discipline adds, and uses real case studies to merge them into a solution. A special focus goes to contextual considerations in healthcare, finance and criminal justice.

The Université Paris-Saclay course detail adds that countries, private companies, standard bodies and international organisations call for governing AI through principles, such as fairness, accountability, transparency and human agency. Its learning objectives describe a growing gap: lawyers and policy makers who misunderstand technical terms may oppose deployment or pass regulation that hinders innovation, while computer scientists who ignore the socio-political and legal context may build non-compliant solutions that exacerbate biases.

Every official topic is taught on this page. Its extensions beyond the syllabus are the review chapter 0, the proofs of the impossibility and privacy theorems, the self-study chapter 6 (fairness-constrained learning, causal fairness, privacy accounting, federated learning, generative AI, unlearning) and the research chapter 7.

Where this course fits in the programme

Official prerequisites: probability and statistics at M1 level, and the basics of machine learning. The course is a Learning course of 2.5 ECTS and 10 h; master-mva.com lists it in semester 2.

• Preparation — the refreshers Probability and Statistics (conditioning, confidence intervals), Linear Algebra, Analysis and Optimization (Cauchy–Schwarz, Lagrange multipliers) and Scientific Python, NumPy and PyTorch (every code block).
• Semester 1 courses that prepare it — Introduction to Statistical Learning (classification risk, ROC curves, scoring), Convex Optimization and Applications in Machine Learning (duality, used in ch. 6.1), Deep Learning (the models DP-SGD trains), Fundamentals of Reproducible Research and Free Software (documentation and audits) and the Turing Seminar.
• Related courses of semester 2 — Regulating AI (the GDPR and the AI Act in depth), Multimodal Explainable AI (XAI) (attribution methods beyond ch. 4), Biostatistics (causal inference behind ch. 6.2), Bayesian Machine Learning, Sequential Learning (feedback loops as bandits), Training and Deploying Large-Scale Models and Artificial Intelligence and the Environment (the footprint of ch. 1.4).
• After the course — fairness, privacy and federated learning are frequent topics of the Research Internship.

Course outline

• Review: probability, classification metrics and convexity — confusion rates, base rates, calibration, intervals for proportions, Laplace noise, duality and Pólya’s urn.
• Why responsible machine learning? — benefits and harms, principles and binding law, sources of harm, feedback loops, energy and generative AI.
• Fairness: legal notions, criteria and impossibility — discrimination law, independence, separation and sufficiency, proxies, impossibility theorems and the COMPAS case.
• Privacy: from anonymisation to differential privacy — re-identification and membership attacks, differential privacy, its mechanisms, composition and DP-SGD.
• Transparency: interpretable models, explanations and recourse — legal duties of explanation, Shapley values, counterfactual explanations, limits.
• Accountability, human agency and context — obligations and documentation, statistical audits, human oversight and the reject option, three sectors.
• Complements — post-processing and fairness-constrained learning, causal fairness, privacy accounting, federated learning, generative AI and unlearning (self-study).
• Research directions and open problems — open questions for each pillar (self-study).

Session plan

10 h in class = 5 sessions of 2 h. The volume is the Université Paris-Saclay programme table (10 h of lectures, 2.5 ECTS). Its course detail announces five sessions of 2.5 h, which would make 12.5 h, and master-mva.com lists the course in semester 2 while the table places it in semester 1.

This plan follows the 10 h and keeps five sessions: an opening session on law, computer science and policy, then one pillar per session, as the course description announces (“in each session, we will dive into one of the components”). The split of the chapters is a proposal.

Level 1 = foundations (definitions, first models), 2 = core results and methods, 3 = advanced material and sector cases; the level never decreases.

Self-study, outside the 10 h (about 8 h in total): ch. 0 Review, to read before session 1 (≈ 2 h) · ch. 6 Complements, after session 5 (≈ 4.5 h, with Ex. 16–17) · ch. 7 Research directions and open problems (≈ 1.5 h). The project or the exam preparation comes on top (see Projects and assessment).

How to study this module

• Read ch. 0 before session 1, then each chapter before its session; a chapter uses only earlier ones.
• After each session, do the exercises of its row. ★ applies a definition to numbers, ★★ asks for a short proof or a program, ★★★ is open-ended.
• Run every Python block: NumPy, SciPy and scikit-learn on a CPU, synthetic data, fixed seeds, a few seconds each.
• Keep two columns in your notes for each pillar: the mathematical definition, and the legal or policy text it answers to. Most exam questions sit between the two.
• Prepare the debate of each session, as the course description announces debates on how to balance benefits and harms:
  • Session 1: should a city use a predictive tool to allocate patrols? Use Theorem 1.4 and Proposition 1.5.
  • Session 2: which criterion of Definition 2.2 should a bail-recommendation tool satisfy, given Theorem 2.6?
  • Session 3: is a statistics office right to add noise to census tables? Weigh Theorem 3.5 against the accuracy users need.
  • Session 4: should a bank disclose its scoring model to refused applicants? Weigh recourse (Proposition 4.6) against gaming.
  • Session 5: when must a hospital keep a human in the loop? Use Theorem 5.4 and Method 5.5.

0. Review: probability, classification metrics and convexity

Every pillar of this course is measured with the vocabulary of binary classification: error rates by group, calibrated scores, confidence intervals, noise distributions. This chapter fixes that vocabulary at the level of a good M1. The refreshers Probability and Statistics and Linear Algebra, Analysis and Optimization give more detail.

0.1 Rates of a binary decision

TPR, FPR and FNR condition on the truth; PPV conditions on the decision. Chapter 2 shows that this difference is the source of the impossibility theorems of fairness.

0.2 Scores, thresholds and calibration

0.3 Logistic regression

The logistic model scores s_w(x) = \sigma(w^\top x) with \sigma(u) = 1/(1 + e^{-u}). It is fitted by minimising the average log-loss \ell(w; x, y) = -y \log \sigma(w^\top x) - (1 - y) \log\big(1 - \sigma(w^\top x)\big), which is convex in w, with gradient \nabla_w \ell(w; x, y) = \big(\sigma(w^\top x) - y\big)\, x .

Since |\sigma(w^\top x) - y| \leq 1, each per-example gradient has norm at most \|x\|_2. Chapter 3 clips such gradients to bound the influence of one person on training.

0.4 Confidence intervals for proportions

Let \hat p be the mean of n independent Bernoulli(p) variables.

• Hoeffding’s inequality holds for every n: P\big(|\hat p - p| \geq t\big) \leq 2 \exp(-2 n t^2).
• The Wald interval uses the central limit theorem, with z = 1.96 at level 95 %: \hat p \pm z \sqrt{\hat p (1 - \hat p) / n}.
• The delta method gives the variance of a smooth function g(\hat p) as about g'(p)^2\, p (1 - p)/n. For g = \log, this is (1 - p)/(n p).

0.5 Laplace and Gaussian noise

By the triangle inequality, shifting the centre by \Delta changes the density by a bounded factor: \frac{\mathrm{Lap}(z - \mu; b)}{\mathrm{Lap}(z - \mu'; b)} \leq \exp\!\left(\frac{|\mu - \mu'|}{b}\right) \quad \text{for every } z . This bounded ratio is exactly what differential privacy needs (ch. 3).

A Gaussian Z \sim \mathcal{N}(0, \sigma^2) has lighter tails, P(Z > t) \leq e^{-t^2/(2\sigma^2)}, but no bounded density ratio under a shift. This is why the Gaussian mechanism of ch. 3 needs a small failure probability \delta.

0.6 Constrained optimisation

Fairness constraints (ch. 6.1) turn learning into a constrained problem: minimise f(\theta) subject to g(\theta) \leq 0. Its Lagrangian is L(\theta, \lambda) = f(\theta) + \lambda\, g(\theta), \qquad \lambda \geq 0 .

For every \lambda \geq 0, the minimum of L(\cdot, \lambda) is a lower bound on the constrained minimum (weak duality). When f and g are convex and some \theta satisfies g(\theta) < 0 (Slater’s condition), the best such bound is attained and equals the constrained minimum (strong duality).

0.7 Pólya’s urn

Chapter 1 models a feedback loop with the simplest reinforcement process of probability.

1. Why responsible machine learning?

Machine learning now informs decisions on credit, jobs, care and liberty. This chapter asks who writes the rules, where harm enters, and how a deployed model amplifies its own bias.

1.1 Law, computer science and public policy

Machine learning now excels at speech recognition, translation, decision support, fault detection and image classification, hence the hope that automation solves old societal problems. Yet lawyers may over-regulate what they misunderstand, and engineers may ship non-compliant systems that amplify bias.

The 2019 guidelines list seven requirements: human agency and oversight; robustness and safety; privacy and data governance; transparency; diversity, non-discrimination and fairness; societal and environmental well-being; accountability.

The AI Act is risk-based. It bans a few practices (Article 5), such as social scoring. Most duties fall on high-risk systems (Article 6): safety components of regulated products (Annex I) and uses listed in Annex III, such as hiring, credit and law enforcement. Chatbots and synthetic content carry transparency duties (Article 50); the rest is minimal-risk. In the 2024 text, bans apply from February 2025 and most high-risk duties from August 2026 or 2027 (Article 113).

1.2 Where harm enters a model

Automation is often presented as a simple substitution of a machine for a person. Dekker and Woods (2002) argue that it transforms the work, with new monitoring tasks and new errors. A model also fixes a norm, a label or a threshold in code, where it is hard to contest. Zuboff (2019) analyses the business model built on behavioural predictions.

1.3 Feedback loops

A deployed model changes the data it is retrained on. Ensign et al. (2018) model predictive policing as an urn.

The loop runs away because incidents are recorded only where the patrol goes. Inverse-propensity weighting of the discoveries, with visit probabilities kept above a floor, gives unbiased rate estimates; allocating by them recovers the proportional share.

import numpy as np

rng = np.random.default_rng(0)
R, T = 400, 20_000                          # independent runs, days

def urn(lam_a, lam_b, ipw=False):
    """Daily patrol sent to A with prob. pi; an incident is seen with prob. lam."""
    a, b = np.ones(R), np.ones(R)           # urn: discovered incidents (+1 prior)
    sa, sb = np.zeros(R), np.zeros(R)       # inverse-propensity sums
    for t in range(1, T + 1):
        if ipw and t > 1:
            la, lb = sa / (t - 1), sb / (t - 1)
            pi = np.clip(la / np.maximum(la + lb, 1e-12), 0.05, 0.95)
        else:
            pi = a / (a + b)
        go_a = rng.random(R) < pi
        seen = rng.random(R) < np.where(go_a, lam_a, lam_b)
        a += go_a & seen
        b += ~go_a & seen
        sa += (go_a & seen) / pi             # unbiased for lam_a
        sb += (~go_a & seen) / (1 - pi)      # unbiased for lam_b
    return pi

for lam in [(0.25, 0.25), (0.30, 0.20)]:
    s = urn(*lam)
    print(f"urn, rates {lam}: mean share of A {s.mean():.3f}, sd {s.std():.3f}, "
          f"runs with share > 0.9: {np.mean(s > 0.9):.2f}")
s = urn(0.30, 0.20, ipw=True)
print(f"IPW, rates (0.3, 0.2): mean share of A {s.mean():.3f}, sd {s.std():.3f} (target 0.600)")

With equal rates the final shares have standard deviation 0.292, close to 1/\sqrt{12} \approx 0.289 for a uniform limit. With rates 0.3 and 0.2, 68 % of runs exceed a share of 0.9. The weighted rule gives 0.600 \pm 0.008 .

1.4 Energy and generative AI

Strubell et al. (2019) estimated such costs in natural language processing, where hyperparameter searches multiply them; see Artificial Intelligence and the Environment.

Generative AI sharpens every pillar: synthetic media to be marked (Article 50), training data that models memorise and leak (ch. 6.5), fluent but false content in decision support.

2. Fairness: legal notions, criteria and impossibility

Law defines discrimination by treatment and impact, statistics by conditional independence. This chapter connects the two and proves that the main criteria conflict.

2.1 Discrimination in law

EU law separates direct discrimination, less favourable treatment on a protected ground, from indirect discrimination: a neutral rule that puts a protected group at a particular disadvantage, unless objectively justified by a legitimate aim with appropriate and necessary means (Directive 2000/43/EC, Article 2). US law speaks of disparate treatment and disparate impact (Griggs v. Duke Power Co., 1971).

US agencies flag adverse impact when a group’s selection rate falls below 80 % of the highest group’s rate: the four-fifths rule of the 1978 Uniform Guidelines. In France, Article 225-1 of the Code pénal lists the protected criteria.

Article 9 of the GDPR restricts the processing of data such as ethnic origin or health, yet measuring discrimination requires them. Article 10(5) of the AI Act allows it, under safeguards, to detect and correct bias in high-risk systems.

2.2 Group criteria

Independence echoes the statistics of indirect discrimination; sufficiency asks that a score mean the same thing for everyone.

2.3 Unawareness is not enough

Removing A avoids direct discrimination but not indirect discrimination: postcode, employment history or vocabulary can encode the attribute.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 40_000
A = (rng.random(n) < 0.3).astype(int)                 # protected attribute, 30 % in group 1
Y = (rng.random(n) < np.where(A == 1, 0.3, 0.5)).astype(int)   # base rates 0.5 and 0.3
X = np.column_stack([1.5 * Y + rng.standard_normal(n),          # informative feature
                     -A + 0.5 * Y + rng.standard_normal(n)])    # proxy of A, weakly informative
tr, te = slice(0, n // 2), slice(n // 2, n)
clf = LogisticRegression().fit(X[tr], Y[tr])          # "unaware": A is not an input
yhat, a, y = clf.predict(X[te]), A[te], Y[te]
for g in (0, 1):
    m = a == g
    print(f"group {g}: selection {yhat[m].mean():.3f}  TPR {yhat[m & (y == 1)].mean():.3f}  "
          f"FPR {yhat[m & (y == 0)].mean():.3f}  PPV {y[m & (yhat == 1)].mean():.3f}")
r = [yhat[a == g].mean() for g in (0, 1)]
print(f"disparate impact ratio {min(r) / max(r):.3f}, accuracy {(yhat == y).mean():.3f}")

The model selects 49.7 % of group 0 and 28.2 % of group 1. The disparate impact ratio is 0.567 < 0.8, and TPR differ too, 0.776 against 0.653. Chapter 6.1 corrects this score.

2.4 Impossibility theorems

3. Privacy: from anonymisation to differential privacy

This chapter shows why removing identifiers fails, defines differential privacy as a limit on what any observer can infer, and builds private algorithms up to training.

3.1 Why removing names is not enough

Sweeney (2000) estimated that ZIP code, birth date and sex single out 87 % of the US population; Sweeney (2002) recounts her linkage of “anonymised” hospital records to a voter list. Narayanan and Shmatikov (2008) re-identified Netflix Prize users from a few dated ratings. Under Recital 26 of the GDPR, data still linkable to a person remain personal data.

k-anonymity fails against richer auxiliary data, and models leak too. Shokri et al. (2017) showed membership inference: deciding from a model whether a person was in its training set. Dinur and Nissim (2003) showed that too many accurate answers to counting queries let an attacker reconstruct almost the whole database.

3.2 Differential privacy

No assumption is made on the observer’s knowledge or computing power: the guarantee survives linkage with any other data.

3.3 Mechanisms

import numpy as np

rng = np.random.default_rng(0)
n, pi_true = 10_000, 0.12                       # 12 % of respondents hold the sensitive trait
x = rng.random(n) < pi_true

# randomized response with eps = ln 3: tell the truth with probability q = 3/4
eps = np.log(3); q = np.exp(eps) / (1 + np.exp(eps))
answers = np.where(rng.random(n) < q, x, ~x)
pi_hat = (answers.mean() - (1 - q)) / (2 * q - 1)
r = answers.mean()
se = np.sqrt(r * (1 - r) / n) / (2 * q - 1)
print(f"randomized response: q = {q:.2f}, estimate {pi_hat:.4f} +/- {1.96 * se:.4f} (truth {pi_true})")

# Laplace mechanism for a mean of incomes clipped to [0, 100 000]
incomes = np.clip(rng.lognormal(10.3, 0.6, size=1000), 0, 100_000)
sens = 100_000 / len(incomes)                   # sensitivity of the mean (replace one record)
for e in [0.1, 1.0]:
    noisy = incomes.mean() + rng.laplace(scale=sens / e, size=5)
    print(f"eps = {e}: true mean {incomes.mean():.0f}, noise sd {np.sqrt(2) * sens / e:.0f}, "
          f"five releases {np.round(noisy).astype(int)}")

Randomized response estimates 0.129 ± 0.018 for a true 0.12. The mean income of 1,000 people is released with noise of standard deviation 141 at \varepsilon = 1. Five releases cost 5\varepsilon by Proposition 3.7, since an observer can average them.

3.4 Private training: DP-SGD

Clipping bounds the \ell_2 sensitivity of the sum by C when one record is added or removed (2C if replaced), so each step is a Gaussian mechanism (Theorem 3.6). Subsampling amplifies its privacy, and the steps compose; ch. 6.3 computes the total budget.

import numpy as np

rng = np.random.default_rng(0)
n, d = 2000, 20
w_true = rng.standard_normal(d) / np.sqrt(d) * 3
X = rng.standard_normal((n, d))
y = (rng.random(n) < 1 / (1 + np.exp(-X @ w_true))).astype(float)
Xtr, ytr, Xte, yte = X[:1000], y[:1000], X[1000:], y[1000:]

def dp_sgd(sigma, C=1.0, q=0.02, T=1000, lr=0.2, seed=1):
    """Poisson-subsampled DP-SGD for logistic regression (Abadi et al., 2016)."""
    g = np.random.default_rng(seed)
    w, m = np.zeros(d), len(ytr)
    for _ in range(T):
        idx = np.flatnonzero(g.random(m) < q)                  # each record kept w.p. q
        p = 1 / (1 + np.exp(-Xtr[idx] @ w))
        grads = (p - ytr[idx])[:, None] * Xtr[idx]             # per-example gradients
        norms = np.maximum(np.linalg.norm(grads, axis=1) / C, 1.0)
        noisy = (grads / norms[:, None]).sum(0) + sigma * C * g.standard_normal(d)
        w -= lr * noisy / (q * m)                               # expected batch size q m
    return np.mean(((Xte @ w) > 0) == yte)

for sigma in [0.0, 1.0, 2.0, 4.0, 8.0]:
    print(f"noise multiplier {sigma}: test accuracy {dp_sgd(sigma):.3f}")
print(f"Bayes classifier: {np.mean(((Xte @ w_true) > 0) == yte):.3f}")

On 1,000 training records, test accuracy goes from 0.809 without noise to 0.783 at \sigma = 4 and 0.721 at \sigma = 8; the Bayes classifier reaches 0.822. Privacy costs accuracy, and more so on small datasets.

4. Transparency: interpretable models, explanations and recourse

This chapter states the legal duties of transparency, then builds two explanation tools with proofs: Shapley values and counterfactual explanations.

4.1 Transparency for whom?

An affected person needs reasons and a way to change the outcome; a deployer, instructions and known limits (AI Act, Article 13); an auditor, documentation and logs (ch. 5). The GDPR requires “meaningful information about the logic involved” in solely automated decisions (Articles 13–15 and 22). The AI Act adds a right to an explanation of a high-risk system’s role in a decision (Article 86).

Rudin (2019) argues that high-stakes decisions should use interpretable models whenever they are as accurate.

4.2 Shapley values

To explain f(x), SHAP (Lundberg and Lee, 2017) uses a game in which the features of S are fixed at the values of x. The interventional version draws the other features from their marginal law: v_x(S) = E\big[f(x_S, X_{\bar S})\big] - E[f(X)] .

import itertools
import math
import numpy as np

rng = np.random.default_rng(0)
d = 4
background = rng.standard_normal((2000, d))          # reference sample for E[f(x_S, X_rest)]

def shapley(f, x, bg):
    """Exact interventional Shapley values by enumeration of the 2^d coalitions."""
    def v(S):
        Z = bg.copy()
        Z[:, list(S)] = x[list(S)]                   # fix the coordinates in S to those of x
        return f(Z).mean()
    phi = np.zeros(d)
    for j in range(d):
        others = [k for k in range(d) if k != j]
        for r in range(d):
            for S in itertools.combinations(others, r):
                weight = math.factorial(r) * math.factorial(d - r - 1) / math.factorial(d)
                phi[j] += weight * (v(S + (j,)) - v(S))
    return phi, v(()), v(tuple(range(d)))

w = np.array([2.0, -1.0, 0.5, 0.0])
linear = lambda Z: Z @ w + 1.0
x = np.array([1.0, 2.0, -1.0, 3.0])
phi, v0, v1 = shapley(linear, x, background)
print("linear model, Shapley:", np.round(phi, 3))
print("w_j (x_j - mean_j):  ", np.round(w * (x - background.mean(0)), 3))

inter = lambda Z: Z[:, 0] * Z[:, 1] + np.maximum(Z[:, 2], 0)   # interaction and a kink
phi, v0, v1 = shapley(inter, x, background)
print("interaction model, Shapley:", np.round(phi, 3), f" sum {phi.sum():.3f} = f(x) - E f = {v1 - v0:.3f}")

The linear case matches Proposition 4.4 to three decimals. For the product x_1 x_2, each factor receives about half of the interaction (1.01 and 1.04), and the values sum to f(x) - E[f(X)] = 1.648. With correlated features, the interventional and conditional games differ (Janzing et al., 2020).

4.3 Counterfactual explanations and recourse

4.4 Limits of explanations

Post-hoc explanations can be manipulated: Slack et al. (2020) built biased classifiers whose LIME and SHAP explanations look innocuous. Disclosure also invites strategic gaming (Hardt, Megiddo, Papadimitriou and Wootters, 2016).

5. Accountability, human agency and context

This chapter turns accountability and human agency into duties, statistical audits and an optimal rule for deferring to a human.

5.1 Obligations and documentation

The AI Act assigns duties by role. Providers of high-risk systems run risk management (Article 9), data governance (Article 10), documentation and logging (Articles 11–12), human oversight (Article 14), and accuracy and robustness (Article 15). Deployers monitor use (Article 26), and some assess impacts on fundamental rights (Article 27). The GDPR adds accountability (Article 5(2)) and impact assessments (Article 35).

Model cards (Mitchell et al., 2019) and datasheets for datasets (Gebru et al., 2021) document intended use, data, metrics by group and known limits.

5.2 Auditing with statistics

Audits compare many subgroups, so their intervals need a multiple-testing correction. Gender Shades (Buolamwini and Gebru, 2018) found error rates of up to 34.7 % for darker-skinned women against at most 0.8 % for lighter-skinned men in commercial gender classifiers.

5.3 Human agency and oversight

People over-trust automation (Parasuraman and Riley, 1997). Article 14 of the AI Act asks that overseers understand the system, stay aware of automation bias, and can override or stop it. Article 22 of the GDPR gives a right to human intervention.

import numpy as np

rng = np.random.default_rng(0)
n = 100_000
eta = 1 / (1 + np.exp(-3.0 * rng.standard_normal(n)))   # true P(Y = 1 | x), known here
y = rng.random(n) < eta
conf, pred = np.maximum(eta, 1 - eta), eta >= 0.5
for c in [0.5, 0.3, 0.2, 0.1]:                           # cost of deferring to a human
    keep = conf >= 1 - c                                 # Chow's rule
    risk = np.mean(np.where(keep, pred != y, c))         # error cost 1, deferral cost c
    print(f"c = {c:.1f}: automated share {keep.mean():.3f}, "
          f"error when automated {np.mean(pred[keep] != y[keep]):.3f}, total risk {risk:.3f}")

At c = 0.2 the machine decides 64.5 % of cases with error 0.066, and total risk falls from 0.162 to 0.113. The rule helps only if the human beats the machine on deferred cases.

5.4 Three sectors

• Healthcare. Obermeyer et al. (2019) found that an algorithm using health costs as a proxy for needs underrated Black patients’ needs; remedying this would raise the share of Black patients receiving extra help from 17.7 % to 46.5 %.
• Finance. Credit scoring is high-risk under Annex III of the AI Act; a refused applicant needs reasons and recourse (ch. 4.3).
• Criminal justice. Article 5 of the AI Act bans predicting offences solely from profiling. In France, Article 33 of Law 2019-222 forbids reusing judges’ identities to predict their practices.

6. Complements: fair learning, privacy accounting, federated learning, generative AI

This self-study chapter extends chapters 2 and 3 in the directions a project or a research internship needs. It covers correcting an unfair score, the causal view of fairness, tight privacy budgets, federated and local privacy, generative models and the right to erasure.

6.1 Correcting a score: post-, pre- and in-processing

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 40_000
A = (rng.random(n) < 0.3).astype(int)                 # protected attribute, 30 % in group 1
Y = (rng.random(n) < np.where(A == 1, 0.3, 0.5)).astype(int)   # base rates 0.5 and 0.3
X = np.column_stack([1.5 * Y + rng.standard_normal(n),          # informative feature
                     -A + 0.5 * Y + rng.standard_normal(n)])    # proxy of A, weakly informative
tr, te = slice(0, n // 2), slice(n // 2, n)
clf = LogisticRegression().fit(X[tr], Y[tr])          # "unaware": A is not an input
s, a, y = clf.predict_proba(X[te])[:, 1], A[te], Y[te]

def report(name, yhat):
    print(name)
    for g in (0, 1):
        m = a == g
        sel, tpr = yhat[m].mean(), yhat[m & (y == 1)].mean()
        fpr, ppv = yhat[m & (y == 0)].mean(), y[m & (yhat == 1)].mean()
        print(f"  group {g}: selection {sel:.3f}  TPR {tpr:.3f}  FPR {fpr:.3f}  PPV {ppv:.3f}")
    r = [yhat[a == g].mean() for g in (0, 1)]
    print(f"  disparate impact ratio {min(r) / max(r):.3f}, accuracy {(yhat == y).mean():.3f}")

report("single threshold 0.5:", (s >= 0.5).astype(int))
# equal opportunity by post-processing: per-group threshold giving TPR = 0.80
t = {g: np.quantile(s[(a == g) & (y == 1)], 0.20) for g in (0, 1)}
print(f"group thresholds: {t[0]:.3f}, {t[1]:.3f}")
report("group thresholds (TPR = 0.80 in both groups):", (s >= np.where(a == 1, t[1], t[0])).astype(int))

Thresholds of 0.467 for group 0 and 0.336 for group 1 equalise TPR at 0.80. The FPR become close, 0.231 and 0.226, so equalised odds almost holds too. PPV stays apart, 0.782 against 0.607, as Theorem 2.6 predicts. The disparate impact ratio rises from 0.567 to 0.770, and accuracy moves from 0.792 \ \text{to} \ 0.784 .

The next theorem shows that group thresholds are not an accident: they are optimal under a parity constraint.

import numpy as np

rng = np.random.default_rng(0)
n, c = 200_000, 0.5                                  # utility of a positive decision: eta - c
A = rng.random(n) < 0.4
eta = np.where(A, rng.beta(3, 5, n), rng.beta(5, 4, n))   # P(Y = 1 | x) by group

def utility(d):
    return np.mean(d * (eta - c))

d_star = eta > c                                     # unconstrained optimum
print(f"unconstrained: utility {utility(d_star):.4f}, selection rates "
      f"{d_star[~A].mean():.3f} / {d_star[A].mean():.3f}")

best = None
for r in np.linspace(0.05, 0.95, 181):               # common selection rate r in both groups
    d = np.zeros(n, bool)
    for g in (False, True):
        m = A == g
        d[m] = eta[m] >= np.quantile(eta[m], 1 - r)  # group threshold: top r of the group
    if best is None or utility(d) > best[0]:
        best = (utility(d), r)
print(f"parity with group thresholds: utility {best[0]:.4f} at common rate {best[1]:.3f}")
# same rates, but one threshold for group 0 and a lottery in group 1
r = best[1]
d = np.where(A, rng.random(n) < r, eta >= np.quantile(eta[~A], 1 - r))
print(f"parity with a lottery in group 1: utility {utility(d):.4f}")

Without constraint, the optimal rule selects 63.6 % of group 0 and 22.8 % of group 1, with utility 0.0668. Parity with group thresholds keeps 0.0570, at a common rate of 0.46. Reaching the same rates with a lottery in group 1 keeps only 0.0305. The price of a fairness constraint depends on how it is met.

Pre-processing changes the data before training.

In-processing adds a constraint or a penalty to training. Agarwal et al. (2018) reduce fair classification under linear constraints to a sequence of cost-sensitive problems, a saddle-point method on the Lagrangian of ch. 0.6. Zemel et al. (2013) learn representations that keep information about Y while hiding A.

6.2 Causal fairness

Group criteria compare distributions. Causal criteria ask what would have happened to the same person under another value of A.

6.3 Privacy accounting

The \delta term of Definition 3.2 comes from Dwork, Kenthapadi, McSherry, Mironov and Naor (2006). Basic composition (Proposition 3.7) adds the \varepsilon of every step, which is far too pessimistic for thousands of SGD steps.

For small \varepsilon the budget grows like \sqrt{k} rather than k. Rényi differential privacy tracks this growth more tightly.

These accountants use the add-or-remove notion of neighbours, standard for DP-SGD; a replacement is a removal followed by an addition. Subsampling helps further. For pure DP, running an \varepsilon-DP mechanism on a Poisson subsample of rate q is \log(1 + q(e^{\varepsilon} - 1))-DP (Balle, Barthe and Gaboardi, 2018), roughly q\varepsilon. For the Gaussian mechanism with noise multiplier \sigma and integer \alpha, Mironov, Talwar and Zhang (2019) give the RDP of one subsampled step: \rho(\alpha) = \frac{1}{\alpha - 1} \log \sum_{k = 0}^{\alpha} \binom{\alpha}{k} (1 - q)^{\alpha - k} q^k \exp\!\Big(\frac{k^2 - k}{2\sigma^2}\Big).

import numpy as np
from scipy.special import gammaln, logsumexp

def rdp_sampled_gaussian(q, sigma, alpha):
    """RDP of one step of the Poisson-subsampled Gaussian mechanism, integer alpha >= 2."""
    k = np.arange(alpha + 1)
    log_binom = gammaln(alpha + 1) - gammaln(k + 1) - gammaln(alpha - k + 1)
    terms = log_binom + (alpha - k) * np.log1p(-q) + k * np.log(q) + (k * k - k) / (2 * sigma**2)
    return logsumexp(terms) / (alpha - 1)

def epsilon(q, sigma, T, delta, alphas=range(2, 257)):
    """Compose T steps in RDP, then convert to (eps, delta)-DP (Mironov, 2017)."""
    return min(T * rdp_sampled_gaussian(q, sigma, a) + np.log(1 / delta) / (a - 1) for a in alphas)

q, T, delta = 0.02, 1000, 1e-5
for sigma in [1.0, 2.0, 4.0, 8.0]:
    # without subsampling, each step is a Gaussian mechanism with RDP alpha / (2 sigma^2)
    no_sub = min(T * a / (2 * sigma**2) + np.log(1 / delta) / (a - 1) for a in range(2, 257))
    print(f"sigma = {sigma}: eps = {epsilon(q, sigma, T, delta):6.2f} with subsampling q = {q}, "
          f"{no_sub:7.1f} without")

For the DP-SGD runs of ch. 3.4, with q = 0.02, 1,000 steps and \delta = 10^{-5}, the budgets are 4.95, 1.75, 0.80 and 0.39 for \sigma = 1, 2, 4, 8. Without subsampling they would be 1011.5, 261.5, 74.0 and 27.1. So the model with test accuracy 0.783 is, for add-or-remove neighbours, (0.80,\ 10^{-5})\text{-differentially private}.

6.4 Federated learning and local privacy

Raw data never leave the clients, but updates leak: Zhu, Liu and Han (2019) reconstruct training images from shared gradients. Secure aggregation (Bonawitz et al., 2017) lets the server see only the sum of the updates, and adding noise to that sum gives central differential privacy. Without a trusted aggregator, each client must add noise itself: this is local differential privacy, as in randomized response.

Duchi, Jordan and Wainwright (2013) show that the local rate cannot be improved beyond constants: local privacy costs a factor \sqrt{n} in accuracy. Shuffling the local reports before analysis recovers part of it (Erlingsson et al., 2019).

import numpy as np

rng = np.random.default_rng(0)
eps, reps = 1.0, 2000
for n in [100, 10_000]:
    x = rng.random(n)                                       # values in [0, 1]
    central = rng.laplace(scale=1 / (n * eps), size=reps)   # curator adds noise to the mean
    local = rng.laplace(scale=1 / eps, size=(reps, n)).mean(1)   # each user perturbs x_i
    print(f"n = {n:6d}: RMSE central {np.sqrt(np.mean(central**2)):.5f} "
          f"(theory {np.sqrt(2) / (n * eps):.5f}), local {np.sqrt(np.mean(local**2)):.5f} "
          f"(theory {np.sqrt(2 / n) / eps:.5f})")

With 10,000 people and \varepsilon = 1, the central error is 0.00014 and the local error 0.0139, a factor of 100, which is \sqrt{n}.

6.5 Generative AI

Large generative models memorise. Carlini et al. (2021) extracted hundreds of verbatim training sequences from GPT-2, including names and contact details. Training or fine-tuning with DP-SGD bounds such extraction through Theorem 3.3, at a cost in quality that the accounting of ch. 6.3 quantifies.

The AI Act adds duties for general-purpose models. Providers keep technical documentation, inform downstream providers, adopt a copyright policy and publish a summary of the training content (Article 53). Models trained with more than 10^{25} floating-point operations are presumed to carry systemic risk (Article 51). They must be evaluated and adversarially tested, and serious incidents reported (Article 55). Outputs of generative systems must be marked as artificially generated in a machine-readable way (Article 50).

6.6 Erasure and machine unlearning

The GDPR limits data to their stated purpose and to what is necessary (Article 5), and gives a right to erasure (Article 17). For a trained model, exact erasure means obtaining the model that training without the record would have produced; retraining from scratch achieves it at full cost.

With S = 20 and k = 10 deletions, about 8.03 shards are retrained, 40 % of the data instead of all of it. Aggregating weaker per-shard models may cost accuracy: the trade-off is between erasure cost and quality.

7. Research directions and open problems

This self-study chapter maps active questions for each pillar. Treat every item as a pointer to read, not a settled result; each comes with a paper whose code is usually public.

7.1 Fairness beyond fixed groups

• Many overlapping groups. Hébert-Johnson, Kim, Reingold and Rothblum (2018) define multicalibration: calibration on every subgroup that a class of functions can identify. It interpolates between group criteria and individual guarantees.
• Unknown groups. Hashimoto, Srivastava, Namkoong and Liang (2018) protect every group above a given size without observing group labels, through distributionally robust optimisation.
• Dynamics. Liu, Dean, Rolf, Simchowitz and Hardt (2018) show that a fairness constraint can harm the protected group in the long run, through its effect on future scores. Perdomo, Zrnic, Mendler-Dünner and Hardt (2020) formalise predictions that change the distribution they predict (performative prediction), the general setting of the feedback loops of ch. 1.3.

7.2 Privacy

• Auditing the guarantee. Jagielski, Ullman and Oprea (2020), and Nasr, Song, Thakurta, Papernot and Carlini (2021), measure the privacy actually achieved by DP-SGD with crafted membership attacks, which gives empirical lower bounds to set against the accountant of ch. 6.3.
• Better accounting. Gaussian differential privacy (Dong, Roth and Su, 2022) describes privacy by a whole trade-off curve between the errors of the test of Theorem 3.3. Privacy amplification by iteration (Feldman, Mironov, Talwar and Thakurta, 2018) protects records that are processed early, when only the last model is released.
• Decentralised learning. Cyffers and Bellet (2022) show that in decentralised algorithms, where each user only sees what passes through their neighbours, privacy can be amplified compared with local privacy. How far this extends to realistic graphs and attacks is open.
• Privacy and fairness interact. Bagdasaryan, Poursaeed and Shmatikov (2019) found that DP-SGD reduces accuracy most on under-represented groups, because clipping and noise hit rare patterns hardest.

7.3 Transparency and accountability

• Limits of attribution. Bilodeau, Jaques, Koh and Kim (2024) prove that common feature-attribution methods, SHAP among them, can fail to beat random guessing at inferring simple model behaviour on rich model classes.
• Human–AI teams. Madras, Pitassi and Zemel (2018) learn when a model should defer to a human, beyond Chow’s rule. Bansal et al. (2021) found that AI explanations increased the chance that people accept a recommendation, whether or not it was correct. Which explanations help oversight, rather than trust, is open.
• Audit frameworks. Raji et al. (2020) propose an end-to-end internal audit process. Turning the documents of ch. 5.1 into checkable evidence for the conformity assessments of the AI Act is an active area.
• Footprint. Luccioni, Viguier and Ligozat (2023) estimate the life-cycle emissions of the BLOOM language model, a template for reporting the cost of ch. 1.4.

7.4 Open problems

1. Give a fairness criterion that remains satisfied over time under the feedback loops of ch. 1.3.
2. Close the gap between the privacy budgets proved for DP-SGD and the leakage measured by the best attacks.
3. Design explanations that provably help a human overseer decide when to override a model (ch. 5.3).
4. Make machine unlearning verifiable by a third party, without retraining from scratch.
5. Train generative models with meaningful privacy for every person in a web-scale corpus.

Exercises

1. ★ Base rates and positive predictive value. A screening test has TPR 0.95 and FPR 0.05. (a) Compute its PPV and its selection rate where the prevalence is 2 %, then where it is 10 %. (b) What does this say about using one alert threshold for two hospitals with different case mixes?

   Solution: (a) By Proposition 0.2, at 2 % the PPV is 0.019 / (0.019 + 0.049) \approx 0.279 and the selection rate 0.068. At 10 % the PPV is 0.095/0.14 \approx 0.679 and the selection rate 0.14.

   (b) The error rates are the same, but an alert means something very different: about 28 % against 68 % of alerts are true cases. Staff who learn to trust alerts in one hospital will over-trust them in the other.

2. ★ Group metrics and the four-fifths rule. Group a has TP = 300, FN = 100, FP = 100, TN = 500; group b has TP = 150, FN = 50, FP = 50, TN = 750. (a) Compute base rates, selection rates, TPR, FPR and PPV. (b) Which criteria of Definition 2.2 hold? (c) Check Theorem 2.6 on these numbers.

   Solution: (a) Base rates 0.4 and 0.2; selection rates 0.4 and 0.2; TPR 0.75 in both; FPR 100/600 \approx 0.167 and 50/800 = 0.0625; PPV 0.75 in both.

   (b) Equal opportunity and predictive parity hold; separation fails (FPR differ) and independence fails, with a disparate impact ratio of 0.5, far below 0.8.

   (c) For group b, (0.2/0.8)(0.25/0.75)(0.75) = 0.0625; for group a, (0.4/0.6)(1/3)(0.75) \approx 0.167. Equal PPV and FNR with different base rates force different FPR.

3. ★ The energy of hyperparameter search. One training run uses 8 GPUs at 0.4 kW for 72 h in a data centre of PUE 1.3. (a) Compute its energy. (b) A search runs 50 such trainings. Compute the total energy and emissions at 0.06 and at 0.5 kg CO2e per kWh. (c) What should a paper report?

   Solution: (a) By Example 1.7, 8 \times 0.4 \times 72 \times 1.3 = 299.5 kWh. (b) 14,976 kWh, that is 899 kg CO2e at 0.06 and 7,488 kg at 0.5. The final run alone is only 18 kg or 150 kg.

   (c) The whole search, the hardware, the location and the grid intensity: reporting only the final run understates the cost fiftyfold, a point made by Strubell et al. (2019).

4. ★ Randomized response in a survey. Respondents tell the truth with probability q = 0.8. (a) Which \varepsilon does this give? (b) Among 2,500 answers, 38 % say yes. Estimate the true proportion with a 95 % interval. (c) Compare the width with a direct survey of the same size.

   Solution: (a) \varepsilon = \ln(0.8/0.2) = \ln 4 \approx 1.386. (b) By Proposition 3.4, \hat\pi = (0.38 - 0.2)/0.6 = 0.30. Its standard error is \sqrt{0.38 \times 0.62 / 2500}/0.6 \approx 0.0162, so the interval is 0.30 \pm 0.032.

   (c) A direct survey has standard error \sqrt{0.21/2500} \approx 0.0092: privacy widens the interval by a factor of about 1.77, or needs about 3.1 times more respondents for the same width.

5. ★ A count released several times. A count of patients is released with the Laplace mechanism at \varepsilon = 0.5. (a) Give the noise scale and standard deviation, and the probability that the error exceeds 5. (b) The same count is released 10 times with fresh noise. What is the total privacy loss, and what can an attacker do? (c) How should 10 releases be budgeted?

   Solution: (a) The sensitivity is 1, so b = 2, the standard deviation is 2\sqrt{2} \approx 2.83, and P(|Z| > 5) = e^{-2.5} \approx 0.082.

   (b) By Proposition 3.7, 5-DP. Averaging the releases gives standard deviation 2.83/\sqrt{10} \approx 0.89, so the count is almost exact. (c) Either release once and reuse the result (post-processing is free), or give each release \varepsilon = 0.05, that is b = 20 and standard deviation 28.3.

6. ★ Shapley values, correlated features. (a) For f(x) = 2x_1 - x_2 + 0.5x_3 + b, with feature means (0, 1, 2) and x = (1, 3, 0), compute the interventional Shapley values and check efficiency.

   (b) Now f(x) = 2x_1, with X_2 = X_1 exactly, E[X_1] = 0 and x = (1, 1). Compute the interventional and the conditional Shapley values.

   Solution: (a) By Proposition 4.4, \phi = (2, -2, -1), and f(x) - E[f(X)] = (b - 1) - b = -1, which is their sum.

   (b) Interventional: \phi = (2, 0), since f ignores x_2. Conditional: v(\{1\}) = v(\{2\}) = v(\{1, 2\}) = 2 and v(\emptyset) = 0, so \phi = (1, 1). The conditional game credits an unused feature that merely correlates with a used one (Janzing et al., 2020).

7. ★ Recourse with several costs. In Example 4.7, (a) compute the minimal \ell_2 change if age were actionable; (b) compute it if only debt is actionable; (c) show that, for the \ell_1 cost, the cheapest change moves a single feature, and find it.

   Solution: (a) \|w\|_2^2 = 2.33, so \delta^* = (0.206, -0.309, 0.129) with cost 0.6/\sqrt{2.33} \approx 0.393. (b) \delta^* = (0, -0.5, 0), cost 0.5.

   (c) By Hölder, w_M^\top \delta \leq \|w_M\|_\infty \|\delta\|_1, so acceptance needs \|\delta\|_1 \geq 0.6/\|w_M\|_\infty. Moving only the feature of largest |w_j|, debt, attains it: lower debt by 0.5. The \ell_1 cost gives sparse, easier-to-act-on advice.

8. ★ An audit with small samples. A hiring tool selects 45 % of 200 applicants of group a and 33 % of 120 applicants of group b. (a) Give 95 % intervals for the difference and for the ratio (Proposition 5.2). (b) What can the auditor conclude? (c) How many applicants per group would make the half-width of the log-ratio interval 0.1, at these rates?

   Solution: (a) Difference 0.12 \pm 0.109, that is [0.011, 0.229]. Ratio 0.733, with log-ratio standard error 0.152, so the interval is [0.545, 0.987].

   (b) The gap is significant at 5 %, but the interval for the ratio contains 0.8: the four-fifths rule is inconclusive. (c) n = 1.96^2 (0.55/0.45 + 0.67/0.33)/0.1^2 \approx 1250 per group.

9. ★★ The Pólya urn. (a) Prove the martingale identity of Proposition 0.8. (b) With a_0 = b_0 = 1 and equal rates, show that the first n discoveries are all in A with probability 1/(n + 1). (c) Check the standard deviation 0.292 of the simulation in ch. 1.3. (d) With a_0 = 10 and b_0 = 1, compute P(X_\infty < 1/2) and interpret.

   Solution: (a) See step 1 of the proof of Proposition 0.8. (b) The probability is the product \frac{1}{2} \cdot \frac{2}{3} \cdots \frac{n}{n + 1} = \frac{1}{n + 1}.

   (c) \mathrm{Beta}(1, 1) is uniform, with standard deviation 1/\sqrt{12} \approx 0.289. (d) P(\mathrm{Beta}(10, 1) < 1/2) = 0.5^{10} \approx 0.001: although the true rates are equal, past records almost surely keep most patrols in A for ever.

10. ★★ Independence against separation. (a) Extend Theorem 2.5 to any number of groups. (b) A classifier has TPR 0.8 and FPR 0.1 in two groups with base rates 0.5 and 0.3; compute the selection rates and the disparate impact ratio. (c) Describe all decisions that satisfy both criteria.

    Solution: (a) Apply the two-group argument to each pair of groups with distinct base rates.

    (b) By Proposition 0.2, 0.1 + 0.7 \times 0.5 = 0.45 and 0.1 + 0.7 \times 0.3 = 0.31, a ratio of 0.689. (c) Exactly the decisions with \mathrm{TPR} = \mathrm{FPR} in every group, such as a lottery with the same rate for everyone: they carry no information on Y.

11. ★★ Calibration and balance. (a) Prove that a calibrated score satisfies E[S \mid A = a] = p_a. (b) Prove the identity of step 2 of Theorem 2.7. (c) A calibrated score has E[S \mid Y = 1] = 0.7 in two groups with base rates 0.5 and 0.3. Compute E[S \mid Y = 0] in each, and say which balance condition fails.

    Solution: (a) Calibration within group a gives E[Y \mid S, A = a] = S; take expectations. (b) Expand E[S \mid A = a] over Y and use (a).

    (c) x = p(1 - y)/(1 - p) gives 0.5 \times 0.3/0.5 = 0.3 and 0.3 \times 0.3/0.7 \approx 0.129. Balance for the negative class fails: negatives of the first group get higher scores on average.

12. ★★ Composition, post-processing, group privacy. (a) Prove that a randomised post-processing g preserves (\varepsilon, \delta)-DP. (b) Prove that an \varepsilon-DP algorithm protects groups of k records with k\varepsilon. (c) A census release is 0.5-DP per person; what does it guarantee for a household of four?

    Solution: (a) Write g(z) = h(z, W) with W independent of the data. For fixed W = w, the set \{z : h(z, w) \in R\} gives the inequality of Definition 3.2; integrating over w preserves it.

    (b) Datasets differing in k records are linked by a chain of k neighbours; multiply the k ratio bounds. (c) A guarantee of \varepsilon = 2 for the household: odds about it can move by a factor e^2 \approx 7.4.

13. ★★ Differential privacy as a test (Python). (a) Apply Theorem 3.3 to a set R and to its complement to show that every test satisfies \mathrm{TPR} - \mathrm{FPR} \leq \frac{e^{\varepsilon} - 1 + 2\delta}{e^{\varepsilon} + 1}. (b) Check both bounds by simulation for a count released with the Laplace mechanism at \varepsilon = 0.5. (c) Explain why the first bound is an equality for thresholds above the larger count.

    Solution: (a) Definition 3.2, with D and D' exchanged and applied to the complement of R, gives 1 - \mathrm{FPR} \leq e^{\varepsilon}(1 - \mathrm{TPR}) + \delta . Add it to \mathrm{TPR} \leq e^{\varepsilon}\mathrm{FPR} + \delta and rearrange.

    import numpy as np
    
    rng = np.random.default_rng(0)
    eps, m = 0.5, 1_000_000
    out_D = 100 + rng.laplace(scale=1 / eps, size=m)      # count on D
    out_D1 = 101 + rng.laplace(scale=1 / eps, size=m)     # count on D' (one more record)
    gap, adv = -np.inf, -np.inf
    for t in np.linspace(95, 106, 45):                    # tests "guess D' if output > t"
        fpr, tpr = np.mean(out_D > t), np.mean(out_D1 > t)
        gap, adv = max(gap, tpr - np.exp(eps) * fpr), max(adv, tpr - fpr)
    print(f"max over tests of TPR - e^eps FPR: {gap:.4f}  (Theorem 3.3: <= 0)")
    print(f"best advantage TPR - FPR: {adv:.4f}  (bound (e^eps - 1)/(e^eps + 1) = "
          f"{(np.exp(eps) - 1) / (np.exp(eps) + 1):.4f})")

    (b) The first maximum is 0.0000, up to sampling noise. The best advantage is 0.221, below the bound 0.245; it equals the total-variation distance 1 - e^{-0.25} of the two Laplace laws.

    (c) For t \geq 101, both tails are exponential with the same rate, and their ratio is exactly e^{\varepsilon}: the Laplace mechanism makes Theorem 3.3 tight.

14. ★★ Membership inference and overfitting (Python). (a) Show that the attack “member if and only if the model classifies the example correctly” has advantage TPR − FPR equal to the gap between training and test accuracy (Yeom et al., 2018). (b) Measure this gap, and the advantage of a margin-threshold attack, for logistic regression with weak to strong regularisation.

    Solution: (a) TPR is the probability that a member is classified correctly, the training accuracy; FPR is the same for a non-member, the test accuracy.

    import numpy as np
    from sklearn.linear_model import LogisticRegression
    from sklearn.metrics import roc_curve
    
    rng = np.random.default_rng(0)
    d, n = 100, 200
    w = rng.standard_normal(d) / np.sqrt(d)
    def sample(k):
        X = rng.standard_normal((k, d))
        return X, (rng.random(k) < 1 / (1 + np.exp(-2 * X @ w))).astype(int)
    Xin, yin = sample(n)                                  # members: the training set
    Xout, yout = sample(n)                                # non-members, same distribution
    
    for C in [10.0, 0.1, 0.001]:                          # weak to strong regularisation
        clf = LogisticRegression(C=C, max_iter=5000).fit(Xin, yin)
        acc_in, acc_out = clf.score(Xin, yin), clf.score(Xout, yout)
        # attack score: signed margin of the true label (high = looks like a member)
        s_in = (2 * yin - 1) * clf.decision_function(Xin)
        s_out = (2 * yout - 1) * clf.decision_function(Xout)
        fpr, tpr, _ = roc_curve(np.r_[np.ones(n), np.zeros(n)], np.r_[s_in, s_out])
        print(f"C = {C:6.3f}: train acc {acc_in:.2f}, test acc {acc_out:.2f}, "
              f"gap {acc_in - acc_out:.2f}, best margin-threshold advantage {np.max(tpr - fpr):.2f}")

    (b) The gaps are 0.32, 0.21 and 0.09, and the best margin thresholds reach 0.36, 0.21 and 0.14; these thresholds are tuned on the evaluation set, so they are optimistic. Overfitting leaks membership. Yeom et al. (2018) also show that \varepsilon-DP training bounds the advantage by e^{\varepsilon} - 1.

15. ★★ Deferral with asymmetric costs. A false negative costs c_{\mathrm{FN}}, a false positive c_{\mathrm{FP}} and a deferral c_d. (a) Derive the optimal rule, generalising Theorem 5.4. (b) When does it ever defer? (c) Apply it to c_{\mathrm{FN}} = 5, c_{\mathrm{FP}} = 1 and c_d = 0.5.

    Solution: (a) Given x, the three costs are \eta c_{\mathrm{FN}} (predict 0), (1 - \eta) c_{\mathrm{FP}} (predict 1) and c_d; choose the smallest. Deferral wins when \frac{c_d}{c_{\mathrm{FN}}} < \eta < 1 - \frac{c_d}{c_{\mathrm{FP}}}.

    (b) This interval is non-empty if and only if c_d < \frac{c_{\mathrm{FP}}\, c_{\mathrm{FN}}}{c_{\mathrm{FP}} + c_{\mathrm{FN}}}. (c) Predict 0 for \eta \leq 0.1, defer for 0.1 < \eta < 0.5, predict 1 for \eta \geq 0.5; indeed 0.5 < 5/6.

16. ★★ Reweighing. In the population of Example 2.4, P(A = 1) = 0.3, with base rates 0.5 in group 0 and 0.3 in group 1. (a) Compute the four weights of Proposition 6.4. (b) Check that both groups have base rate 0.44 after reweighing. (c) Does a classifier trained on the reweighted data satisfy independence?

    Solution: (a) P(Y = 1) = 0.44; the weights are w(0, 1) = 0.88, w(0, 0) = 1.12, w(1, 1) \approx 1.467 and w(1, 0) = 0.8.

    (b) Group 1: 0.09 \times 1.467 = 0.132 and 0.21 \times 0.8 = 0.168, so the base rate is 0.132/0.3 = 0.44; group 0: 0.308/0.7 = 0.44.

    (c) Not necessarily: reweighing makes the training labels independent of A, but the classifier still uses proxies in X; independence of \hat Y must be measured on test data.

17. ★★★ Local, central and shuffled privacy. (a) As in Proposition 6.12, derive the error of randomized response on bits as a function of \varepsilon and n. (b) For one million users and \varepsilon = 1, compare central and local errors for a mean. (c) Read Erlingsson et al. (2019) and explain how shuffling changes the trust model, and what it buys.

    Hints and expected result: (a) The variance of \hat\pi is r(1 - r)/(n(2q - 1)^2), and 2q - 1 = \tanh(\varepsilon/2) \approx \varepsilon/2 for small \varepsilon, so the error scales like 1/(\varepsilon\sqrt{n}).

    (b) Central 1.4 \times 10^{-6}, local 1.4 \times 10^{-3}: a factor 1,000. (c) A shuffler that hides who sent which report turns \varepsilon_0-local reports into a central guarantee that improves with n; the trust moves from the server to the shuffler. Expected: a one-page summary with the amplification statement and its assumptions.

18. ★★★ A contextual review. A hospital wants a sepsis alert trained on electronic health records. Write the two-page review of Method 5.5, with a model card (Mitchell et al., 2019).

    Hints and expected result: check the label (billing codes or a clinical definition: measurement bias, ch. 1.2); choose a criterion (equal TPR across groups is natural when missing a case is the costly error, ch. 2); privacy (health data fall under Article 9 of the GDPR, so an impact assessment under Article 35 is needed; consider DP-SGD, ch. 3.4).

    Then explanation for clinicians (ch. 4), and oversight with the asymmetric deferral rule of Exercise 15. Law: medical software can fall under Regulation (EU) 2017/745 on medical devices, and so be high-risk under Article 6(1) of the AI Act. A good answer states each choice, its alternative and its cost.

Projects and assessment

The official assessment differs between the two sources. master-mva.com announces an end-of-term exam; the Université Paris-Saclay course detail lists participation, continuous assessment and a project, of type “Evaluation Continue non Intégrale”. Neither displays weights. The proposals below cover both: project subjects sized for a short course, most of them paper studies in the MVA tradition, and an exam proposal that mixes a course question, a computation and proofs.

Solo projects

1. [Solo · ≈ 20 h] Paper study: “Equality of opportunity in supervised learning” (Hardt, Price and Srebro, 2016). Can post-processing equalise error rates on real census data, and at what cost?
   • Data: the ACSIncome task of Folktables (Ding et al., 2021) for one US state, with sex or race as A.
   • Train a score, draw the ROC curve of each group and its convex hull, and solve the linear program of Proposition 6.1 for equalised odds and for equal opportunity.
   • Success criterion: the figure of the feasible region, and group TPR within 0.01 of each other on test data, with intervals. Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters 2 and 6.1.
2. [Solo · ≈ 20 h] Paper study: “Deep learning with differential privacy” (Abadi et al., 2016), with an audit. How much accuracy does a privacy budget cost, and how tight is the budget?
   • Implement DP-SGD (Algorithm 3.8) in PyTorch for a small network on MNIST or on the scikit-learn digits, with the accountant of ch. 6.3.
   • Plot test accuracy against \varepsilon for at least four budgets, then run the loss-threshold membership attack of Yeom et al. (2018) on each model.
   • Success criterion: every measured advantage below the bound of Exercise 13. Deliverables: code, report, talk. Chapters 3 and 6.3.
3. [Solo · ≈ 20 h] Paper study: “Actionable recourse in linear classification” (Ustun, Spangher and Liu, 2019). Who can change a refusal?
   • Data: the Statlog German Credit data of the UCI repository. Fit a logistic score, mark immutable features, and compute minimal recourse with Proposition 4.6, then with integer constraints using scipy.optimize.milp.
   • Measure the share of refused applicants with feasible recourse, and the cost distribution in each group.
   • Success criterion: a table of recourse rates and costs by group, with one documented case. Deliverables: code, report, talk. Chapters 2 and 4.

Group projects

1. [Group of 2–3 · ≈ 30 h each] Paper study: “Runaway feedback loops in predictive policing” (Ensign et al., 2018), extended to recommendation.
   • Member 1: the urn of Definition 1.3; check Theorem 1.4 and Proposition 1.5 over 1,000 runs and measure the speed of runaway.
   • Member 2: the inverse-propensity correction of ch. 1.3, its variance, and the role of the exploration floor.
   • Member 3: a recommendation loop on synthetic users, measuring how concentrated exposure becomes (Gini index) with and without correction.
   • Success criterion: a corrected allocation within 0.02 of the proportional share, and a measured reduction of exposure concentration. Deliverables: code, a 6–8 page report, a 20-minute talk. Chapters 0 and 1.
2. [Group of 2–3 · ≈ 30 h each] An end-to-end fairness audit, with documentation. Data: the ACSEmployment or ACSIncome task of Folktables (Ding et al., 2021).
   • Member 1: a datasheet (Gebru et al., 2021), base rates, and the proxies of A (ch. 2.3).
   • Member 2: metrics by group and by intersectional subgroup, with the intervals of Proposition 5.2 and a multiple-testing correction.
   • Member 3: mitigation by reweighing (Proposition 6.4), by the reductions approach of Agarwal et al. (2018) and by post-processing (Proposition 6.1), then a model card (Mitchell et al., 2019).
   • Success criterion: every criterion of Definition 2.2 reported with intervals before and after mitigation, the accuracy cost, and documents a non-specialist can read. Deliverables: code, report, talk. Chapters 2, 5 and 6.1.
3. [Group of 2–3 · ≈ 30 h each] Federated learning with privacy, and an attack. Split the scikit-learn digits among 20 clients with skewed label distributions.
   • Member 1: federated averaging (Algorithm 6.11) and its accuracy against centralised training.
   • Member 2: client-level privacy: clip each update, add Gaussian noise to the sum, and account for it with ch. 6.3.
   • Member 3: the gradient-inversion attack of Zhu, Liu and Han (2019) on a small network, with and without noise.
   • Success criterion: an accuracy–\varepsilon curve, and reconstruction error as a function of the noise, showing the budget below which reconstruction fails. Deliverables: code, report, talk. Chapters 3, 6.3 and 6.4.

Written exam (proposal)

Two hours, no documents, a non-programmable calculator allowed; four questions of 5 points. master-mva.com announces an end-of-term exam without giving its format, so this is a proposal.

1. Question (5 points). (a) Define direct and indirect discrimination in EU law, and relate each to a statistical criterion or a practice of chapter 2. (b) State independence, separation and sufficiency. (c) Describe the risk levels of the AI Act, with one high-risk system and one prohibited practice. Expected: ch. 2.1 and Definition 2.2; removing A addresses direct discrimination, while independence measures indirect discrimination; ch. 1.1 and ch. 5.4, for instance credit scoring (Annex III) and predicting offences solely from profiling (Article 5).

2. Question (5 points). Group a: TP = 240, FN = 60, FP = 70, TN = 630. Group b: TP = 80, FN = 20, FP = 90, TN = 810. (a) Compute base rates, TPR, FPR, PPV and selection rates. (b) Which criteria hold? Check Theorem 2.6. (c) The count of positives in group b is released at \varepsilon = 0.2; give the noise scale and standard deviation.

   Expected: (a) base rates 0.3 and 0.1; TPR 0.8 and FPR 0.1 in both; PPV 240/310 \approx 0.774 and 80/170 \approx 0.471; selection 0.31 and 0.17. (b) Separation holds; predictive parity and independence fail (ratio 0.548); (0.1/0.9)(0.529/0.471)(0.8) \approx 0.1. (c) Scale 5, standard deviation 5\sqrt{2} \approx 7.07.

3. Question (5 points). State and prove the Laplace mechanism (Theorem 3.5). Deduce the bound of Theorem 3.3 on a membership attacker, and compute the largest TPR it allows at FPR 0.05 for \varepsilon = 1. Expected: the density ratio bounded by the triangle inequality and integrated over R; the test as a set R; \mathrm{TPR} \leq e \times 0.05 \approx 0.136.

4. Question (5 points). State and prove Chow’s rule (Theorem 5.4). With c = 0.15, decide for \eta(x) \in \{0.05, 0.3, 0.6, 0.9\}. Then explain what Article 14 of the AI Act asks of the humans who receive the deferred cases. Expected: the three conditional costs and their minimum; predict 0, defer, defer, predict 1; awareness of automation bias, correct interpretation of outputs, and the ability to override or stop the system.

Grading rubric (proposal)

The official sources display no weights. The split below is a proposal for grading a project report and its presentation.

References

Official sources of the course

• master-mva.com, course page “Responsible machine learning”, 2026–2027: https://www.master-mva.com/cours/responsible-machine-learning/. The page links a course presentation: http://nvayatis.perso.math.cnrs.fr/RMLcourse.html.
• Université Paris-Saclay, M2 Mathematics, Vision, Learning, programme table and course detail “Responsible machine learning” (ECTS, hourly volume, organisation and assessment).
• S. W. A. Dekker, D. D. Woods, “MABA-MABA or Abracadabra? Progress on human–automation co-ordination”, Cognition, Technology & Work 4(4), 240–244, 2002. Cited by the course description.
• S. Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power, PublicAffairs, 2019. Cited by the course description.
• E. Strubell, A. Ganesh, A. McCallum, “Energy and policy considerations for deep learning in NLP”, ACL, 2019. Cited by the course description.
• The course description also cites “Carr, 2017” without further detail; this page does not guess which work is meant.

Books and lecture notes

• S. Barocas, M. Hardt, A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities, MIT Press, 2023.
• C. Dwork, A. Roth, “The algorithmic foundations of differential privacy”, Foundations and Trends in Theoretical Computer Science 9(3–4), 211–407, 2014.
• M. Kearns, A. Roth, The Ethical Algorithm: The Science of Socially Aware Algorithm Design, Oxford University Press, 2019.
• C. Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd edition, 2022, https://christophm.github.io/interpretable-ml-book/.
• H. M. Mahmoud, Pólya Urn Models, Chapman & Hall/CRC, 2008.

Legal and policy texts

• Regulation (EU) 2016/679 (General Data Protection Regulation), Official Journal of the European Union L 119, 2016.
• Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal of the European Union, 2024.
• Council Directive 2000/43/EC implementing the principle of equal treatment between persons irrespective of racial or ethnic origin, 2000.
• Regulation (EU) 2017/745 on medical devices, 2017.
• Code pénal (France), Article 225-1; Loi n° 2019-222 du 23 mars 2019 de programmation 2018-2022 et de réforme pour la justice, Article 33.
• Griggs v. Duke Power Co., 401 U.S. 424, 1971; Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. Part 1607, 1978.
• OECD, Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449, 2019, revised 2024.
• High-Level Expert Group on Artificial Intelligence, Ethics Guidelines for Trustworthy AI, European Commission, 2019.
• UNESCO, Recommendation on the Ethics of Artificial Intelligence, 2021.
• ISO/IEC 42001:2023, Information technology — Artificial intelligence — Management system, 2023.
• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023.

Harms and feedback loops (ch. 1)

• H. Suresh, J. Guttag, “A framework for understanding sources of harm throughout the machine learning life cycle”, EAAMO, 2021.
• D. Ensign, S. A. Friedler, S. Neville, C. Scheidegger, S. Venkatasubramanian, “Runaway feedback loops in predictive policing”, FAT*, PMLR 81, 160–171, 2018.

Fairness (ch. 2 and 6.1–6.2)

• A. Chouldechova, “Fair prediction with disparate impact: a study of bias in recidivism prediction instruments”, Big Data 5(2), 153–163, 2017.
• J. Kleinberg, S. Mullainathan, M. Raghavan, “Inherent trade-offs in the fair determination of risk scores”, ITCS, 2017.
• J. Angwin, J. Larson, S. Mattu, L. Kirchner, “Machine bias”, ProPublica, 23 May 2016.
• W. Dieterich, C. Mendoza, T. Brennan, “COMPAS risk scales: demonstrating accuracy equity and predictive parity”, Northpointe Inc., 2016.
• M. Hardt, E. Price, N. Srebro, “Equality of opportunity in supervised learning”, NeurIPS, 2016.
• S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, A. Huq, “Algorithmic decision making and the cost of fairness”, KDD, 2017.
• F. Kamiran, T. Calders, “Data preprocessing techniques for classification without discrimination”, Knowledge and Information Systems 33(1), 1–33, 2012.
• A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, H. Wallach, “A reductions approach to fair classification”, ICML, 2018.
• R. Zemel, Y. Wu, K. Swersky, T. Pitassi, C. Dwork, “Learning fair representations”, ICML, 2013.
• M. J. Kusner, J. Loftus, C. Russell, R. Silva, “Counterfactual fairness”, NeurIPS, 2017.

Privacy (ch. 3 and 6.3–6.6)

• L. Sweeney, “Simple demographics often identify people uniquely”, Carnegie Mellon University, Data Privacy Working Paper 3, 2000.
• L. Sweeney, “k-anonymity: a model for protecting privacy”, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10(5), 557–570, 2002.
• A. Narayanan, V. Shmatikov, “Robust de-anonymization of large sparse datasets”, IEEE Symposium on Security and Privacy, 2008.
• R. Shokri, M. Stronati, C. Song, V. Shmatikov, “Membership inference attacks against machine learning models”, IEEE Symposium on Security and Privacy, 2017.
• I. Dinur, K. Nissim, “Revealing information while preserving privacy”, PODS, 2003.
• C. Dwork, F. McSherry, K. Nissim, A. Smith, “Calibrating noise to sensitivity in private data analysis”, TCC, 2006.
• C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, M. Naor, “Our data, ourselves: privacy via distributed noise generation”, EUROCRYPT, 2006.
• S. L. Warner, “Randomized response: a survey technique for eliminating evasive answer bias”, Journal of the American Statistical Association 60(309), 63–69, 1965.
• M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, L. Zhang, “Deep learning with differential privacy”, ACM CCS, 2016.
• S. Yeom, I. Giacomelli, M. Fredrikson, S. Jha, “Privacy risk in machine learning: analyzing the connection to overfitting”, IEEE CSF, 2018.
• C. Dwork, G. N. Rothblum, S. Vadhan, “Boosting and differential privacy”, FOCS, 2010.
• I. Mironov, “Rényi differential privacy”, IEEE CSF, 2017.
• I. Mironov, K. Talwar, L. Zhang, “Rényi differential privacy of the sampled Gaussian mechanism”, arXiv:1908.10530, 2019.
• B. Balle, G. Barthe, M. Gaboardi, “Privacy amplification by subsampling: tight analyses via couplings and divergences”, NeurIPS, 2018.
• B. McMahan, E. Moore, D. Ramage, S. Hampson, B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data”, AISTATS, 2017.
• L. Zhu, Z. Liu, S. Han, “Deep leakage from gradients”, NeurIPS, 2019.
• K. Bonawitz et al., “Practical secure aggregation for privacy-preserving machine learning”, ACM CCS, 2017.
• J. C. Duchi, M. I. Jordan, M. J. Wainwright, “Local privacy and statistical minimax rates”, FOCS, 2013.
• Ú. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, A. Thakurta, “Amplification by shuffling: from local to central differential privacy via anonymity”, SODA, 2019.
• N. Carlini et al., “Extracting training data from large language models”, USENIX Security Symposium, 2021.
• L. Bourtoule et al., “Machine unlearning”, IEEE Symposium on Security and Privacy, 2021.

Transparency (ch. 4)

• L. S. Shapley, “A value for n-person games”, in Contributions to the Theory of Games II, Annals of Mathematics Studies 28, Princeton University Press, 1953.
• S. M. Lundberg, S.-I. Lee, “A unified approach to interpreting model predictions”, NeurIPS, 2017.
• D. Janzing, L. Minorics, P. Blöbaum, “Feature relevance quantification in explainable AI: a causal problem”, AISTATS, 2020.
• S. Wachter, B. Mittelstadt, C. Russell, “Counterfactual explanations without opening the black box: automated decisions and the GDPR”, Harvard Journal of Law & Technology 31(2), 2018.
• B. Ustun, A. Spangher, Y. Liu, “Actionable recourse in linear classification”, FAT*, 2019.
• C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”, Nature Machine Intelligence 1, 206–215, 2019.
• D. Slack, S. Hilgard, E. Jia, S. Singh, H. Lakkaraju, “Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods”, AIES, 2020.
• M. Hardt, N. Megiddo, C. Papadimitriou, M. Wootters, “Strategic classification”, ITCS, 2016.

Accountability and human oversight (ch. 5)

• M. Bovens, “Analysing and assessing accountability: a conceptual framework”, European Law Journal 13(4), 447–468, 2007.
• M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru, “Model cards for model reporting”, FAT*, 2019.
• T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, K. Crawford, “Datasheets for datasets”, Communications of the ACM 64(12), 86–92, 2021.
• J. Buolamwini, T. Gebru, “Gender shades: intersectional accuracy disparities in commercial gender classification”, FAT*, PMLR 81, 77–91, 2018.
• R. Parasuraman, V. Riley, “Humans and automation: use, misuse, disuse, abuse”, Human Factors 39(2), 230–253, 1997.
• C. K. Chow, “On optimum recognition error and reject tradeoff”, IEEE Transactions on Information Theory 16(1), 41–46, 1970.
• D. Madras, T. Pitassi, R. Zemel, “Predict responsibly: improving fairness and accuracy by learning to defer”, NeurIPS, 2018.
• Z. Obermeyer, B. Powers, C. Vogeli, S. Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations”, Science 366(6464), 447–453, 2019.

Research directions (ch. 7)

• Ú. Hébert-Johnson, M. P. Kim, O. Reingold, G. N. Rothblum, “Multicalibration: calibration for the (computationally-identifiable) masses”, ICML, 2018.
• T. Hashimoto, M. Srivastava, H. Namkoong, P. Liang, “Fairness without demographics in repeated loss minimization”, ICML, 2018.
• L. T. Liu, S. Dean, E. Rolf, M. Simchowitz, M. Hardt, “Delayed impact of fair machine learning”, ICML, 2018.
• J. C. Perdomo, T. Zrnic, C. Mendler-Dünner, M. Hardt, “Performative prediction”, ICML, 2020.
• M. Jagielski, J. Ullman, A. Oprea, “Auditing differentially private machine learning: how private is private SGD?”, NeurIPS, 2020.
• M. Nasr, S. Song, A. Thakurta, N. Papernot, N. Carlini, “Adversary instantiation: lower bounds for differentially private machine learning”, IEEE Symposium on Security and Privacy, 2021.
• J. Dong, A. Roth, W. J. Su, “Gaussian differential privacy”, Journal of the Royal Statistical Society: Series B 84(1), 3–37, 2022.
• V. Feldman, I. Mironov, K. Talwar, A. Thakurta, “Privacy amplification by iteration”, FOCS, 2018.
• E. Cyffers, A. Bellet, “Privacy amplification by decentralization”, AISTATS, 2022.
• E. Bagdasaryan, O. Poursaeed, V. Shmatikov, “Differential privacy has disparate impact on model accuracy”, NeurIPS, 2019.
• B. Bilodeau, N. Jaques, P. W. Koh, B. Kim, “Impossibility theorems for feature attribution”, Proceedings of the National Academy of Sciences 121(2), 2024.
• G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, D. Weld, “Does the whole exceed its parts? The effect of AI explanations on complementary team performance”, CHI, 2021.
• I. D. Raji et al., “Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing”, FAT*, 2020.
• A. S. Luccioni, S. Viguier, A.-L. Ligozat, “Estimating the carbon footprint of BLOOM, a 176B parameter language model”, Journal of Machine Learning Research 24(253), 1–15, 2023.

Data sets and software

• F. Ding, M. Hardt, J. Miller, L. Schmidt, “Retiring Adult: new datasets for fair machine learning”, NeurIPS, 2021 (the Folktables package).
• H. Hofmann, “Statlog (German Credit Data)”, UCI Machine Learning Repository, 1994.
• Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, “Gradient-based learning applied to document recognition”, Proceedings of the IEEE 86(11), 2278–2324, 1998 (MNIST).
• F. Pedregosa et al., “Scikit-learn: machine learning in Python”, Journal of Machine Learning Research 12, 2825–2830, 2011.