Privacy and Uncertainty Quantification

Contents

• Introduction and overview
• 0. Reminders: probability, concentration, DP as seen in M1
• 1. Differential privacy: the foundation
• 2. Mechanisms: Laplace, Gaussian, exponential
• 3. Composition: advanced and Rényi DP
• 4. Local versus global, and DP-SGD
• 5. DP in industry and attacks
• 6. Uncertainty: sources and vocabulary
• 7. Calibration: diagrams, ECE, Brier
• 8. Bayesian deep learning: MC dropout versus ensembles
• 9. Conformal prediction: theorem and algorithm
• 10. Quantile and conformalized regression
• 11. UQ for LLMs
• 12. Applications: deciding under uncertainty
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who must ship a model with guarantees on privacy and on calibration. Empirical first: the module starts from concrete failures — a classifier with 91% accuracy but an ECE of 0.18 (it is over-confident), a DP-SGD fine-tuning at ε = 2 that leaves the model unusable — and only then names the tools (mechanisms, composition, conformal prediction).

Differential privacy and uncertainty quantification are the two families of guarantees that a model needs before it can be deployed responsibly; the module treats them in two parts, privacy first (ch. 1–5), then uncertainty (ch. 6–12).

What you will be able to do

• State the definition of \varepsilon-differential privacy and prove that a mechanism satisfies it (Laplace, Gaussian, exponential);
• Compose mechanisms (sequential and parallel theorems) and tighten the privacy budget with Rényi DP;
• Choose between local and global DP, and explain precisely what DP-SGD modifies in SGD;
• Read a reliability diagram and quantify the miscalibration of a classifier (ECE, Brier score);
• Compare MC dropout and deep ensembles for estimating the uncertainty of a deep network;
• Implement “split” conformal prediction and prove its marginal coverage guarantee;
• Produce adaptive intervals via conformalized quantile regression;
• Audit the uncertainty of an LLM and argue for a decision under uncertainty (alert threshold, abstention).

Official objectives and syllabus

Official syllabus: course description not published — this module appears neither in the S3 track nor in the S4 track of the 2026-2027 SynapseS catalog (both currently empty), and the master’s wiki indicates that the detailed M2 curriculum is still under development: description, teaching hours and instructor are therefore “not published”. The themes below are those carried by the module in the master’s brochure, as restated by these course materials, mapped to the sections that cover them.

The exact split between brochure content and the materials’ own extensions can only be settled once the official course description is published; in any case, chapter 0 (reminders: probability, concentration, DP as seen in M1) is part of the common structure of the repository’s course materials.

Where this course fits in the program

M2 module offered in period 1, outside the recommended track: it is aimed at all students who want to add demonstrable guarantees to their profile, whatever their specialty.

Prerequisites: the limit theorems, confidence intervals and tests of Refresher Statistics (ch. 4 and 6), and the honest comparison of two models of Machine Learning (4.9). They are the language of the coverage guarantees of ch. 9. Statistics in Action (bootstrap, multiple testing) is optional, a reference for the same logic. The first encounter with differential privacy in ch. 5 of Emerging ML helps if you took it; otherwise ch. 0–1 are self-contained.

Downstream modules: the privacy part of Explainability, Security, Privacy of LLMs (memorization, membership inference, DP of LLMs, ch. 7–8) relies directly on the mechanisms and composition proved here. On the production side, the drift monitoring of ML Ops and LLM Engineering Principles (ch. 5) tells when the data have moved; that module does not treat calibration or conformal prediction, and ch. 7–10 here say what to re-check when it fires (recalibration, a recomputed conformal quantile).

Canonical home. The DP proofs of ch. 1–4 (definition, post-processing, the Laplace mechanism, composition, Rényi DP, amplification by subsampling) are the reference derivations of these notes; the Gaussian mechanism, advanced composition and the subsampled-Gaussian accountant of DP-SGD are sketched or cited with their sources. ESP of LLMs ch. 8, Security & Robustness ch. 7–8, Advanced GNN 7.4 and Boosting → Tabular ch. 8 use these results without re-proving them: this page is where the proofs are.

Ch. 5 is the short introduction to membership inference that those pages can cite; the full attack (shadow models, LiRA) is Security & Robustness Def. 8.3.

Course outline

• Reminders: probability, concentration, DP as seen in M1 — puts the Markov/Hoeffding inequalities and the DP definition back in shape before going further.
• Differential privacy: the foundation — definitions (neighboring, mechanism), the three structural properties (post-processing, group privacy, convexity) and sensitivity, the quantity that sets the noise of ch. 2.
• Mechanisms: Laplace, Gaussian, exponential — sensitivity, noise calibration and the exponential mechanism for discrete responses.
• Composition: advanced and Rényi DP — why naive compositions waste the budget, and how Rényi DP tightens it.
• Local versus global, and DP-SGD — two trust models, then the algorithm that trains networks while preserving privacy.
• DP in industry and attacks — real deployments, the \varepsilon budget in practice, and attacks (reconstruction, inference) that motivate the theory.
• Uncertainty: sources and vocabulary — aleatoric versus epistemic, and why modern networks are poorly calibrated.
• Calibration: diagrams, ECE, Brier — measuring the gap between stated confidence and observed frequency.
• Bayesian deep learning: MC dropout versus ensembles — two practical approximations of epistemic uncertainty, with their trade-offs.
• Conformal prediction: theorem and algorithm — a distribution-free coverage guarantee in a few lines of code.
• Quantile and conformalized regression — intervals whose width adapts to the query point.
• UQ for LLMs — verbalized confidence, semantic coherence and the limits of the previous methods on generations.
• Applications: deciding under uncertainty — assembling the tools to arbitrate abstention, alert and action in a real system.
• Exercises — from proofs of mechanisms to conformal implementations on real data.
• Projects and assessment — solo and group subjects, a written-exam proposal and a rubric.
• References — the primary sources (Dwork, Rényi DP, Angelopoulos) to go to the text.

Session plan

24 h in class (assumed) = 8 sessions of 3 h. The fiche publishes no teaching hours: 24 h is the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS) and is assumed here; the session split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced material (recent research, open problems); the level never goes down from one session to the next.

The uncertainty part opens with a short vocabulary chapter (ch. 6), taught in the same session as calibration (ch. 7); that session works at the level of the core privacy chapters before it, since temperature scaling and the Brier decomposition are core methods.

Self-study, outside the 24 h: no chapter — every chapter is covered in class; the project (Ex. 11) takes about 10 h of personal work over the module.

How to study this module

• Redo the proofs on scratch paper: every theorem fits on one page, and it is by rewriting them that one retains the exact hypotheses.
• Implement Laplace noise and “split” conformal yourself (a few dozen lines): the guarantees become tangible when you see the coverage histogram.
• Whenever an \varepsilon appears, ask yourself which pair of neighboring datasets it distinguishes — that is the only way to give it meaning.
• Work through the two parts in order but in parallel in your head: chapters 6–12 replay in UQ the same moves as in privacy (guarantee, budget, utility trade-off).
• Do the exercises on your own first (the page gives no solutions; the worked examples of each chapter are the models to follow), and compare your intervals with those of others: coverage is verified, not declared.

0. Reminders: probability, concentration, DP as seen in M1

Both parts of the module prove guarantees, and these proofs use a small toolbox from M1: concentration inequalities, the first definition of differential privacy, and the language of tests and intervals. This chapter lists what is assumed and where it is used.

1. Differential privacy: the foundation

A hospital wants to publish statistics about its patients, or to train a model on their records, without revealing anything about any single patient. This chapter gives the definition that makes this promise precise and checkable.

The historical motivation (reconstructed by Dinur-Nissim, 2003): publishing “anonymized” statistics about a database is often enough to reconstruct the individual data by cross-referencing enough queries — naive anonymization (removing name/SSN) does not resist.

The answer of Dwork, McSherry, Nissim, Smith (2006): define protection as a property of the mechanism (the randomized algorithm), not of the data.

2. Mechanisms: Laplace, Gaussian, exponential

The definition of chapter 1 says what a private mechanism must satisfy, not how to build one. This chapter gives the three standard constructions: add noise scaled to the sensitivity (Laplace, Gaussian), or sample a discrete answer with probabilities that depend smoothly on its quality (exponential mechanism).

3. Composition: advanced and Rényi DP

A real analysis never releases a single statistic: it asks many queries, or, in training, takes thousands of gradient steps on the same data. This chapter computes how the privacy losses of successive mechanisms add up, and how Rényi DP keeps the total budget small.

4. Local versus global, and DP-SGD

Two practical questions remain: whom must the individuals trust with their raw data, and how does one train a neural network under differential privacy? This chapter answers the first with the global/local distinction and the second with DP-SGD.

import torch, torch.nn as nn
from opacus import PrivacyEngine

train_ds = torch.utils.data.TensorDataset(          # toy stand-in for real data
    torch.randn(10_000, 32), torch.randint(0, 2, (10_000,)))
model = nn.Linear(32, 2)
loader = torch.utils.data.DataLoader(train_ds, batch_size=256)
opt = torch.optim.SGD(model.parameters(), lr=0.05)

privacy_engine = PrivacyEngine()
model, opt, loader = privacy_engine.make_private_with_epsilon(
    module=model, optimizer=opt, data_loader=loader,
    target_epsilon=3.0,          # targeted (epsilon, delta=1e-5) budget
    target_delta=1e-5, epochs=20,
    max_grad_norm=1.0,           # C: per-example clipping
)
# Ordinary training loop; at each step:
#   grad_i clipped to C, noise N(0, sigma^2 C^2), sigma set by the
#   RDP accountant + amplification by subsampling.
for epoch in range(20):
    for x, y in loader:
        opt.zero_grad()
        loss = nn.functional.cross_entropy(model(x), y)
        loss.backward(); opt.step()
print(privacy_engine.get_epsilon(delta=1e-5))   # epsilon actually spent

5. DP in industry and attacks

The theory of chapters 1–4 is deployed at national and planetary scale, and it is motivated by attacks that really work. This chapter describes three deployments and the two attack families that justify paying the price of a formal guarantee.

6. Uncertainty: sources and vocabulary

A medical classifier announces 0.97 confidence on a patient unlike any in its training set; without calibration or a coverage guarantee, the clinic cannot set the threshold below which the model should abstain. This chapter opens the second part of the module: it names the sources of such uncertainty and fixes the vocabulary of the guarantees we will prove.

7. Calibration: diagrams, ECE, Brier

Before correcting a model’s confidence, one must measure how wrong it is. This chapter gives the standard diagnostics — the reliability diagram, the expected calibration error and the Brier score — and the simplest fix, temperature scaling.

import numpy as np
def ece(conf, y, bins=15):
    edges = np.linspace(0, 1, bins + 1); idx = np.digitize(conf, edges[1:-1])
    n = len(conf); total = 0.0
    for b in range(bins):
        m = idx == b
        if m.sum() == 0: continue
        total += (m.sum()/n) * abs(y[m].mean() - conf[m].mean())
    return total

def fit_temperature(logits, y, grid=np.linspace(0.5, 10, 200)):
    # NLL on validation, one dimension, non-parametric and sufficient for labs
    best_T, best = 1.0, np.inf
    for T in grid:
        z = logits / T
        p = np.exp(z - z.max(1, keepdims=True))
        p /= p.sum(1, keepdims=True)
        nll = -np.log(np.clip(p[np.arange(len(y)), y], 1e-12, None)).mean()
        if nll < best: best, best_T = nll, T
    return best_T

8. Bayesian deep learning: MC dropout versus ensembles

Calibration corrects the confidence of one model, but a single model cannot tell what it has not learned. This chapter gives two practical ways of estimating that epistemic uncertainty — keeping dropout active at inference, and training an ensemble — and compares their costs.

9. Conformal prediction: theorem and algorithm

Question: can we build prediction sets with coverage guaranteed without any distributional assumption? Answer (Vovk): yes, provided one has exchangeable data and accepts marginal coverage. This is the most useful result of the UQ part.

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(1)
n_tr, n_cal, n_te, alpha = 2000, 3000, 5000, 0.1
X = rng.uniform(-2, 2, size=(n_tr + n_cal + n_te, 1))
y = 2 * X[:, 0] + rng.normal(0, 1, len(X))              # toy i.i.d. data
X_tr, X_cal, X_te = np.split(X, [n_tr, n_tr + n_cal])   # exchangeable split
y_tr, y_cal, y_te = np.split(y, [n_tr, n_tr + n_cal])

model = LinearRegression().fit(X_tr, y_tr)              # never the calibration set
s_cal = np.abs(y_cal - model.predict(X_cal))            # non-conformity scores
k = int(np.ceil((n_cal + 1) * (1 - alpha)))             # 9 for n=9; 2701 for n=3000
qhat = np.sort(s_cal)[k - 1]
lo, hi = model.predict(X_te) - qhat, model.predict(X_te) + qhat

# empirical check of the coverage (exercise 7): it fluctuates around 0.90,
# since the guarantee holds on average over the calibration draw
cov = ((y_te >= lo) & (y_te <= hi)).mean()
print(f"qhat = {qhat:.3f}, observed coverage = {cov:.3f} (target >= {1-alpha})")

10. Quantile and conformalized regression

Split conformal regression returns intervals of the same width everywhere, even where the noise is small. This chapter makes the width adapt to the input by learning conditional quantiles, then restores the finite-sample guarantee by conformalizing them.

11. UQ for LLMs

LLMs add three difficulties to the calibration problem: the output space is combinatorial (sequences, not classes); RLHF training seems to degrade verbal calibration; and generative use (a written answer) does not naturally expose token probabilities.

The M2 module Explainability, Security, Privacy of LLMs continues this part on the security and privacy side (jailbreaks ch. 5, memorization and training extraction ch. 7, membership inference and DP of LLMs ch. 8); cf. also the Large Language Models module, ch. 9 (hallucination, abstention).

12. Applications: deciding under uncertainty

A probability, an interval or a privacy budget is only useful once it drives a decision: treat or not, raise an alert, abstain and hand over to a human. This chapter turns the guarantees of the module into decision rules.

Exercises

1. ★ Laplace mechanism.

   1. Re-prove thm 2.1 for k = 1.

   2. Counting with ε = ln 20: probability of an error ≥ 2? ≥ 5?

   3. 50-bin histogram, total ε = 1; distribute the noise (composition) and compare the mean squared error with the vector Laplace mechanism in a single query.

2. ★ Local DP.

   1. Prove the debiasing formula 4.2 and its variance factor.

   2. In RAPPOR with 128 bits, 2 successive RRs: what ε per bit?

   3. Sizing: precision of ±2 points on a proportion, p = 0.75 — how many users?

3. ★ Calibration.

   1. Recompute the ECE 7.2 with 10 equal-width bins and comment on the sensitivity.

   2. Implement the reliability diagram.

   3. Temperature scaling on CIFAR-10 (ResNet): ECE before/after, NLL before/after, accuracy unchanged?

   4. Show that the constant 0.75 beats the classifier of example 7.4 in both Brier score and log-loss (about 0.56 against 0.93 nats). Then replace the wrong prediction (0.9, y{=}0) by (1.0, y{=}0): which score stays finite, and which one punishes the wrong certainty without bound?

4. ★★ Composition.

   1. Redo the computation 3.4 for \sigma = 12, k = 256, \delta = 10^{-5}.

   2. Inverse: what \sigma for k = 10^4 DP-SGD steps, ε = 2, δ = 10⁻⁵, q = 0.01 (approximation without amplification, then with the q^2 factor)?

   3. Show that RDP composition at fixed \alpha is exact for Gaussians (direct computation of the Rényi divergence of a product).

5. ★★ DP-SGD. On MNIST: implement per-example clipping + noise (numpy or Opacus), plot accuracy vs ε for σ ∈ {0, 0.5, 1, 2}, comment on the frontier; ablation on C (0.1 to 10); verify that without clipping the mechanism is no longer DP.

6. ★★ MC dropout versus ensembles. 1D heteroscedastic regression: train

   1. one network + MC dropout (S = 50 passes);

   2. 5 networks (deep ensemble); compare NLL, RMSE, coverage of the bands at ±2 predicted standard deviations;

   3. test both out of distribution (extrapolation);

   4. compare training and inference times, and conclude in a budget-constrained situation.

7. ★★ Conformal.

   1. Re-prove the uniformity of the rank with ties (convention of thm 9.2).

   2. Empirically verify the coverage over 1000 repetitions (n = 1000, α = 0.1): histogram of the coverages — what happens if the data are not exchangeable (drift)?

   3. Effect of the calibration size: n \in \{20, 100, 1000\};

   4. implement per-class Mondrian and measure the coverage inequality across classes.

8. ★★ CQR. Reproduce example 10.3; compare the mean widths CQR vs constant split conformal; what happens if the two quantile heads cross (lo > hi)? Fix it (monotonicity constraint or sorting).

9. ★★ Decision. Screening: prevalence 5%, costs c_{FP} = 1, c_{FN} = 20.

   1. Optimal threshold on calibrated probabilities.

   2. Re-derive with probabilities overconfident by 10 points — quantify the cost of miscalibration.

   3. Add the abstention option with a delegation cost; plot risk-coverage.

10. ★★★ LLM UQ.

    1. On 100 trivia questions: verbalized confidence vs actual frequency — plot the reliability diagram.

    2. Self-consistency K = 10: correlation between agreement and correctness.

    3. Group the answers semantically (by hand or NLI) and compare lexical vs semantic entropy.

    4. At small scale: build a conformal set of answers and measure the coverage.

11. ★★★ Project. Complete “privacy + UQ” pipeline: a sensitive dataset (e.g. Adult):

    1. publish useful statistics under budget ε = 2 (mechanisms, composition);
    2. train a private predictive model (DP-SGD);
    3. calibrate and conformalize its outputs;
    4. write the guarantee sheet (ε, δ, neighbour convention and accountant; coverage level and calibration-set size; ECE before and after; the exchangeability assumption and what would break it) and the limitations discussion.

    Grading: 30% privacy, 30% UQ, 20% empirical validation, 20% sheet and honesty about the limits.

    This guarantee sheet is the format required by the trust work package of every Year 2 transverse project (projects page).

Projects and assessment

The fiche of this module displays no assessment: its 2026–2027 SynapseS description is not published (see the table of the introduction). Everything below is therefore a proposal: the written exam is a mock final, and the projects are practice that an instructor may adopt as continuous assessment. Every subject exercises objective O8 (trustworthy AI: privacy and uncertainty guarantees) and O7 (hands-on practice: audited pipelines); the graph subjects add O4 and the RAG subject adds O3.

Each project report includes the guarantee sheet of Ex. 11, which is also the format of the trust work package of the Year 2 transverse projects: Y2.1 (conformal abstention), Y2.2 (conformal under shift) and Y2.5 (privacy and calibration of fraud alerts).

Solo projects

1. [Solo · ≈ 20 h] Paper study: an RDP accountant from scratch (Mironov, 2017; Mironov, Talwar and Zhang, 2019). Question: how much do Rényi DP and subsampling save over advanced composition, and does a short NumPy accountant agree with a production one?
   • Implement the Gaussian RDP curve, additive composition and the conversion to (\varepsilon, \delta) over a grid of orders (Definition 3.3). Then add the RDP of the sampled Gaussian mechanism of Mironov, Talwar and Zhang, a binomial sum for integer orders (ch. 3, 4).
   • Reproduce Example 3.4, Proposition 4.4 and Ex. 4b. Cross-check against the RDP accountant of Opacus, with the same integer orders, for q \in \{0.001, 0.01, 0.1\} and \sigma \in \{0.5, 1, 2\}, up to 10^4 steps. No data is needed.
   • Opacus converts RDP to (\varepsilon, \delta) with the slightly tighter formula of Balle, Barthe, Gaboardi, Hsu and Sato (2020), not with that of Definition 3.3: implement both and report the gap.
   • Deliverables: code with unit tests, a 4–6 page report with curves of \varepsilon against the number of steps, a 10-minute talk. Success: the RDP curve, and \varepsilon under the same conversion, within 1 % of Opacus on the whole grid; \varepsilon^* \approx 5.75 in Example 3.4 with the conversion of Definition 3.3.
2. [Solo · ≈ 25 h] Conformal prediction for GNN node classification. Question: does split conformal keep its 90 % coverage on a citation graph, where the nodes are linked rather than i.i.d.? The subject connects this module to Graph ML and Advanced GNN.
   • Train a two-layer GCN (Kipf and Welling, 2017; Graph ML ch. 5) on Cora and PubMed, with the Planetoid splits of PyTorch Geometric. Calibrate on held-out labelled nodes: split conformal with the score of Definition 9.1, then per-class Mondrian conformal (ch. 9).
   • Compare with naive sets that keep the most probable classes of an MC-dropout average (S = 50 passes) or of a 5-GCN ensemble until their mass reaches 90 % (ch. 8). Reproduce the split-conformal baseline of Huang, Jin, Candès and Leskovec (2023).
   • Break exchangeability on purpose: calibrate on high-degree nodes only, or on one community, and relate the coverage loss to the homophily of the test nodes (ch. 12, “On graphs”).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Success: mean coverage within one point of 90 % over 50 random calibration/test splits, with the per-class gaps and the shift experiment reported with intervals.
3. [Solo · ≈ 20 h] Edge-level versus node-level DP for graph statistics. Question: what does it cost to protect a person rather than one of their friendships? The subject applies ch. 1–3 to the graph setting of Advanced GNN 7.4.
   • Derive the sensitivities (Definition 1.3) of the edge count, the degree histogram and the triangle count, under edge neighbours and under node neighbours. For node-DP, first project the graph to a maximum degree D, and bound how much one node can change the projected graph: this is the subtle step.
   • Release the statistics with the Laplace and Gaussian mechanisms (ch. 2), and an edge-DP top-10 of the highest-degree nodes with the exponential mechanism. Keep the whole budget ledger with RDP composition (ch. 3).
   • Data: the ego-Facebook and ca-GrQc graphs of SNAP (Leskovec and Krevl, 2014).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Success: curves of relative error for \varepsilon from 0.1 to 10 under both neighbour notions, the gap explained by the ratio of sensitivities, and a justified choice of D.

Group projects

1. [Group of 3 · ≈ 30 h each] A private and calibrated readmission model. Question: can one hospital model be private, calibrated and covered at once, and what does each guarantee cost? The subject extends Ex. 11 to the Diabetes 130-US hospitals data (Strack and coauthors, 2014; UCI repository, about 100,000 encounters).
   • Member 1: privacy — the privacy unit is the patient, so keep one encounter per patient (about 70,000 rows, as in Strack and coauthors) and row-level DP then protects patients (Definition 1.1). Cohort statistics under a budget \varepsilon = 1 with a budget ledger (ch. 1–3); DP-SGD with Opacus for \varepsilon \in \{1, 3, 8\} and \delta = 10^{-6} (ch. 4).
   • Member 2: attack — a loss-threshold attack and a shadow-model membership inference (ch. 5; Security & Robustness Def. 8.3), before and after DP, reported as the true-positive rate at 1 % false positives.
   • Member 3: uncertainty and decision — temperature scaling and ECE (ch. 7), conformal sets for readmission and CQR intervals for the length of stay (ch. 9–10), a cost-sensitive threshold with abstention to a clinician (ch. 12).
   • Deliverables: code, a 6–8 page report, the guarantee sheet of Ex. 11, a 15-minute talk. Success: a privacy–utility–coverage frontier over the three budgets, an attack advantage that shrinks with \varepsilon, and coverage within one point of 90 % on held-out patients.
2. [Group of 3 · ≈ 30 h each] An abstaining RAG assistant with semantic uncertainty. Question: can a retrieval-augmented LLM tell when not to answer, with a guarantee on the errors of what it does answer? The subject connects this module to Large Language Models ch. 7 and 9, and prepares Y2.1.
   • Data and models: TriviaQA (Joshi, Choi, Weld and Zettlemoyer, 2017), with its Wikipedia evidence documents as the retrieval corpus. Kuhn, Gal and Farquhar (2023) used the closed-book setting, which is the no-retrieval baseline here. Use an open instruction model of 1–3 B parameters on a free GPU, and an NLI model for entailment, DeBERTa-large fine-tuned on MNLI as in that paper.
   • Member 1: the RAG pipeline (BM25 and a dense retriever), K = 10 sampled answers per question, verbalized confidence (Definition 11.1), correctness by exact match.
   • Member 2: uncertainty scores — token log-probability, self-consistency and semantic entropy with bidirectional entailment (Definition 11.2); AUROC for correctness and ECE (ch. 7).
   • Member 3: abstention — a threshold chosen on 500 calibration questions by conformal risk control (Definition 12.2), with the loss “answered and wrong” at \alpha = 0.1; the risk–coverage curve; 20 canary strings planted in the document store and an extraction test (ch. 5).
   • Deliverables: code, a 6–8 page report with the guarantee sheet, a 15-minute talk. Success: over 20 random calibration/test splits, the mean rate of answered-and-wrong questions stays at or below 10 %; the AUROC of semantic and lexical entropy is reported with intervals, with and without retrieval; canary leakage is measured.
3. [Group of 3 · ≈ 30 h each] Paper study: conformal fraud alerts under drift on the Elliptic graph (Weber and coauthors, 2019). Question: does a coverage guarantee survive a regime shift in a transaction graph, and what restores it? The subject prepares Y2.5 and uses the drift monitoring of MLOps ch. 5.
   • Member 1: models — reproduce the temporal split of the paper (train on time steps 1–34, test on 35–49) with a random forest and a GCN (Graph ML ch. 5); calibrate both (ch. 7).
   • Member 2: guarantees over time — split conformal alert sets calibrated on the last training steps; the coverage per time step through the dark-market shutdown that the paper reports near step 43; recalibration on a rolling window, and Mondrian conformal per class (ch. 9).
   • Member 3: decisions — a cost-sensitive alert threshold under a budget of analyst reviews per time step, abstention to the analyst, and the risk–coverage curve (ch. 12).
   • Deliverables: code, a 6–8 page report with the guarantee sheet, a 15-minute talk. Success: the illicit-class F1 of the paper is reproduced within a few points, the coverage drop after the shift is measured, and the report says how much the rolling window recovers and at what cost in set size.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet allowed, a non-programmable calculator, four questions. The fiche displays no exam, so this is a mock.

1. Question (4 points). (a) State \varepsilon-DP with the neighbour convention of this page, and the post-processing property. Why may a model trained with DP-SGD be published with all its weights? (b) Distinguish aleatoric and epistemic uncertainty, and say which of the two the variance between MC-dropout passes estimates. (c) What does the split conformal guarantee promise, and what does it not promise? Give one deployment in which it fails.

   Expected: (a) Definition 1.1 with add/remove neighbours, and Proposition 1.2(1). The weights are a post-processing of the noised gradients, whose whole sequence is (\varepsilon, \delta)-DP by composition and amplification (ch. 3–4). (b) Definition 6.1; the variance between passes is the epistemic part (Result 8.2).

   3. Marginal coverage of at least 1 - \alpha, on average over the calibration draw and the new point (Theorem 9.2). It promises nothing per group or per input. It fails without exchangeability: calibration in one hospital and deployment in another, or after a drift.

2. Question (5 points). (a) A counting query is released with the Laplace mechanism at \varepsilon = 0.5. Give the noise scale and the probability of an error of at least 4. (b) A Gaussian counting query is released 100 times with \sigma = 10\,\Delta_2. With \delta = 10^{-5}, compute the RDP account and its optimal order. (c) An \varepsilon = 2 mechanism runs on a Poisson subsample with q = 0.02. What is its guarantee?

   Expected: (a) the scale is b = 1/\varepsilon = 2 and \mathbb P(|Z| \geq 4) = e^{-2} \approx 0.135 (Theorem 2.1). (b) Each release is (\alpha, \alpha/200)-RDP, and the 100 releases compose into (\alpha, \alpha/2)-RDP (Definition 3.3). The conversion gives \varepsilon(\alpha) = \frac{\alpha}{2} + \frac{\ln 10^5}{\alpha - 1}, minimal at \alpha - 1 = \sqrt{2 \ln 10^5} \approx 4.8, where \varepsilon \approx 2.90 + 2.40 = 5.3; advanced composition would give about 53. (c) By Proposition 4.4, \varepsilon' = \ln(1 + 0.02\,(e^2 - 1)) \approx 0.12.

3. Question (5 points). Let the scores s_1, \dots, s_{n+1} be exchangeable with no ties, and let \hat q be the k-th smallest of the first n, where k = \lceil (n+1)(1-\alpha) \rceil \leq n.

   1. Prove that \mathbb P(s_{n+1} \leq \hat q) = k/(n+1). (b) Deduce that the coverage lies between 1 - \alpha and 1 - \alpha + 1/(n+1). (c) Give k and the coverage when (n, \alpha) = (19, 0.1). What happens with eight calibration points?

   Expected: (a) by exchangeability, the rank of s_{n+1} among the n+1 scores is uniform on \{1, \dots, n+1\}, and the event \{s_{n+1} \leq \hat q\} is “this rank is at most k” (Theorem 9.2).

   2. By definition of the ceiling, (n+1)(1-\alpha) \;\leq\; k \;<\; (n+1)(1-\alpha) + 1; divide by n+1.

   3. Then k = 18 and the coverage is exactly 18/20 = 0.90. With eight points, k = \lceil 8.1 \rceil = 9 exceeds n, and the set is the whole output space (Example 9.3).

4. Question (6 points). A binary model outputs the logits z = (3, 0), for classes 1 and 0, on a group of patients, 70 % of whom have class 1. (a) Compute the announced probability of class 1, and the temperature that calibrates it on this group. (b) Prove that, for calibrated probabilities and costs c_{\text{FP}}, c_{\text{FN}}, predicting 1 minimizes the expected cost exactly when \hat p \;\geq\; \frac{c_{\text{FP}}}{c_{\text{FP}} + c_{\text{FN}}}.

   3. Predicting 1 triggers a costly protocol, with c_{\text{FP}} = 3 and c_{\text{FN}} = 1. Compare the decisions and expected costs before and after calibration.

   Expected: (a) at T = 1, \hat p = 1/(1 + e^{-3}) \approx 0.95. Calibration asks 1/(1 + e^{-3/T}) = 0.7, so T = 3/\ln(7/3) \approx 3.5 (Procedure 7.5).

   2. Predicting 1 costs c_{\text{FP}}(1 - \hat p) on average, predicting 0 costs c_{\text{FN}}\,\hat p; the first is smaller exactly above the threshold (Definition 12.2).

   3. The threshold is 3/4. Before calibration, 0.95 \geq 0.75: the model treats everyone, at an expected cost of 3 \times 0.3 = 0.9 per patient. After calibration, 0.7 < 0.75: it treats no one, at a cost of 1 \times 0.7 = 0.7. Miscalibration costs 0.2 per patient.

Grading rubric (proposal)

References

• C. Dwork, F. McSherry, K. Nissim, A. Smith, “Calibrating Noise to Sensitivity in Private Data Analysis”, TCC 2006 — the founding act.
• C. Dwork, A. Roth, “The Algorithmic Foundations of Differential Privacy”, Foundations and Trends in Theoretical Computer Science 9, 2014 — the handbook.
• C. Dwork, G. Rothblum, S. Vadhan, “Boosting and Differential Privacy”, FOCS 2010 — advanced composition.
• I. Mironov, “Rényi Differential Privacy”, CSF 2017 — arXiv:1702.07476.
• M. Bun, T. Steinke, “Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds”, TCC 2016.
• Y.-X. Wang, B. Balle, S. Kasiviswanathan, “Subsampled Rényi Differential Privacy and Analytical Moments Accountant”, AISTATS 2019 — arXiv:1808.00087.
• M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, L. Zhang, “Deep Learning with Differential Privacy”, CCS 2016 — arXiv:1607.00133 (DP-SGD, moments accountant).
• Ú. Erlingsson, V. Pihur, A. Korolova, “RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response”, CCS 2014 — arXiv:1407.6981.
• S. Warner, “Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias”, JASA 60, 1965.
• A. Evfimievski et al., “Limiting Privacy Breaches in Privacy Preserving Data Mining”, PODS 2003 — modern randomized response, local DP.
• J. Abowd, “The U.S. Census Bureau Adopts Differential Privacy”, KDD 2018.
• Apple Differential Privacy Team, “Learning with Privacy at Scale”, Apple Machine Learning Journal 1(8), 2017.
• R. Shokri, M. Stronati, C. Song, V. Shmatikov, “Membership Inference Attacks Against Machine Learning Models”, IEEE S&P 2017.
• N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, D. Song, “The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks”, USENIX Security 2019 — arXiv:1802.08232.
• G. Brier, “Verification of Forecasts Expressed in Terms of Probability”, Monthly Weather Review 78, 1950; A. Murphy, “A New Vector Partition of the Probability Score”, J. Applied Meteorology 10, 1973.
• M. P. Naeini, G. F. Cooper, M. Hauskrecht, “Obtaining Well Calibrated Probabilities Using Bayesian Binning”, AAAI 2015.
• C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017 — arXiv:1706.04599.
• B. Lakshminarayanan, A. Pritzel, C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles”, NeurIPS 2017 — arXiv:1612.01474.
• Y. Gal, Z. Ghahramani, “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning”, ICML 2016 — arXiv:1506.02142.
• A. G. Wilson, P. Izmailov, “Bayesian Deep Learning and a Probabilistic Perspective of Generalization”, NeurIPS 2020 — arXiv:2002.08791.
• V. Vovk, A. Gammerman, G. Shafer, Algorithmic Learning in a Random World, Springer, 2005 (2nd ed. 2022) — the founding book of conformal prediction.
• J. Lei, M. G’Sell, A. Rinaldo, R. Tibshirani, L. Wasserman, “Distribution-Free Predictive Inference for Regression”, JASA 113, 2018 — arXiv:1604.04173.
• Y. Romano, E. Patterson, E. Candès, “Conformalized Quantile Regression”, NeurIPS 2019 — arXiv:1905.03222.
• A. N. Angelopoulos, S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification”, 2023 — arXiv:2107.07511.
• A. N. Angelopoulos, S. Bates, E. Candès, M. Jordan, L. Lei, “Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control”, The Annals of Applied Statistics, 2025 (preprint October 2021) —arXiv:2110.01052 (high-probability risk control by multiple testing).
• A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, T. Schuster, “Conformal Risk Control”, 2022, ICLR 2024 — arXiv:2208.02814 (control of an expected risk).
• B. Balle, G. Barthe, M. Gaboardi, “Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences”, NeurIPS 2018 — arXiv:1807.01647.
• S. Farquhar, J. Kossen, L. Kuhn, Y. Gal, “Detecting Hallucinations in Large Language Models Using Semantic Entropy”, Nature 630, 2024.
• K. Huang, Y. Jin, E. Candès, J. Leskovec, “Uncertainty Quantification over Graph with Conformalized Graph Neural Networks”, NeurIPS 2023.
• Project sources: I. Mironov, K. Talwar, L. Zhang, “Rényi Differential Privacy of the Sampled Gaussian Mechanism”, 2019 — arXiv:1908.10530; B. Balle, G. Barthe, M. Gaboardi, J. Hsu, T. Sato, “Hypothesis Testing Interpretations and Rényi Differential Privacy”, AISTATS 2020 — arXiv:1905.09982 (the RDP-to-DP conversion used by Opacus);T. N. Kipf, M. Welling, “Semi-Supervised Classification with Graph Convolutional Networks”, ICLR 2017 — arXiv:1609.02907; J. Leskovec, A. Krevl, “SNAP Datasets: Stanford Large Network Dataset Collection”, 2014; B. Strack et al., “Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records”, BioMed Research International, 2014 (the Diabetes 130-US hospitals data, UCI repository); M. Joshi, E. Choi, D. Weld, L. Zettlemoyer, “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, ACL 2017 — arXiv:1705.03551; M. Weber et al., “Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Forensics”, KDD Workshop on Anomaly Detection in Finance, 2019 — arXiv:1908.02591 (the Elliptic data).
• R. Koenker, G. Bassett, “Regression Quantiles”, Econometrica 46, 1978.
• S. Lin, J. Hilton, O. Evans, “Teaching Models to Express Their Uncertainty in Words”, TMLR 2022 — arXiv:2205.14334.
• K. Tian, W. Yao, D. Yu, L. Mou, “Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback”, EMNLP 2023 (Findings) — arXiv:2305.14975.
• M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, B. Hooi, “Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs”, ICLR 2024 — arXiv:2306.13063.
• L. Kuhn, Y. Gal, S. Farquhar, “Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation”, ICLR 2023 — arXiv:2302.09664.
• B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, E. W. Steyerberg, on behalf of Topic Group 6 (Evaluating diagnostic tests and prediction models) of the STRATOS initiative, “Calibration: the Achilles heel of predictive analytics”, BMC Medicine 17:230, 2019.
• Internal follow-ups: M1 modules “Refresher Statistics” (ch. 8: Bayesian statistics), “Machine Learning” (ch. 8: generalization; ch. 4: evaluation), “Emerging ML” (ch. 3–5: federated, Byzantine, privacy: first encounter with DP), “Statistics in Action” (ch. 7–8: multiple testing, bootstrap — the empirical-quantile logic of conformal), M2 “Large Language Models” (ch. 9–10); the M2 module “Explainability, Security, Privacy of LLMs” (ch. 5–8) continues the security and privacy part for LLMs.
• Tools: Opacus (PyTorch / Meta), TensorFlow Privacy, OpenDP, Google DP library; awesome-conformal-prediction for the conformal ecosystem, MAPIE (scikit-learn-contrib).