Refresher in Statistics

Introduction and overview

Independent study notes organised around the catalog label “Refresher in Statistics”, Year 1 · Period 1 — not an official syllabus or course-selection plan.

Undergraduate refresherPreparation for ML

The question this page answers

When does a number computed on a finite sample justify a claim about future data?

A test with 99% sensitivity can still leave only a 17% probability of disease after a positive result when prevalence is 1% and specificity is 95%: the denominator matters. A sample mean can look stable while its uncertainty estimate is wrong because observations share a source. Both problems return when evaluating a classifier or asking whether adding a graph actually helps a language model.

The approach is empirical first: calculate the counts, simulate the estimator, inspect the residuals — then state the model and its assumptions.

Source labels and study budget

• Catalog record: the SynapseS 2026–2027 sheet for APM_51438_EP reports 12 h CTD (lectures–tutorials), period X-3A-P1 (17 Sep–18 Dec 2026), English, with Marine Le Morvan listed as responsible; total workload and ECTS are not displayed, so none is asserted. Its description, prerequisite and syllabus tabs are empty, and its assessment section for the LLGA track is empty too (details).
• Suggested study budget: about 36 h for a full pass — the 12 h in class plus about 24 h of self-study, chapter by chapter in the session plan. This is an adjustable study estimate, not a teaching allocation or ECTS equivalence.
• Working background: basic algebra and calculus, vectors and matrices, Python/NumPy. See the Refresher in Computer Science if arrays are the obstacle.

What you will be able to do

• Handle probabilities, random variables, common distributions and random-vector notation.
• State the LLN and CLT, and recognize when their assumptions fail.
• Build estimators by moments or maximum likelihood; evaluate bias, variance, MSE.
• Construct confidence intervals and tests; distinguish what is concluded from what is not.
• Fit and diagnose a linear regression.
• Compute Bayesian updates; separate posterior uncertainty from frequentist coverage.
• Find the stationary law of a finite Markov chain and see why the Gibbs sampler targets it.
• Name the statistical tool behind an ML formula and the extra assumptions it needs.

Session plan

12 h in class = 4 sessions of 3 h. The volume is the one of the SynapseS 2026–2027 sheet (12 h CTD); the sheet publishes no session-by-session programme, so the split below is our proposal for covering the core of these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced theory; the level never goes down from one session to the next.

Self-study, outside the 12 h (about 24 h; ≈ 36 h in all): ch. 3 Common distributions (≈ 3 h, Ex. 5; section 3.4 optional) · ch. 6 Intervals and tests (≈ 5 h, Ex. 6, 8, 13; required before Period 2) · ch. 7 Linear regression (≈ 5 h, Ex. 14–15; required before Period 2) · ch. 8 Bayesian statistics (≈ 3 h, Ex. 7) · ch. 9 From statistics to ML (≈ 3 h) · exit checkpoint and capstone evaluation kit (≈ 1 h) · ch. 10 Missing data (≈ 4 h, Ex. 16).

Read ch. 3 before session 3: chapters 4 and 5 use its table and the gamma sums of section 3.2.

Notation

Site-wide conventions: see the notation page; deviations on this page:

• \mathbb{P} is a probability measure and P the data distribution: a dataset D_n=\{(X_i,Y_i)\}_{i=1}^n is drawn iid from P. In section 2.7, P is a transition matrix, as for random walks on graph pages.
• With one argument, \ell(\theta) is the log-likelihood, maximized, and L(\theta) the likelihood (chapters 3, 5, 8, 9). A loss has two arguments, \ell(\hat y,y), prediction first, and is minimized. The risk and empirical risk are R(f) and \hat R_n(f) (chapter 9).
• Chapter 7 writes p for the number of regression coefficients, intercept included, so X\in\mathbb{R}^{n\times p}; elsewhere d counts features.
• \pi is a prior density in chapter 8 and a stationary law in section 2.7.

Before you start

Try the three positioning checks below before the first session and before choosing a route. A correct answer includes the reasoning or the assumption, not just the final number.

1. Conditioning. A population of 10,000 people has a disease prevalence of 1%, and a test has sensitivity 99% and specificity 95%. How many true and false positives should you expect, and what fraction of the positive results concern infected people? If the denominator is unclear, read Bayes’ formula and the worked tree.
2. Sampling uncertainty. You observe independent Gaussian variables with known \sigma=2, sample size n=25 and mean \bar x=10.4. Compute a 95% confidence interval for the mean, and say what happens to its width at n=100. Would the same calculation be justified for 25 strongly related nodes of a single graph? Review convergence and the CLT and confidence intervals.
3. Likelihood and evaluation. For Bernoulli observations (1,0,1,1,0), write the likelihood and find its maximizer. Does a lower training negative log-likelihood establish that a larger model predicts better? Review MLE derivations, ERM and cross-entropy.

Reading route

This page is a reading proposal, not a compulsory linear course.

• Core — the 12 h in class. The four sessions of the session plan cover conditioning and Bayes’ formula, random variables and moments, the LLN, the CLT and the bootstrap and estimation. Read the distribution table before session 3 and keep it as a reference. Useful checks are exercises 1–4 and 9–12.
• Self-study before Machine Learning — about 17 h. Chapter 3 (≈ 3 h), then confidence intervals and p-values in chapter 6 (≈ 5 h), least squares and regression diagnostics in chapter 7 (≈ 5 h), and the bridge to ML in chapter 9 with the exit checkpoint (≈ 4 h). Exercises 5, 6, 8 and 13–15 check these chapters. Chapters 6 and 7 are required before Period 2: its courses assume tests, intervals and regression.
• Deepen — about 7 h, as the next problem requires. Bayesian statistics (≈ 3 h, exercise 7) and missing data (≈ 4 h, exercise 16), together with the proofs of the in-class chapters, Gaussian conditioning and the measure-theoretic layer.
• Skip or defer — when the checks already pass. Do not restart from σ-algebras just because they come first: if the three positioning checks are secure, skim chapters 1 and 2 (except Markov chains) and spend the time saved on chapter 9 and the exit checkpoint. Exact ANOVA, the full distribution table and the missing-data papers can remain references until needed.

When a section is weak, reproduce its numerical example, run the code, and then attempt the corresponding exercise before reading the solution. The completion criterion is evidence you can reproduce — a calculation, a simulation, a failure mode — not hours logged.

Contents

• Introduction and overview
  • The question this page answers
  • Session plan
  • Notation
• Before you start
• Reading route
• 1. The language of probability
  • Bayes’ formula
• 2. Random variables
  • Expectation, moments and MGFs
  • Conditional expectation
  • Finite Markov chains
• 3. Common probability distributions
  • Gaussian conditioning
  • Going further: the measure-theoretic layer
• 4. Convergence and limit theorems
  • Empirical quantiles and the bootstrap
• 5. Parametric estimation
  • Score and Fisher information
  • MLE derivations
• 6. Confidence intervals and tests
  • Confidence intervals
  • p-values and multiple testing
• 7. Linear regression
  • Least squares
  • Goodness of fit and diagnostics
• 8. Bayesian statistics
  • Credible interval vs confidence interval
• 9. From statistics to machine learning
  • Empirical risk minimization
  • Likelihood to cross-entropy
• 10. Missing data: mechanisms, baselines and research
  • Missingness mechanisms
  • Selected research
• Exercises
• Exit checkpoint and connections
  • Graph × LLM transfer check
  • Capstone evaluation kit
  • Where to go next—and what remains
• Sources and scope
• Projects and assessment
• References

1. The language of probability

Before the σ-algebra: a number. A spam filter with 95% recall on mail that is only 2% spam yields, per 10,000 messages, ≈190 caught spam but ≈490 false positives if specificity is 95%. The model is not malfunctioning: the conditional probability was misread. Likewise, 99% aggregate accuracy can hide 20% error on a rare class. ML is a calculus of uncertainty; this chapter gives the vocabulary to avoid swapping denominators.

1.1. Conditioning and the law of total probability

The multiplication formula follows directly: \mathbb{P}(A \cap B) = \mathbb{P}(A \mid B)\mathbb{P}(B), iterated for three events as \mathbb{P}(C \mid A \cap B)\,\mathbb{P}(B \mid A)\,\mathbb{P}(A). This is the skeleton of Markov models and LLM autoregression.

1.2. Independence

1.3. Bayes’ formula

1.4. Simpson’s paradox

2. Random variables

Chapter 1 gives probabilities to events, but data are numbers: a loss, a count, a measurement. This chapter models such numbers as random variables, describes them by their distribution, moments and first inequalities, and ends with finite Markov chains.

A standard model for a dataset is an n-tuple X_1, \dots, X_n that is iid (independent, identically distributed) according to an unknown distribution P. This is an assumption to check, not a property automatically shared by rows, graph nodes, tokens or repeated measurements.

2.1. Cumulative distribution function

2.2. Variables with a density: change of variables

2.3. Expectation, moments, moment generating function

2.4. Basic inequalities

2.5. Covariance, correlation, independence: untangling

2.6. Random vectors and conditional expectation

For a vector with joint density, marginals integrate out the other variables: f_{X_1}(x_1) = \int f\,dx_2 \dots dx_d. The conditional density is f_{X\mid Y}(x \mid y) = f(x,y)/f_Y(y) when f_Y(y)>0, and Bayes reads in densities: f_{Y\mid X}(y \mid x) \propto f_{X\mid Y}(x\mid y)\, f_Y(y).

2.7. Finite Markov chains

In a random walk on a graph or a bigram language model, the next value depends on the past only through the present.

Detailed balance is checked one pair of states at a time; samplers are built to satisfy it for a target \pi.

2.8. Generating and manipulating distributions in Python

import numpy as np

rng = np.random.default_rng(seed=42)          # modern generator (PCG64)
x = rng.normal(loc=0.0, scale=1.0, size=1000) # 1000 draws from N(0,1)
u = rng.uniform(0, 1, size=100_000)

# Inverse method (theorem 2.4 + section 2.1): uniform U -> exponential
lam = 2.0
expo = -np.log(1.0 - u) / lam
print(expo.mean().round(4), (1/lam))          # ~0.5 = E[X] of E(2)

# Histogram (empirical density) and the model density
import matplotlib.pyplot as plt
plt.hist(x, bins=40, density=True, alpha=.6, label='sample')
grid = np.linspace(-4, 4, 200)
plt.plot(grid, np.exp(-grid**2/2)/np.sqrt(2*np.pi), 'r', label='N(0,1) density')
plt.legend(); plt.show()

print(x.mean(), x.var(ddof=1))   # empirical mean, unbiased variance
print(np.quantile(x, [0.25, 0.5, 0.75]))  # empirical quantiles (chap. 3)

3. Common probability distributions

A dozen named distributions cover most of the models met in Machine Learning. These are reusable modeling “bricks”. Know their support, parameters, expectation and variance, and recognize when they appear. Use the table as a reference rather than a memorization gate.

Conventions: Gamma uses shape and scale throughout these notes; the Laplace (double-exponential) distribution is distinct from Poisson and exponential — its link to \ell_1 regularization appears in chapter 8.

3.1. Gaussians: marginals and conditionals

3.2. Sums of variables: stability under convolution

3.3. Order statistics

import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(0, 1, size=(10_000, 9))
print(x.max(axis=1).mean().round(3))   # ~0.900 = n/(n+1)
q = np.quantile(rng.normal(size=100_000), [0.05, 0.25, 0.5, 0.75, 0.95])
print(q.round(3))                    # ~[-1.645 -0.674 0 0.674 1.645]

3.4. Going further: the measure-theoretic layer

Self-study, optional. These remarks complete chapters 1, 2 and 4; the in-class sessions use at most their statements. Items 6 and 7 are best read after chapter 4.

1. Continuity along monotone sequences. If A_n\uparrow A, then \mathbb{P}(A_n)\uparrow\mathbb{P}(A). Set C_1=A_1 and C_n=A_n\setminus A_{n-1}: these sets are disjoint with union A, and \textstyle\sum_{k\leq n}\mathbb{P}(C_k)=\mathbb{P}(A_n), so σ-additivity concludes. Taking complements gives the decreasing case, A_n\downarrow A; theorem 2.3 uses both.

2. Why a σ-algebra (Vitali). On [0,1] there are subsets to which no length can be given: Vitali’s construction, which uses the axiom of choice, rules out a translation-invariant, σ-additive length on all subsets. This is why a probability is defined on a σ-algebra \mathcal{F} rather than on every subset of \Omega.

3. Borel sets and measurability. The Borel σ-algebra of \mathbb{R} is the smallest σ-algebra containing the intervals. A map X is measurable when X^{-1}(B)\in\mathcal{F} for every Borel set B. Checking the half-lines \{X\leq t\} suffices (definition 2.1), because they generate the Borel sets; for the same reason, independence of variables (definition 1.6) can be checked on intervals.

   Two σ-algebras \mathcal{G} and \mathcal{H} are independent when every G\in\mathcal{G} and H\in\mathcal{H} satisfy \mathbb{P}(G\cap H)=\mathbb{P}(G)\,\mathbb{P}(H).

4. The law as a pushforward, and the converse of theorem 2.3. The distribution of X is the pushforward measure \mathbb{P}_X=\mathbb{P}\circ X^{-1} on the Borel sets. Conversely, every nondecreasing, right-continuous F with limits 0 and 1 is the CDF of a unique law: Carathéodory’s extension theorem extends the set function (a,b]\mapsto F(b)-F(a) from intervals to all Borel sets.

5. Tonelli–Fubini. For a measurable h\geq0, a double integral can be computed in either order (Tonelli), and the same holds for integrable h (Fubini). When X\perp Y, the joint law is the product measure \mathbb{P}_X\otimes\mathbb{P}_Y, and integrating xy in either order gives \mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y] (proposition 2.7).

6. Almost sure convergence and Borel–Cantelli. The events A_n(\varepsilon)=\bigcup_{k\geq n}\{|X_k-X|>\varepsilon\} decrease to “deviations above \varepsilon infinitely often”, which has probability 0 under a.s. convergence, so \mathbb{P}(A_n(\varepsilon))\to0 by item 1. Since \{|X_n-X|>\varepsilon\}\subseteq A_n(\varepsilon), a.s. convergence implies convergence in probability (proposition 4.2).

   The Borel–Cantelli lemmas say that if \sum_n\mathbb{P}(B_n)<\infty, almost surely only finitely many B_n occur; if the B_n are independent and \sum_n\mathbb{P}(B_n)=\infty, almost surely infinitely many occur. For the counterexample of proposition 4.2, \sum 1/n=\infty: the jumps X_n=n recur forever, so there is no a.s. convergence.

   The same tools prove the strong law (\bar X_n\to\mu a.s. when \mathbb{E}|X_1|<\infty), from which the integrable-only weak law of theorem 4.3 follows, and Glivenko–Cantelli: the empirical CDF converges uniformly, \sup_t|F_n(t)-F(t)|\to0\quad\text{a.s.}

7. Characteristic functions and Lévy’s theorem. The characteristic function \varphi_X(t)=\mathbb{E}[e^{itX}] always exists and characterizes the law. Lévy’s continuity theorem: X_n\to X in distribution if and only if \varphi_{X_n}\to\varphi_X pointwise. With finite variance, \varphi_Z(t/\sqrt n)=1-t^2/(2n)+o(1/n), so \varphi_{S_n}(t)\to e^{-t^2/2}, which proves the CLT (theorem 4.4) without the MGF assumption of its proof sketch.

4. Convergence and limit theorems

ML optimizes on a finite sample and hopes for good performance on future data. Limit theorems describe how empirical averages behave under a sampling model — they do not by themselves justify selecting a predictor after inspecting that sample (chapter 9).

For bounded variables, Hoeffding’s exponential bound is stated and proved in Machine Learning ch. 0, Theorem 0.3.

4.1. Empirical quantiles and the bootstrap

The empirical CDF F_n(t) = \frac1n \sum_i \mathbf{1}_{X_i \leq t} is a plug-in estimator of F, consistent at each t by the LLN. Quantiles follow by inversion.

To approximate uncertainty for a nonlinear statistic (median, ratio, AUC), the nonparametric bootstrap resamples B times with replacement, recomputes the statistic, and reads off the bootstrap quantiles:

import numpy as np
rng = np.random.default_rng(0)
x = rng.standard_t(df=3, size=200)  # symmetric, heavy-tailed distribution

B = 5_000
boot_med = np.array([np.median(rng.choice(x, size=len(x), replace=True))
                     for _ in range(B)])
lo, hi = np.quantile(boot_med, [0.025, 0.975])
print(f"95% bootstrap CI of the median: [{lo:.3f}, {hi:.3f}]")
# Percentile interval = [2.5%, 97.5%] bootstrap quantiles; an approximation,
# not exact in finite samples. Validity depends on the statistic (bootstrap
# can fail for extrema) and on the independence structure: time series and
# related graph observations may need block or cluster resampling instead.

import numpy as np
rng = np.random.default_rng(0)
for n in [1, 5, 30, 200]:
    # 5000 repetitions of the mean of n independent uniforms [0,1]
    means = rng.uniform(0, 1, size=(5000, n)).mean(axis=1)
    std = means.std()
    print(f"n={n:4d}  observed std = {std:.4f} "
          f"(theory 1/sqrt(12 n) = {1/np.sqrt(12*n):.4f})")

5. Parametric estimation

Assume the data distribution belongs to a family \{p_\theta:\theta\in\Theta\} and seek \theta from an iid sample. This chapter develops bias/variance, likelihood and information; chapter 6 turns them into intervals and tests.

5.1. Score and Fisher information

5.2. MLE of the usual distributions, fully derived

5.3. Method of moments and plug-in

6. Confidence intervals and tests

Chapter 5 produces point estimates; this chapter attaches an uncertainty statement to them. A confidence interval gives a range of parameter values with a guaranteed coverage, and a test decides whether the data contradict a reference hypothesis.

6.1. Confidence intervals: construction by pivot quantity

6.2. Hypothesis testing: the Neyman-Pearson framework

A test compares H_0 (null hypothesis) with H_1 (alternative). Choose a test statistic T and a rejection region R:

Power depends on the particular alternative. The p-value is the probability, under the null model, of a statistic at least as extreme as the observed one; reject when p\leq\alpha. A more stringent threshold lowers power; more independent observations can raise it against a fixed nonzero effect — but cannot repair a biased design or a misspecified null.

6.3. Tests on means: z, t, Welch

from scipy import stats
import numpy as np
rng = np.random.default_rng(3)
a = rng.normal(10.2, 2.0, 50)      # group A
b = rng.normal( 9.4, 2.4, 60)      # group B (different variance)

t_stat, p_value = stats.ttest_ind(a, b, equal_var=False)   # Welch
print(f"t = {t_stat:.3f}, p = {p_value:.4f}")
# If p < 0.05, reject equality of means at the preselected 5% level.
# p does NOT mean "probability that the means are equal".

6.4. Chi-square tests

6.5. One-way ANOVA

import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
g1, g2, g3 = rng.normal(5.0, 1, 30), rng.normal(5.3, 1, 30), rng.normal(5.1, 1, 30)
F, p = stats.f_oneway(g1, g2, g3)
print(f"F = {F:.3f}, p = {p:.4f}")
# Read the simulated result rather than assuming a particular p-value.
# Also inspect group means and effect sizes.

6.6. p-values: correct reading and multiple testing

7. Linear regression

A complete predictive baseline in one model: loss, optimization, uncertainty, diagnostics. Write y=X\beta^\star+\varepsilon, \qquad X\in\mathbb{R}^{n\times p},\quad \beta^\star\in\mathbb{R}^p,\quad y,\varepsilon\in\mathbb{R}^n. p counts all coefficients, intercept included. For inference, treat X as fixed with full column rank, n>p, and \varepsilon\sim\mathcal{N}(0,\sigma^2I_n); least-squares fitting itself needs no Gaussian noise.

7.1. Least squares: the geometry of projection

7.2. Inference on the coefficients

import numpy as np
from scipy import stats
rng = np.random.default_rng(1)
n, p = 200, 3
X = np.column_stack([np.ones(n), rng.uniform(0, 10, n), rng.normal(0, 1, n)])
beta_star = np.array([1.0, 2.0, -0.5])
y = X @ beta_star + rng.normal(0, 2.0, n)

# Least squares + full inference by hand (theorem 7.3)
XtX_inv = np.linalg.inv(X.T @ X)
beta_hat = XtX_inv @ X.T @ y
resid = y - X @ beta_hat
sigma2_hat = resid @ resid / (n - p)
se = np.sqrt(np.diag(XtX_inv) * sigma2_hat)
t_stat = beta_hat / se
pvals = 2 * stats.t.sf(np.abs(t_stat), df=n - p)
print(np.c_[beta_hat, se, t_stat, pvals].round(4))
# A realized |t| near 1 gives a two-sided p near 0.3.
# For a prespecified pure-noise coefficient, however, the null p-value varies:
# it is not guaranteed to be nonsignificant in a particular sample.

# Same estimation with scikit-learn; this does not provide the inference table
from sklearn.linear_model import LinearRegression
print(LinearRegression().fit(X[:, 1:], y).coef_.round(3))

The explicit inverse mirrors the formulas. For numerical fitting, prefer QR/SVD-based least squares such as np.linalg.lstsq(X, y, rcond=None), especially when columns are nearly dependent.

7.3. Goodness of fit and diagnostics

8. Bayesian statistics

After 8 successes in 10 trials, the empirical frequency is 0.8 but a uniform Beta prior gives a posterior mean of 0.75: the difference is a modeling choice about uncertainty, not an arithmetic disagreement. The frequentist treats the parameter as fixed; the Bayesian represents uncertainty about it with a prior and updates that distribution with the data. This chapter is optional deepening, useful for posterior inference, latent-variable models and uncertainty analysis.

8.1. Conjugacy: Bernoulli-Beta

from scipy import stats
post = stats.beta(9, 3)
print(post.mean().round(3), post.ppf([0.025, 0.975]).round(3))

8.2. Gaussian-Gaussian conjugacy

8.3. Credible interval vs confidence interval

8.4. MAP vs MLE

8.5. The Bayesian approach in generation

9. From statistics to machine learning

Each statistical pillar becomes a learning question: what is estimated, what loss is minimized, what is random, and what evidence supports performance on new units?

9.1. Empirical risk minimization (ERM)

9.2. Bias-variance in prediction, by computation

import numpy as np
rng = np.random.default_rng(0)

def f(x):                       # true function: modulated sine
    return np.sin(2 * x) + 0.3 * x

n_train, n_rep = 40, 300        # small samples, many repetitions
x_test = np.linspace(-3, 3, 200)
noise = 0.15**2

for deg in range(1, 13):
    preds = np.zeros((n_rep, x_test.size))
    for r in range(n_rep):
        x = rng.uniform(-3, 3, n_train)
        y = f(x) + rng.normal(0, 0.15, n_train)
        c = np.polyfit(x, y, deg)          # polynomial of degree `deg`
        preds[r] = np.polyval(c, x_test)
    sq_b = ((f(x_test) - preds.mean(axis=0))**2).mean()  # bias²
    v = preds.var(axis=0).mean()                         # variance
    print(f"deg={deg:2d}  bias^2={sq_b:.4f}  var={v:.4f}  "
          f"err={sq_b + v + noise:.4f}")
# Low degree: large bias. High degree: unstable variance, worst at boundaries.
# Select complexity by validation, not by decreasing training error.

9.3. MLE to cross-entropy: the exact bridge

A from-scratch affine MNIST classifier is a concrete instance: x\in\mathbb{R}^{784}, logits z=Wx+b, softmax, categorical cross-entropy — the statistical objective without hidden layers.

9.4. Four correspondences to take away

• Empirical risk: LLN/CLT describe a fixed predictor’s loss; selection-aware arguments address learned ones.
• Bias-variance: estimation error becomes a prediction trade-off plus observation noise (simulation 9.3).
• Likelihood → loss: probabilistic models yield squared-error and cross-entropy; VAEs and diffusion optimize variational bounds or denoising objectives (preview: VAEs in Deep Learning ch. 9, diffusion in its section 9.4 and in Multimodal Generative AI ch. 3).
• Conditioning → prediction: \mathbb{E}[Y\mid X] is the squared-loss target; LLMs estimate next-token laws conditioned on context — sequence dependence and evaluation come later.

10. Missing data: mechanisms, baselines and research

Before choosing an imputer, ask what is missing, why, and whether the target is population estimation or prediction from incomplete inputs. This research extension is optional: read it when missingness affects the task.

10.1. Missingness mechanisms: MCAR, MAR, MNAR

10.2. The bias of naive solutions: worked examples

10.3. Doing better: regression imputation and beyond

10.4. Selected research on prediction with missing values

Exercises

1. ★ Practical Bayes. A test detecting a virus affecting 1% of the population has sensitivity 99% and specificity 95%. A patient tests positive. What is the probability of infection? What if two tests are positive, assuming their outcomes are independent conditional on infection status?

   One positive: \mathbb{P}(M\mid+) =\frac{0.99\times0.01}{0.99\times0.01+0.05\times0.99} \approx0.167. Two positives under the stated conditional independence: \mathbb{P}(M\mid++) =\frac{0.99^2\times0.01}{0.99^2\times0.01+0.05^2\times0.99} =\frac{0.009801}{0.012276} \approx0.798. The probability rises from about 17% to 80%. Repeating the calculation with squared likelihoods is unjustified if the tests share errors beyond their common dependence on infection status.

2. ★ Reverse Simpson. Build a contingency table with 2 groups and 2 treatments in which treatment A is better overall but worse within each group. Explain the mechanism.

   Swap the treatment labels in example 1.11:

   Group	A	B
   Mild	470/500 = 94%	95/100 = 95%
   Severe	8/50 = 16%	80/200 = 40%
   Overall	478/550 ≈ 87%	175/300 ≈ 58%

   A receives many more mild cases, so the aggregate comparison reverses the within-group comparison. Randomization can remove this assignment imbalance in expectation; observational adjustment needs a justified causal model.

3. ★ Normal approximation. A factory produces parts of which 8% are defective. In a batch of 500 independent parts, approximate the probability of more than 55 defects.

   N\sim\mathcal{B}(500,0.08) has mean 40 and variance 36.8. Without continuity correction, \mathbb{P}(N>55)\approx1-\Phi\!\left(\frac{55-40}{\sqrt{36.8}}\right)\approx0.007. Since N>55 means N\geq56, a continuity correction uses 55.5 and gives approximately 0.0053. Compare both with scipy.stats.binom.sf(55, 500, 0.08); a tail approximation can have appreciable relative error.

4. ★ Numerical Chebyshev. Request time has mean 200 ms and standard deviation 40 ms. (a) Bound the probability of exceeding 320 ms without specifying a distribution. (b) Compute it under a Gaussian model.

   1. Without any distributional assumption, the inclusion \{X>320\}\subseteq\{|X-200|\geq120\} lets Chebyshev bound \mathbb{P}(X>320)\leq\frac{40^2}{120^2}=\frac19\approx0.111.
   2. Under a Gaussian model, \mathbb{P}(X>320)=1-\Phi(3)\approx0.00135: the distribution-free bound is about 82 times larger. Chebyshev is conservative here, and the smaller number is only as credible as the Gaussian tail assumption.

5. ★ Stable sums. Using MGFs, show that the sum of two independent Poissons is Poisson and that two independent gammas with the same scale sum to a gamma. Deduce the sum and mean distributions for iid exponentials.

   Poisson: e^{\lambda_1(e^t-1)}e^{\lambda_2(e^t-1)} =e^{(\lambda_1+\lambda_2)(e^t-1)}. Gamma: (1-bt)^{-a_1}(1-bt)^{-a_2}=(1-bt)^{-(a_1+a_2)}. Thus \sum_iX_i\sim\Gamma(n,1/\lambda) for iid \mathcal{E}(\lambda) variables, and \bar X_n\sim\Gamma(n,1/(n\lambda)). Keep scale and rate conventions separate.

6. ★ Full t interval in code. Write mean_ci(x, alpha=0.05) returning the t confidence interval for a mean. Over 10,000 Gaussian samples of size 20, check the proportion containing the true mean.

   import numpy as np
   from scipy import stats
   
   def mean_ci(x, alpha=0.05):
       x = np.asarray(x)
       n = len(x)
       m, s = x.mean(), x.std(ddof=1)
       tq = stats.t.ppf(1 - alpha/2, df=n - 1)
       return m - tq*s/np.sqrt(n), m + tq*s/np.sqrt(n)
   
   rng = np.random.default_rng(0)
   mu, n_rep = 3.0, 10_000
   covered = 0
   for _ in range(n_rep):
       sample = rng.normal(mu, 1.0, 20)
       lo, hi = mean_ci(sample)   # both endpoints from the SAME sample
       covered += (lo <= mu <= hi)
   
   print(covered / n_rep)         # close to 0.95
   print(np.sqrt(0.95 * 0.05 / n_rep))  # Monte Carlo SE ~0.0022

   Each repetition builds one interval from one sample; the Monte Carlo fluctuation (±0.002 here) explains deviations from exactly 0.95. Next: try skewed or dependent observations and name the assumption that changed.

7. ★ Sequential Bayesian updating. Start with a Beta(2,2) prior on click probability p. Observe 3 clicks, then 1 failure, then 6 clicks. Give each posterior, the final 90% credible interval and the MAP estimate. Distinguish the MAP from posterior-predictive click probability.

   The successive posteriors are Beta(5,2), Beta(5,3) and Beta(11,3). The final MAP is \frac{11-1}{11+3-2}=\frac{10}{12}=\frac56\approx0.833. The posterior-predictive probability of the next click is instead the posterior mean 11/14\approx0.786. The equal-tail 90% credible interval is scipy.stats.beta(11, 3).ppf([0.05, 0.95]), approximately [0.59,0.93]. Thompson sampling uses a draw from this posterior, not just its mode.

8. ★ Sampling distribution and coverage in code. Simulate 10,000 Gaussian samples of size 50, plot their sample means with the theoretical density, and check coverage of \bar X_n\pm1.96s/\sqrt n.

   import numpy as np, matplotlib.pyplot as plt
   from scipy import stats
   rng = np.random.default_rng(0)
   samples = rng.normal(5.0, 2.0, size=(10_000, 50))
   means = samples.mean(axis=1)
   plt.hist(means, bins=60, density=True, alpha=.6)
   grid = np.linspace(4, 6, 300)
   plt.plot(grid, stats.norm(5.0, 2.0/np.sqrt(50)).pdf(grid), 'r')
   plt.show(); plt.close()
   lo, hi = means - 1.96*samples.std(axis=1, ddof=1)/np.sqrt(50), means + 1.96*samples.std(axis=1, ddof=1)/np.sqrt(50)
   print(((lo <= 5.0) & (5.0 <= hi)).mean())   # close to, but below, 0.95
   
   # Exact Gaussian t procedure, since sigma was estimated
   tq = stats.t.ppf(0.975, df=49)
   se = samples.std(axis=1, ddof=1) / np.sqrt(50)
   lo_t, hi_t = means - tq*se, means + tq*se
   print(((lo_t <= 5.0) & (5.0 <= hi_t)).mean())  # close to 0.95

   Gaussian sample means are exactly Gaussian. With estimated s, the z interval covers ≈94.4% here; the exact t critical value is 2.010, not 1.96 — ddof=1 fixes the variance estimator, not the critical value.

9. ★ A two-state chain and a random walk. (a) The weather is sunny (state 1) or rainy (state 2), with transition matrix P=\begin{pmatrix}0.8&0.2\\0.4&0.6\end{pmatrix}. Find the stationary law, the probability of rain in two days given sun today, and the factor by which the gap to the stationary law shrinks at each step. (b) On a connected undirected graph with m edges and degrees d_v, show that the random walk P=D^{-1}A satisfies detailed balance with \pi_v=d_v/(2m).

   1. Detailed balance 0.2\,\pi_1=0.4\,\pi_2 with \pi_1+\pi_2=1 gives \pi=(2/3,1/3). In two steps from state 1, (P^2)_{12}=0.8\times0.2+0.2\times0.6=0.28. The second eigenvalue is \operatorname{tr}P-1=0.4: the gap is multiplied by 0.4 at each step, as in example 2.19. Check with np.linalg.matrix_power(P, 50)[0], which is close to (0.667,0.333).

   2. For an edge \{u,v\}, \pi_uP_{uv}=\frac{d_u}{2m}\cdot\frac1{d_u}=\frac1{2m}=\pi_vP_{vu}, and both sides are 0 for a non-edge. Proposition 2.18 then makes \pi stationary: this is Proposition 0.6 of Graph ML, behind random-walk embeddings and PageRank.

10. ★★ MLE of the exponential distribution. For iid X_i\sim\mathcal{E}(\lambda), show that \hat\lambda_{\mathrm{MLE}}=1/\bar X_n and compute its bias.

    \ell(\lambda)=n\log\lambda-\lambda\sum_iX_i gives \hat\lambda=n/\sum_iX_i. With the shape-scale convention, \sum_iX_i\sim\Gamma(n,1/\lambda), \qquad \bar X_n\sim\Gamma\big(n,1/(n\lambda)\big). For n>1, \mathbb{E}[1/\bar X_n]=n\lambda/(n-1), so the bias is \lambda/(n-1). For n=1, the estimator has infinite expectation.

11. ★★ Poisson MLE and Fisher. For iid X_i\sim\mathcal{P}(\lambda), \lambda>0: (a) derive the MLE; (b) compute Fisher information using both identities of theorem 5.6; (c) check the Cramér-Rao bound.

    1. The MLE is \hat\lambda=\bar X_n. (b) For one observation the score is s(\lambda)=X/\lambda-1, so its variance gives I(\lambda)=\mathrm{Var}(X/\lambda)=\frac{\lambda}{\lambda^2}=\frac1\lambda, and the curvature identity gives the same value: -\mathbb{E}[\ell''(\lambda)]=\mathbb{E}[X]/\lambda^2=1/\lambda. Finally, for (c): since \mathrm{Var}(\bar X_n)=\lambda/n=1/(nI(\lambda)), the unbiased estimator attains the bound in finite samples.

12. ★★ Change of variable. If X\sim\mathcal{N}(0,1), determine the densities of Y=X^2 and Z=|X|.

    Squaring is not injective: split into positive and negative branches and sum their contributions: f_Y(y)=\frac1{\sqrt{2\pi y}}e^{-y/2},\qquad y>0. This is \chi^2_1, equivalently \Gamma(1/2,2) in shape-scale notation. For the half-normal variable, f_Z(z)=2\varphi(z) for z\geq0, and zero for z<0.

13. ★★ Applied Neyman-Pearson. For iid X_i\sim\mathcal{E}(\lambda), derive the likelihood-ratio test of \lambda=\lambda_0 against \lambda=\lambda_1>\lambda_0.

    \frac{L_1}{L_0} =\left(\frac{\lambda_1}{\lambda_0}\right)^n \exp\left(-(\lambda_1-\lambda_0)\sum_iX_i\right). The ratio increases as S=\sum_iX_i decreases, so reject for S\leq c. Under H_0, S\sim\Gamma(n,1/\lambda_0) in shape-scale notation.

    Choose c as its lower \alpha quantile, so \mathbb{P}_{\lambda_0}(S\leq c)=\alpha.

14. ★★ Regression by hand. Derive the normal equations and show uniqueness when the columns of X are linearly independent.

    Set \nabla_\beta\|y-X\beta\|^2=2X^\top(X\beta-y)=0, giving X^\top X\beta=X^\top y. For nonzero a, a^\top X^\top Xa=\|Xa\|^2>0 iff X has independent columns. Thus X^\top X is positive definite and invertible. Reproduce the same result geometrically using theorem 7.1.

15. ★★ Gauss-Markov in dimension 1. In y_i=\mu+\varepsilon_i, with zero-mean uncorrelated errors of common variance \sigma^2, show that the sample mean has minimum variance among linear unbiased estimators \sum_iw_iy_i.

    Unbiasedness requires \sum_iw_i=1. Cauchy-Schwarz gives \mathrm{Var}\Big(\sum_iw_iy_i\Big) =\sigma^2\sum_iw_i^2 \geq\sigma^2\frac{(\sum_iw_i)^2}{n} =\frac{\sigma^2}{n} =\mathrm{Var}(\bar Y). Equality holds iff all weights equal 1/n. This is the one-parameter case of theorem 7.2.

16. ★★★ Missing data. In a MAR setting where X_2 is more often missing for large X_1, explain why consistently estimating and imputing \mathbb{E}[X_2\mid X_1] can recover \mathbb{E}[X_2] but underestimate \mathrm{Var}(X_2). State the required overlap or extrapolation assumption. What addition addresses the missing conditional variance?

    MAR plus overlap allows the observed rows to identify the conditional law at relevant X_1 values, subject to adequate estimation. A deterministic cutoff instead needs a justified extrapolation model, such as a correctly specified linear Gaussian conditional mean.

    Let m(X_1)=\mathbb{E}[X_2\mid X_1]. Replacing a missing value by m(X_1) preserves its conditional mean and hence the marginal mean by the tower property. But it removes residual variation around that mean. Stochastic imputation adds a draw from the estimated conditional residual law; calibrated conditional variance, not arbitrary noise, is required. Proper multiple imputation also accounts for uncertainty in the fitted imputation model.

Exit checkpoint and connections

Before starting Machine Learning — after the four sessions and the self-study chapters 3, 6, 7 and 9 — produce three pieces of evidence:

1. A conditioning calculation with the denominator visible (exercise 1).
2. A reproducible coverage experiment, one sample per interval, assumptions stated (exercise 6).
3. A loss/model derivation plus an evaluation argument: why training loss is not held-out performance.

Graph × LLM transfer check

Toy evaluation, not a reported benchmark. A text-only baseline and a graph-assisted model answer the same 30 questions in three source groups (10 each). Deployment concerns new groups; questions within a group may be dependent.

Checkpoint. Compute the overall gain. Does the table establish a reliable improvement on new groups? Specify the split, baseline and uncertainty unit for a larger experiment, and identify a visible failure case.

Capstone evaluation kit

Self-study, about 1 h with the checkpoint above. The projects below, the Period 2 team projects and the end-of-year capstones all end with a claim of the form “system B beats system A on new units”. This kit turns the transfer check into a protocol for such claims.

1. Split by deployment unit, then by time. Name the unit the claim is about — a document, a user, a source, a graph component, a molecular scaffold — and put all its rows on one side of the split. With timestamps, train on the past and test on the future. Fit preprocessing, imputation (chapter 10) and graph construction on the training side only; Graph ML Example 8.1 shows the leak this prevents for edges.
2. Seeds. Re-run every configuration with at least 3 seeds in Year 1 and at least 5 in Year 2. Report the mean and the spread across seeds, and never keep only the best seed.
3. Paired, cluster-level intervals. Evaluate both systems on the same items, take per-item differences, and bootstrap whole units, not rows — the group-level uncertainty of the transfer check. Report the gain with its 95% interval. McNemar’s test, on the pairs where exactly one system is right, is the matching paired test for accuracies. With fewer than about 20 units, report the per-unit results instead of an interval.
4. Multiple testing. Count every variant compared on the evaluation data: prompts, graphs, hyperparameters. Control the family with Bonferroni or Benjamini–Hochberg (section 6.6), or label the analysis exploratory.
5. Winner’s curse. The best of K validation scores is biased upward: in example 3.6 the maximum of 9 uniform scores averages 0.9. Select on validation, write down the analysis plan, then open the test set once.
6. Structure control. For any claim that a graph helps, rerun the pipeline with shuffled edges (degree-preserving) and with no edges. A gain that survives the shuffle does not come from the graph’s edges.

import numpy as np

def paired_cluster_bootstrap(diff, group, B=10_000, seed=0):
    """Mean per-item difference and a 95% CI that resamples whole groups."""
    rng = np.random.default_rng(seed)
    labels = np.unique(group)
    sums = np.array([diff[group == g].sum() for g in labels])
    sizes = np.array([(group == g).sum() for g in labels])
    idx = rng.integers(0, len(labels), size=(B, len(labels)))
    boot = sums[idx].sum(axis=1) / sizes[idx].sum(axis=1)
    return diff.mean(), np.quantile(boot, [0.025, 0.975])

rng = np.random.default_rng(1)
group = np.repeat(np.arange(40), 15)        # 40 sources, 15 questions each
gain = rng.normal(0.3, 1.0, 40)[group]      # the graph helps some sources only
a = rng.random(600) < 0.6                   # text-only correct?
b = rng.random(600) < 1 / (1 + np.exp(-(np.log(1.5) + gain)))  # graph-assisted
diff = b.astype(float) - a.astype(float)
print(paired_cluster_bootstrap(diff, group))     # this seed: 0.005, [-0.082, 0.090]
row = rng.integers(0, 600, size=(10_000, 600))   # the wrong, row-level bootstrap
print(np.quantile(diff[row].mean(axis=1), [0.025, 0.975]))  # [-0.050, 0.058]: too narrow

Each piece is taught in depth elsewhere: the reference pipeline of Machine Learning section 4.6, with its leakage and nested cross-validation sections, and its section 4.9 on comparing two models (paired tests, McNemar); the minimal experimental protocol of Advanced Deep Learning Definition 1.1; the honest-benchmarking rules of Boosting to Foundation Models ch. 9; data and model tests in ML Ops ch. 2.

Where to go next—and what remains

Main route: refreshers as needed → Machine Learning → Deep Learning, then the graph and language branches. Bayesian inference, missing-data research and further probability are selective extensions, not universal prerequisites. Return to the reading route when a module exposes a gap.

Sources and scope

Source layers stay separate: catalog information and this site’s study recommendations are not the same thing. Consult official sources for enrolment and assessment requirements.

The record used is the SynapseS 2026–2027 sheet for APM_51438_EP (captured as supplied; the live catalog was not rechecked):

Because the sheet displays no description, objectives or syllabus, no “official topic → chapter” correspondence table can be drawn up: the ten-chapter outline is not checked against an official list.

The ≈36 h full-pass budget (the 12 h in class plus ≈ 24 h of self-study, see the session plan) is an optional estimate — neither a conversion of the 12 h CTD nor an ECTS allocation; the session split is likewise our proposal. The chapter sequence, simulations, ML bridge, transfer check and selected papers are editorial study choices, not official prerequisites or equivalences.

Projects and assessment

Official assessment: not displayed. The SynapseS 2026–2027 sheet of APM_51438_EP gives the grade format — numeric /20, with a US letter grade — but its assessment section for the LLGA track is empty: no exam, project or resit rule is shown (sources). Everything below is therefore a proposal sized for a 12 h refresher: solo subjects of 12–15 h, group subjects of 20–25 h per member, a two-hour written exam and a rubric.

The projects exercise two master objectives: O1 (ML and DL foundations: likelihood, estimation, uncertainty) and O7 (hands-on practice: an honest evaluation protocol). The graph-versus-text study adds O5 (Graph × LLM), and the prompt A/B test adds O3 (large language models). Both group subjects apply the capstone evaluation kit that the end-of-year projects reuse.

Solo projects

1. [Solo · ≈ 12 h] Coverage under dependence: when iid standard errors lie. How far below 95% do the usual intervals for a mean fall when observations are heavy-tailed or share a source, and which resampling scheme repairs them?
   • Implement: a simulator of the coverage of the z, t, percentile-bootstrap and cluster-bootstrap intervals, for iid Gaussian, Student t_3 and clustered data (clusters of size m, intra-cluster correlation \rho from 0 to 0.6), then for node features on a stochastic block model.
   • Prove: from the variance formula with covariances (chapter 4), the design effect 1+(m-1)\rho and the coverage 2\Phi\big(1.96/\sqrt{1+(m-1)\rho}\big)-1 that it predicts for the iid interval.
   • Data: synthetic, with networkx.stochastic_block_model for the graph case. Link: the leakage-free splits of Graph ML Example 8.1.
   • Deliverables: code, 4-page report with coverage-versus-\rho curves, 10-minute talk. Chapters: ch. 4, 6; exercises 6 and 8.
   • Success criterion: with 2,000 replications, the measured iid coverage matches the prediction within 3 points (about 69% for m=10, \rho=0.3), and the cluster bootstrap reaches at least 92% from 30 clusters on, while its under-coverage with 10 clusters is shown.
2. [Solo · ≈ 12 h] From likelihood to loss: count and click models from scratch. Do hand-written maximum-likelihood fits give the same coefficients, standard errors and losses as library code, and when do Fisher, bootstrap and delta-method intervals disagree?
   • Implement: logistic and Poisson regression by Newton’s method in NumPy; standard errors from the observed Fisher information; bootstrap intervals; a delta-method interval for an odds ratio or a rate ratio e^{\beta_j}.
   • Prove: the fitted objectives are the binary cross-entropy and the Poisson negative log-likelihood (theorem 9.4), and with an intercept the Poisson score equation makes predicted and observed totals equal.
   • Data: simulated well-specified data first; then the breast-cancer data bundled with scikit-learn (569 tumours) and the French motor-insurance claim counts freMTPL2freq (OpenML id 41214), with exposure as sample weight — equivalent to a log-exposure offset — and one reference level dropped per categorical feature.
   • Trap: the 30 breast-cancer features separate the two classes linearly, so the unpenalized logistic MLE does not exist. Show that the Newton coefficients grow without bound, then fit the 10 standardized “mean” features, where the MLE exists.
   • Deliverables: code, 4-page report, 10-minute talk. Chapters: ch. 4 (delta method, bootstrap), 5, 9.
   • Success criterion: coefficients agree with scikit-learn (LogisticRegression(penalty=None) and PoissonRegressor(alpha=0), both with solver="newton-cholesky" and a tight tol) to 10^{-6}; Fisher and bootstrap standard errors agree within 10% on the simulated data, and the report measures and explains the gap on claim counts (overdispersion).
3. [Solo · ≈ 15 h] Paper study: which imputation to predict with missing values? Does a better imputation give a better predictor, and does keeping the missingness mask matter? A paper study of Le Morvan, Josse, Scornet and Varoquaux (NeurIPS 2021), with Le Morvan and Varoquaux (2024) for the “diminishing returns” claim — papers 3 and 4 of section 10.4.
   • Implement: mean imputation with and without mask indicators and iterative (MICE-style) imputation with scikit-learn’s IterativeImputer, each followed by a linear model and by gradient boosting, against the native missing-value support of HistGradientBoostingRegressor; MCAR, MAR and self-masked MNAR masks at 30%.
   • Data: Gaussian synthetic data as in example 10.2, then California housing (sklearn.datasets.fetch_california_housing) with the same masks.
   • Deliverables: code, 4–6 page report, 10-minute talk. Chapters: ch. 10, 6, and the evaluation kit.
   • Success criterion: every imputer is fitted inside the training folds; held-out R^2 differences come with intervals over 5 seeds × 5 folds; the report states whether the 2024 claim holds on these data.

Group projects

1. [Group of 3 · ≈ 25 h each] Does the graph really help? A rigorous text-only versus graph-assisted evaluation. On multi-hop questions, does a knowledge-graph view in the prompt of a small LLM raise accuracy on new sources, beyond an equally long graph with shuffled edges? It scales up the transfer check and seeds the evaluation of the Graph × LLM capstones.
   • Data: the 2WikiMultiHopQA dev set (Ho, Duong Nguyen, Sugawara and Aizawa, COLING 2020), 600–1,000 questions clustered by their first gold Wikipedia article. The graph view is extracted from the context paragraphs, never from the gold evidence triples, which serve only for error analysis.
   • Implement: three prompts for one open model (for example Qwen2.5-1.5B-Instruct on a free GPU tier): text only; text plus a 2-hop triple graph; text plus the same graph with degree-preserving shuffled edges.
   • Statistics: paired per-question differences, cluster bootstrap by article (the kit’s code), McNemar’s test, Holm or Benjamini–Hochberg across variants, and a one-page analysis plan written before the test questions are scored.
   • Split of work: A builds the retrieval and graph pipeline; B runs the models with fixed seeds and logs per-question outputs; C owns the analysis plan, the bootstrap and the shuffled control.
   • Deliverables: repository, analysis plan, 6-page report, 15-minute talk. Chapters: ch. 4, 6, 9 and the exit checkpoint. Links: Graph ML ch. 10 (GraphRAG), Large Language Models ch. 7.
   • Success criterion: the gain, or its absence, is reported with a 95% cluster interval, next to the shuffled control and after correction across variants — exactly as the plan said.
2. [Group of 2 · ≈ 20 h each] Bayesian versus frequentist prompt A/B testing, and the winner’s curse. Among eight prompts for a labelled task, which is best, how many labelled queries does each design spend to find it, and how optimistic is the winning score?
   • Data: SST-2 sentiment (Socher et al., 2013; GLUE validation split, 872 sentences) or BoolQ (Clark et al., 2019; 3,270 dev questions), with one small open instruct model and fixed decoding.
   • Implement: (a) a fixed-horizon design — every prompt on the same items, paired comparisons with the baseline prompt, Bonferroni and Benjamini–Hochberg; (b) Beta–Bernoulli Thompson sampling over prompts (chapter 8), stopped when the posterior probability of the leader exceeds 0.95 or a fixed query budget is spent, since near-equal prompts may never reach 0.95.
   • Winner’s curse: measure the optimism of the selected prompt — validation accuracy minus accuracy on a held-out set touched once — and compare it with a simulation of eight equally good prompts (example 3.6).
   • Split of work: A builds the evaluation harness and design (a); B builds design (b) and the simulation. Link: Large Language Models ch. 9 (benchmarks and contamination).
   • Deliverables: code, 4–6 page report, 15-minute joint talk. Chapters: ch. 3, 6, 8.
   • Success criterion: both designs report their decision and their query count; the measured optimism comes with an interval and is compared with the simulated null.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, a non-programmable calculator; four independent questions. Use z_{0.975}=1.96.

1. Question (5 points). A colleague writes: “the 95% confidence interval for the accuracy gain is [0.004,0.076], so the gain lies in it with probability 0.95; and p=0.03 means a 3% chance that the gain is zero.” Correct both statements, then describe a setting in which the first reading becomes legitimate.

   Expected: coverage is a property of the procedure over repeated samples (definition 6.1); the realized interval contains the fixed gain or not. A p-value is a tail probability computed under H_0; turning it into a probability of H_0 needs a prior, as in the medical test of example 1.8. With a prior, a 95% credible interval satisfies the first reading (definition 8.5, example 8.6).

2. Question (6 points). Two models answer the same 400 independent questions. Model A is right on 300 and model B on 316; A alone is right on 20 questions and B alone on 36. (a) Give the accuracy gain and a 95% interval from the paired differences. (b) Test equal accuracy with McNemar’s statistic, in its normal approximation. (c) What changes if the 400 questions come from 20 documents of 20 questions?

   Expected: (a) the gain is 16/400=0.04; the differences are +1 (36 times), -1 (20 times) or 0, with variance 0.14-0.04^2=0.1384, so \mathrm{SE}=\sqrt{0.1384/400}\approx0.0186, \qquad 0.04\pm1.96\times0.0186=[0.004,\,0.076].

   2. Under H_0, B’s wins among the 56 discordant pairs follow \mathcal{B}(56,1/2), so z=(36-20)/\sqrt{56}\approx2.14 and p\approx0.03 (exact binomial: 0.044): reject at 5%. (c) The questions are no longer independent: average the differences per document and build the interval from the 20 document means (a t_{19} interval or a cluster bootstrap). With positively correlated questions within a document it is wider, and the conclusion may change.

3. Question (5 points). (a) Prove that if a probability vector \pi is in detailed balance with a transition matrix P, then \pi^\top P=\pi^\top. (b) For the two-state matrix below, with 0<a,b<1, find \pi and prove that \mu_t(1)-\pi_1=(1-a-b)^t\,(\mu_0(1)-\pi_1). P=\begin{pmatrix}1-a&a\\b&1-b\end{pmatrix}

   Expected: (a) sum \pi_iP_{ij}=\pi_jP_{ji} over i and use that row j sums to 1 (proposition 2.18). (b) Detailed balance a\pi_1=b\pi_2 gives \pi=\big(b/(a+b),\,a/(a+b)\big). One step gives \mu_{t+1}(1)=(1-a)\mu_t(1)+b\,(1-\mu_t(1))=b+(1-a-b)\,\mu_t(1); subtract the same identity for the fixed point \pi_1 and iterate. Since |1-a-b|<1, the law converges from any start.

4. Question (4 points). Let U_1,\dots,U_K be iid uniform on [0,1]. Prove that \mathbb{E}[\max_kU_k]=K/(K+1). Explain why reporting the best of K validation scores overstates performance, and give the protocol that fixes it.

   Expected: \mathbb{P}(\max_kU_k\leq t)=t^K (proposition 3.5), so the density is Kt^{K-1} and \mathbb{E}[\max_kU_k]=\int_0^1Kt^K\,dt=\frac{K}{K+1}. When variants are equally good, the selected score is the maximum of noisy estimates and is biased upward — 0.9 for K=9 (example 3.6). Select on validation (or nested cross-validation), then evaluate the selected model once on an untouched test set, and count the K comparisons.

Total: 5 + 6 + 5 + 4 = 20 points.

Grading rubric (proposal)

References

Reading recommendations, not a prescribed catalog bibliography. Choose a reference for a specific gap rather than treating the list as additional compulsory reading.

• L. Wasserman, All of Statistics: A Concise Course in Statistical Inference, Springer, 2004 — a compact reference across probability, inference and statistical learning.
• G. Casella, R. Berger, Statistical Inference, 2nd ed., Duxbury, 2001 — deeper derivations and conditions for estimation and testing.
• J. Rice, Mathematical Statistics and Data Analysis, 3rd ed., Duxbury, 2007 — a data-oriented complement to the worked examples.
• C. Robert, The Bayesian Choice, Springer, 2007 — optional depth on Bayesian decision theory and inference.
• A. Gelman, J. Carlin, H. Stern, D. Dunson, A. Vehtari, D. Rubin, Bayesian Data Analysis, 3rd ed., CRC Press, 2013 — modeling and posterior analysis beyond conjugate examples.
• R. Little, D. Rubin, Statistical Analysis with Missing Data, 3rd ed., Wiley, 2019 — missingness mechanisms, likelihood methods and multiple imputation.
• Selected papers for chapter 10:
  • Linear predictor on linearly-generated data with missing values: non consistency and solutions, AISTATS 2020.
  • NeuMiss networks: differentiable programming for supervised learning with missing values, NeurIPS 2020.
  • What’s a good imputation to predict with missing values?, NeurIPS 2021.
  • Imputation for prediction: beware of diminishing returns, 2024.
  • Public research publication list — further bibliography, not authorship or endorsement of these independent notes.