Statistics in Action

Contents

• Introduction and overview
• 0. Review: the statistician’s survival kit
• 1. The life cycle of a statistical study
• 2. Experimental designs: randomization, blocks, power
• 3. Case study: agricultural trials and blocks
• 4. Observational versus experimental: biases
• 5. Causal inference: potential outcomes and DiD
• 6. Regression in practice: diagnostics and transformations
• 7. Multiple testing: Bonferroni and Benjamini-Hochberg
• 8. The bootstrap in practice
• 9. Communicating: intervals, p-values, honest visualizations
• 10. A/B testing: sequential and targeted effects
• 11. Course research notes (Naulet, MaIAGE)
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a product / data engineer who ships A/B tests and wants an honest pipeline.

Empirical first: a “significant” p-value after peeking; underpowered design → business false negative — then we name experimental design, randomization, bootstrap, multiplicity.

The statistician’s craft on real data: a differentiator in an AI master’s.

What you will be able to do

• Run the complete cycle of a statistical study, from the initial question to communicable conclusions.
• Design an experimental plan: randomization, blocks, and a power computation suited to the objective.
• Identify the biases of observational studies and the classic pitfalls of inference.
• Apply the basics of causal inference: potential outcomes and difference-in-differences (DiD).
• Practice regression for real: diagnostics, transformations, reasoned decisions.
• Correct multiple tests (Bonferroni, Benjamini-Hochberg) according to the goal.
• Build intervals and tests by bootstrap on arbitrary data.
• Run a modern A/B test: sequential, targeted effects — and report the results honestly.

Official objectives and syllabus (2026-2027 catalog)

According to the APM_52066_EP — “Statistique en Action” description: the official objective is to show students how statistics are used in practice to answer a precise question, by introducing a series of important model-based approaches. Students learn to select and use the appropriate strategic methodologies, on concrete examples of real datasets from different fields — medicine, genomics, ecology, among others.

All analyses are carried out in R; knowledge of R programming is not required (only basic scripting).

The sheet also lists a detailed programme by topic, without a calendar: statistical tests (comparing two populations, power analysis, multiple testing), regression models (linear and nonlinear, diagnostics, model comparison), mixed-effects models (linear and nonlinear), mixture models and model-based clustering (Gaussian mixtures, stochastic block models, EM algorithm) and classification. The course unit appears in the 2026-2027 catalog via the “P2 – Applied Mathematics” track of the engineering cycle, without appearing in the SynapseS composition of the LLGA track S2.

Complement to the official syllabus — the themes structuring these materials (experimental designs and power computation, multiple testing, sequential A/B testing, honest communication of uncertainties) extend the official description; they correspond to the historical topics of the MAP566 course associated with this course unit.

Where this course fits in the program

A 36 h course unit of period 2 (X-3A-P2) in other tracks of the École, kept here for reference: it is not in the 2026–2027 LLGA lists, so no LLGA module requires it, but it remains the methodological “reality check” of these notes. It is taught by Zacharie Naulet (École Polytechnique & INRAE, MaIAGE team).

Official prerequisites: not communicated (the “Prerequisites” tab of the SynapseS description is empty); the description only specifies that no knowledge of R is required, basic scripting being sufficient.

• Prerequisites — the background of Refresher in Statistics (tests, intervals, estimation) and the framework of Machine Learning; the review section reactivates the “survival kit” actually used afterwards.
• Downstream modules — the culture of intervals, bootstrap and uncertainty built here directly feeds Privacy and Uncertainty Quantification (M2), which deepens the uncertainty quantification of models.

Course outline

• Review: the statistician’s survival kit — the only results from the statistics course that are really used, every week.
• The life cycle of a statistical study — from question to report: the map of the craft.
• Experimental designs: randomization, blocks, power — deciding before collecting how many and how.
• Case study: agricultural trials and blocks — the historical origin of experimental designs, on a complete case.
• Observational versus experimental: biases — why we do not always randomize, and what that costs.
• Causal inference: potential outcomes and DiD — approaching a causal effect with imperfect data.
• Regression in practice: diagnostics and transformations — using, checking and repairing a regression model.
• Multiple testing: Bonferroni and Benjamini-Hochberg — what to do when testing many hypotheses at once.
• The bootstrap in practice — intervals without closed formulas, on almost any statistic.
• Communicating: intervals, p-values, honest visualizations — making results readable without distorting them.
• A/B testing: sequential and targeted effects — the randomized trial as digital platforms practice it, and the synthesis of the whole module.
• Course research notes (Naulet, MaIAGE) — a tour of the research work of Zacharie Naulet.
• Exercises and References — systematic training and sources to go further.

Session plan

36 h in class = 12 sessions of 3 h. The volume is the one of the SynapseS 2026–2027 sheet (36 h in person: 18 h of lectures and 18 h of small-class sessions); the sheet publishes no session-by-session programme, so the split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced material (sequential inference, recent research); the level never goes down from one session to the next.

Self-study, outside the 36 h (about 2 h in total): ch. 11 Course research notes (Naulet, MaIAGE) — exchangeability, species sampling, bootstrap for graphs (≈ 2 h).

How to study this module

• Validate the “survival kit” of the review section: it, and not an advanced chapter, determines your ease throughout the module.
• Work through each section on a real dataset: this course’s specificity is that obstacles come from the data, not from formulas.
• Simulate: the power of a test, the inflation of false positives under multiple testing and the coverage of the bootstrap are understood in a few lines of code.
• Before every analysis, write the plan — hypotheses, sample size, criterion: the reflex this module seeks to instill.
• Apply the same rigor to your machine learning models: evaluating a model is a statistical study in its own right.
• Redo the exercises of each section before moving on to the next; the project (last exercise) brings them all together.

0. Review: the statistician’s survival kit

This chapter gathers the few results of the statistics refresher that the module uses every week, and fixes the vocabulary — model, estimator, bias, variance — in which every later question is asked.

1. The life cycle of a statistical study

Concrete scene: an A/B test stopped too early “proves” a lift that does not exist. This chapter shows how to structure the question before touching the data, so that such errors are ruled out by the plan rather than detected afterwards.

A real statistical study does not begin with a dataset: it begins with a question. The diagram below structures the whole module; each chapter tackles a link where studies typically fail.

2. Experimental designs: randomization, blocks, power

Before any data are collected, two decisions fix what a study can prove: how the treatments are assigned to the units, and how many units are needed. This chapter answers both — randomization and blocks for the first, a power computation for the second.

3. Case study: agricultural trials and blocks

Agricultural experimentation is the cradle of experimental design: Fisher at Rothamsted in the 1920s invented randomization, blocks and analysis of variance precisely because plots are heterogeneous — the same problem as your users or your patients.

It is also the daily ground of INRAE, the instructor’s institution: multi-year multi-location variety trials, complete blocks and alpha-lattices (Patterson & Williams, 1976) to screen hundreds of varieties.

4. Observational versus experimental: biases

Randomizing is often impossible (one does not assign smoking or a degree at random), and the data then come from observation. This chapter shows what goes wrong when the groups compared were not formed by chance, and how to recognize it.

5. Causal inference: potential outcomes and DiD

“Does the treatment cause the improvement?” needs a precise meaning before it can be answered. This chapter gives one — the comparison of two potential outcomes of the same unit — and shows how randomization, or failing that a difference-in-differences, estimates it.

6. Regression in practice: diagnostics and transformations

A fitted regression always produces coefficients and p-values, even when its assumptions are badly violated. This chapter gives the checks that tell whether those numbers can be trusted, and the repairs to apply when they cannot.

7. Multiple testing: Bonferroni and Benjamini-Hochberg

The modern statistician tests thousands of hypotheses at once (genomics, multi-metric A/B testing, subgroup analyses). Testing m true null hypotheses each at \alpha = 5\,\% produces on average \alpha m false positives — with m = 1000: 50 “discoveries” in pure noise. One must choose what to control.

8. The bootstrap in practice

For a median, a ratio or a trimmed mean there is no simple formula for the standard error. The bootstrap obtains the uncertainty of almost any statistic by simulation from the data themselves; this chapter explains why it works and when it does not.

import numpy as np
rng = np.random.default_rng(0)

x = np.array([118, 132, 127, 141, 122, 135, 128, 240])
n, B = len(x), 5000
med_star = np.median(rng.choice(x, size=(B, n), replace=True), axis=1)
mhat = np.median(x)

# 1. percentile: quantiles of the bootstrap law
lo, hi = np.quantile(med_star, [0.025, 0.975])
# 2. basic (reverse percentile): symmetrization around the estimate
lo_b, hi_b = 2*mhat - hi, 2*mhat - lo
print(f"median = {mhat}, percentile CI [{lo:.0f}, {hi:.0f}], basic CI [{lo_b:.0f}, {hi_b:.0f}]")
# 3. bootstrap bias: E*[m*] - mhat
print(f"estimated bootstrap bias: {med_star.mean() - mhat:+.2f} ms")

9. Communicating: intervals, p-values, honest visualizations

A correct analysis is useless if the report misleads its reader. This chapter fixes what a p-value does and does not say, why an interval tells more than a test, and how to draw figures that do not distort the result.

10. A/B testing: sequential and targeted effects

Online platforms run thousands of randomized experiments a year, watched in real time by teams eager to conclude. This chapter brings together the design, multiple-testing and causal tools of the module to answer the questions specific to that setting: when may one stop, and for whom does the variant work?

import numpy as np
rng = np.random.default_rng(52)

# ---- Bayesian A/B test: Beta posteriors, decision by expected loss ----
conv = np.array([1274, 1378])          # conversions
n    = np.array([12000, 12000])        # visits
a, b = 1.0, 1.0                        # Beta(1, 1) prior
post = [rng.beta(conv[i] + a, n[i] - conv[i] + b, size=200_000) for i in range(2)]

d = post[1] - post[0]                  # absolute uplift drawn a posteriori
loss_if_B = np.where(d < 0, -d, 0.0)   # loss if we deploy B and A was better

print(f"P(B > A)            : {(d > 0).mean():.3f}")
print(f"gain median         : {np.median(d):.5f}")
print(f"95 % CI of the gain : {np.quantile(d, [0.025, 0.975])}")
print(f"E[loss | deploy B] : {loss_if_B.mean():.2e}")
# Typical outputs: P(B>A) = 0.984; median uplift = 0.0087;
# CI = [0.0007, 0.0166]; E[loss] = 2.3e-05.
# Decision: the cost of a wrong choice of B (2 conversions out of 10^5) is negligible
# compared with the expected gain (+0.87 point) -- we deploy B.

11. Course research notes (Naulet, MaIAGE)

Zacharie Naulet is a junior professor (INRAE, MaIAGE team — Mathematics and Computer Science applied to Food and the Environment) and teaches at École Polytechnique; his PhD (Université Paris-Saclay) was on nonparametric Bayesian methods for signal and image processing. His themes — nonparametric Bayesian statistics, exchangeability, hidden state space models, applications to biological data and networks — directly feed several chapters of this module: the bootstrap (ch. 8), multiple testing and the estimation of \pi_0 (ch. 7), and, beyond this module, random-graph models.

Exercises

1. ★ Randomization by hand. Data 3, 4, 5, 6, 7, 8, split 3+3. Enumerate the 20 allocations, compute all values of \hat d, recover p = 0.10 and the exact distribution (stem-and-leaf display). Which split gives exactly \hat d = 2.33?

2. ★ Power. (a) Redo the computation of 2.5 for \delta/\sigma = 0.4 and 85 % power.

   (b) A two-sided test has \alpha = 5\,\% and n = 50 per group: plot \pi(\delta) for \delta/\sigma \in [0; 1] and read off the smallest detectable effect size with 80 % power.

   (c) For proportions (p_A = 0.10, expected effect +1 point), adapt the formula by replacing 2\sigma^2 with p_A(1-p_A) + p_B(1-p_B) and compute n.

3. ★ Simpson. Verify all the cells of table 4.2 (e.g. 289/350 = 82.6 %), then recompute the overall rates by standardizing on A’s profile and on B’s; comment on the sensitivity to the profile choice.

4. ★ Regression. On the cars dataset (or an equivalent): fit d \sim s, plot the 4 diagnostics, apply Box-Cox, refit; identify the point with the largest Cook’s distance, redo without it; compare the classical and HC1 standard errors (statsmodels).

5. ★ Communication. You have \hat d = 7.0, 95 % CI = [0.4; 13.6], p = 0.037, n = 80 (Example 9.2). Write in 5 lines: (a) a misleading “press” summary; (b) an honest summary; (c) list the three graphical manipulations that would reverse the reader’s impression, and their fixes.

6. ★★ DiD. With table 5.4: compute the naive before/after comparison, the cross-sectional after comparison, and the DiD; check that the DiD equals the interaction coefficient of the model y = \beta_0 + \beta_1 \text{NJ} + \beta_2 \text{After} + \beta_3 (\text{NJ} \times \text{After}) + e . Simulate 500 restaurants with common AR(1) shocks and show that naive standard errors are 3–5× too small without clustering (Bertrand et al. 2004).

7. ★★ BH in practice. (a) Apply Bonferroni, Holm and BH to the p-values of example 7.4 for q = \alpha = 0.10 and q = 0.01. (b) Simulation: generate m = 1000 p-values (m_0 = 900 uniform, 100 false nulls with p ~ Beta(1, 8)), repeat 500 times, estimate the empirical FWER and FDR of each procedure and verify the calibration \mathrm{FDR} \approx q\, m_0/m.

8. ★★ Peeking. Simulate 1000 A/B tests under H_0 (two Bernoulli p = 0.10, n = 10^5 per arm): (a) a single look at midpoint — false positive rate; (b) daily looks for 28 days with stopping if p < 0.05: compare the rate with the independence (Šidák) value 1-0.95^{28} \approx 76\,\% (the union bound, 28 × 5 % = 140 %, says nothing here) and with the inflation reported by Johari et al.; (c) implement a Gaussian mSPRT and verify its “always” validity.

9. ★★ Bayesian A/B. Take up the code of ch. 10: (a) analyze the sensitivity to the Beta(1,1) vs Beta(10,10) prior; (b) turn the decision rule into “expected loss < cost of a rollout”; (c) add the per-segment analysis (mobile/desktop) with BH correction of the interaction tests; (d) relate to the frequentist Wald CI — when do they diverge?

10. ★★ RCBD. Add to the data of Example 3.1 a moderate block × variety (interaction) effect, redo the ANOVA, test the interaction; build the same trial under a completely randomized design (same values, assignment blind to the blocks) and show the loss of F. Write the simultaneous (Tukey) confidence interval for the 6 comparisons.

11. ★★★ Project. INRAE or open data from a blocked trial (or kaggle: agricultural crop yield): from plan to conclusions — map the potential biases, justify the plan, ANOVA + diagnostics, bootstrap of the contrasts, multiple-comparison correction, final one-page report with honest visualizations and limits. Grading: 40 % plan, 40 % correct analysis, 20 % communication.

Projects and assessment

Official assessment (SynapseS 2026–2027 sheet of APM_52066_EP, read on 24 October 2026): “a group project, an individual written test without documents on that project, and a final exam”, with no weights displayed. The 2024–2025 and 2025–2026 sheets announced “1 or 2 group projects + 1 lab-session report + a final exam” instead.

The unit is not in the LLGA lists (notes kept for reference), so everything below is a proposal in that format, sized for a 36 h course: group subjects of about 25 h per member for the group project, solo subjects of about 20 h for students who work alone, a two-hour final exam, and a rubric for the project reports. The individual written test can question each member on the group’s analysis plan and on one derivation it uses, for instance theorem 2.4 for the sizing.

The projects exercise two master objectives: O1 (ML foundations, here the statistics under every ML evaluation: power, intervals, multiplicity, causal identification) and O7 (hands-on practice, from the analysis plan to an honest report). The LLM leaderboard adds O3 (large language models); the bias audit adds O8 (trustworthy and responsible AI) and prepares the end-of-year project Y1.3.

Solo projects

1. [Solo · ≈ 20 h] Peeking simulator and a sequential A/B toolkit. How far does the type-I error inflate under daily looks, and what do an always-valid test, a Bayesian stopping rule and variance reduction cost or save?
   • Implement: a simulator of K looks (theorem 10.2, exercise 8); a Gaussian mSPRT with always-valid p-values (definition 10.3); the expected-loss rule of the ch. 10 code, applied at every look; CUPED with a pre-period covariate (definition 10.4).
   • Prove: under the null hypothesis the mixture ratio is a nonnegative martingale of mean 1, hence always valid by Ville’s inequality; CUPED multiplies the variance by 1-\rho^2, where \rho is the correlation between the metric and its pre-period value.
   • Data: simulated Bernoulli streams; then Kevin Hillstrom’s MineThatData e-mail experiment (2008: 64,000 customers randomized between two e-mail campaigns and no e-mail, with past-year spend as pre-period covariate), replayed in a random arrival order.
   • Deliverables: code, 4–6 page report, 10-minute talk. Chapters: ch. 2, 7, 8, 10.
   • Success criterion: over 1,000 null experiments, the 28-look rate is close to 27 % and the mSPRT stays at or below 5 %, both with Monte Carlo intervals; the report gives the power and the median sample size at stopping against the fixed horizon, and the CUPED gain on the Hillstrom data with a bootstrap interval, next to 1-\rho^2.
2. [Solo · ≈ 20 h] Paper study: how much should we trust difference-in-differences? Reproduce the central experiment of Bertrand, Duflo and Mullainathan (2004), then apply its lesson to Card and Krueger (1994).
   • Implement: the DiD as the interaction regression of exercise 6, with naive, HC1 and clustered standard errors; an event-study check of parallel trends on the simulated panels (the Card and Krueger survey has a single pre-period); the placebo-law simulation — 50 “states” × 20 years with AR(1) shocks of coefficient 0.8, a fake law passed in half of the states at a random year, 1,000 replications.
   • Prove: in the 2 × 2 design, the OLS interaction coefficient equals the double difference of cell means (definition 5.3).
   • Data: the Card and Krueger survey of 410 fast-food restaurants in New Jersey and Pennsylvania (public on David Card’s data page); the simulated panels.
   • Deliverables: code, 4–6 page report, 10-minute talk. Chapters: ch. 4, 5, 6, 8.
   • Success criterion: the DiD of about +2.75 FTE is recovered with an interval clustered by restaurant (with two states, clustering by state is impossible, a limit the report discusses); in the placebo simulation, naive standard errors reject a true null far more often than 5 % (about 38 % in a check run made for these notes) while clustering by state brings the rate within [3 %, 8 %].

Group projects

1. [Group of 3 · ≈ 25 h each] An honest LLM leaderboard. Which published rankings of four to six small open LLMs survive per-item uncertainty and the correction for many pairwise comparisons? A paper study of E. Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” (2024), carried out with the tools of this course.
   • Data: per-item outcomes on ARC-Challenge (Clark et al., 2018; 1,172 test questions), GSM8K (Cobbe et al., 2021; 1,319 test problems) and BoolQ (Clark et al., 2019; 3,270 dev questions), from EleutherAI’s lm-evaluation-harness run with --log_samples on models of at most 2B parameters (free GPU tier).
   • Implement: the smallest detectable paired gain of each benchmark, before scoring (theorem 2.4 for paired differences); paired bootstrap intervals for every score difference (ch. 8); McNemar tests, then Holm against Benjamini–Hochberg over all model pairs and benchmarks (ch. 7); a report whose figures follow definition 9.3.
   • Split of work: A runs the models with fixed prompts and seeds and logs per-item outputs; B owns the sizing, the bootstrap and the variance over 3 prompt templates; C owns the multiple-testing plan, written before scoring, and the report.
   • Deliverables: repository, analysis plan, 6-page report, 15-minute talk. Chapters: ch. 2, 7, 8, 9. Links: Large Language Models 9.6 and Machine Learning 4.9.
   • Success criterion: every difference comes with its paired 95 % interval and n; the report states which adjacent ranks stay separated after each correction, and how many orderings published in the model cards do not survive.
2. [Group of 3 · ≈ 25 h each] A blocked field trial from plan to report. How much did blocking buy in real trials, and which treatment differences survive honest multiple comparisons? It extends exercise 11 to two classic, public trials.
   • Data: Yates’ Rothamsted oats experiment (1935: 6 blocks, 3 varieties, 4 nitrogen doses, 72 plots; MASS::oats in R, or statsmodels.datasets.get_rdataset("oats", "MASS")), and the alpha-lattice oat trial agridat::john.alpha (24 genotypes, 3 replicates of 6 incomplete blocks; John & Williams, 1995).
   • Implement: the analysis of variance with blocks (ch. 3), with the split-plot error strata for the oats; the four diagnostic plots and a Box-Cox check (ch. 6); bootstrap intervals for the contrasts (ch. 8); Tukey against Benjamini–Hochberg for the pairwise comparisons (ch. 7); the efficiency of blocking against a completely randomized design.
   • Split of work: A reconstructs the design and sizes the next trial (ch. 2: plots needed to detect a 5 % yield gain); B runs the models and the diagnostics; C owns the bootstrap, the multiple comparisons and the one-page summary.
   • Deliverables: code, 6-page report ending with the one-page summary of exercise 11, 15-minute talk. Chapters: ch. 2, 3, 6, 7, 8, 9.
   • Success criterion: the classic split-plot analysis of the oats is reproduced (nitrogen F ≈ 37.7 on 3 and 45 degrees of freedom, as in Venables & Ripley’s Modern Applied Statistics with S); the efficiency of blocking comes with a bootstrap interval; every claimed difference states its correction and its interval.
3. [Group of 3 · ≈ 25 h each] Bias audit of a recommendation dataset. Which selection and survivorship biases shape logged ratings, and could a recommender change look good for the wrong reasons? The audit prepares the evaluation of the end-of-year project Y1.3 (cold-start news recommendation).
   • Data: MovieLens-1M (Harper & Konstan, 2015: 1,000,209 ratings by 6,040 users with age, gender and occupation); MIND-small (Wu et al., 2020) for the Y1.3 setting.
   • Implement: a map of the biases (ch. 4) — ratings exist only for chosen items, heavy raters dominate, users who leave stop rating; a search for Simpson reversals between segments; a simulated randomized change on the logged users with a planted segment-dependent effect, analysed by segment × treatment interaction tests with Benjamini–Hochberg (ch. 7, definition 10.4); a written discussion of interference (SUTVA, ch. 5).
   • Split of work: A owns the bias map and the Simpson search; B the simulator and the interaction tests; C the interference discussion, the figures and the report.
   • Deliverables: repository, 6-page report, 15-minute talk. Chapters: ch. 4, 5, 7, 9, 10.
   • Success criterion: over 200 simulated experiments, the planted segments are found with an average false discovery proportion at most q; every bias of the map comes with a number (share of ratings, gap, reversal) and its consequence for an offline evaluation.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, a non-programmable calculator; four independent questions. Use z_{0.975}=1.96 and z_{0.80}=0.842.

1. Question (5 points). A product team checked its A/B test every morning and stopped on day 9, when the p-value first fell below 0.05; it reports “a significant +2 % on conversion (p = 0.04)”. (a) Why is the error rate of this procedure not 5 %? Bound it for 9 looks. (b) Give two valid ways to run the next test. (c) A colleague proposes to stop instead when the 95 % credible interval of a flat-prior Bayesian analysis excludes 0. Does this fix the problem?

   Expected: (a) The procedure rejects if any of the 9 looks crosses the threshold (theorem 10.2): the rate is at least 5 % and, for two-sided Gaussian tests, at most 1-0.95^9 \approx 37\,\%. Positive correlation between looks makes it smaller — about 19 % for equally spaced looks — but far above 5 %.

   (b) A fixed horizon sized in advance (theorem 2.4) with a single look; or an always-valid p-value such as the mSPRT (definition 10.3); group-sequential boundaries also qualify. (c) No: with a flat prior the credible interval is close to the Wald interval, a fixed-horizon boundary, so the same inflation occurs. Validity needs a mixture likelihood ratio (a Bayes factor) as stopping boundary, which shrinks like \sqrt{\log t / t}.

2. Question (6 points). Two LLMs are scored on the same benchmark items; a pilot shows that they disagree on 20 % of the items. (a) How many items give 80 % power to detect an accuracy gain of 3 points at the 5 % level (two-sided), with a paired analysis? (b) ARC-Challenge has 1,172 test questions: which gain can it detect with the same power? (c) Five models are compared pairwise on it; the ten p-values are 0.001, 0.008, 0.012, 0.030, 0.041, 0.09, 0.20, 0.33, 0.52, 0.81. Which pairs do Bonferroni, Holm and Benjamini–Hochberg declare different at level 0.05?

   Expected: (a) The per-item difference takes the values −1, 0 or 1, with variance 0.20 - 0.03^2 = 0.1991. Theorem 2.4, written for one sample of paired differences, gives n = \frac{(1.96 + 0.842)^2 \times 0.1991}{0.03^2} \approx 1737 \text{ items.}

   (b) Solving for the gain, \delta \approx 2.802\sqrt{0.2/1172} \approx 0.037: about 3.7 points, so smaller gains are invisible on this benchmark. (c) Bonferroni (threshold 0.005) and Holm (0.005, then 0.05/9 ≈ 0.0056, which 0.008 fails) reject one pair. BH compares p_{(k)} with 0.005\,k: 0.001, 0.008 and 0.012 pass, 0.030 > 0.020 and every later one fails, so k^* = 3 pairs (theorem 7.3).

3. Question (5 points). Under the null hypothesis the observations are i.i.d. with density f_0; the mSPRT of definition 10.3 mixes the alternatives f_\theta with a prior density h. (a) Show that its mixture ratio \Lambda_t is a nonnegative martingale of mean 1 under the null. (b) Ville’s inequality states that such a martingale exceeds a level c > 0 at some time with probability at most 1/c. Deduce that the p-value of definition 10.3 is always valid.

   Expected: (a) For a fixed \theta, the product L_t(\theta) of the ratios f_\theta(x_s)/f_0(x_s) over s \leq t satisfies \mathbb{E}_0\big[L_{t+1}(\theta) \mid x_{1:t}\big] = L_t(\theta) \int f_\theta(x)\,dx = L_t(\theta); integrating against h keeps the martingale property (Tonelli: everything is nonnegative), and \Lambda_0 = 1.

   (b) For any stopping time T, the event p_T \leq \alpha means that \Lambda_s \geq 1/\alpha for some s \leq T, so \mathbb{P}_0(p_T \leq \alpha) \leq \mathbb{P}_0\big(\sup_t \Lambda_t \geq 1/\alpha\big) \leq \alpha .

4. Question (4 points). A company runs 200 A/B tests a year, and 10 % of the ideas tested truly work. (a) With 80 % power at the 5 % level and a single look, how many “wins” are expected, and which fraction of them is real? (b) Daily peeking raises the false-positive rate to 27 % and the power to 90 %: same questions. What should the company report next to each win?

   Expected: (a) 20 real ideas give 16 true wins and 180 null ideas give 9 false wins: 25 wins, of which 64 % are real (the arithmetic of example 9.4). (b) 18 true and 0.27 \times 180 \approx 49 false wins: only about 27 % of 67 wins are real, so peeking turns most “wins” into noise. Next to each win: the effect with its interval and n, the planned horizon and the number of looks (definitions 9.3 and 10.1).

Total: 5 + 6 + 5 + 4 = 20 points.

Grading rubric (proposal)

References

• R. A. Fisher, The Design of Experiments, Oliver & Boyd, 1935 — randomization, blocks, the lady tasting tea.
• G. E. P. Box, W. G. Hunter, J. S. Hunter, Statistics for Experimenters, 2nd ed., Wiley, 2005.
• A. Dean, D. Voss, Design and Analysis of Experiments, Springer, 1999.
• H. D. Patterson, E. R. Williams, “A new class of resolvable incomplete block designs”, Biometrika 63, 1976 — alpha-lattices.
• J. Neyman, “On the application of probability theory to agricultural experiments (1923)”, trans. Statistical Science 5, 1990; D. B. Rubin, “Estimating causal effects of treatments in randomized and nonrandomized studies”, J. Educational Psychology 66, 1974.
• G. Imbens, D. Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences, Cambridge University Press, 2015.
• S. Wager, S. Athey, “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests”, J. Amer. Statist. Assoc. 113, 2018 — causal forests.
• D. Card, A. Krueger, “Minimum Wages and Employment”, American Economic Review 84, 1994.
• M. Bertrand, E. Duflo, S. Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?”, Q. J. Economics 119, 2004.
• H. White, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity”, Econometrica 48, 1980 — the HC0 sandwich estimator.
• C. R. Charig et al., “Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy”, BMJ 292, 1986 — the Simpson’s paradox data.
• E. J. Elton, M. J. Gruber, C. R. Blake, “Survivorship Bias in Mutual Fund Performance”, Journal of Finance 51, 1996.
• M. Mangel, S. M. Samuels, “Analyzing the Wald airplane data”, Statistical Science 17, 2002.
• B. Efron, “Bootstrap Methods: Another Look at the Jackknife”, Annals of Statistics 7, 1979; B. Efron, R. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall, 1993; A. van der Vaart, Asymptotic Statistics, Cambridge University Press, 1998 (ch. 23).
• Y. Benjamini, Y. Hochberg, “Controlling the False Discovery Rate”, J. R. Statist. Soc. B 57, 1995; Y. Benjamini, D. Yekutieli, “The control of the false discovery rate in multiple testing under dependency”, Annals of Statistics 29, 2001; J. Storey, “A direct approach to false discovery rates”, J. R. Statist. Soc. B 64, 2002; R. J. Simes, “An improved Bonferroni procedure for multiple tests of significance”, Biometrika 73, 1986.
• P. Armitage, C. K. McPherson, B. C. Rowe, “Repeated significance tests on accumulating data”, J. R. Statist. Soc. A 132, 1969 — the exact cost of peeking; Z. Šidák, “Rectangular confidence regions for the means of multivariate normal distributions”, J. Amer. Statist. Assoc. 62, 1967; H. Robbins, “Statistical methods related to the law of the iterated logarithm”, Annals of Mathematical Statistics 41, 1970 — the mixture likelihood ratio.
• R. Johari, P. Koomen, L. Pekelis, D. Walsh, “Peeking at A/B Tests: Why it matters, and what to do about it”, KDD 2017 (doi:10.1145/3097983.3097992); long version “Always Valid Inference: Continuous Monitoring of A/B Tests”, Operations Research 70, 2022 (doi:10.1287/opre.2021.2135).
• A. Deng, Y. Xu, R. Kohavi, T. Walker, “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Period Data” (CUPED), KDD 2013.
• R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, N. Pohlmann, “Online Controlled Experiments at Large Scale”, KDD 2013; R. Kohavi, D. Tang, Y. Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020.
• R. L. Wasserstein, N. A. Lazar, “The ASA’s Statement on p-Values”, The American Statistician 70, 2016.
• V. Amrhein, S. Greenland, B. McShane, “Scientists rise up against statistical significance”, Nature 567, 2019.
• J. P. A. Ioannidis, “Why Most Published Research Findings Are False”, PLoS Medicine 2, 2005.
• N. L. Kerr, “HARKing: Hypothesizing After the Results are Known”, Personality and Social Psychology Review 2, 1998; A. Gelman, E. Loken, “The garden of forking paths”, 2013 (working paper).
• G. Cumming, “The New Statistics: Why and How”, Psychological Science 25, 2014.
• J. M. Hoenig, D. M. Heisey, “The Abuse of Power”, The American Statistician 55, 2001.
• APM_52066_EP “Statistics in action” datasheet, SynapseS catalog of École Polytechnique.
• Research group (Z. Naulet, INRAE MaIAGE / École Polytechnique): research page: znaulet.pages-forge.inrae.fr; C. Balocchi, S. Favaro, Z. Naulet, Bayesian Nonparametric Inference for “Species-sampling” Problems (Statistical Science); S. Favaro, Z. Naulet, Near-optimal estimation of the unseen under regularly varying tail populations (Bernoulli 2023); Optimal estimation of high-order missing masses, and the rare-type match problem (IEEE Trans. IT); V. Veitch, E. Sharma, Z. Naulet, D. Roy, Exchangeable modelling of relational data; Z. Naulet, J. Rousseau, F. Caron, Asymptotic Analysis of Statistical Estimators related to MultiGraphex Processes under Misspecification; Z. Naulet, D. Roy, E. Sharma, V. Veitch, Bootstrap estimators for the tail-index and for the count statistics of graphex processes; F. Camerlenghi, S. Favaro, Z. Naulet, F. Panero, Optimal disclosure risk assessment (Annals of Statistics 2021); K. Abraham, E. Gassiat, Z. Naulet, Frontiers to the learning of nonparametric hidden Markov models; E. Gassiat, I. Kaddouri, Z. Naulet, Clustering risk in Non-parametric Hidden Markov and I.I.D. Models; Z. Naulet, E. Barat, Bayesian nonparametric estimation for Quantum Homodyne Tomography; Adaptive Bayesian density estimation in sup-norm; Tails assumptions and posterior concentration rates for mixtures of Gaussians; Y. Mahdaviyeh, Z. Naulet, Risk of the Least Squares Minimum Norm Estimator under the Spike Covariance Model; I. Kaddouri, Z. Naulet, E. Gassiat, On the impossibility of detecting a late change-point in the preferential attachment random graph model; L. Bianco, C. Keribin, Z. Naulet, SubSearch: Robust Estimation and Outlier Detection for Stochastic Block Models.
• E. Dolera, S. Favaro, “De Finetti’s theorem: rate of convergence in Kolmogorov distance”, Bernoulli 26, 2020 — arXiv:1802.02244 (standard cited in ch. 11).