Turing Seminar: An Introduction to AGI Safety

Contents

• Introduction and overview
• 0. Review: probability, information and optimisation
• 1. Capabilities: from GPT-2 to frontier models
• 2. Forecasting general-purpose AI
• 3. Benefits and risks of general-purpose AI
• 4. Specification: proxies, Goodhart’s law and preference learning
• 5. Robustness and generalization failures
• 6. Interpretability: understanding what models compute
• 7. Evaluating dangerous capabilities
• 8. Governance and international coordination
• 9. AI control and scalable oversight
• 10. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with a good M1 in mathematics, computer science or physics who has trained a neural network and wants to understand, with numbers and proofs where they exist, why the safety of general-purpose AI is a technical problem.

Empirical first: pick the best of n answers with a learned score, and the true quality of the pick first rises with n, then falls back towards that of an average answer when the score’s errors are heavy-tailed (ch. 4, after Gao, Schulman and Hilton, 2023). A linear classifier that is 98 % accurate at telling 3 from 8 in the scikit-learn digits changes half of its correct answers when each pixel moves by about 15 % of its range (ch. 5). This course explains such failures, and the tools built to detect and prevent them.

The Turing Seminar is an Opening course: it asks where the fast progress of general-purpose AI is leading, what could go wrong, and what engineers can do about it. The course has three parts: capabilities and forecasts (ch. 1–2), benefits and risks (ch. 3–5), and solutions, namely understanding, evaluating and regulating models (ch. 6–8). The course is given by Loucas Pillaud-Vivien (ENPC, Cermics) and Léo Dana (Inria, Sierra), as listed on master-mva.com.

Related MVA pages: Foundations of Large Language Models for the models themselves, Regulating AI for the law, and Responsible Machine Learning and Multimodal Explainable AI for fairness and explanations.

What you will be able to do

• Compute the training compute of a model and derive the compute-optimal split between parameters and data from a parametric scaling law.
• Explain why smooth improvements can look like sudden “emergent” jumps, and read a capability trend critically.
• Build a forecast from a doubling trend with its uncertainty, and score forecasts with a proper scoring rule.
• Classify the risks of general-purpose AI as misuse, malfunction or systemic, and reason about layered defences whose failures are correlated.
• State Goodhart’s law in its regressional and extremal forms, and prove the formula for the KL divergence of best-of-n sampling.
• Derive the optimal policy of KL-regularised preference learning and the loss of direct preference optimisation.
• Compute the smallest adversarial perturbation of a linear model, and explain goal misgeneralization with the implicit bias of gradient descent.
• Explain superposition, train a toy model that exhibits it, and describe probes, sparse autoencoders and activation patching.
• Design a dangerous-capability evaluation with honest statistics: pass@k, confidence bounds, elicitation and its limits.
• Model a race between developers as a game, and explain how verification and repeated interaction make cooperation possible.
• Compare AI control protocols, such as upfront auditing and trusted monitoring, with an explicit attacker model.
• Locate a research question in AI safety, and plan a contribution to it with machine-learning skills.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com starts from the rapid progress of AI, “from GPT-2 to GPT-5”. Its first part reviews these advances and where they might lead: informed by scaling laws and expert forecasts, it argues that Artificial General Intelligence (AGI) could emerge within this decade. The second part discusses the benefits and risks of future AGIs.

The description traces the risks to two sources: our limited understanding of the models we create, and the lack of international coordination to avoid competitive races and ensure safe deployment. It adds that assessing which dangerous capabilities a model has, and how they can be elicited, requires careful evaluation, often of closed-source systems.

The final part examines solutions for controlling AIs in a robust way: understanding, evaluating and regulating them, with an explicit discussion of how engineers can contribute through research and machine-learning expertise. The Université Paris-Saclay course detail names the course “Séminaire Turing - Sûreté et interprétabilité des IA à usage général”, that is, safety and interpretability of general-purpose AI.

Every official topic except the expert interventions is taught on this page. Its extensions beyond the syllabus are the review chapter 0, the quantitative models that make the debates precise (compute-optimal allocation, proper scoring, best-of-n, races as repeated games), chapter 9 on AI control and scalable oversight, and the research chapter 10.

Where this course fits in the programme

Official prerequisites: not displayed. It is an Opening course of semester 1 (period 1), with 2.5 ECTS and 12 h. It asks for no specialised knowledge, but the notes use the probability, linear algebra and Python of a good M1.

• Preparation — the refreshers Probability and Statistics (conditioning, concentration), Linear Algebra, Analysis and Optimization (norms, Lagrange multipliers, gradient descent) and Scientific Python, NumPy and PyTorch (every code block).
• Related courses of semester 1 — Foundations of Large Language Models (the transformer and its training), Deep Learning and Theoretical Foundations of Deep Learning (implicit bias, generalization), Fundamentals of Reproducible Research and Free Software (the debate on open models).
• Courses of semester 2 that build on it — Regulating AI (the legal frame of ch. 8), Responsible Machine Learning, Multimodal Explainable AI (attribution and interpretability), Reinforcement Learning (reward design, ch. 4), Large Language Models for Code and Proof (verification), Training and Deploying Large-Scale Models (compute) and Artificial Intelligence and the Environment. A safety topic is a natural Research Internship.

Course outline

• Review: probability, information and optimisation — KL divergence, Gaussian conditioning, power-law fits, concentration and dual norms, in the form used later.
• Capabilities: from GPT-2 to frontier models — training compute, scaling laws, compute-optimal allocation, emergence and test-time compute.
• Forecasting general-purpose AI — definitions of AGI, trend extrapolation, expert surveys, and how to score and combine forecasts.
• Benefits and risks of general-purpose AI — what these systems already do for science and work, a taxonomy of risks, and defence in depth.
• Specification: proxies, Goodhart’s law and preference learning — why optimising a proxy fails, best-of-n, RLHF and DPO.
• Robustness and generalization failures — adversarial examples, jailbreaks, goal misgeneralization.
• Interpretability: understanding what models compute — probes, superposition, sparse autoencoders, causal interventions.
• Evaluating dangerous capabilities — elicitation, pass@k, statistics of evaluations, access to closed models, sandbagging, safety frameworks.
• Governance and international coordination — races as games, verification and compute, regulation, the role of engineers.
• AI control and scalable oversight — auditing games, trusted monitoring, debate, weak-to-strong generalization (self-study).
• Research directions and open problems — open questions in each area and how to start working on them (self-study).

Session plan

12 h in class = 6 sessions of 2 h. The volume is the Université Paris-Saclay programme table (12 h of lectures, 2.5 ECTS). master-mva.com announces 7 sessions of 2 h plus a project, with readings before each session, and paper presentations, expert interventions, research projects, debates and discussions during the sessions. This plan fits the official 12 h into six sessions of 2 h; the split of the chapters between them is a proposal, and an extra session, where held, can host the project presentations.

Level 1 = foundations (definitions, orders of magnitude, first models), 2 = core results and methods, 3 = advanced material and current research; the level never decreases.

Self-study, outside the 12 h (about 7 h in total): ch. 0 Review: probability, information and optimisation, to read before session 1 (≈ 2 h) · ch. 9 AI control and scalable oversight, after session 6 (≈ 3 h, with Ex. 15–16) · ch. 10 Research directions and open problems (≈ 2 h, with Ex. 17). The project comes on top (see Projects and assessment).

How to study this module

• Read ch. 0 before session 1. Then read each chapter before its session, as the official organisation asks for readings before each session; each chapter uses only earlier ones.
• Pair each session with the matching part of the official resources: AI Safety Atlas and the International AI Safety Report for sessions 1–3, Hendrycks’ textbook for sessions 2–4, the BlueDot Impact alignment and governance courses for sessions 4–6.
• Prepare the paper of each session as if you had to present it in 10 minutes: the question, the method, one figure, one limitation.
• After each session, do the exercises of its row; ★ checks the definitions, ★★ asks for a short proof or a computation.
• Run every Python block: NumPy, SciPy, scikit-learn and PyTorch on a CPU, fixed seeds, a few seconds each.
• Keep a list of claims you heard with their evidence. The field moves fast, and many statements are forecasts or hypotheses rather than results; the notes say which is which.
• Prepare the debates with the tools of the chapters:
  • Session 2: could AGI emerge within this decade? State a definition (Definition 2.1 or another), the evidence (a trend, a survey, an argument), and a probability that could be scored with Theorem 2.4. Then ask which family of risk of Definition 3.1 deserves most of a fixed research budget, and which layer of Proposition 3.2 to add first.
  • Session 6: should the weights of frontier models be released? Weigh research access, reproducibility and the spread of power against irreversibility and the removal of safeguards by fine-tuning, with ch. 7.3 and Theorem 8.2.

0. Review: probability, information and optimisation

Five M1 tools return throughout the course: the Kullback–Leibler divergence (ch. 1, 2 and 4), Gaussian conditioning (ch. 4), power laws on log–log axes (ch. 1 and 2), concentration of averages (ch. 7) and dual norms (ch. 5). This chapter states them in the form used later. The refreshers Probability and Statistics and Linear Algebra, Analysis and Optimization give more detail.

0.1 Entropy, cross-entropy and KL divergence

A language model is trained to minimise the cross-entropy between the data distribution and its own predictions. The decomposition above shows that the best achievable loss is the entropy of the data, an “irreducible” term that reappears in the scaling laws of ch. 1.

0.2 Gaussian conditioning

For V \sim \mathcal{N}(0, \sigma_V^2) and independent noise X \sim \mathcal{N}(0, \sigma_X^2), the proxy U = V + X gives the shrinkage formula used for Goodhart’s law in ch. 4: \mathbb{E}[V \mid U = u] = \frac{\sigma_V^2}{\sigma_V^2 + \sigma_X^2}\, u .

0.3 Power laws and doubling times

Fitting \log y against \log x by least squares is the maximum-likelihood fit when the errors are multiplicative and log-normal. A power law fitted over a few decades of x can be extrapolated only as far as the mechanism that produced it is believed to continue; ch. 1 and 2 return to this caveat.

0.4 Concentration of an average

The proof bounds the moment generating function of each bounded variable (Hoeffding’s lemma) and applies Markov’s inequality; see Boucheron, Lugosi and Massart (2013), ch. 2. Setting the right-hand side to \delta gives the sample size used in ch. 7: n \ge \frac{\log(2/\delta)}{2t^2}.

0.5 Dual norms

In dimension d, a vector with entries of size about c has \|w\|_1 \approx dc but \|w\|_2 \approx \sqrt{d}\,c. A perturbation of tiny size in every coordinate can therefore move a linear score a lot. This is the root of the adversarial examples of ch. 5.

0.6 The logistic function

The logistic function and its inverse, the log-odds, are \sigma(t) = \frac{1}{1 + e^{-t}}, \qquad \sigma^{-1}(q) = \log\frac{q}{1 - q}. It turns a score difference into a probability, as in the preference models of ch. 4. Its log-derivative is 1 - \sigma(t). Hence the loss -\log\sigma(t) has derivative \sigma(t) - 1, which increases with t, so the loss is convex in t.

1. Capabilities: from GPT-2 to frontier models

From 2019 to 2025, language models went from plausible paragraphs to olympiad problems and software agents. This chapter measures that progress in compute and explains why the loss improved predictably.

1.1 Milestones and training compute

GPT-2 (Radford et al., 2019) had 1.5 billion parameters; GPT-3 (Brown et al., 2020) had 175 billion, trained on 300 billion tokens. Later reports, such as GPT-4 (OpenAI, 2023), disclose no parameter counts; the AI Index Report 2025 tracks the public trends.

GPT-5 (OpenAI, 2025) gives no size either. Its system card describes one system that routes each query to a fast model or to a deeper reasoning model, the test-time axis of ch. 1.4.

1.2 Scaling laws

Kaplan et al. (2020) found such power laws over several orders of magnitude. How should a fixed budget C be spent?

import numpy as np

# Parametric fit of Hoffmann et al. (2022), approach 3, rounded constants
E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28

def loss(N, D):
    return E + A / N**alpha + B / D**beta

def optimal_allocation(C):
    """Minimise loss(N, D) subject to 6 N D = C (Theorem 1.4)."""
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
    N = G * (C / 6) ** (beta / (alpha + beta))
    return N, C / (6 * N)

for C in [1e21, 5.76e23, 1e25]:
    N, D = optimal_allocation(C)
    print(f"C = {C:.2e} FLOP: N* = {N:.2e} params, D* = {D:.2e} tokens, "
          f"D*/N* = {D/N:5.1f}, loss = {loss(N, D):.3f}")

# Brute-force check on a grid for the budget of Gopher and Chinchilla
C = 5.76e23
Ns = np.logspace(9, 12, 20001)
print("grid minimiser:", f"{Ns[loss(Ns, C / (6 * Ns)).argmin()]:.2e}")

# Fit a power law L(C) = a C^(-b) to noisy synthetic points on log-log axes
rng = np.random.default_rng(0)
Cs = np.logspace(18, 22, 9)
L_obs = 40.0 * Cs ** (-0.05) * np.exp(0.01 * rng.standard_normal(Cs.size))
slope, intercept = np.polyfit(np.log(Cs), np.log(L_obs), 1)
print(f"fitted exponent b = {-slope:.4f}, prefactor a = {np.exp(intercept):.1f}")

At the Chinchilla budget, 5.76 \times 10^{23} FLOP, the code finds 32 billion parameters and 93 tokens per parameter. The Chinchilla model itself has 70 billion parameters and 1.4 trillion tokens, 20 tokens per parameter, as the paper’s other two methods give.

Besiroglu et al. (2024) found that the published constants fit the paper’s own data poorly; their refit agrees with the other two methods. Constants are fragile; the robust lesson is that parameters and data grow together.

1.3 Emergence and the choice of metric

Wei et al. (2022) reported “emergent” abilities: near-zero accuracy, then a sharp rise with scale. Schaeffer, Miranda and Koyejo (2023) traced many such jumps to the metric.

For k = 20, raising p from 0.80 to 0.90, 0.95 and 0.99 moves exact match from 0.012 to 0.12, 0.36 and 0.82, while the per-token log-likelihood shows no jump. An abrupt curve needs a check of its metric.

1.4 Reasoning models and test-time compute

A second axis of scaling appeared in 2024: compute at inference. Models trained by reinforcement learning to reason at length, such as DeepSeek-R1 (DeepSeek-AI, 2025), improve with longer reasoning, and adaptive test-time compute can beat a larger model (Snell et al., 2024).

Agents add tools and long horizons (ch. 2). Each new axis of scaling is one more way for capabilities to appear before they are evaluated (ch. 7).

2. Forecasting general-purpose AI

Could AGI emerge within this decade? This chapter makes the question precise, builds forecasts from trends and surveys, and scores them.

2.1 What is AGI?

There is no agreed definition. Morris et al. (2024) cross levels of performance, from “emerging” to “superhuman”, with generality.

Such a definition is the resolution criterion of a forecast, and must be fixed first.

2.2 Extrapolating trends

Kwa et al. (2025) measure the length of tasks, timed on human professionals, that models complete with 50 % success. This “time horizon” doubled about every seven months from 2019 to 2025.

The trend may slow if compute, data or energy run short, or speed up if AI accelerates AI research; the width of the range is part of the answer.

2.3 Expert surveys

Grace et al. (2024) surveyed 2,778 researchers who had published at top AI venues. The aggregate gave a 10 % chance that unaided machines outperform humans in every possible task by 2027, and 50 % by 2047, thirteen years earlier than their survey a year before. Asking about the automation of all occupations gave much later dates.

2.4 Scoring and combining forecasts

A proper rule rewards honesty: exaggerating or downplaying a risk never pays.

import numpy as np

rng = np.random.default_rng(0)

# Example 2.2: doubling-time extrapolation with an uncertain doubling time
h0, target = 1.0, 167.0                       # current horizon and target, in hours
k = np.log2(target / h0)                      # doublings needed
T = np.exp(rng.normal(np.log(7.0), 0.3, 100_000))   # doubling time (months), median 7
years = k * T / 12
print(f"doublings {k:.2f}; years 5/50/95 %:", np.round(np.quantile(years, [0.05, 0.5, 0.95]), 2))

# Proposition 2.5: 15 noisy forecasters against their average
n_events, m = 2000, 15
p_true = rng.uniform(0.05, 0.95, n_events)
y = rng.random(n_events) < p_true
logit = np.log(p_true / (1 - p_true))
q = 1 / (1 + np.exp(-(logit + rng.normal(0, 1.0, (m, n_events)))))
brier_each = ((q - y) ** 2).mean(axis=1)
print(f"Brier: mean individual {brier_each.mean():.4f}, best individual {brier_each.min():.4f}, "
      f"average forecast {((q.mean(axis=0) - y) ** 2).mean():.4f}")

The average forecast (Brier 0.189) beats the best of the 15 forecasters (0.210).

3. Benefits and risks of general-purpose AI

Generality makes these systems valuable and risky at once. This chapter reviews both and models layered defences.

3.1 Benefits

• Biomedical research. AlphaFold 2 (Jumper et al., 2021) predicts protein structures with near-experimental accuracy in many cases; two of its authors, Demis Hassabis and John Jumper, shared the 2024 Nobel Prize in Chemistry.
• Language barriers. The NLLB project (NLLB Team, 2022) translates between 200 languages, many of them low-resource.
• Administrative burdens. Experiments find productivity gains in writing and customer support, largest for novices (Noy and Zhang, 2023; Brynjolfsson, Li and Raymond, 2025).

3.2 A taxonomy of risks

The International AI Safety Report (Bengio et al., 2025) groups risks into three families.

Hendrycks, Mazeika and Woodside (2023) add the AI race (ch. 8) and organisational failures. Ch. 4–6 study malfunctions, ch. 7 misuse potential, ch. 8 races.

3.3 Defence in depth

Systems stack imperfect safeguards: training, filters, monitoring, access control, as in the “Swiss cheese” model of Reason (1990).

4. Specification: proxies, Goodhart’s law and preference learning

We cannot write down what we want, so we optimise a measurable proxy: a reward, a reward model, human ratings. This chapter shows when optimising a proxy stops helping, and derives the objective used to train chat models.

4.1 Proxies and Goodhart’s law

“When a measure becomes a target, it ceases to be a good measure” is Strathern’s (1997) phrasing of Goodhart’s law. An agent trained on the score of the boat race CoastRunners learned to circle and hit the same targets instead of finishing (Clark and Amodei, 2016). Sycophancy is a proxy failure of models trained on human approval (Sharma et al., 2024).

4.2 Regressional and extremal Goodhart

This is Proposition 0.3. Heavy-tailed noise changes the picture.

The most extreme proxy values are then explained by the errors, not by the value.

4.3 Best-of-n and its KL divergence

The simplest optimiser samples n outputs from a model \pi and keeps the best by proxy score.

import numpy as np

rng = np.random.default_rng(0)
trials = 4000

def best_of_n(noise, n):
    """Mean true value V of the sample with the highest proxy U = V + X."""
    V = rng.standard_normal((trials, n))
    U = V + noise((trials, n))
    return V[np.arange(trials), U.argmax(axis=1)].mean()

gauss = lambda s: rng.standard_normal(s)
heavy = lambda s: rng.standard_t(2, s)          # Student t, 2 degrees of freedom
print("    n  KL (nats)  E[V], Gaussian noise  E[V], t(2) noise")
for n in [1, 4, 16, 64, 256, 1024]:
    kl = np.log(n) - (n - 1) / n                 # Theorem 4.4
    print(f"{n:5d}  {kl:8.2f}   {best_of_n(gauss, n):12.3f}   {best_of_n(heavy, n):14.3f}")

With Gaussian noise, the true value of the pick grows: 0.74 at n = 4, 2.30 at n = 1024. With heavy-tailed noise it peaks near 0.69 at n = 16 and falls to 0.11 at n = 1024, as Proposition 4.3 predicts.

Gao, Schulman and Hilton (2023) observed this rise and fall when optimising language models against a learned reward model, judged by a larger “gold” one.

4.4 Learning from human preferences

Reinforcement learning from human feedback (RLHF) learns the proxy from comparisons (Christiano et al., 2017; Ouyang et al., 2022).

The KL term limits the optimisation pressure measured by Theorem 4.4.

Rafailov et al. (2023) invert it: the reward is \beta\log(\pi^*/\pi_{\mathrm{ref}}) plus a term in x alone, which cancels in the Bradley–Terry difference. This gives direct preference optimisation (DPO), with no separate reward model.

DPO removes the reward model, not the proxy: preferences can still be noisy or sycophantic.

5. Robustness and generalization failures

A model can be right on the test set and wrong under small changes of its input. This chapter quantifies adversarial fragility, shows why safety training can be bypassed, and how correct data can teach the wrong goal.

5.1 Adversarial examples

Imperceptible perturbations change the predictions of image classifiers (Szegedy et al., 2014). Goodfellow, Shlens and Szegedy (2015) blamed near-linearity in high dimension and proposed the fast gradient sign method (FGSM), the step \delta = \varepsilon\,\mathrm{sign}(\nabla_x \ell) on the loss \ell.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)             # 8x8 images, pixels in [0, 16]
mask = (y == 3) | (y == 8)                       # binary task: 3 against 8
X, y = X[mask] / 16.0, (y[mask] == 8).astype(int)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, random_state=0)
clf = LogisticRegression(C=1.0, max_iter=5000).fit(Xtr, ytr)
w, b = clf.coef_[0], clf.intercept_[0]

f = Xte @ w + b
correct = (f > 0) == (yte == 1)
eps_inf = np.abs(f) / np.abs(w).sum()            # Proposition 5.1
print(f"test accuracy {correct.mean():.3f}")
print(f"median l_inf flip radius {np.median(eps_inf[correct]):.3f} (pixel range 1)")
delta = -1.0001 * eps_inf[:, None] * np.sign(w)[None, :] * np.sign(f)[:, None]
print("all flipped:", bool(np.all(np.sign((Xte + delta) @ w + b) != np.sign(f))))

The classifier is 98.3 % accurate, yet half of its correct answers flip when each pixel moves by at most 0.149 (ignoring the range [0, 1]). As \|w\|_1 grows with the dimension, the radius shrinks for larger images.

5.2 Jailbreaks and repeated attempts

Safety training teaches refusals. Wei, Haghtalab and Steinhardt (2023) explain jailbreaks by competing objectives, helpfulness against harmlessness, and by safety training generalizing less far than capabilities. Zou et al. (2023) found adversarial suffixes, optimised on open models, that transfer to closed ones.

Robustness must be measured against an attacker’s budget, not per query (ch. 7).

5.3 Goal misgeneralization

In CoinRun, the coin always sat at the end of the level during training. When Langosco et al. (2022) moved it, agents ran to the end and ignored it. The data could not tell the two goals apart; the implicit bias of training chose.

Learned goals cannot be read from behaviour alone, which motivates looking inside models (ch. 6).

6. Interpretability: understanding what models compute

Behavioural tests show what a model does on the inputs we try. Interpretability reads how it computes, to find hidden goals or failure modes before they show up. This chapter covers the main tools and the obstacle of superposition.

6.1 Features as directions

Mechanistic interpretability looks for features, properties the model represents, and circuits that connect them (Olah et al., 2020). The linear representation hypothesis takes features to be directions in activation space (Park, Choe and Veitch, 2024).

A probe shows that a property is decodable, not that the model uses it; control tasks separate the two (Hewitt and Liang, 2019).

6.2 Superposition

Orthogonal features would be at most d in a layer of width d. Models seem to represent far more, in superposition: nearly orthogonal directions, which high dimension provides.

The price is interference between features. Elhage et al. (2022) showed with toy models that superposition appears when features are sparse, so that interference rarely matters.

import numpy as np
import torch

torch.manual_seed(0)
torch.set_num_threads(1)
n, m, S = 5, 2, 0.9           # 5 features, 2 dimensions, each feature absent w.p. 0.9

W = torch.nn.Parameter(0.1 * torch.randn(m, n))
b = torch.nn.Parameter(torch.zeros(n))
opt = torch.optim.Adam([W, b], lr=1e-2)
for step in range(5000):
    x = torch.rand(1024, n) * (torch.rand(1024, n) > S)     # sparse features in [0, 1]
    x_hat = torch.relu(x @ W.T @ W + b)                     # h = W x, x' = ReLU(W^T h + b)
    loss = ((x - x_hat) ** 2).sum(dim=1).mean()
    opt.zero_grad(); loss.backward(); opt.step()

Wd = W.detach().numpy()
angles = np.sort(np.degrees(np.arctan2(Wd[1], Wd[0])) % 360)
print("loss", round(loss.item(), 4))
print("feature norms  ", np.round(np.linalg.norm(Wd, axis=0), 2))
print("angle gaps (deg)", np.round(np.diff(np.r_[angles, angles[0] + 360]), 1))
print("bias", np.round(b.detach().numpy(), 2))

Five features fit in two dimensions as a regular pentagon: norms near 1.1, gaps near 72°, and a bias near −0.22 that filters interference. With dense features (S = 0, Ex. 12) mainly two survive.

6.3 Sparse autoencoders

Columns of W_d are candidate features, and the \ell_1 penalty keeps few active at once. Bricken et al. (2023) and Cunningham et al. (2024) found interpretable features this way; Templeton et al. (2024) extracted millions from a production model, some related to safety, such as deception.

6.4 Causal interventions

Meng et al. (2022) located factual associations this way, and Wang et al. (2023) found a circuit for indirect-object identification in GPT-2 small. Adding a direction to activations steers behaviour (Turner et al., 2023; Zou et al., 2023, on representation engineering).

Interpretability cannot yet verify that a frontier model has no hidden goal; its explanations must be tested by interventions.

7. Evaluating dangerous capabilities

Safe deployment requires knowing what a model can be made to do. This chapter defines dangerous-capability evaluations and the statistics to read them.

7.1 What to evaluate

What matters is what a motivated actor could obtain, so an evaluation reports its best elicitation. A capability not found is not a capability absent.

7.2 Elicitation by repeated sampling

The pass@k metric (Chen et al., 2021) asks whether one of k samples succeeds: 1 - (1 - p)^k, as in Proposition 5.2.

The plug-in 1 - (1 - c/n)^k is biased low, by concavity: it understates the capability.

import numpy as np
from math import comb

def pass_at_k(n, c, k):
    """Unbiased estimator of 1 - (1-p)^k from n samples with c successes (Theorem 7.2)."""
    return 1.0 - comb(n - c, k) / comb(n, k) if n - c >= k else 1.0

rng = np.random.default_rng(0)
p, n, k, reps = 0.05, 50, 10, 20000
c = rng.binomial(n, p, reps)
unbiased = np.array([pass_at_k(n, ci, k) for ci in c])
plug_in = 1 - (1 - c / n) ** k
print(f"true pass@{k} = {1 - (1 - p) ** k:.4f}")
print(f"mean unbiased = {unbiased.mean():.4f}, mean plug-in = {plug_in.mean():.4f}")

# Hoeffding (Proposition 0.5): trials for a +/- 0.05 estimate with 95 % confidence
print("Hoeffding n >=", int(np.ceil(np.log(2 / 0.05) / (2 * 0.05 ** 2))))
for n0 in [30, 100, 300]:                       # Proposition 7.3
    print(f"0 successes in {n0}: p_max = {1 - 0.05 ** (1 / n0):.4f}, 3/n = {3 / n0:.4f}")

The plug-in averages 0.375 against a true 0.401; the unbiased estimator is on target. Hoeffding’s bound guarantees a rate within 0.05, with 95 % confidence, after 738 trials, and 300 clean attempts only show p \le 1\,\%, weak protection by Example 5.3.

7.3 Access to closed models

Most frontier models are closed. Evaluators may get API access only, or log-probabilities, fine-tuning and activations. Casper et al. (2024) argue that black-box access is insufficient for rigorous audits: deeper access elicits more and allows the checks of ch. 6.

7.4 Sandbagging and safety frameworks

Models can be prompted or fine-tuned to underperform selectively on dangerous-capability evaluations (van der Weij et al., 2024), and frontier models underperformed strategically in some test scenarios (Meinke et al., 2024).

Frontier safety frameworks, such as Anthropic’s Responsible Scaling Policy (2023), OpenAI’s Preparedness Framework (2023) and Google DeepMind’s Frontier Safety Framework (2024), set capability thresholds and the safeguards required beyond them: if-then commitments that make evaluations decisive.

8. Governance and international coordination

Safety measures are adopted only if developers and states have reasons to adopt them. This chapter models races, shows why verification enables cooperation, and surveys regulation.

8.1 Races as games

Armstrong, Bostrom and Shulman (2016) model developers trading safety for speed in a race. In the simplest version, each of two labs chooses Safe (S) or Fast (F).

Repeat the game forever with discount factor \delta \in (0, 1), and consider grim trigger: play S until the other plays F, then F forever.

A strategy pair is a subgame-perfect equilibrium when it is a Nash equilibrium after every history. By the one-shot deviation principle, it suffices to check that no player gains by deviating in a single period and then following the strategy again.

With T = 5, R = 3, P = 1, cooperation needs \delta \ge 0.5. The proof assumes deviations are seen at once; with imperfect verification the required patience rises (Ex. 14).

8.2 Verification and compute

Compute is a natural target for governance: detectable, excludable, quantifiable, with a concentrated supply chain (Sastry et al., 2024), and predictive of capability (ch. 1).

8.3 Regulation and international coordination

The EU AI Act classifies uses by risk; its obligations for general-purpose models apply from 2 August 2025. Providers of models with systemic risk must evaluate them, including adversarial testing, report serious incidents and ensure cybersecurity, as detailed by the voluntary General-Purpose AI Code of Practice of July 2025.

The Bletchley Park summit (November 2023) produced the Bletchley Declaration, and states created AI safety institutes, such as the United Kingdom’s, renamed AI Security Institute in 2025. At Seoul (May 2024), companies signed the Frontier AI Safety Commitments. The International AI Safety Report, chaired by Yoshua Bengio, appeared before the Paris AI Action Summit of February 2025.

8.4 What engineers can do

The bottlenecks are technical: evaluations that elicit well (ch. 7), interpretability that finds hidden goals (ch. 6), training that resists proxy failures (ch. 4–5), verification of training runs and hardware. Engineers build and stress-test these tools in labs, safety institutes, academia and non-profits; ch. 10 lists open problems.

9. AI control and scalable oversight

The previous chapters try to make models aligned and to check it. This self-study chapter asks two complementary questions. First, can we deploy a capable model safely even if it might be trying to subvert us? Second, how can weaker overseers supervise stronger models? Both are studied with explicit attacker models, in the spirit of security engineering.

9.1 The control setting

The red team plays the worst case: it knows the protocol but not which tasks will be audited. A catastrophe happens if an attack, such as code with a hidden backdoor, is used without being audited; an audited attack is caught, and the deployment is stopped.

9.2 Upfront auditing

The simplest protocol audits the first M = fN tasks, then trusts U. The attacker attacks each task independently with probability p.

9.3 Trusted monitoring

A better protocol asks T to give each output of U a suspicion score, and audits the fraction f of the most suspicious ones.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
N, f = 1000, 0.02
M = int(f * N)                                  # the first M tasks are audited

# Upfront auditing: closed form (Theorem 9.2) against simulation
p_star = 1 - f ** (1 / (N - M))
red_win = (1 - f) * f ** (f / (1 - f))
games = 20000
attacks = rng.random((games, N)) < p_star
first = np.where(attacks.any(axis=1), attacks.argmax(axis=1), N)
red_sim = np.mean((first >= M) & (first < N))
print(f"optimal attack rate p* = {p_star:.5f}")
print(f"red win: formula {red_win:.4f}, simulation {red_sim:.4f}, safety {1 - red_win:.4f}")

# Trusted monitoring (Proposition 9.4): audit the f most suspicious outputs
tau = norm.ppf(1 - f)
for mu in [1.0, 2.0, 3.0, 4.0]:
    print(f"separation mu = {mu}: catch probability c = {norm.sf(tau - mu):.3f}")

The simulation confirms the formula (0.903 against 0.905). With a monitor that separates attacks by \mu = 3 standard deviations, trusted monitoring catches 83 % of attacks, so its safety is 83 %, against 9.5 % for upfront auditing with the same budget.

Two refinements matter in practice. Defer to trusted replaces the moderately suspicious outputs of U by outputs of T, trading usefulness for safety. Untrusted monitoring uses a copy of U as a better monitor, but the copy may collude with the policy: a common-mode failure in the sense of Proposition 3.2.

9.4 Scalable oversight: debate

When a model’s answers are too hard for humans to check, training on human judgement rewards answers that merely look right (ch. 4). Scalable oversight looks for protocols in which a weak judge can still reward the truth.

The proof reduces true quantified Boolean formulas to debates in which each debater chooses the quantified variables of one kind; the judge checks the final assignment. The guarantee needs optimal play, which training does not ensure. Empirically, Khan et al. (2024) found that non-expert judges, human or model, answer reading-comprehension questions more accurately with debate than without. Recursive reward modelling, in which models help humans evaluate other models, is a related proposal (Leike et al., 2018).

9.5 Weak-to-strong generalization

Burns et al. (2023) study an analogy: fine-tune a strong model on labels produced by a weak one, and ask how much of the strong model’s ability is recovered.

PGR equals 1 when weak supervision is as good as ground truth, and 0 when the student only imitates its supervisor. Burns et al. (2023) report substantial but incomplete recovery on NLP tasks. Ex. 15 shows the mechanism on synthetic data: random supervisor errors are averaged out, while systematic ones are learned.

10. Research directions and open problems

This self-study chapter maps active research questions, area by area, and ends with practical ways for an engineer or applied mathematician to contribute. The field moves quickly: treat every item as a pointer to read, not as a settled result. Two broad surveys of open problems are Anwar et al. (2024) and, for an earlier and more general list, Hendrycks et al. (2021).

10.1 Scaling and forecasting

• Data and compute limits. Villalobos et al. (2024) estimate when the stock of public human-written text will be exhausted at current scaling rates. Whether synthetic data and reinforcement learning remove this limit is open.
• Robust scaling laws. The replication by Besiroglu et al. (2024) shows how much fitted exponents depend on the data and the fitting procedure. Laws for downstream capabilities, rather than loss, remain much less reliable.
• Measuring agents. Task-length measures such as that of Kwa et al. (2025) depend on the task suite. Building benchmarks that track real-world autonomy, and do not saturate within a year, is an open problem.

10.2 Alignment and specification

• Hidden or learned goals. Hubinger et al. (2019) analysed the hypothesis that training could produce a model that pursues its own objective while appearing aligned. Later work has built model organisms of such behaviour: backdoored “sleeper agents” whose behaviour persisted through standard safety training (Hubinger et al., 2024), and a model that selectively complied with a training objective, reasoning that this avoided being modified (Greenblatt et al., 2024, on alignment faking; not the control paper of ch. 9).
• Emergent misalignment. Betley et al. (2025) found that fine-tuning a model on a narrow task, writing insecure code without saying so, made it broadly misaligned on unrelated prompts. Why such generalization happens is not understood.
• Reward hacking in reasoning models. Baker et al. (2025) monitored the chains of thought of a reasoning model to detect reward hacking. They found that penalising the flagged thoughts led the model to hide its intent while still hacking: optimisation pressure against a monitor is a Goodhart problem (ch. 4).
• Theory of reward hacking. Skalse et al. (2022) give a formal definition of a proxy reward that cannot be hacked and show that such proxies are rare. Pan, Bhatia and Steinhardt (2022) measure how misspecified rewards produce phase transitions in behaviour as agents grow more capable.

10.3 Interpretability

• Beyond sparse autoencoders. Ameisen et al. (2025) and Lindsey et al. (2025) trace computational graphs through replacement models of a production language model. Whether such explanations are faithful and complete enough for safety cases is open; Sharkey et al. (2025) survey the open problems of mechanistic interpretability.
• Chain-of-thought monitoring. Korbak et al. (2025) argue that reasoning written in natural language is a fragile opportunity for oversight, which training choices could destroy. Measuring monitorability, and keeping it, is a new research area.
• Unlearning. Li et al. (2024) introduced the WMDP benchmark of hazardous knowledge and a method to remove it. Checking that removed knowledge cannot be recovered by fine-tuning connects unlearning with evaluation (ch. 7).

10.4 Evaluation, control and verification

• Statistics of evaluations. Miller (2024) applies standard statistical practice to language-model evaluations: standard errors, clustered questions, paired comparisons. Elicitation gaps and sandbagging (ch. 7.4) have no accepted measurement method.
• Control of agents. Extending the protocols of ch. 9 from single outputs to long agentic trajectories, with tools and memory, is open, as is estimating the catch probability c of Proposition 9.4 for realistic attacks.
• Guarantees. Dalrymple et al. (2024) propose systems with quantitative safety guarantees relative to a world model and a formal specification. How far formal methods can scale to general-purpose systems is unknown.

10.5 Governance

• Verification of agreements. Theorem 8.2 needs deviations to be observable. Hardware and software mechanisms that let a third party verify claims about training runs, without revealing secrets, are an engineering problem.
• Thresholds as proxies. Compute thresholds (Definition 8.3) will need to track algorithmic progress, or be complemented by capability-based triggers from evaluations.
• Expert assessments. Bengio et al. (2024) set out a consensus statement on managing extreme risks; the International AI Safety Report (Bengio et al., 2025) synthesises the evidence for policymakers. Turning such assessments into verifiable commitments is the open step.

10.6 How engineers can contribute

The official course description insists on this question. Concrete routes for an MVA student:

• Reproduce and extend a paper. Most results above come with code. A careful reproduction with error bars, then one ablation, is a real contribution and the format of the projects below.
• Use the open tooling. TransformerLens (Nanda and Bloom, 2022) supports interpretability experiments on open models, and the Inspect framework of the UK AI Security Institute supports evaluations.
• Bring mathematics. Statistics of evaluations, game theory of control and governance, optimisation theory of reward hacking, high-dimensional geometry of representations: each section of these notes has an open problem that needs an applied mathematician.
• Follow structured programmes. The BlueDot Impact courses on alignment and governance, listed by the course as an official resource, give reading lists and cohorts.

10.7 Open problems

1. Give a capability-based trigger, measurable before deployment, that would replace or complement a compute threshold.
2. Find a training signal for tasks that humans cannot evaluate, with a guarantee in the spirit of Theorem 9.6 that survives imperfect play.
3. Decide, from weights and activations, whether a model contains a backdoor of the kind studied by Hubinger et al. (2024).
4. Bound the elicitation gap: how much capability can remain hidden after a given elicitation budget?
5. Design control protocols for agents whose catch probability can be estimated with confidence intervals.
6. Explain emergent misalignment: why does narrow fine-tuning change broad dispositions?

Exercises

1. ★ Training compute and thresholds. (a) Compute the training compute of a model with 8 \times 10^{9} parameters trained on 1.5 \times 10^{13} tokens. (b) How many days does it take on 1,000 accelerators that each sustain 4 \times 10^{14} FLOP/s? (c) On the same data, what is the largest model below the threshold of Definition 8.3?

   Solution: (a) By Proposition 1.1, C = 6 \times 8 \times 10^{9} \times 1.5 \times 10^{13}, that is 7.2 \times 10^{23} FLOP.

   (b) 7.2 \times 10^{23} / (4 \times 10^{17}) = 1.8 \times 10^{6} s, about 20.8 days. (c) N < 10^{25}/(9 \times 10^{13}) \approx 1.1 \times 10^{11} parameters.

2. ★ Emergence from a smooth metric. (a) For answers of k = 10 tokens, which per-token accuracy p gives an exact-match accuracy of 0.5 under Proposition 1.6? (b) If p improves from 0.80 to 0.98 across model sizes, how does exact match change? (c) Which transformation of exact match makes the curve smooth again?

   Solution: (a) p = 0.5^{1/10} \approx 0.933. (b) From 0.8^{10} \approx 0.107 to 0.98^{10} \approx 0.817: an apparent jump. (c) Its logarithm, k\log p, which is linear in \log p.

3. ★ Scoring forecasts. Forecaster A announces 0.9 and B announces 0.6 for the same event. (a) Compute both Brier and log losses if the event happens, and if it does not. (b) If the true probability is 0.7, who has the lower expected Brier loss?

   Solution: (a) If it happens: Brier 0.01 for A and 0.16 for B; log 0.105 and 0.511. If not: Brier 0.81 and 0.36; log 2.303 and 0.916.

   (b) By Theorem 2.4, A expects (0.9 - 0.7)^2 + 0.21 = 0.25 and B expects (0.6 - 0.7)^2 + 0.21 = 0.22. B is closer to the truth, so B wins in expectation.

4. ★ Extrapolating a doubling trend. A task horizon is 2 hours today and doubles every 7 months. (a) When does it reach a 40-hour work week? (b) And with a doubling time of 4 months? (c) Which input of the forecast matters most?

   Solution: (a) \log_2 20 \approx 4.32 doublings, so about 30 months. (b) About 17 months. (c) The time is proportional to T, so a 40 % error on the doubling time is a 40 % error on the date; the assumption that the trend continues matters even more.

5. ★ Defence in depth. Two layers fail independently with probability 0.05 each. (a) Compute the breach probability of Proposition 3.2 for \rho = 0, 0.001 and 0.01. (b) With \rho = 0.01, compare adding a third independent layer that fails 20 % of the time with reducing \rho to 0.001.

   Solution: (a) 0.0025, then 0.001 + 0.999 \times 0.0025 \approx 0.0035, then 0.01 + 0.99 \times 0.0025 \approx 0.0125.

   (b) The third layer gives 0.01 + 0.99 \times 0.0005 \approx 0.0105; reducing \rho gives 0.0035. Decorrelating the layers is three times more effective here.

6. ★ Repeated attempts and the rule of three. (a) A jailbreak works with probability 0.002 per attempt. How many attempts give a 50 % chance of success? (b) A red team makes 500 attempts and none succeeds. Give the 95 % upper bound on p. (c) What could an attacker with 1,000 attempts achieve if p equals that bound?

   Solution: (a) k \ge \log 0.5/\log 0.998 \approx 346.2, so 347 attempts. (b) 1 - 0.05^{1/500} \approx 0.0060, close to 3/500 (Proposition 7.3). (c) Since (1 - p_{\max})^{500} = 0.05, the attacker succeeds with probability 1 - 0.05^{2} = 0.9975: a clean red-team result can be compatible with near-certain success for a determined attacker.

7. ★★ Compute-optimal ratio. (a) From Theorem 1.4, show that the optimal number of tokens per parameter is \frac{D^*}{N^*} = G^{-2}\left(\frac{C}{6}\right)^{\frac{\alpha - \beta}{\alpha + \beta}}. (b) With \alpha = 0.34 and \beta = 0.28, by how much does the ratio grow from 10^{21} to 10^{25} FLOP? Check against the output of the code of ch. 1.2. (c) What happens when \alpha = \beta?

   Solution: (a) Divide the two formulas of Theorem 1.4: the exponents give (\alpha - \beta)/(\alpha + \beta) and the constants G^{-1}/G.

   (b) The exponent is 0.06/0.62 \approx 0.097, so the ratio grows by 10^{4 \times 0.097} \approx 2.44; the code prints 50.1 and 122.1, a ratio of 2.44. (c) The ratio is constant: parameters and data scale exactly together.

8. ★★ Best-of-n. (a) Show that the KL divergence of Theorem 4.4 is at most \log n, and explain why without computation. (b) Find the n for which it is about 3 nats. (c) Explain why best-of-n moves away from the base model only slowly, and what this implies for reinforcement learning, which has no such limit.

   Solution: (a) (n-1)/n \ge 0. Directly, the density ratio n F^{n-1} is at most n, so its logarithm is at most \log n.

   (b) \log 54 - 53/54 \approx 3.008, so n \approx 54. (c) The KL grows like \log n: each extra nat costs a factor e in samples. RL can move the policy arbitrarily far, so its optimisation pressure on the proxy, and the risk of the over-optimisation seen in ch. 4.3, must be limited by the KL penalty of Definition 4.5.

9. ★★ From Bradley–Terry to DPO. (a) Write the negative log-likelihood of a reward model on preference pairs, and show that adding any function h(x) to r(x, y) does not change it. (b) Derive the DPO loss of the key box of ch. 4.4 from Theorem 4.6.

   (c) For \beta = 0.1, with ratios \pi/\pi_{\mathrm{ref}} equal to 1.5 on y_w and 0.8 on y_l, compute the loss of this pair, and compare with its value at the start, when \pi = \pi_{\mathrm{ref}}.

   Solution: (a) The loss is -\sum \log\sigma\big(r(x, y_w) - r(x, y_l)\big); the term h(x) cancels in the difference.

   (b) Theorem 4.6 gives the reward in terms of the optimal policy: r = \beta\log(\pi^*/\pi_{\mathrm{ref}}) + \beta\log Z(x). Substituted into the loss of (a), the term \beta \log Z(x) cancels, and the policy replaces the reward.

   (c) The margin is 0.1\,(\log 1.5 - \log 0.8) \approx 0.0629, so the loss is \log(1 + e^{-0.0629}) \approx 0.662, against \log 2 \approx 0.693 at the start.

10. ★★ The \ell_2 flip radius. (a) Show that the smallest \|\delta\|_2 that flips f(x) = \langle w, x\rangle + b > 0 is f(x)/\|w\|_2, and give the minimiser.

    (b) Show that the ratio of the \ell_\infty radius to the \ell_2 radius lies in [1/\sqrt{d}, 1]. (c) Adapt the code of ch. 5.1 to print the median \ell_2 radius.

    Solution: (a) By Cauchy–Schwarz (Proposition 0.6), f(x + \delta) \ge f(x) - \|w\|_2\|\delta\|_2. Equality holds for \delta = -f(x)\, w/\|w\|_2^2.

    (b) The ratio is \|w\|_2/\|w\|_1, and \|w\|_2 \le \|w\|_1 \le \sqrt{d}\,\|w\|_2. (c) Add print(np.median((np.abs(f) / np.linalg.norm(w))[correct])): about 0.79, while a typical image has norm about 3.9.

11. ★★ Gaussian against heavy-tailed Goodhart (Python). Estimate \mathbb{E}[V \mid U \approx u] for U = V + X, with V standard normal and X either standard normal or Student with 2 degrees of freedom. Compare with Propositions 4.2 and 4.3.

    Solution:

    import numpy as np
    
    rng = np.random.default_rng(1)
    n = 2_000_000
    V = rng.standard_normal(n)
    for name, X in [("gaussian", rng.standard_normal(n)), ("t(2)", rng.standard_t(2, n))]:
        U = V + X
        row = []
        for u in [1, 2, 3, 4, 5, 6]:
            sel = (U > u) & (U < u + 0.5)            # bin [u, u + 0.5)
            row.append(f"{V[sel].mean():5.2f}")
        print(f"{name:9s} E[V | U ~ u] for u = 1..6:", " ".join(row))

    The Gaussian row, 0.62, 1.11, 1.62, 2.09, 2.56, 3.15, follows (u + 0.25)/2 at the bin centres, as Proposition 4.2 predicts. The Student row, 0.55, 0.83, 0.87, 0.76, 0.60, 0.51, peaks and then decreases towards 0, as Proposition 4.3 predicts.

12. ★★ Superposition capacity (Python). (a) Draw 2,000 random unit vectors in \mathbb{R}^{1024} and compute their largest pairwise overlap; compare with the guarantee of Theorem 6.2 for \varepsilon = 0.2. (b) Train the toy model of ch. 6.2 with dense features, S = 0, and describe the result.

    Solution:

    import numpy as np
    import torch
    
    rng = np.random.default_rng(0)
    d, m = 1024, 2000
    U = rng.standard_normal((m, d))
    U /= np.linalg.norm(U, axis=1, keepdims=True)
    G = np.abs(U @ U.T)
    np.fill_diagonal(G, 0)
    print(f"{m} random unit vectors in R^{d}: max overlap {G.max():.3f}")
    print(f"guarantee of Theorem 6.2 for eps = 0.2: {np.exp(d * 0.2 ** 2 / 4):.0f} vectors")
    
    torch.manual_seed(0)
    torch.set_num_threads(1)
    n, k, S = 5, 2, 0.0                              # dense features
    W = torch.nn.Parameter(0.1 * torch.randn(k, n))
    b = torch.nn.Parameter(torch.zeros(n))
    opt = torch.optim.Adam([W, b], lr=1e-2)
    for step in range(5000):
        x = torch.rand(1024, n) * (torch.rand(1024, n) > S)
        loss = ((x - torch.relu(x @ W.T @ W + b)) ** 2).sum(dim=1).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    print("dense features, norms:", np.round(np.linalg.norm(W.detach().numpy(), axis=0), 2))

    (a) The largest overlap among the 2,000 random vectors is 0.153. For \varepsilon = 0.2 the theorem guarantees 28,001 vectors, but for \varepsilon = 0.153 only about e^{6} \approx 400, fewer than the 2,000 observed. The cap bound overestimates the Gaussian-like tail by a factor of order \varepsilon\sqrt{d}, here about 13, but the exponential growth e^{d\varepsilon^2/4} is the right order.

    (b) The norms are about 0.9, 0.37, 0.91, 0.35 and 0.37: with dense features, interference is costly, and the model mainly keeps two features, as Elhage et al. (2022) found.

13. ★★ Estimating pass@k. Twenty samples contain two successes. (a) Compute the unbiased estimates of pass@1, pass@5 and pass@10 of Theorem 7.2, and the plug-in estimates. (b) Prove that the plug-in estimator is biased downwards.

    Solution: (a) pass@1 = 0.1, and \text{pass@}5 = 1 - \binom{18}{5}\Big/\binom{20}{5} = 1 - \frac{8568}{15504} \approx 0.447, against 1 - 0.9^5 \approx 0.410 for the plug-in. Likewise pass@10 = 1 - 43758/184756 \approx 0.763, against 1 - 0.9^{10} \approx 0.651.

    (b) The map t \mapsto 1 - (1 - t)^k is concave on [0, 1], so Jensen’s inequality gives \mathbb{E}[1 - (1 - c/n)^k] \le 1 - (1 - p)^k, strictly for k \ge 2 and 0 < p < 1.

14. ★★ Cooperation with imperfect verification. In the repeated game of Theorem 8.2, a lab that plays F is detected only with probability q in each period; until detected, it keeps playing F and earns T; once detected, both play F forever. (a) Show that grim trigger remains an equilibrium if and only if \frac{\delta}{1 - \delta} \ge \frac{T - R}{q\,(R - P)}. (b) With T = 5, R = 3 and P = 1, find the threshold on \delta when q is 1, 0.5 or 0.25.

    Solution: (a) The value W of deviating satisfies W = T + \delta\big(qP/(1 - \delta) + (1 - q)W\big). Compare W with R/(1 - \delta) and rearrange: \delta q (R - P) \ge (1 - \delta)(T - R). For q = 1 this is Theorem 8.2.

    (b) The right-hand side is 2/(2q), so \delta \ge 0.5, 2/3 and 0.8. Better verification makes cooperation possible for less patient players.

15. ★★★ Weak-to-strong generalization (Python). On synthetic data, train a strong model on labels from a weak supervisor, and compute the PGR of Definition 9.7 for two kinds of supervision: a weak model that sees only 4 informative features, whose errors are a simple function of the input, and labels with 30 % random flips.

    Solution:

    import numpy as np
    from sklearn.datasets import make_classification
    from sklearn.linear_model import LogisticRegression
    from sklearn.ensemble import HistGradientBoostingClassifier
    from sklearn.model_selection import train_test_split
    
    X, y = make_classification(n_samples=12000, n_features=20, n_informative=10, n_redundant=0,
                               flip_y=0.0, class_sep=1.0, shuffle=False, random_state=0)
    Xw, Xr, yw, yr = train_test_split(X, y, test_size=0.75, random_state=0)
    Xs, Xte, ys, yte = train_test_split(Xr, yr, test_size=0.4, random_state=0)
    rng = np.random.default_rng(0)
    
    weak = LogisticRegression(max_iter=1000).fit(Xw[:, :4], yw)   # sees 4 informative features
    labels = {
        "systematic": weak.predict(Xs[:, :4]),                   # errors: a simple function of x
        "random": np.where(rng.random(len(ys)) < 0.3, 1 - ys, ys),  # errors independent of x
    }
    strong = lambda: HistGradientBoostingClassifier(random_state=0)
    a_ceil = (strong().fit(Xs, ys).predict(Xte) == yte).mean()
    for name, lab in labels.items():
        a_weak = (lab == ys).mean()                               # accuracy of the supervision
        a_w2s = (strong().fit(Xs, lab).predict(Xte) == yte).mean()
        pgr = (a_w2s - a_weak) / (a_ceil - a_weak)
        print(f"{name:10s} weak {a_weak:.3f}  student {a_w2s:.3f}  ceiling {a_ceil:.3f}  PGR {pgr:.2f}")

    With systematic errors, the student reaches 0.608 from a supervisor at 0.598: PGR 0.03, since a flexible student learns the supervisor’s errors exactly. With random errors, it reaches 0.806 from 0.703: PGR 0.45, since random errors average out.

    The open question is which regime human supervision of superhuman models resembles; human errors are systematic, which is what makes the problem hard. Extensions: early stopping, an auxiliary confidence loss as in Burns et al. (2023).

16. ★★★ Upfront auditing against an imperfect attacker. In Theorem 9.2, suppose that an attack after the audit period succeeds only with probability s, and otherwise fails unnoticed. (a) Write the attacker’s winning probability. (b) With p = \lambda/N and N \to \infty, find the optimal \lambda and the limit value. (c) Evaluate it for f = 0.02 and s = 0.25, and discuss which protocol of ch. 9.3 you would add.

    Hints and expected result: (a) (1 - p)^{M}\big(1 - (1 - ps)^{N - M}\big). (b) With a = s(1 - f), the limit is e^{-\lambda f}(1 - e^{-\lambda a}), maximised at \lambda^* = \log\big((a + f)/f\big)/a, with value \left(\frac{f}{a + f}\right)^{f/a}\frac{a}{a + f}. For s = 1 this is Theorem 9.2. (c) About 0.749 (\lambda^* \approx 10.5), so safety is about 25 %; the exact optimum of (a) for N = 1000 is 0.748. Trusted monitoring with c above 0.75 would do better, and the two can be combined.

17. ★★★ Position paper: releasing frontier model weights. Write a two-page position for or against releasing the weights of a model just above the threshold of Definition 8.3. State your definition of the risk (Definition 3.1), your evidence, a probability that could be scored (Theorem 2.4), the safeguards you would require, and the strongest argument of the other side.

    Hints and expected result: a good answer separates benefits (research access, reproducibility, competition) from risks (irreversibility, removal of safeguards by fine-tuning, transfer of attacks found on open models, ch. 5.2). It uses evaluations of marginal risk with honest statistics (ch. 7), and names the evidence that would change its conclusion.

Projects and assessment

The official assessment is 100 % project, with the details given in the course syllabus (master-mva.com); Université Paris-Saclay describes it as “Evaluation Continue non Intégrale”. The syllabus itself is not reproduced in the catalogue, so the subjects below are proposals sized for a seminar project: a precise question, code or a proof, a short report and a talk. Each one reproduces or extends published work, in the tradition of MVA projects. The official assessment has no written exam; the exam proposal is for self-assessment.

Solo projects

1. [Solo · ≈ 20 h] Paper study: “Toy Models of Superposition” (Elhage et al., 2022). When do sparse features share dimensions?
   • Implement the model of ch. 6.2 with an importance weight on each feature in the loss, as in the paper. Reproduce the phase diagram for two features in one dimension: represented, superposed or ignored, as a function of sparsity and relative importance.
   • Reproduce the geometry for five features in two dimensions, and measure how the bias filters interference.
   • Success criterion: a phase diagram with the three regions, with boundaries stable over 5 seeds. Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters 0 and 6.
2. [Solo · ≈ 20 h] Paper study: reward-model over-optimisation (Gao, Schulman and Hilton, 2023), synthetic version. Does best-of-n against a learned proxy follow the law d(\alpha - \beta d) that the paper reports, with d the square root of the KL divergence and \alpha, \beta fitted coefficients (not those of ch. 1)?
   • Gold reward: a fixed random two-layer network on Gaussian inputs. Proxy: a Bradley–Terry model (Definition 4.5) fitted on 100 to 10,000 noisy comparisons.
   • Measure the gold reward of best-of-n against d, using Theorem 4.4 for the KL.
   • Success criterion: the rise and fall, a fitted curve with confidence bands, and the peak location as a function of the number of comparisons. Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters 0 and 4.
3. [Solo · ≈ 20 h] Forecasting study: the growth of training compute. Re-estimate the doubling time of the training compute of notable models from the public database of notable AI models maintained by Epoch AI, and compare with Sevilla et al. (2022).
   • Fit log-compute against time (Definition 0.4), with bootstrap intervals, and test for a change of slope.
   • Turn the fit into a probabilistic forecast of when a model crosses 10^{26} FLOP, scored after the fact with Theorem 2.4.
   • Success criterion: a doubling time with a 90 % interval and a written forecast. Deliverables: notebook, 4-page report, 10-minute talk. Chapters 1 and 2.

Group projects

1. [Group of 2–3 · ≈ 30 h each] Paper study: AI control (Greenblatt et al., 2024) in a toy setting. Generated programming tasks, small Python functions with unit tests, are solved by a scripted untrusted policy that sometimes inserts a backdoor, such as a special input with a wrong output.
   • Member 1: the task generator and the red-team policies (attack rate, subtlety of the backdoor).
   • Member 2: the trusted monitor, a small classifier on features of the code, and its ROC curve.
   • Member 3: the protocols of ch. 9: upfront auditing, trusted monitoring, defer to trusted.
   • Success criterion: a safety–usefulness frontier with confidence intervals, which matches Theorem 9.2 and Proposition 9.4 in the cases they cover. Deliverables: code, a 6–8 page report, a 20-minute talk. Chapters 3, 7 and 9.
2. [Group of 2–3 · ≈ 30 h each] Paper study: “Progress measures for grokking via mechanistic interpretability” (Nanda et al., 2023). Train a one-layer transformer on addition modulo 113 on a CPU or a free GPU tier, and reverse-engineer it.
   • Member 1: training, and the grokking curve of train and test accuracy.
   • Member 2: Fourier analysis of the embeddings and logits; linear probes for the key frequencies (Definition 6.1).
   • Member 3: causal tests (Definition 6.5): ablate the key frequencies, then all others.
   • Success criterion: test accuracy collapses when the few key frequencies are removed and is preserved when all the others are. Deliverables: code, a 6–8 page report, a 20-minute talk. Chapters 0 and 6.
3. [Group of 2–3 · ≈ 30 h each] An honest evaluation: the elicitation gap of a small open model. Measure how much accuracy elicitation adds for a small open-weight instruction model (0.5 to 1.5 billion parameters) on the GSM8K test set (Cobbe et al., 2021).
   • Member 1: the evaluation harness, with zero-shot, few-shot and chain-of-thought prompting.
   • Member 2: repeated sampling, unbiased pass@k (Theorem 7.2) and majority voting.
   • Member 3: statistics (Hoeffding and bootstrap intervals, paired comparisons) and a sandbagging test: prompt the model to underperform and try to detect it.
   • Success criterion: an elicitation curve with error bars, and a detection rate for sandbagging. Deliverables: code, a 6–8 page report, a 20-minute talk. Chapters 5 and 7.

Written exam (proposal)

Two hours, no documents, a non-programmable calculator allowed; four questions of 5 points.

1. Question (5 points). Give the three families of risk of Definition 3.1 with one example each. Then name the four variants of Goodhart’s law, illustrate two of them, and explain why two safeguards built from the same model may protect less than their individual failure rates suggest. Expected: Definition 3.1 and Definition 4.1 with correct examples (for instance, CoastRunners as extremal, and sycophancy as a policy exploiting the gap between approval and quality); Proposition 3.2, where a common mode \rho sets a floor on the breach probability.

2. Question (5 points). (a) Compute the training compute of a model with 3 \times 10^{10} parameters trained on 6 \times 10^{12} tokens, and compare it with the EU threshold. (b) Under Theorem 1.4 with \alpha = 0.34 and \beta = 0.28, by which factors should N and D grow when compute grows by 100? (c) Ten samples contain one success: estimate pass@3.

   Expected: (a) 1.08 \times 10^{24} FLOP, below 10^{25}. (b) 100^{0.45} \approx 7.9 for parameters and 100^{0.55} \approx 12.6 for tokens. (c) 1 - \binom{9}{3}/\binom{10}{3} = 1 - 84/120 = 0.3.

3. Question (5 points). State and prove Theorem 4.6 on the KL-regularised optimum. Deduce the DPO loss, and explain why the partition function does not need to be computed. Expected: the identity J(\pi) = \beta\log Z - \beta\,\mathrm{KL}(\pi\|\pi^*) and Gibbs’ inequality; inversion of the optimum; cancellation of \beta \log Z(x) in the Bradley–Terry difference.

4. Question (5 points). Prove Theorem 4.4 on the KL divergence of best-of-n. Find the n that gives about 2 nats, and explain with Proposition 4.3 why a larger n can lower the true quality of the selected answer.

   Expected: the density nF^{n-1}, \mathbb{E}[\log W] = -1/n; n \approx 19, since \log 19 - 18/19 \approx 1.997; with heavy-tailed proxy errors, the most extreme proxy scores are explained by the errors, so the true value reverts to the mean.

Grading rubric (proposal)

The official grade is 100 % project. The split below is a proposal for grading a project report and its presentation.

References

Official sources and resources of the course

• master-mva.com, course page “Turing Seminar – An introduction to AGI Safety”, 2026–2027: https://www.master-mva.com/cours/seminaire-turing/. The page links the course syllabus and the recorded seminars of the 2024 edition on YouTube.
• Université Paris-Saclay, M2 Mathematics, Vision, Learning, programme table and course detail “Séminaire Turing - Sûreté et interprétabilité des IA à usage général” (ECTS and hourly volume).
• M. Grey, C.-R. Segerie, AI Safety Atlas, French Center for AI Safety (CeSIA). Most sessions of the seminar are based on its chapters, according to the course page.
• D. Hendrycks, Introduction to AI Safety, Ethics, and Society, Center for AI Safety, 2024.
• BlueDot Impact, courses on AI Alignment and AI Governance, https://bluedot.org/courses.
• N. Maslej et al., The AI Index 2025 Annual Report, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, 2025.
• Y. Bengio et al., International AI Safety Report, UK Department for Science, Innovation and Technology, January 2025.

Books and background

• N. Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014.
• S. Russell, Human Compatible: Artificial Intelligence and the Problem of Control, Viking, 2019.
• B. Christian, The Alignment Problem: Machine Learning and Human Values, W. W. Norton, 2020.
• S. Boucheron, G. Lugosi, P. Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence, Oxford University Press, 2013.
• K. Ball, “An elementary introduction to modern convex geometry”, in S. Levy (ed.), Flavors of Geometry, MSRI Publications 31, Cambridge University Press, 1997.
• J. Reason, Human Error, Cambridge University Press, 1990.
• M. Strathern, “‘Improving ratings’: audit in the British University system”, European Review 5(3), 305–321, 1997.

Surveys and agendas

• D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané, “Concrete problems in AI safety”, arXiv:1606.06565, 2016.
• D. Hendrycks, N. Carlini, J. Schulman, J. Steinhardt, “Unsolved problems in ML safety”, arXiv:2109.13916, 2021.
• D. Hendrycks, M. Mazeika, T. Woodside, “An overview of catastrophic AI risks”, arXiv:2306.12001, 2023.
• R. Ngo, L. Chan, S. Mindermann, “The alignment problem from a deep learning perspective”, ICLR, 2024.
• U. Anwar et al., “Foundational challenges in assuring alignment and safety of large language models”, Transactions on Machine Learning Research, 2024.
• Y. Bengio et al., “Managing extreme AI risks amid rapid progress”, Science 384(6698), 842–845, 2024.

Capabilities, scaling and forecasting (ch. 1–2)

• A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, “Language models are unsupervised multitask learners”, OpenAI technical report, 2019.
• T. Brown et al., “Language models are few-shot learners”, NeurIPS, 2020.
• OpenAI, “GPT-4 technical report”, arXiv:2303.08774, 2023.
• OpenAI, “GPT-5 system card”, August 2025.
• J. Kaplan et al., “Scaling laws for neural language models”, arXiv:2001.08361, 2020.
• J. Hoffmann et al., “Training compute-optimal large language models”, NeurIPS, 2022.
• T. Besiroglu, E. Erdil, M. Barnett, J. You, “Chinchilla scaling: a replication attempt”, arXiv:2404.10102, 2024.
• J. Sevilla, L. Heim, A. Ho, T. Besiroglu, M. Hobbhahn, P. Villalobos, “Compute trends across three eras of machine learning”, IJCNN, 2022.
• J. Wei et al., “Emergent abilities of large language models”, Transactions on Machine Learning Research, 2022.
• R. Schaeffer, B. Miranda, S. Koyejo, “Are emergent abilities of large language models a mirage?”, NeurIPS, 2023.
• DeepSeek-AI, “DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning”, arXiv:2501.12948, 2025.
• C. Snell, J. Lee, K. Xu, A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling model parameters”, arXiv:2408.03314, 2024.
• M. R. Morris et al., “Position: Levels of AGI for operationalizing progress on the path to AGI”, ICML, 2024.
• T. Kwa et al., “Measuring AI ability to complete long tasks”, arXiv:2503.14499, 2025.
• K. Grace et al., “Thousands of AI authors on the future of AI”, arXiv:2401.02843, 2024.
• G. W. Brier, “Verification of forecasts expressed in terms of probability”, Monthly Weather Review 78(1), 1–3, 1950.
• T. Gneiting, A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation”, Journal of the American Statistical Association 102(477), 359–378, 2007.
• P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, M. Hobbhahn, “Will we run out of data? Limits of LLM scaling based on human-generated data”, ICML, 2024.

Benefits and risks (ch. 3)

• J. Jumper et al., “Highly accurate protein structure prediction with AlphaFold”, Nature 596, 583–589, 2021.
• NLLB Team, “No language left behind: scaling human-centered machine translation”, arXiv:2207.04672, 2022.
• S. Noy, W. Zhang, “Experimental evidence on the productivity effects of generative artificial intelligence”, Science 381(6654), 187–192, 2023.
• E. Brynjolfsson, D. Li, L. Raymond, “Generative AI at work”, Quarterly Journal of Economics 140(2), 889–942, 2025.

Specification and preference learning (ch. 4)

• D. Manheim, S. Garrabrant, “Categorizing variants of Goodhart’s law”, arXiv:1803.04585, 2018.
• J. Clark, D. Amodei, “Faulty reward functions in the wild”, OpenAI blog, 2016.
• M. Sharma et al., “Towards understanding sycophancy in language models”, ICLR, 2024.
• L. Gao, J. Schulman, J. Hilton, “Scaling laws for reward model overoptimization”, ICML, 2023.
• P. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, “Deep reinforcement learning from human preferences”, NeurIPS, 2017.
• L. Ouyang et al., “Training language models to follow instructions with human feedback”, NeurIPS, 2022.
• R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, C. Finn, “Direct preference optimization: your language model is secretly a reward model”, NeurIPS, 2023.
• J. Skalse, N. Howe, D. Krasheninnikov, D. Krueger, “Defining and characterizing reward hacking”, NeurIPS, 2022.
• A. Pan, K. Bhatia, J. Steinhardt, “The effects of reward misspecification: mapping and mitigating misaligned models”, ICLR, 2022.

Robustness and generalization (ch. 5)

• C. Szegedy et al., “Intriguing properties of neural networks”, ICLR, 2014.
• I. Goodfellow, J. Shlens, C. Szegedy, “Explaining and harnessing adversarial examples”, ICLR, 2015.
• A. Wei, N. Haghtalab, J. Steinhardt, “Jailbroken: how does LLM safety training fail?”, NeurIPS, 2023.
• A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models”, arXiv:2307.15043, 2023.
• R. Shah et al., “Goal misgeneralization: why correct specifications aren’t enough for correct goals”, arXiv:2210.01790, 2022.
• L. Langosco, J. Koch, L. Sharkey, J. Pfau, D. Krueger, “Goal misgeneralization in deep reinforcement learning”, ICML, 2022.
• E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, S. Garrabrant, “Risks from learned optimization in advanced machine learning systems”, arXiv:1906.01820, 2019.
• E. Hubinger et al., “Sleeper agents: training deceptive LLMs that persist through safety training”, arXiv:2401.05566, 2024.
• R. Greenblatt et al., “Alignment faking in large language models”, arXiv:2412.14093, 2024.
• J. Betley et al., “Emergent misalignment: narrow finetuning can produce broadly misaligned LLMs”, ICML, 2025.

Interpretability (ch. 6)

• C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, S. Carter, “Zoom in: an introduction to circuits”, Distill, 2020.
• K. Park, Y. J. Choe, V. Veitch, “The linear representation hypothesis and the geometry of large language models”, ICML, 2024.
• G. Alain, Y. Bengio, “Understanding intermediate layers using linear classifier probes”, ICLR workshop, 2017.
• J. Hewitt, P. Liang, “Designing and interpreting probes with control tasks”, EMNLP, 2019.
• N. Elhage et al., “Toy models of superposition”, Transformer Circuits Thread, 2022.
• T. Bricken et al., “Towards monosemanticity: decomposing language models with dictionary learning”, Transformer Circuits Thread, 2023.
• H. Cunningham, A. Ewart, L. Riggs, R. Huben, L. Sharkey, “Sparse autoencoders find highly interpretable features in language models”, ICLR, 2024.
• A. Templeton et al., “Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet”, Transformer Circuits Thread, 2024.
• K. Meng, D. Bau, A. Andonian, Y. Belinkov, “Locating and editing factual associations in GPT”, NeurIPS, 2022.
• K. Wang, A. Variengien, A. Conmy, B. Shlegeris, J. Steinhardt, “Interpretability in the wild: a circuit for indirect object identification in GPT-2 small”, ICLR, 2023.
• A. M. Turner et al., “Activation addition: steering language models without optimization”, arXiv:2308.10248, 2023.
• A. Zou et al., “Representation engineering: a top-down approach to AI transparency”, arXiv:2310.01405, 2023.
• N. Nanda, L. Chan, T. Lieberum, J. Smith, J. Steinhardt, “Progress measures for grokking via mechanistic interpretability”, ICLR, 2023.
• E. Ameisen et al., “Circuit tracing: revealing computational graphs in language models”, Transformer Circuits Thread, 2025.
• J. Lindsey et al., “On the biology of a large language model”, Transformer Circuits Thread, 2025.
• L. Sharkey et al., “Open problems in mechanistic interpretability”, arXiv:2501.16496, 2025.
• T. Korbak et al., “Chain of thought monitorability: a new and fragile opportunity for AI safety”, arXiv:2507.11473, 2025.
• B. Baker et al., “Monitoring reasoning models for misbehavior and the risks of promoting obfuscation”, arXiv:2503.11926, 2025.

Evaluations (ch. 7)

• T. Shevlane et al., “Model evaluation for extreme risks”, arXiv:2305.15324, 2023.
• M. Phuong et al., “Evaluating frontier models for dangerous capabilities”, arXiv:2403.13793, 2024.
• M. Chen et al., “Evaluating large language models trained on code”, arXiv:2107.03374, 2021.
• K. Cobbe et al., “Training verifiers to solve math word problems”, arXiv:2110.14168, 2021.
• S. Casper et al., “Black-box access is insufficient for rigorous AI audits”, ACM FAccT, 2024.
• T. van der Weij, F. Hofstätter, O. Jaffe, S. F. Brown, F. R. Ward, “AI sandbagging: language models can strategically underperform on evaluations”, arXiv:2406.07358, 2024.
• A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, M. Hobbhahn, “Frontier models are capable of in-context scheming”, arXiv:2412.04984, 2024.
• E. Miller, “Adding error bars to evals: a statistical approach to language model evaluations”, arXiv:2411.00640, 2024.
• N. Li et al., “The WMDP benchmark: measuring and reducing malicious use with unlearning”, ICML, 2024.
• Anthropic, Responsible Scaling Policy, 2023. OpenAI, Preparedness Framework (Beta), 2023. Google DeepMind, Frontier Safety Framework, 2024.

Governance (ch. 8)

• S. Armstrong, N. Bostrom, C. Shulman, “Racing to the precipice: a model of artificial intelligence development”, AI & Society 31(2), 201–206, 2016.
• G. Sastry et al., “Computing power and the governance of artificial intelligence”, arXiv:2402.08797, 2024.
• Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), 2024.
• European Commission, General-Purpose AI Code of Practice, 2025.
• The Bletchley Declaration by Countries Attending the AI Safety Summit, 1–2 November 2023.
• Frontier AI Safety Commitments, AI Seoul Summit, May 2024.

Control and oversight (ch. 9–10)

• R. Greenblatt, B. Shlegeris, K. Sachan, F. Roger, “AI control: improving safety despite intentional subversion”, ICML, 2024.
• G. Irving, P. Christiano, D. Amodei, “AI safety via debate”, arXiv:1805.00899, 2018.
• A. Khan et al., “Debating with more persuasive LLMs leads to more truthful answers”, ICML, 2024.
• J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, S. Legg, “Scalable agent alignment via reward modeling: a research direction”, arXiv:1811.07871, 2018.
• C. Burns et al., “Weak-to-strong generalization: eliciting strong capabilities with weak supervision”, arXiv:2312.09390, 2023; ICML, 2024.
• D. Dalrymple et al., “Towards guaranteed safe AI: a framework for ensuring robust and reliable AI systems”, arXiv:2405.06624, 2024.

Software

• N. Nanda, J. Bloom, TransformerLens, open-source library for mechanistic interpretability, 2022.
• UK AI Security Institute, Inspect, open-source framework for large language model evaluations, 2024.