Deep Learning

Contents

• Introduction and overview
• 0. Review: chain rule, likelihood and gradient descent
• 1. A history of deep learning and its relation to cognitive science
• 2. Feedforward networks
• 3. Optimisation and regularisation
• 4. Representation learning and siamese networks
• 5. Generative adversarial networks and transfer learning
• 6. Recurrent networks and LSTMs for natural language processing
• 7. Object detection
• 8. Self-supervised methods
• 9. Deep reinforcement learning
• 10. Attention and transformers
• 11. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with an M1 in mathematics, computer science or physics, who knows some Python and wants to build, train and understand deep networks for images, text and sequential decisions.

Empirical first: in 2012, a convolutional network cut the top-5 error of the ImageNet challenge from 26.2 % (the second-best entry) to 15.3 % (Krizhevsky, Sutskever and Hinton, 2012). In 2015, one network architecture, trained from pixels and scores only, played 49 Atari games at a level comparable to a professional human games tester (Mnih et al., 2015). This course explains how such systems are built and trained, and where they still fail.

The course covers the fundamental tools of deep learning, then its main families of models in three domains: computer vision, natural language processing and deep reinforcement learning. It is organised around three practical mini-projects, one per domain.

It shares tools with Object Recognition and Computer Vision, ALTEGRAD and Theoretical Foundations of Deep Learning, taught in the same semester. Its official follow-up is Deep Learning in Practice, in semester 2.

What you will be able to do

• Trace the history of neural networks from the threshold neuron to deep learning, and explain what cognitive science contributed to it.
• Derive backpropagation, and implement multilayer perceptrons and convolutional networks in NumPy and PyTorch.
• Choose an optimiser, an initialisation and regularisers, and diagnose under- and overfitting.
• Learn embeddings with autoencoders and siamese networks, and use them for verification and one-shot recognition.
• Train a generative adversarial network, prove the characterisation of its equilibrium, and recognise mode collapse.
• Transfer a pre-trained network to a new task by feature extraction or fine-tuning.
• Model text with recurrent networks and LSTMs, explain vanishing gradients, and add attention to an encoder–decoder.
• Build and evaluate an object detector: anchors, box regression, non-maximum suppression, focal loss and mean average precision.
• Pre-train representations without labels, with pretext tasks, contrastive and non-contrastive objectives.
• Formulate a control problem as a Markov decision process, and solve it with deep Q-learning or policy gradients.
• Carry out mini-projects in object recognition, natural language processing and reinforcement learning.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com starts from a fact: with more computational power and data, but also with new training algorithms and whole new approaches, deep learning produced many breakthroughs in recent years in object and spoken language recognition, text generation and robotics.

The class covers the fundamental aspects and the recent developments of deep learning in three domains: computer vision, natural language processing and deep reinforcement learning. Its learning objective is to introduce students to deep learning through practical projects, so that by the end of the course they can implement and develop new methods in these domains.

The same page announces that each section of the course is a 1 h 30 lecture followed by a 1 h 30 lab, in which students start the mini-projects of the evaluation and receive feedback. It gives an eight-item table of contents. The Université Paris-Saclay course detail gives 24 h of lectures and an assessment type, and displays no list of contents. Each item, and each part of a composite item, has a row below.

Beyond the official items, these notes add a review of the mathematical tools (ch. 0), a self-study chapter on attention and transformers (ch. 10), the modern successors of the recurrent models of ch. 6, and a self-study chapter on research directions (ch. 11).

Where this course fits in the programme

There is no official prerequisite; the course page expects basic knowledge of Python. The course is taught in the first period of semester 1 by Vincent Lepetit (Université de Bordeaux) and Maria Vakalopoulou, as listed on master-mva.com.

• Preparation — the refreshers Scientific Python, NumPy and PyTorch (the language of the labs), Linear Algebra, Analysis and Optimization (Jacobians, gradient descent) and Probability and Statistics (likelihood, entropy, Markov chains).
• Related courses in semester 1 — Object Recognition and Computer Vision, ALTEGRAD, Theoretical Foundations of Deep Learning, Probabilistic Graphical Models and Deep Generative Models, Foundations of Large Language Models, Convex Optimization, Introduction to Statistical Learning, Deep Learning and Signal Processing, 3D Computer Vision and Robotics.
• Courses that build on it in semester 2 — Deep Learning in Practice (named by the official page), Reinforcement Learning, Generative Models for Images, Deep Learning for Image Restoration and Synthesis, Representation Learning for Computer Vision and Medical Imaging, Deep Learning for Medical Imaging, Algorithms for Speech and Natural Language Processing, Point Clouds and 3D Modelling and Training and Deploying Large-Scale Models.
• Research — the mini-projects are a first contact with experimental research, a natural step towards the research internship.

Course outline

• Review: chain rule, likelihood and gradient descent — the M1 tools used everywhere else, with automatic differentiation.
• A history of deep learning and its relation to cognitive science — from the threshold neuron and the perceptron to deep networks as models of the brain.
• Feedforward networks — multilayer perceptrons, universal approximation, backpropagation, convolutional and residual layers.
• Optimisation and regularisation — stochastic gradient methods and their convergence, initialisation, weight decay, dropout and normalisation.
• Representation learning and siamese networks — autoencoders, metric learning with contrastive and triplet losses, one-shot recognition.
• Generative adversarial networks and transfer learning — the adversarial game and its equilibrium, then reusing a trained network on a new task.
• Recurrent networks and LSTMs for natural language processing — language models, vanishing gradients, gated cells and attention.
• Object detection — evaluation, two-stage and one-stage detectors, focal loss and set prediction.
• Self-supervised methods — pretext tasks, contrastive learning and mutual information, non-contrastive methods.
• Deep reinforcement learning — Markov decision processes, deep Q-learning and policy gradients.
• Attention and transformers — self-attention and the transformer block (self-study).
• Research directions and open problems — scale, generative models, robustness, sample-efficient control and the brain (self-study).

Session plan

24 h in class = 8 sessions of 3 h.

The volume is that of the Université Paris-Saclay programme table (24 h of lectures). On master-mva.com each section of the course is a 1 h 30 lecture followed by a 1 h 30 lab, and the table of contents has eight items, so the plan gives one 3 h session to each item, in the official order. This split is a proposal.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods and their proofs, 3 = advanced material close to current research; the level never decreases from one session to the next.

Self-study, outside the 24 h (about 4 h in total): ch. 10 Attention and transformers (≈ 2 h, with Ex. 17) · ch. 11 Research directions and open problems (≈ 2 h).

How to study this module

• Read the chapters in order: each one uses only the previous ones. A reader fluent in multivariate calculus and maximum likelihood can skim ch. 0.
• After each session, do the exercises listed in its row of the session plan, then run the Python block of the chapter and change one parameter.
• The Python environment is the one of the scientific Python refresher: NumPy, scikit-learn and PyTorch on CPU. Every code block runs in less than 20 seconds on a laptop CPU, with synthetic data or the digits data set bundled with scikit-learn.
• Start each mini-project in the lab where it is announced (see Projects and assessment): the labs exist to get early feedback.
• Keep a page of formulas: backpropagation, the SGD bound, the triplet loss, the GAN equilibrium, the LSTM cell, IoU and focal loss, InfoNCE, the Bellman equation and the policy gradient.

0. Review: chain rule, likelihood and gradient descent

Deep learning fits a parametrised function by gradient descent on a negative log-likelihood. This chapter recalls the three M1 tools behind that sentence, and the automatic differentiation that makes them cheap.

Evaluated from the right, this product only needs vector–Jacobian products, never a matrix–matrix product. This is reverse-mode automatic differentiation: its cost is a small constant times the cost of evaluating the function (Griewank and Walther, 2008). Backpropagation (ch. 2) is its special case for neural networks.

Minimising the average cross-entropy over the data therefore drives the model towards the data distribution. This is why classifiers are trained with the cross-entropy loss (ch. 2).

The code checks this gradient against PyTorch’s reverse mode and against finite differences, then runs gradient descent with step 1/L.

import numpy as np
import torch
torch.set_num_threads(1)
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 3))
y = (X @ np.array([1.0, -2.0, 0.5]) + 0.5 * rng.normal(size=200) > 0).astype(float)
w = rng.normal(size=3)

def loss(w):                                    # mean cross-entropy of logistic regression
    z = X @ w
    return np.mean(np.logaddexp(0, z) - y * z)

g_hand = X.T @ (1 / (1 + np.exp(-X @ w)) - y) / len(y)        # chain rule: (sigma(z) - y) x
wt = torch.tensor(w, requires_grad=True)                       # reverse-mode autodiff
torch.nn.functional.binary_cross_entropy_with_logits(torch.tensor(X) @ wt, torch.tensor(y)).backward()
eps = 1e-6
g_fd = np.array([(loss(w + eps * e) - loss(w - eps * e)) / (2 * eps) for e in np.eye(3)])
print("autograd vs hand:", np.abs(g_hand - wt.grad.numpy()).max())
print("finite diff vs hand:", np.abs(g_hand - g_fd).max())

L = np.linalg.eigvalsh(X.T @ X / len(y)).max() / 4             # smoothness constant
for t in range(301):                                           # gradient descent, step 1/L
    if t % 100 == 0:
        print(f"step {t:3d}  loss {loss(w):.4f}")
    w -= (1 / L) * X.T @ (1 / (1 + np.exp(-X @ w)) - y) / len(y)

Reverse mode agrees with the formula to machine precision, finite differences to 10^{-10}. The loss falls from 1.125 to 0.187 in 300 steps, decreasing at every step as Proposition 0.5 guarantees.

1. A history of deep learning and its relation to cognitive science

Most ideas of deep learning are older than its name: they were proposed between 1943 and 1998, many of them by psychologists and neuroscientists. This chapter follows that history through the results that shaped it, and the exchanges with cognitive science that still continue.

The threshold neuron and the perceptron

With binary inputs, one unit computes AND (w = (1, 1), b = -1.5), OR (b = -0.5) or NOT (w = -1, b = 0.5), so networks of such units compute every Boolean function. Hebb (1949) added a learning principle: a connection strengthens when the two neurons it links are active together.

Minsky and Papert showed what one layer cannot do. With no algorithm to train hidden layers, interest in neural networks declined for more than a decade.

The code runs the perceptron on separable data, compares its mistakes with Theorem 1.3, then fails on XOR.

import numpy as np
rng = np.random.default_rng(1)
w_star = np.array([0.6, 0.8])                      # unit normal of a separating line
X = rng.uniform(-1, 1, size=(500, 2))
X = X[np.abs(X @ w_star) > 0.1]                    # keep a margin of at least 0.1
y = np.sign(X @ w_star)
R, gamma = np.linalg.norm(X, axis=1).max(), np.abs(X @ w_star).min()

def perceptron(X, y, epochs=100):
    w, mistakes = np.zeros(X.shape[1]), 0
    for _ in range(epochs):
        errors = 0
        for x, t in zip(X, y):
            if t * (w @ x) <= 0:                   # mistake: move w towards t x
                w, errors = w + t * x, errors + 1
        mistakes += errors
        if errors == 0:
            return w, mistakes, True
    return w, mistakes, False

w, k, ok = perceptron(X, y)
print(f"{len(X)} points, mistakes {k}, bound (R/gamma)^2 = {(R / gamma) ** 2:.0f}, converged {ok}")
X_xor = np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]], float)   # last column = bias
y_xor = np.array([-1, 1, 1, -1])
print("XOR converged:", perceptron(X_xor, y_xor)[2])
relu = lambda u: np.maximum(u, 0)
print("two-layer XOR:", relu(X_xor[:, 0] + X_xor[:, 1]) - 2 * relu(X_xor[:, 0] + X_xor[:, 1] - 1))

The perceptron converges after 7 mistakes on 439 points, far below the worst-case bound of 192. On XOR it never converges, while the two-layer formula is exact.

Backpropagation, connectionism and the road to depth

Reverse-mode differentiation was published by Linnainmaa in 1970 and proposed for networks by Werbos in 1974. The 1986 paper of Rumelhart, Hinton and Williams made it known, by showing that hidden units learn useful internal features.

The connectionist programme of the Parallel Distributed Processing volumes (Rumelhart, McClelland and the PDP Research Group, 1986) held that cognition emerges from many simple units with learned connections, against symbolic AI. Fodor and Pylyshyn (1988) objected that such networks lack the systematic compositionality of thought. The debate continues about what large networks generalise (Lake et al., 2017).

In the 1990s and 2000s, kernel methods often outperformed neural networks. Depth came back with unsupervised pre-training (2006), then with large labelled data sets such as ImageNet (Deng et al., 2009), GPUs and the ReLU.

What deep learning took from, and gave back to, the study of the brain

• Vision. Hubel and Wiesel (1962) found simple cells in the cat’s visual cortex, tuned to an oriented edge at one position, and complex cells, tolerant to the exact position. The neocognitron alternates the two operations, and the convolution–pooling stacks of ch. 2 inherit this design.
• Models of the cortex. The upper layers of convolutional networks trained only for object recognition predict the responses of neurons in the inferior temporal cortex of monkeys better than earlier models (Yamins et al., 2014). Networks became quantitative hypotheses about the brain (Yamins and DiCarlo, 2016).
• Learning signals. Dopamine neurons fire like the temporal-difference error of reinforcement learning (Schultz, Dayan and Montague, 1997), the quantity used again in ch. 9.
• Plausibility. Backpropagation sends errors back through the transposed forward weights, which neurons do not seem to have. Fixed random feedback weights still allow learning (Lillicrap et al., 2016); how the brain assigns credit remains open (ch. 11).

2. Feedforward networks

A feedforward network composes simple layers into one function, trained end to end. This chapter defines the model, shows why it can approximate any continuous function, and computes its gradient. Convolutional and residual layers then adapt it to images.

Without activations, the network would be a composition of affine maps, hence affine: depth would add nothing.

The theorem guarantees existence, not efficiency: the width may grow exponentially with the dimension. Depth helps, since some functions computed by deep narrow networks need exponentially wide shallow ones (Telgarsky, 2016).

The code implements Theorem 2.5 for one hidden layer of 32 ReLUs, checks it against finite differences, and trains the network on the two-moons data set.

import numpy as np
from sklearn.datasets import make_moons
rng = np.random.default_rng(0)
X, y = make_moons(500, noise=0.15, random_state=0)
params = [rng.normal(0, 1.0, (2, 32)), np.zeros(32),            # He init: variance 2 / fan-in
          rng.normal(0, np.sqrt(2 / 32), (32, 2)), np.zeros(2)]

def loss_and_grads(params, X, y):
    W1, b1, W2, b2 = params
    z1 = X @ W1 + b1
    a1 = np.maximum(z1, 0)                                      # hidden ReLU layer
    z2 = a1 @ W2 + b2                                           # logits
    p = np.exp(z2 - z2.max(1, keepdims=True))
    p /= p.sum(1, keepdims=True)                                # softmax
    n = len(y)
    loss = -np.log(p[np.arange(n), y]).mean()
    d2 = p.copy()
    d2[np.arange(n), y] -= 1
    d2 /= n                                                     # dL/dz2 = (p - Y) / n
    d1 = (d2 @ W2.T) * (z1 > 0)                                 # back through the ReLU
    return loss, [X.T @ d1, d1.sum(0), a1.T @ d2, d2.sum(0)]

loss, grads = loss_and_grads(params, X, y)
eps = 1e-6
for P, G in zip(params, grads):                                 # check one entry per array
    i = tuple(rng.integers(s) for s in P.shape)
    old = P[i]
    P[i] = old + eps; lp = loss_and_grads(params, X, y)[0]
    P[i] = old - eps; lm = loss_and_grads(params, X, y)[0]
    P[i] = old
    print(f"backprop {G[i]:+.7f}   finite difference {(lp - lm) / (2 * eps):+.7f}")
for step in range(3000):                                        # full-batch gradient descent
    loss, grads = loss_and_grads(params, X, y)
    for P, G in zip(params, grads):
        P -= 0.5 * G
z = np.maximum(X @ params[0] + params[1], 0) @ params[2] + params[3]
print(f"final loss {loss:.3f}, training accuracy {(z.argmax(1) == y).mean():.3f}")

Backpropagation and finite differences agree to seven digits. After 3000 steps the network separates the two moons, with 99.8 % training accuracy.

Convolutional and residual layers

An image of 224 \times 224 pixels with 3 colours has 150 528 inputs: a dense first layer of 1000 units would need 150 million weights. Convolutional layers share their weights across positions.

Stacking layers enlarges the receptive field, the set of input pixels that influence one output: with stride 1, L layers of 3 \times 3 kernels see a (2L + 1) \times (2L + 1) window.

A convolutional network therefore detects a pattern wherever it occurs (LeCun et al., 1989). With strides and zero padding, equivariance holds only approximately.

The identity term carries the gradient back to early layers even when the Jacobians of the blocks are small. It made networks of more than a hundred layers trainable, and is part of nearly every modern architecture.

3. Optimisation and regularisation

Training minimises a non-convex loss over millions of parameters, and a low training loss does not guarantee a low test error. This chapter gives the optimisers and their guarantee, then the regularisers that close the gap between training and test.

The first term is the rate of gradient descent; the second is a noise floor proportional to the step. Decaying the learning rate lowers the floor. So does enlarging the batch: the variance of a mini-batch of m independent draws is that of one example divided by m. On a non-convex loss, the theorem only guarantees an approximate stationary point.

With v = 1/n instead, the variance would halve at each layer, and shrink by 2^{30} \approx 10^9 after 30 layers. For \tanh, Glorot and Bengio (2010) proposed v = 2 / (n_{\text{in}} + n_{\text{out}}), which balances the forward and the backward pass.

Regularisation

A network with more parameters than examples can fit noise. Regularisers restrict, explicitly or implicitly, the functions that training reaches.

With Adam, adding \alpha \theta to the gradient is not weight decay, because the update rescales each coordinate. AdamW subtracts \eta \alpha \theta directly from the weights (Loshchilov and Hutter, 2019).

Dropout multiplies each unit, during training, by an independent mask m_j / (1 - p) with m_j \sim \mathrm{Bernoulli}(1 - p), so that its expectation is unchanged; at test time the mask is removed (Srivastava et al., 2014).

Two more regularisers are implicit. Early stopping ends training when the validation error starts to rise; on a quadratic loss it shrinks flat directions like weight decay (Exercise 9). Data augmentation trains on transformed copies of the examples — crops, flips, colour changes for images — and so encodes the invariances the model should have.

The code fits 30 noisy samples of \sin 3x with a two-hidden-layer network of width 128, with and without regularisation, and measures the test error against the true function.

import torch
from torch import nn
torch.set_num_threads(1)
torch.manual_seed(0)
f = lambda x: torch.sin(3 * x)
x_tr = torch.rand(30, 1) * 2 - 1
y_tr = f(x_tr) + 0.3 * torch.randn(30, 1)            # 30 noisy observations
x_te = torch.linspace(-1, 1, 400).view(-1, 1)

def run(p_drop=0.0, wd=0.0, steps=2000):
    torch.manual_seed(1)
    net = nn.Sequential(nn.Linear(1, 128), nn.ReLU(), nn.Dropout(p_drop),
                        nn.Linear(128, 128), nn.ReLU(), nn.Dropout(p_drop), nn.Linear(128, 1))
    opt = torch.optim.AdamW(net.parameters(), lr=5e-3, weight_decay=wd)
    best = (float("inf"), 0)
    for t in range(steps):
        net.train()
        loss = ((net(x_tr) - y_tr) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
        if t % 50 == 0:
            net.eval()
            with torch.no_grad():
                err = ((net(x_te) - f(x_te)) ** 2).mean().item()
            best = min(best, (err, t))
    return loss.item(), err, best

for name, kw in [("none", {}), ("weight decay 1.0", {"wd": 1.0}), ("dropout 0.5", {"p_drop": 0.5})]:
    tr, te, (best, t) = run(**kw)
    print(f"{name:17s} train MSE {tr:.3f}  test MSE {te:.3f}  (best {best:.3f} at step {t})")

Without regularisation, the training error (0.019) falls below the noise variance (0.09): the network fits the noise, and its test error is 0.076. Weight decay brings it down to 0.011, dropout to 0.021. The unregularised run was best after only 50 steps (0.008): early stopping would have caught that point.

4. Representation learning and siamese networks

The hidden layers of a trained network compute a representation of its input, and much of the success of deep learning comes from reusing such representations. This chapter learns representations on purpose: by reconstructing the input, and by comparing inputs.

Autoencoders

The encoder is only determined up to an invertible k \times k map: the autoencoder finds the principal subspace, not the principal axes. Non-linear autoencoders extend PCA to curved manifolds, and variational autoencoders make them generative (Probabilistic Graphical Models).

Siamese networks and metric learning

Autoencoders keep all the information about the input, nuisance factors included. When we know which inputs should be close — two signatures of the same person, two photographs of the same face — the representation can be learned from comparisons.

Dissimilar pairs already farther apart than the margin give no gradient. Most random negatives are such easy cases, so FaceNet trains on semi-hard negatives: farther than the positive, but within the margin.

This is what makes siamese embeddings suited to one-shot recognition: a class is recognised from a single example, by nearest neighbour in the embedding (Koch, Zemel and Salakhutdinov, 2015). Prototypical networks replace the single example by the mean embedding of a few (Snell, Swersky and Zemel, 2017).

The code trains a triplet network on 900 digit images, then tests 10-way one-shot recognition on the 897 others: one reference image per class, then nearest neighbour.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
torch.set_num_threads(1)
torch.manual_seed(0)
rng = np.random.default_rng(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32)
idx = rng.permutation(len(y))
train, test = idx[:900], idx[900:]                          # disjoint images, same ten classes
by_class = {c: train[y[train] == c] for c in range(10)}

net = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 32))
embed = lambda x: nn.functional.normalize(net(x), dim=1)    # points on the unit sphere
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
for step in range(1000):                                    # triplets (anchor, positive, negative)
    c = rng.integers(10, size=128)
    c_neg = (c + rng.integers(1, 10, size=128)) % 10
    a = [rng.choice(by_class[k]) for k in c]
    p = [rng.choice(by_class[k]) for k in c]
    n = [rng.choice(by_class[k]) for k in c_neg]
    ea, ep, en = embed(X[a]), embed(X[p]), embed(X[n])
    d_ap, d_an = ((ea - ep) ** 2).sum(1), ((ea - en) ** 2).sum(1)
    loss = torch.relu(d_ap - d_an + 0.5).mean()             # triplet loss, margin 0.5
    opt.zero_grad(); loss.backward(); opt.step()

def one_shot(F, trials=1000):                               # 10-way, one test image per class
    r, hits = np.random.default_rng(1), 0
    for _ in range(trials):
        support = [r.choice(test[y[test] == k]) for k in range(10)]
        q = r.choice(np.setdiff1d(test, support))
        hits += int(np.argmin(((F[support] - F[q]) ** 2).sum(1))) == y[q]
    return hits / trials

with torch.no_grad():
    Z = embed(X).numpy()
print(f"one-shot 10-way accuracy on test images: pixels {one_shot(X.numpy()):.3f}, "
      f"triplet embedding {one_shot(Z):.3f}")

Nearest neighbour on raw pixels is right 64.8 % of the time, and 95.8 % in the learned 32-dimensional embedding.

Transfer to new classes is harder. Trained on the digits 0–4 only, the same network does worse than pixels on the digits 5–9: five classes are too few to learn a general notion of similarity (Exercise 18). Omniglot, the standard one-shot benchmark, provides 964 characters for training (Lake, Salakhutdinov and Tenenbaum, 2015).

5. Generative adversarial networks and transfer learning

This chapter has two parts, like the official item: learning to generate data through a game between two networks, then reusing a trained network on a new task. The two meet at the end, in domain-adversarial training.

Generative adversarial networks

A generator G maps noise z \sim p_z, for instance Gaussian, to a sample G(z). We want the distribution p_g of G(z) to match the data distribution, without ever writing down a density.

At the equilibrium, D^\star = 1/2 everywhere, and the discriminator loss -V equals \log 4 \approx 1.386.

The code trains a GAN on a mixture of eight Gaussians placed on a circle, a standard test of mode coverage, with the non-saturating loss.

import math
import torch
from torch import nn
torch.set_num_threads(1)
torch.manual_seed(0)
centres = torch.tensor([[math.cos(2 * math.pi * k / 8), math.sin(2 * math.pi * k / 8)] for k in range(8)]) * 2

def real_batch(n):                                          # mixture of 8 Gaussians on a circle
    return centres[torch.randint(8, (n,))] + 0.05 * torch.randn(n, 2)

G = nn.Sequential(nn.Linear(8, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 2))
D = nn.Sequential(nn.Linear(2, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 1))
opt_G = torch.optim.Adam(G.parameters(), lr=1e-3, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=1e-3, betas=(0.5, 0.999))
bce = nn.functional.binary_cross_entropy_with_logits
ones, zeros = torch.ones(256, 1), torch.zeros(256, 1)

def coverage(n=2000):                                       # modes hit, fraction of good samples
    with torch.no_grad():
        x = G(torch.randn(n, 8))
    d, k = torch.cdist(x, centres).min(1)
    good = d < 0.2
    return len(k[good].unique()), good.float().mean().item()

for step in range(3001):
    x_real, x_fake = real_batch(256), G(torch.randn(256, 8))
    loss_D = bce(D(x_real), ones) + bce(D(x_fake.detach()), zeros)   # discriminator ascent
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()
    loss_G = bce(D(x_fake), ones)                                     # non-saturating generator loss
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()
    if step % 1000 == 0:
        modes, good = coverage()
        print(f"step {step:4d}  D loss {loss_D.item():.3f}  modes covered {modes}/8  good samples {good:.2f}")

All eight modes are covered after 1000 steps, and 91 % of the samples lie within 0.2 of a mode centre after 3000. The discriminator loss stays near 1.1, below the equilibrium value 1.386: the game has not reached its equilibrium, as is usual in practice.

Transfer learning

Yosinski et al. (2014) measured how transferable each layer is: the first layers learn general, Gabor-like filters, the last ones task-specific features. Initialising with transferred features and fine-tuning improved generalisation even over training on the target task alone. Off-the-shelf CNN features beat hand-designed pipelines on many recognition tasks (Razavian et al., 2014; Oquab et al., 2014), and better ImageNet models tend to transfer better (Kornblith, Shlens and Le, 2019).

Ben-David et al. (2010) refine this bound with a divergence between the input distributions that only involves the class of predictors, plus the error of the best predictor on both domains. The bound suggests learning features whose distribution is the same on both domains.

Domain-adversarial training does so with a GAN-like game: a domain classifier tries to tell source features from target features, and the encoder learns to fool it through a gradient-reversal layer (Ganin et al., 2016).

6. Recurrent networks and LSTMs for natural language processing

Text is a sequence of variable length, in which the meaning of a word can depend on words far before it. Recurrent networks read such sequences with a state updated at each step. This chapter explains why they are hard to train, and how gated cells fix it.

A uniform model over V has perplexity |V|: perplexity is the size of an equally uncertain uniform choice. A model that gives probability 0.1 to every observed token has perplexity 10.

Conversely, when \|W\| > 1 the product can grow exponentially: the gradients explode. Gradient clipping rescales the gradient when its norm exceeds a threshold (Pascanu, Mikolov and Bengio, 2013). Vanishing gradients have no such simple fix: the network cannot learn dependencies over long lags.

The other paths, through h_{k-1} and the gates, behave like an ordinary recurrent network; the direct path is what lets the errors flow. Initialising the forget-gate bias to 1 or more is a standard trick (Jozefowicz, Zaremba and Sutskever, 2015). The GRU (Cho et al., 2014) merges the cell and the hidden state, with two gates, and behaves similarly.

The code measures, at initialisation, the gradient that reaches the first input of a sequence from the final state, for a vanilla RNN and for LSTMs with forget-gate bias 0 and 3.

import torch
from torch import nn
torch.set_num_threads(1)

def grad_to_first_input(cell, T, forget_bias=None, trials=20):
    """Average norm of d(sum of final hidden state) / d(first input), at initialisation."""
    torch.manual_seed(0)
    net = cell(4, 64, batch_first=True)
    if forget_bias is not None:                    # PyTorch orders the LSTM gates i, f, g, o
        with torch.no_grad():
            net.bias_ih_l0[64:128].fill_(forget_bias)
            net.bias_hh_l0[64:128].fill_(0.0)
    x = nn.functional.one_hot(torch.randint(4, (trials, T)), 4).float().requires_grad_()
    h, _ = net(x)
    h[:, -1].sum().backward()
    return x.grad[:, 0].norm(dim=1).mean().item()

print("   T     RNN    LSTM b_f=0  LSTM b_f=3")
for T in (5, 20, 50, 100):
    print(f"{T:4d}  {grad_to_first_input(nn.RNN, T):.1e}   {grad_to_first_input(nn.LSTM, T, 0.0):.1e}"
          f"    {grad_to_first_input(nn.LSTM, T, 3.0):.1e}")

The RNN gradient falls by twelve orders of magnitude between lags 5 and 50, and underflows to zero in single precision at lag 100. With bias 0, the forget gates sit near 0.5 and the LSTM gradient decays as fast. With bias 3, they sit near 0.95, and the gradient stays of order 0.1 up to lag 100.

Encoder–decoders and attention

Translation maps a sentence to a sentence of another length. A sequence-to-sequence model encodes the source into states h_1, \dots, h_S, and a decoder RNN generates the target one token at a time (Sutskever, Vinyals and Le, 2014). Training feeds the true previous token to the decoder (teacher forcing); generation uses greedy or beam search.

Attention removes the bottleneck of a single vector for the whole source sentence, and its weights align target words with source words. Removing the recurrence and keeping only attention gives the transformer (ch. 10).

Other language tasks reuse the same parts. Sentence classification, such as sentiment analysis, reads the last state or a pooled state of a bidirectional RNN; sequence labelling, such as named entities, reads every state. Word embeddings pre-trained by word2vec (Mikolov et al., 2013) or GloVe (Pennington, Socher and Manning, 2014) initialise the embedding layer, and ELMo uses the states of a bidirectional LSTM language model as contextual features (Peters et al., 2018).

7. Object detection

A classifier says what is in an image; a detector says what is where. It outputs a variable number of boxes, each with a class and a score. This raises three questions: how to evaluate a set of boxes, how to propose candidates, and how to remove duplicates.

IoU is unchanged when both boxes are translated or scaled together, and 1 - \mathrm{IoU} is a distance on sets, the Jaccard distance.

Detectors produce many overlapping boxes around each object, and NMS keeps one of them. Its threshold trades duplicates against missed objects in crowds.

Two-stage detectors

R-CNN (Girshick et al., 2014) classifies with a CNN about 2000 region proposals produced by a bottom-up segmentation algorithm. Fast R-CNN (Girshick, 2015) computes the feature map once, and pools a fixed-size feature for each proposal. Faster R-CNN (Ren et al., 2015) replaces the external proposals by a region proposal network, which scores and refines anchor boxes of several scales and aspect ratios at every position of the feature map.

The same convolutional head can therefore serve every position and, with anchors of several sizes, every scale. Feature pyramid networks (Lin et al., 2017) attach the head to several resolutions of the feature map, small objects to the fine ones.

One-stage detectors and the focal loss

YOLO (Redmon et al., 2016) and SSD (Liu et al., 2016) predict classes and boxes densely in one pass, without proposals: for every cell of a coarse grid in YOLO, for every anchor in SSD. They are fast, but anchor-based one-stage detectors face an extreme imbalance: tens of thousands of anchors per image, almost all of them easy background.

RetinaNet, a one-stage detector with a feature pyramid and the focal loss, matched the accuracy of two-stage detectors (Lin et al., 2017).

Detection as set prediction

DETR (Carion et al., 2020) predicts a fixed-size set of boxes with a transformer, and matches them one-to-one with the ground-truth objects, padded with “no object”, by the Hungarian algorithm (Kuhn, 1955). The matching minimises a cost of class and box errors, and the loss is computed on the matched pairs. A duplicate is matched to “no object” and penalised, so neither anchors nor NMS are needed.

The code implements IoU, NMS and AP on three objects, with six noisy detections each and two false positives.

import numpy as np

def iou(a, B):                                   # box a against boxes B, format (x1, y1, x2, y2)
    w = np.clip(np.minimum(a[2], B[:, 2]) - np.maximum(a[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(a[3], B[:, 3]) - np.maximum(a[1], B[:, 1]), 0, None)
    inter = w * h
    area = lambda b: (b[..., 2] - b[..., 0]) * (b[..., 3] - b[..., 1])
    return inter / (area(a) + area(B) - inter)

def nms(boxes, scores, thr=0.5):
    order, keep = np.argsort(-scores), []
    while len(order):
        i, order = order[0], order[1:]
        keep.append(i)
        order = order[iou(boxes[i], boxes[order]) < thr]   # drop boxes overlapping the winner
    return np.array(keep)

def average_precision(boxes, scores, gt, thr=0.5):
    order, matched, tp = np.argsort(-scores), set(), []
    for i in order:                                         # greedy matching by decreasing score
        o = iou(boxes[i], gt)
        j = int(np.argmax(o))
        hit = o[j] >= thr and j not in matched
        matched |= {j} if hit else set()
        tp.append(hit)
    tp = np.array(tp, float)
    prec = np.cumsum(tp) / np.arange(1, len(tp) + 1)
    rec = np.cumsum(tp) / len(gt)
    prec = np.maximum.accumulate(prec[::-1])[::-1]          # monotone precision envelope
    return np.sum(np.diff(np.concatenate([[0], rec])) * prec)

rng = np.random.default_rng(0)
gt = np.array([[10, 10, 50, 50], [60, 20, 100, 70], [20, 60, 60, 110]], float)
boxes = np.repeat(gt, 6, axis=0) + rng.normal(0, 4, (18, 4))   # 6 noisy detections per object
scores = rng.uniform(0.3, 1.0, 18)
boxes = np.vstack([boxes, [[70, 80, 110, 120], [0, 0, 30, 30]]])  # two false positives
scores = np.append(scores, [0.99, 0.6])
keep = nms(boxes, scores)
print(f"IoU of the first detection with its object: {iou(boxes[0], gt[:1])[0]:.3f}")
print(f"{len(boxes)} detections -> {len(keep)} after NMS")
print(f"AP@0.5 before NMS {average_precision(boxes, scores, gt):.3f}, "
      f"after NMS {average_precision(boxes[keep], scores[keep], gt):.3f}")

Before NMS, the duplicates count as false positives, and AP at IoU 0.5 is 0.733. After NMS, five boxes remain, the three objects and the two false positives, and AP rises to 0.833. The false positive ranked second still costs a sixth: \mathrm{AP} = \tfrac13 (1 + 0.75 + 0.75) = 0.833 .

8. Self-supervised methods

Labels are expensive; raw images and text are not. Self-supervised learning builds a supervised task from unlabelled data, and keeps the encoder it trains as a representation (ch. 4) for downstream tasks.

To tell which rotation was applied, a network must know what an upright object looks like, which requires recognising it. Contrastive learning later overtook hand-designed pretext tasks: it learns invariance to a family of augmentations directly.

The bound never exceeds \log N: certifying a large mutual information takes many negatives (McAllester and Stratos, 2020). In practice, InfoNCE is valued less as an estimator than for the geometry it induces.

SimCLR (Chen et al., 2020) uses InfoNCE with the other views of a batch as negatives, strong augmentations (crops and colour distortion), and a projection head discarded after training. MoCo (He et al., 2020) keeps a queue of negatives encoded by a slowly updated momentum encoder, so that their number no longer depends on the batch size.

Non-contrastive methods and collapse

Every non-contrastive method therefore needs a mechanism against collapse: the asymmetry of predictor and stop-gradient, or explicit terms that keep the variance of each feature up and decorrelate the features. Why the asymmetric methods avoid collapse is only partly understood (Tian, Chen and Ganguli, 2021).

DINO (Caron et al., 2021) distils a momentum teacher into a student across views. Masked autoencoders hide 75 % of the patches of an image and reconstruct them (He et al., 2022). Both scale to large vision transformers.

The code pre-trains an encoder on the digits with InfoNCE, using random shifts, pixel masking and noise as augmentations, without any label. A linear probe is then trained with five labels per class, and tested on the other images, then on the same images shifted one pixel to the right.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
torch.set_num_threads(1)
torch.manual_seed(0)
np.random.seed(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32)

def shift(x, i, j):                               # translate 8x8 images, zero padding
    img = nn.functional.pad(x.view(-1, 1, 8, 8), (1, 1, 1, 1))
    return img[:, :, i:i + 8, j:j + 8].reshape(-1, 64)

def augment(x):                                   # random shift + pixel masking + noise
    img = shift(x, *np.random.randint(0, 3, size=2))
    return img * (torch.rand_like(img) > 0.2) + 0.1 * torch.randn_like(img)

encoder = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 64))
head = nn.Sequential(nn.ReLU(), nn.Linear(64, 64))  # projection head, discarded after training
opt = torch.optim.Adam(list(encoder.parameters()) + list(head.parameters()), lr=1e-3)
tau, n = 0.5, 256
for step in range(1500):                          # SimCLR / InfoNCE: no label is used
    xb = X[torch.randint(len(X), (n,))]
    z = nn.functional.normalize(head(encoder(torch.cat([augment(xb), augment(xb)]))), dim=1)
    s = z @ z.T / tau
    s.fill_diagonal_(-1e9)                        # a view is not its own candidate
    target = torch.cat([torch.arange(n, 2 * n), torch.arange(n)])   # the other view is positive
    loss = nn.functional.cross_entropy(s, target)
    opt.zero_grad(); loss.backward(); opt.step()

X_shift = shift(X, 1, 0)                          # every test image moved one pixel to the right
features = {"pixels": lambda x: x.numpy(), "contrastive encoder": lambda x: encoder(x).detach().numpy()}
for name, F in features.items():
    acc, acc_shift = [], []
    for r in range(20):                           # linear probe trained on 5 labels per class
        g = np.random.default_rng(r)
        lab = np.concatenate([g.choice(np.where(y == c)[0], 5, replace=False) for c in range(10)])
        rest = np.setdiff1d(np.arange(len(y)), lab)
        clf = LogisticRegression(max_iter=3000).fit(F(X[lab]), y[lab])
        acc.append(clf.score(F(X[rest]), y[rest]))
        acc_shift.append(clf.score(F(X_shift[rest]), y[rest]))
    print(f"{name:20s} test accuracy {np.mean(acc):.3f}   on shifted test images {np.mean(acc_shift):.3f}")

On unshifted images, both representations reach similar accuracies: 86.1 % for pixels, 87.2 % for the encoder. On images shifted one pixel to the right, the pixel classifier drops to 31.2 %, while the encoder keeps 81.0 %: contrastive learning has built in the invariance of its augmentations.

9. Deep reinforcement learning

In reinforcement learning an agent learns by acting: it observes a state, chooses an action, receives a reward, and seeks to maximise the rewards collected over time. Deep networks represent its value functions and policies when the states are images or other high-dimensional signals.

Deep Q-networks

Q-learning (Watkins and Dayan, 1992) applies the Bellman operator with samples: after a transition (s, a, r, s'), it moves Q(s, a) towards r + \gamma \max_{a'} Q(s', a'). With a table, it converges to Q^\star if every pair is visited infinitely often, with suitably decreasing steps. For images, a network replaces the table.

The replay buffer breaks the correlation between successive samples and reuses each transition many times. The target network keeps the regression target fixed between refreshes. Both were needed for stable learning on Atari games from raw pixels.

Double DQN reduces this bias by choosing the maximising action with the online network and evaluating it with the target network (van Hasselt, Guez and Silver, 2016).

Policy gradients

Value-based methods need a maximum over actions. For continuous actions or stochastic policies, one optimises a parametrised policy \pi_\theta directly.

The best baselines are close to V^\pi(s_t), which turns G_t - b into an estimate of the advantage Q^\pi - V^\pi. Actor–critic methods learn this baseline with a second network, the critic (Mnih et al., 2016). PPO limits each policy update by clipping the probability ratio between new and old policies (Schulman et al., 2017); it is widely used for control and to fine-tune language models from human feedback (Ouyang et al., 2022).

AlphaGo combined policy and value networks, trained on expert games and by self-play, with Monte Carlo tree search (Silver et al., 2016); AlphaZero learned chess, shogi and Go from self-play alone (Silver et al., 2018). The temporal-difference error that critics learn from, r + \gamma V(s') - V(s), is the signal attributed to dopamine neurons (ch. 1).

The code trains a logistic policy, linear in the four state variables of a cart-pole, with REINFORCE and a constant baseline. In practice, and here, the factor \gamma^t of Theorem 9.6 is dropped, a slight bias that weights all steps equally. Episodes end when the pole leans by more than 12° or the cart leaves the track, and are capped at 200 steps.

import numpy as np
rng = np.random.default_rng(0)

def step(s, a):                                   # cart-pole dynamics (Barto, Sutton and Anderson, 1983)
    x, v, th, om = s
    f = 10.0 if a == 1 else -10.0
    tmp = (f + 0.05 * om ** 2 * np.sin(th)) / 1.1
    acc_th = (9.8 * np.sin(th) - np.cos(th) * tmp) / (0.5 * (4 / 3 - 0.1 * np.cos(th) ** 2 / 1.1))
    acc_x = tmp - 0.05 * acc_th * np.cos(th) / 1.1
    s = s + 0.02 * np.array([v, acc_x, om, acc_th])
    return s, abs(s[0]) > 2.4 or abs(s[2]) > 0.2095  # failure: cart out or pole beyond 12 degrees

def episode(theta, T=200):                        # linear-logistic policy pi(a=1|s) = sigmoid(theta . [s, 1])
    s, feats, actions = rng.uniform(-0.05, 0.05, 4), [], []
    for t in range(T):
        phi = np.append(s, 1.0)
        a = int(rng.random() < 1 / (1 + np.exp(-theta @ phi)))
        feats.append(phi); actions.append(a)
        s, done = step(s, a)
        if done:
            break
    return np.array(feats), np.array(actions)

theta, gamma, lr = np.zeros(5), 0.99, 0.01
for it in range(1, 101):                           # 100 iterations of 10 episodes
    batch = [episode(theta) for _ in range(10)]
    returns = []
    for feats, _ in batch:                        # reward 1 per step: discounted reward-to-go
        G = (1 - gamma ** np.arange(len(feats), 0, -1)) / (1 - gamma)
        returns.append(G)
    b = np.mean(np.concatenate(returns))          # constant baseline (Proposition 9.7)
    grad = np.zeros(5)
    for (feats, actions), G in zip(batch, returns):
        p = 1 / (1 + np.exp(-feats @ theta))
        grad += ((actions - p) * (G - b)) @ feats  # sum_t grad log pi(a_t|s_t) (G_t - b)
    theta += lr * grad / len(batch)
    if it % 20 == 0:
        print(f"iteration {it:2d}: mean episode length {np.mean([len(f) for f, _ in batch]):6.1f} / 200")

The mean episode length grows from about 20 steps under the initial random policy to about 190 out of 200 after 80 iterations, that is 800 episodes. Deep reinforcement learning replaces the linear policy by a network and the state by pixels, and pays for it with far more episodes.

10. Attention and transformers

Self-study chapter. The attention of ch. 6 lets a recurrent decoder read the encoder states. The transformer (Vaswani et al., 2017) removes recurrence and builds the whole network from attention. It processes all positions of a sequence in parallel, and now dominates language processing and much of vision.

Self-attention thus sees a set, not a sequence. Order is injected by adding positional encodings to the token embeddings: sinusoids of several frequencies in the original transformer, learned vectors or relative encodings in later models.

The original transformer normalised after each residual addition; normalising before it, as above, trains more stably in deep stacks. Attention compares all pairs of tokens: per layer it costs O(n^2 d) operations and O(n^2) memory, against O(n d^2) for the perceptron. Long contexts motivate the efficient variants of ch. 11.

Three families followed. BERT pre-trains an encoder by masked language modelling (Devlin et al., 2019; ch. 8). GPT models decode from left to right with a causal mask (Radford et al., 2018). The vision transformer applies the same block to 16 \times 16 patches of an image (Dosovitskiy et al., 2021). These models are the subject of Foundations of Large Language Models and ALTEGRAD.

The code checks Definition 10.1 and Propositions 10.2–10.3 numerically.

import numpy as np
rng = np.random.default_rng(0)
n, d, d_k = 6, 16, 8
X = rng.normal(size=(n, d))                                   # n tokens of dimension d
Wq, Wk, Wv = (rng.normal(0, d ** -0.5, (d, d_k)) for _ in range(3))

def attention(X):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    S = Q @ K.T / np.sqrt(d_k)                                # scaled scores
    A = np.exp(S - S.max(1, keepdims=True))
    A /= A.sum(1, keepdims=True)                              # each row sums to one
    return A @ V, A

out, A = attention(X)
P = np.eye(n)[rng.permutation(n)]                             # a permutation matrix
out_perm, _ = attention(P @ X)
print("rows of A sum to", A.sum(1).round(6))
print("permutation equivariance error:", np.abs(out_perm - P @ out).max())
q = rng.normal(size=(10000, 64)); k = rng.normal(size=(10000, 64))
print("std of q.k for d_k = 64:", (q * k).sum(1).std().round(2), "-> after scaling:",
      ((q * k).sum(1) / 8).std().round(2))

Each row of attention weights sums to one, and permuting the tokens permutes the outputs, to machine precision. For d_k = 64, raw dot products have a standard deviation of about 8 = \sqrt{64}, brought back to about 1 by the scaling.

11. Research directions and open problems

Self-study chapter. Each item below states what is established, what remains open, and where to start reading. The MVA courses that go further are linked.

Scale and foundation models

• Scaling laws. Kaplan et al. (2020) observed that the test loss of language models falls as a power law in the number of parameters, the amount of data and the compute, over several orders of magnitude. Hoffmann et al. (2022) showed that, at fixed compute, many large models were under-trained, and that parameters and training tokens should grow in proportion. Why power laws appear, and which abilities emerge with scale, remain open.
• Foundation models for vision. CLIP aligns images and captions by contrastive learning on 400 million pairs (Radford et al., 2021), and DINOv2 produces self-supervised features that are used frozen (Oquab et al., 2024). Their failures on counting, spatial relations and compositional prompts are an active topic; see Multimodal Explainable AI.

Generative models beyond GANs

• Diffusion models learn to reverse a gradual noising process, and are trained by a simple regression (Ho, Jain and Abbeel, 2020; Song et al., 2021). They replaced GANs for most image generation. Fast sampling, evaluation, and the memorisation of training images (Carlini et al., 2023) are open questions; see Generative Models for Images and Stochastic Calculus in Machine Learning.

Generalisation and robustness

• Generalisation. Standard networks fit random labels on CIFAR-10, so classical capacity measures cannot explain why they generalise on real labels (Zhang et al., 2017). The test error can decrease again beyond the interpolation threshold, a “double descent” (Belkin et al., 2019). See Theoretical Foundations of Deep Learning.
• Adversarial examples. Imperceptible perturbations change the predictions of image classifiers (Szegedy et al., 2014; Goodfellow, Shlens and Szegedy, 2015). Adversarial training improves robustness at a cost in accuracy (Madry et al., 2018); robustness without that cost, and certified guarantees at scale, are open.

Self-supervised learning

• Predicting in representation space. Joint-embedding predictive architectures predict the representations of masked regions rather than their pixels (Assran et al., 2023). A theory of which augmentations and objectives yield useful features, and why non-contrastive methods avoid collapse (ch. 8), is still incomplete.

Deep reinforcement learning

• Sample efficiency. Model-based agents learn a model of the environment and plan in it; MuZero learns the model needed for tree search without being given the rules (Schrittwieser et al., 2020). Offline reinforcement learning learns from fixed data sets without new interaction (Levine et al., 2020). Exploration, generalisation across environments and reward specification remain hard; see Reinforcement Learning.
• Learning from human feedback. Reward models trained on human preferences (Christiano et al., 2017) are used, with PPO, to fine-tune language models (Ouyang et al., 2022). How to specify and verify the objectives of such systems is discussed in the Turing Seminar.

Geometry, 3D and medical images

• Object pose. EPnP solves the perspective-n-point problem in time linear in the number of points (Lepetit, Moreno-Noguer and Fua, 2009). BB8 predicts with a CNN the projections of the corners of an object’s 3D bounding box, then recovers its pose with PnP (Rad and Lepetit, 2017). Neural radiance fields learn a 3D scene from posed images (Mildenhall et al., 2020); see 3D Computer Vision.
• Medical imaging. U-Net set the standard architecture for segmentation from few annotated images (Ronneberger, Fischer and Brox, 2015). Scarce labels, shifts between hospitals and scanners (Proposition 5.6) and calibrated uncertainty are central open problems; see Deep Learning for Medical Imaging and Representation Learning for Computer Vision and Medical Imaging.

Brains and machines

• Credit assignment. Lillicrap et al. (2020) review how the brain might approximate backpropagation, from feedback alignment to local learning rules. Whether networks that predict neural responses (ch. 1) also reproduce behaviour and learning remains debated.

Exercises

Levels: ★ apply the course, ★★ combine or prove a short result, ★★★ open-ended or research-flavoured. The Practice column of the session plan says when to do each one.

1. ★ Gradients through a ReLU layer. Let f(W) = \frac12 \|\mathrm{ReLU}(W x) - y\|^2, with x = (1, 2), \qquad W = \begin{pmatrix} 1 & 1 \\ -1 & 0 \end{pmatrix}, \qquad y = (2, 0). Compute f(W), the gradient \nabla_W f and the gradient with respect to x, using Definition 0.1 and Proposition 0.2.

   Solution: W x = (3, -1), so \mathrm{ReLU}(W x) = (3, 0), the residual is (1, 0) and f = 0.5.

   The Jacobian of the ReLU is \operatorname{diag}(1, 0), so the error at the pre-activation is \delta = (1, 0). Then \nabla_W f = \delta\, x^\top = \begin{pmatrix} 1 & 2 \\ 0 & 0 \end{pmatrix}, \qquad \nabla_x f = W^\top \delta = (1, 1). The second row gets no gradient: the second unit is inactive on this input.

2. ★ Boolean functions and the perceptron. (a) Give threshold neurons (Definition 1.1) for NAND and for the majority of three binary inputs. (b) Run the perceptron rule (Definition 1.2) on OR, with labels \pm 1, a constant 1 appended to each input, the order (0,0), (0,1), (1,0), (1,1) and w = 0 at the start, until a full pass makes no mistake. (c) Compare the number of mistakes with Theorem 1.3 for the unit vector w^\star = (2, 2, -1)/3.

   Solution: (a) NAND: w = (-1, -1), b = 1.5. Majority: w = (1, 1, 1), b = -1.5.

   (b) The weights after each mistake are, pass by pass: \begin{aligned} &\text{pass 1: } (0, 0, -1),\ (0, 1, 0),\ (1, 1, 1); && \text{pass 2: } (1, 1, 0); && \text{pass 3: } (1, 1, -1),\ (1, 2, 0);\\ &\text{pass 4: } (1, 2, -1),\ (2, 2, 0); && \text{pass 5: } (2, 2, -1). \end{aligned} The sixth pass makes no mistake: 9 mistakes in total, and the final weights are (2, 2, -1).

   (c) The margins of w^\star on the four inputs are 1/3, 1/3, 1/3 and 1, so \gamma = 1/3; the longest input is (1, 1, 1), of norm R = \sqrt{3}. The bound is (R / \gamma)^2 = 27 \geq 9 .

3. ★ Backpropagation through a softmax. A network has one hidden ReLU layer with W_1 = I_2 and no biases, then logits z_2 = W_2 a_1 and a softmax. With x = (1, -1), \qquad W_2 = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}, and the second class as label, compute the loss, then the gradients with respect to W_2 and W_1 by Theorem 2.5.

   Solution: z_1 = (1, -1), a_1 = (1, 0), z_2 = (1, 0), so p = (0.731, 0.269) and the loss is -\log 0.269 = 1.313.

   By Proposition 2.4, \delta_2 = p - e_2 = (0.731, -0.731). Then W_2^\top \delta_2 = (0.731, -1.462), and the ReLU mask (1, 0) gives \delta_1 = (0.731, 0): \frac{\partial \mathcal{L}}{\partial W_2} = \delta_2\, a_1^\top = \begin{pmatrix} 0.731 & 0 \\ -0.731 & 0 \end{pmatrix}, \qquad \frac{\partial \mathcal{L}}{\partial W_1} = \delta_1\, x^\top = \begin{pmatrix} 0.731 & -0.731 \\ 0 & 0 \end{pmatrix}. A gradient step lowers the logit of the wrong class and raises that of the right one.

4. ★ Parameters, shapes and receptive fields. A network for 3 \times 32 \times 32 images applies, in order:

   • a 3 \times 3 convolution from 3 to 32 channels, with stride 1 and padding 1, a ReLU, and a 2 \times 2 max-pooling of stride 2;
   • a 3 \times 3 convolution from 32 to 64 channels, with padding 1, a ReLU and the same pooling;
   • global average pooling, and a linear layer to 10 classes.

   (a) Count the parameters of each layer. (b) Give the shape after each layer. (c) What is the receptive field of one output of the second convolution? (d) How many weights would a dense layer need, from the input to as many outputs as the first convolution?

   Solution: (a) By Definition 2.6, the convolutions have 32 \cdot 3 \cdot 9 + 32 = 896 and 64 \cdot 32 \cdot 9 + 64 = 18\,496 parameters, and the linear layer 64 \cdot 10 + 10 = 650: 20 042 in total.

   (b) The shapes are 32 \times 32 \times 32, then 32 \times 16 \times 16, 64 \times 16 \times 16, 64 \times 8 \times 8, a vector of 64, and 10 logits.

   (c) Each layer adds (kernel size − 1) times the current pixel spacing: 3 after the first convolution, 4 after the pooling (spacing 2), then 4 + 2 \cdot 2 = 8. The receptive field is 8 \times 8 pixels.

   (d) 3072 \times 32\,768 \approx 1.0 \times 10^8 weights, five thousand times more than the whole network.

5. ★ Momentum and Adam on a constant gradient. Assume the gradient is the same vector g at every step. (a) Give the momentum buffer of Definition 3.1 after t steps, started at 0, and its limit for \mu = 0.9. (b) Show that, with \epsilon = 0, every Adam step moves each coordinate by exactly \eta, in the direction opposite to the sign of g. (c) What would the first step be without the bias corrections?

   Solution: (a) The buffer is a geometric sum: v_t = (1 + \mu + \dots + \mu^{t-1})\, g = \frac{1 - \mu^t}{1 - \mu}\, g . It tends to 10 g: momentum multiplies the effective step by 10.

   (b) By induction, m_t = (1 - \beta_1^t)\, g and s_t = (1 - \beta_2^t)\, g^2. After the bias corrections, the update is \eta\, g / |g| coordinatewise, whatever the scale of g.

   (c) Without them, the first step would be multiplied by the factor below: the corrections remove a spurious jump at the start. \frac{1 - \beta_1}{\sqrt{1 - \beta_2}} = \frac{0.1}{\sqrt{0.001}} \approx 3.16

6. ★ The GAN optimum on three points. On the set \{a, b, c\}, let p_{\text{data}} = (1/2, 1/2, 0) and p_g = (1/4, 1/4, 1/2). (a) Give the optimal discriminator of Proposition 5.2. (b) Compute C(G) and deduce the Jensen–Shannon divergence from Theorem 5.3. (c) Check the result by computing the divergence directly.

   Solution: (a) D^\star = (2/3, 2/3, 0): the point c is certainly fake.

   (b) Only a and b count in the first expectation, and c contributes \log 1 = 0 to the second: C(G) = \log \tfrac23 + \tfrac12 \log \tfrac13 = -0.955, \qquad \mathrm{JSD} = \frac{C(G) + \log 4}{2} = 0.216 .

   (c) The mixture is M = (3/8, 3/8, 1/4), and the mean of the two divergences below is 0.216: \mathrm{KL}(p_{\text{data}} \,\|\, M) = \log \tfrac43 = 0.288, \qquad \mathrm{KL}(p_g \,\|\, M) = \tfrac12 \log \tfrac23 + \tfrac12 \log 2 = 0.144 .

7. ★ Perplexity and the size of an LSTM. (a) A language model gives the probabilities 0.5, 0.25, 0.1 and 0.4 to the four tokens of a test text. What is its perplexity (Definition 6.1)? (b) How many parameters has an LSTM layer (Definition 6.4) with inputs of dimension 300 and a hidden state of dimension 512? Compare with a vanilla RNN of the same sizes. (c) What is the perplexity of a uniform model over 10 000 tokens?

   Solution: (a) The product of the probabilities is 0.005, so the perplexity is 0.005^{-1/4} = 200^{1/4} \approx 3.76.

   (b) Each of the four gates has a 512 \times 812 matrix acting on u_t = (h_{t-1}, x_t), and a bias: 4 \times (512 \times 812 + 512) = 1\,665\,024 . PyTorch keeps two bias vectors per gate and counts 1 667 072. The vanilla RNN has 512 \times 812 + 512 = 416\,256 parameters, a quarter.

   (c) 10 000, the size of the vocabulary.

8. ★ IoU, NMS and AP by hand. Two objects G_1 = (0, 0, 10, 10) and G_2 = (20, 0, 30, 10) are detected by four boxes, in decreasing order of score: D_1 = (1, 1, 11, 11), D_2 = (2, 0, 12, 10), D_3 = (20, 2, 30, 12) and D_4 = (40, 0, 50, 10). (a) Compute the IoU of each box with each object. (b) Compute AP at IoU 0.5 (Definition 7.2) without NMS. (c) Apply NMS with threshold 0.5 (Algorithm 7.3), and compute AP again.

   Solution: (a) Only three IoUs are non-zero; D_4 meets no object.

   • D_1 with G_1: the intersection is a 9 \times 9 square, so the IoU is 81/119 = 0.681.
   • D_2 with G_1, and D_3 with G_2: the intersection has area 80, and the IoU is 80/120 = 0.667.

   (b) D_1 is correct, D_2 is a duplicate (a false positive), D_3 is correct and D_4 is false. The precisions are 1, 0.5, 0.667 and 0.5, the recalls 0.5, 0.5, 1 and 1. With the precision envelope, \mathrm{AP} = 0.5 \times 1 + 0.5 \times 0.667 = 0.833 .

   (c) \mathrm{IoU}(D_1, D_2) = 0.681 \geq 0.5, so NMS deletes D_2. The list D_1, D_3, D_4 has precisions 1, 1, 0.667 and reaches recall 1 at rank 2, so \mathrm{AP} = 1. The functions of the code of ch. 7 give the same numbers.

9. ★★ Early stopping as weight decay. Run gradient descent with step \eta from \theta_0 = 0 on the quadratic loss of Proposition 3.4, with 0 < \eta \lambda_i \leq 1 for every eigenvalue. (a) Show that after t steps \theta_t = \sum_i \Big[1 - (1 - \eta \lambda_i)^t\Big]\, \big(u_i^\top \theta^\star\big)\, u_i . (b) Show that stopping after t steps shrinks the eigendirections roughly like weight decay with \alpha = 1/(\eta t). (c) Compare the two shrinkage factors for \eta = 1, t = 100 and eigenvalues from 0.001 to 1.

   Solution: (a) The gradient is H(\theta - \theta^\star), so each step multiplies the error by I - \eta H: \theta_{t+1} - \theta^\star = (I - \eta H)(\theta_t - \theta^\star), \qquad \theta_t - \theta^\star = -(I - \eta H)^t\, \theta^\star . Diagonalising H gives the formula.

   (b) When \eta \lambda t \ll 1, the factor is close to \eta \lambda t = \lambda / \alpha, like \lambda / (\lambda + \alpha). When \eta \lambda t \gg 1, both factors are close to 1. Early stopping keeps the high-curvature directions and shrinks the flat ones.

   (c) The factors agree at both ends of the range. In between, early stopping shrinks less, by up to about 0.21 near \lambda = 0.025 (0.920 against 0.714):

   Eigenvalue	Early stopping	Weight decay
   0.001	0.095	0.091
   0.01	0.634	0.500
   0.1	1.000	0.909
   1	1.000	0.990

10. ★★ The autoencoder finds the subspace, not the axes. In the setting of Theorem 4.3: (a) show that if (E, D) is optimal, so is (E A, A^{-1} D) for every invertible k \times k matrix A; (b) with tied weights D = E^\top, show that every optimal encoder has orthonormal columns spanning the principal subspace; (c) explain how to recover the principal axes from an optimal encoder.

    Solution: (a) The product E A A^{-1} D = E D is unchanged, and so is the error.

    (b) The argument takes three steps.

    • At the optimum, P = E E^\top equals the projector V_k V_k^\top, of rank k. So E has rank k, and G = E^\top E is invertible.
    • From P^2 = P, we get E\, G\, E^\top = E E^\top. Multiplying by E^\top on the left and by E on the right gives G^3 = G^2, hence G = I.
    • The columns of E are therefore orthonormal, and they span the range of P, the principal subspace.

    (c) Then E = V_k R, with R = V_k^\top E orthogonal, and the codes X E have the second-moment matrix E^\top X^\top X E = R^\top \operatorname{diag}(\sigma_1^2, \dots, \sigma_k^2)\, R . Diagonalising this k \times k matrix gives R, hence the axes V_k = E R^\top, when the singular values are distinct.

11. ★★ Disjoint supports: why the Jensen–Shannon divergence stops helping. On the real line, the data distribution is a point mass \delta_0, and the generator produces the point mass \delta_\theta.

    (a) Show that the Jensen–Shannon divergence between them is \log 2 for every \theta \neq 0, so that the criterion C(G) of Theorem 5.3 is flat in \theta. (b) What is \mathrm{KL}(\delta_0 \,\|\, \delta_\theta)? (c) Show that the Wasserstein-1 distance is |\theta|, and conclude.

    Solution: (a) The mixture M = \frac12 (\delta_0 + \delta_\theta) gives mass 1/2 to each point, so both divergences to M equal \log 2: \mathrm{KL}(\delta_0 \,\|\, M) = \log \frac{1}{1/2} = \log 2, \qquad C(G) = -\log 4 + 2 \log 2 = 0 \quad (\theta \neq 0). The criterion is constant, and its gradient vanishes.

    (b) It is infinite, since \delta_\theta gives no mass to 0.

    (c) The only coupling moves a unit mass over the distance |\theta|. This distance decreases continuously as \theta \to 0, and gives a useful gradient. Arjovsky, Chintala and Bottou (2017) use this kind of example to motivate the Wasserstein GAN of ch. 5.

12. ★★ Exploding gradients and clipping. A scalar linear recurrent network reads one impulse and is penalised on its final state: h_t = w\, h_{t-1} + x_t, \qquad h_0 = 0, \quad x_1 = 1, \quad x_t = 0 \ (t \geq 2), \qquad \ell = \tfrac12 h_T^2 . (a) Compute h_T and d\ell / dw. (b) Evaluate the derivative for T = 50 at w = 1.1 and at w = 0.9. (c) Show that gradient clipping keeps a descent direction. Compare the steps at w = 1.1, with learning rate 0.01, without clipping and with clipping at norm 1.

    Solution: (a) h_T = w^{T-1}, so \ell = \frac12 w^{2T-2} and d\ell / dw = (T - 1)\, w^{2T-3}.

    (b) 49 \times 1.1^{97} \approx 5.1 \times 10^5 and 49 \times 0.9^{97} \approx 1.8 \times 10^{-3}: a ratio of about 2.8 \times 10^8, more than eight orders of magnitude, as Proposition 6.3 predicts for \|W\| above and below 1.

    (c) Clipping multiplies the gradient by \lambda = \min(1, c / \|g\|) \in (0, 1], so its inner product with the gradient is \lambda \|g\|^2 > 0. Without clipping, the step is about −5 100 and sends w near −5 100, where the loss is astronomical. With clipping, the step is −0.01, and w becomes 1.09.

13. ★★ The gradient of the focal loss. For an object anchor with logit u, let p = \sigma(u) be the probability of the true class, so p_t = p in Definition 7.6. (a) Show that \frac{d\, \mathrm{FL}}{du} = (1 - p)^\gamma \Big(\gamma\, p \log p - (1 - p)\Big). (b) Compare the gradients of the focal loss with \gamma = 2 and of the cross-entropy, at p = 0.99 and at p = 0.3. (c) Interpret.

    Solution: (a) With dp/du = p(1 - p), the product rule gives \frac{d}{du}\Big[-(1 - p)^\gamma \log p\Big] = \Big[\gamma (1 - p)^{\gamma - 1} \log p - \frac{(1 - p)^\gamma}{p}\Big]\, p\, (1 - p), which is the claim. For \gamma = 0 it is -(1 - p), the gradient of the cross-entropy.

    (b) At p = 0.99: -0.01 for the cross-entropy, -3.0 \times 10^{-6} for the focal loss. At p = 0.3: -0.700 and -0.697.

    (c) The easy anchor’s gradient is divided by about 3 300, the hard anchor’s is almost unchanged. The gradient, not only the loss of Example 7.7, concentrates on the hard examples.

14. ★★ How much information can InfoNCE certify?

    (a) Show that \mathcal{L}_N \geq 0, so that the bound of Theorem 8.3 never exceeds \log N.

    (b) Let X be uniform on K values and Y = X, with the critic f(x, y) = c\, \mathbb{1}[x = y] and uniform negatives. As c \to \infty, show that the loss tends to the limit below, where B \sim \mathrm{Bin}(N - 1, 1/K) counts the negatives equal to the positive: \mathcal{L}_N \to \mathbb{E} \log(1 + B) .

    (c) Evaluate the bound for (K, N) equal to (1000, 256), (10, 256) and (1000, 65 536), and compare with I(X; Y) = \log K.

    Solution: (a) The positive term is one of the terms of the denominator, so the fraction is at most 1 and its logarithm at most 0.

    (b) As c \to \infty, the softmax spreads uniformly over the candidates equal to x: the positive and the B matching negatives. The positive therefore gets probability 1 / (1 + B). The loss stays between 0 and \log N, so dominated convergence applies.

    (c) The bounds are 5.38, 2.28 and 6.90 nats, against \log K = 6.91, 2.30 and 6.91. With 256 negatives, the bound is capped near \log 256 = 5.55: certifying \log K nats takes N \gg K (McAllester and Stratos, 2020).

15. ★★ Value iteration on a two-state MDP. Two states s_1, s_2 and two deterministic actions: stay, or move to the other state. Staying in s_2 earns 1, everything else earns 0, and \gamma = 0.9. (a) Compute Q^\star and an optimal policy with Proposition 9.2. (b) Starting from Q_0 = 0, how many iterations of value iteration does Theorem 9.3 require to reach \|Q_k - Q^\star\|_\infty \leq 0.01? (c) Is the bound tight here?

    Solution: (a) Staying in s_2 forever is worth 1/(1 - \gamma) = 10. From s_1, moving then staying is worth \gamma \times 10 = 9. Hence Q^\star(s_1, \cdot) = (8.1,\ 9), \qquad Q^\star(s_2, \cdot) = (10,\ 8.1), for (stay, move): move in s_1, stay in s_2.

    (b) \|Q_0 - Q^\star\|_\infty = 10, so it suffices that 10 \times 0.9^k \leq 0.01, that is k \geq \log 1000 / \log(1/0.9) = 65.6: 66 iterations.

    (c) Yes. By induction Q_k(s_2, \text{stay}) = 10\,(1 - 0.9^k), whose error is exactly 10 \times 0.9^k. Running value iteration confirms that iteration 66 is the first within 0.01.

16. ★★ Baselines on a three-armed bandit. A softmax policy \pi_\theta(a) \propto e^{\theta_a} chooses among three arms with Gaussian rewards of variance 1 and means 10, 10.5 and 11. (a) Show that \nabla_\theta \log \pi_\theta(a) = e_a - \pi_\theta, and that the exact gradient of J(\theta) = \sum_a \pi_\theta(a)\, \bar{r}_a has coordinates \pi_\theta(b)\, (\bar{r}_b - J). (b) Run the code below: it compares the single-sample REINFORCE estimates with and without the baseline b = J (Proposition 9.7), then learns with each. (c) Explain the results.

    import numpy as np
    rng = np.random.default_rng(0)
    means = np.array([10.0, 10.5, 11.0])                  # three arms, unit-variance Gaussian rewards
    
    def softmax(theta):
        p = np.exp(theta - theta.max())
        return p / p.sum()
    
    pi = np.ones(3) / 3                                   # uniform policy, theta = 0
    a = rng.choice(3, size=100000, p=pi)
    r = means[a] + rng.normal(size=a.size)
    score = np.eye(3)[a] - pi                             # grad log pi(a) = e_a - pi
    for name, b in [("no baseline", 0.0), ("baseline J", pi @ means)]:
        g = (r - b)[:, None] * score
        print(f"{name:12s} mean {g.mean(0).round(3)}  total variance {g.var(0).sum():7.2f}")
    print("exact gradient", (pi * (means - pi @ means)).round(3))
    
    for name, use_b in [("no baseline", False), ("baseline", True)]:
        best = []
        for run in range(20):                             # 20 independent runs of 1000 steps
            theta, r_avg = np.zeros(3), 0.0
            for t in range(1, 1001):
                p = softmax(theta)
                a = rng.choice(3, p=p)
                r = means[a] + rng.normal()
                theta += 0.02 * (r - (r_avg if use_b else 0.0)) * (np.eye(3)[a] - p)
                r_avg += (r - r_avg) / t                  # running mean of the rewards
            best.append(softmax(theta)[2])
        print(f"{name:12s} P(best arm) after 1000 steps: mean {np.mean(best):.2f}, worst run {np.min(best):.2f}")

    Solution: (a) The gradient of the expression below is e_a - \pi_\theta, as in Proposition 2.4: \log \pi_\theta(a) = \theta_a - \log \sum_{a'} e^{\theta_{a'}} . Averaging \bar{r}_a (e_a - \pi_\theta) over a \sim \pi_\theta gives the coordinates \pi_\theta(b)\, \bar{r}_b - \pi_\theta(b)\, J.

    (b) Both estimates are unbiased, with mean close to the exact gradient (-0.167, 0, 0.167). Their total variances are 74.3 without the baseline and 0.73 with it. After 1000 steps the best arm has probability 0.60 on average without the baseline, and 0.01 in the worst run; with it, 0.92 on average and 0.87 in the worst run.

    (c) Without a baseline, every reward is about 10, so each update pushes up whichever arm was sampled, by a large amount. These pushes cancel only on average, and a run can lock onto a bad arm. Subtracting J keeps only the informative part of the reward.

17. ★★★ A causal attention layer learns a delay. This exercise goes with the self-study ch. 10.

    (a) Implement one head of causal self-attention (Definitions 10.1 and 10.4), with learned positional embeddings. Train it to output, at each position t \geq 3, the token found at position t - 3. (b) Check causality: changing future tokens must leave earlier outputs unchanged. (c) Remove the positional embeddings and explain the result with Proposition 10.2. (d) Open question: what does it take to learn an induction head, which predicts the token that followed the previous occurrence of the current token?

    import torch
    from torch import nn
    torch.set_num_threads(1)
    V, T, d, lag = 8, 16, 32, 3
    
    class CausalAttention(nn.Module):
        def __init__(self, positions=True):
            super().__init__()
            self.tok = nn.Embedding(V, d)
            self.pos = nn.Parameter(0.5 * torch.randn(T, d)) if positions else None
            self.q, self.k, self.v = (nn.Linear(d, d, bias=False) for _ in range(3))
            self.out = nn.Linear(d, V)
        def forward(self, s):
            x = self.tok(s) + (self.pos if self.pos is not None else 0)
            S = self.q(x) @ self.k(x).transpose(1, 2) / d ** 0.5
            mask = torch.triu(torch.ones(T, T, dtype=torch.bool), 1)   # True above the diagonal
            A = torch.softmax(S.masked_fill(mask, float("-inf")), dim=-1)
            return self.out(A @ self.v(x)), A
    
    def train(positions):
        torch.manual_seed(0)
        model = CausalAttention(positions)
        opt = torch.optim.Adam(model.parameters(), lr=3e-3)
        for step in range(1500):                          # target at position t: the token at t - lag
            s = torch.randint(V, (64, T))
            logits, _ = model(s)
            loss = nn.functional.cross_entropy(logits[:, lag:].reshape(-1, V), s[:, :-lag].reshape(-1))
            opt.zero_grad(); loss.backward(); opt.step()
        return model
    
    for positions in (True, False):
        model = train(positions)
        s = torch.randint(V, (500, T))
        with torch.no_grad():
            logits, A = model(s)
            s2 = s.clone(); s2[:, 10:] = torch.randint(V, (500, T - 10))   # change the future of t = 9
            leak = (model(s2)[0][:, :10] - logits[:, :10]).abs().max().item()
        acc = (logits[:, lag:].argmax(-1) == s[:, :-lag]).float().mean().item()
        offset = (torch.arange(T)[:, None] - A.mean(0).argmax(1)[:, None]).squeeze()[lag:]
        print(f"positions {positions!s:5s}: accuracy {acc:.3f} (chance {1 / V:.3f}), "
              f"future leak {leak:.1e}, most attended offset {offset.tolist()}")

    Hints and expected result: (a) With positions, the accuracy is 1.000, and at every position the most attended key lies exactly 3 steps back.

    (b) Changing the tokens after position 9 changes the outputs up to position 9 by exactly 0.

    (c) Without positions the accuracy falls to 0.330. The output at t then depends only on the current token and on the set of visible tokens, not on their order (Proposition 10.2 applied to the causal prefix). The best guess is a visible token, which beats the chance level 0.125 only at early positions, where few tokens are visible.

    (d) Olsson and coauthors (2022) find that induction heads are formed by two heads in successive layers: a previous-token head, like the one trained here, then a head that matches on its output. Try two layers on sequences with repeated random segments.

18. ★★★ One-shot recognition of new classes. The code of ch. 4 trains and tests on the same ten digit classes. (a) Train the triplet network on the digits 0–4 only, and evaluate 5-way one-shot recognition on the digits 5–9, against nearest neighbour on pixels. (b) Vary the number of training steps, and compare with one-shot accuracy on the training classes. (c) Look for a change that makes the embedding beat pixels on new classes: augmentations (ch. 8), a smaller embedding, fewer or more training classes.

    Hints and expected result: build the dictionary of training indices from all the images of the digits 0–4. Let the one-shot function draw its support and its query among all the images of the digits 5–9, which training never sees.

    (a)–(b) Pixels reach 0.71. The embedding reaches 0.51, 0.50 and 0.47 after 50, 200 and 1000 steps, while it is at 0.99–1.00 on the training classes: longer training specialises the metric to the classes it has seen. Other seeds and sampling choices move these numbers by a few points, not the conclusion.

    (c) On the 2-way task 8 against 9, pixels reach about 0.77. Training on 2, 4 or 8 other classes gives about 0.63–0.72, and random one-pixel shifts do not reliably help (about 0.64–0.76). With ten classes in total, the digits cannot show the benefit of many training classes; this is why one-shot benchmarks such as Omniglot (964 training characters) and miniImageNet (Vinyals and coauthors, 2016) exist.

19. ★★★ Long lags: RNN against LSTM. A network reads a sequence of T random symbols out of 4, and must output the first symbol from its final state. (a) Run the code below for a vanilla RNN and for LSTMs with forget-gate bias 0 and 3, at lags 5, 20 and 30. (b) Repeat with other seeds, and find the longest lag that each model learns in 300 steps. (c) Relate the results to Propositions 6.3 and 6.5. (d) Open question: add attention over all the hidden states (Definition 6.6), and test T = 100.

    import torch
    from torch import nn
    torch.set_num_threads(1)
    
    class Model(nn.Module):
        def __init__(self, cell, forget_bias=None):
            super().__init__()
            self.rnn = cell(4, 32, batch_first=True)
            self.out = nn.Linear(32, 4)
            if forget_bias is not None:                  # PyTorch gate order: i, f, g, o
                with torch.no_grad():
                    self.rnn.bias_ih_l0[32:64].fill_(forget_bias)
        def forward(self, x):
            h, _ = self.rnn(x)
            return self.out(h[:, -1])                     # read the final state only
    
    def run(cell, T, forget_bias=None, steps=300):
        torch.manual_seed(0)
        model = Model(cell, forget_bias)
        opt = torch.optim.Adam(model.parameters(), lr=1e-2)
        for step in range(steps):                         # target: the FIRST symbol of the sequence
            s = torch.randint(4, (64, T))
            loss = nn.functional.cross_entropy(model(nn.functional.one_hot(s, 4).float()), s[:, 0])
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            opt.step()
        s = torch.randint(4, (1000, T))
        with torch.no_grad():
            return (model(nn.functional.one_hot(s, 4).float()).argmax(1) == s[:, 0]).float().mean().item()
    
    print("  T    RNN   LSTM b_f=0  LSTM b_f=3    (chance 0.25)")
    for T in (5, 20, 30):
        print(f"{T:3d}  {run(nn.RNN, T):.2f}   {run(nn.LSTM, T, 0.0):.2f}        {run(nn.LSTM, T, 3.0):.2f}")

    Hints and expected result: (a) At lag 5 all three models reach accuracy 1.00. At lags 20 and 30, the RNN (0.38 and 0.23) and the LSTM with bias 0 (0.27 and 0.26) stay near chance, while the LSTM with bias 3 reaches 1.00.

    (b) Single runs vary at longer lags: with the seed above, the LSTM with bias 3 is still at chance at lag 50 after 300 steps, and reaches 0.49 after 1000 steps.

    (c) With bias 3, the forget gates start near \sigma(3) \approx 0.95, and 0.95^{30} \approx 0.2 of the error signal survives along the cell path. With bias 0 the factor is about 0.5^{30} \approx 10^{-9}, and the RNN’s Jacobian products decay as fast, as measured in ch. 6.

20. ★★★ The alternating game need not converge. Near its equilibrium, a GAN with a linear discriminator and a point-mass generator reduces to the bilinear game \min_x \max_y\, x y, the Dirac GAN of Mescheder, Geiger and Nowozin (2018). Its unique equilibrium is (0, 0).

    (a) Show that simultaneous gradient descent–ascent with step \eta multiplies the distance to the equilibrium by \sqrt{1 + \eta^2} at each step. (b) Show that alternating updates, as in the GAN code of ch. 5, neither converge nor diverge when 0 < \eta < 2. (c) Show that the extragradient method converges for 0 < \eta < 1. (d) Check with the code. (e) Open question: compare with optimistic updates (Daskalakis and coauthors, 2018) and with the gradient penalties studied by Mescheder and coauthors (2018).

    import numpy as np
    eta, steps = 0.1, 200
    for name in ["simultaneous", "alternating", "extragradient"]:
        x, y = 1.0, 0.0                                   # min_x max_y x*y, equilibrium (0, 0)
        for t in range(steps):
            if name == "simultaneous":
                x, y = x - eta * y, y + eta * x
            elif name == "alternating":
                x = x - eta * y
                y = y + eta * x
            else:                                         # extrapolate, then update from the old point
                xh, yh = x - eta * y, y + eta * x
                x, y = x - eta * yh, y + eta * xh
        print(f"{name:14s} distance to equilibrium after {steps} steps: {np.hypot(x, y):.3f}")

    Solution: (a) The update is (x, y) \mapsto (x - \eta y,\ y + \eta x). Its matrix is I + \eta J, with J the rotation by a quarter turn, so it multiplies every norm by \sqrt{1 + \eta^2}: the iterates spiral outwards.

    (b) The alternating update x' = x - \eta y, then y' = y + \eta x', has the matrix A = \begin{pmatrix} 1 & -\eta \\ \eta & 1 - \eta^2 \end{pmatrix}, \qquad \det A = 1, \qquad \operatorname{tr} A = 2 - \eta^2 . For 0 < \eta < 2, the trace lies in (-2, 2), so the two eigenvalues are complex conjugates of product 1, of modulus 1. The iterates circle on an ellipse around the equilibrium.

    (c) The extragradient matrix is (1 - \eta^2) I + \eta J, a rotation scaled by the factor below, which is smaller than 1 for 0 < \eta < 1: \sqrt{(1 - \eta^2)^2 + \eta^2} = \sqrt{1 - \eta^2 + \eta^4} .

    (d) With \eta = 0.1 and 200 steps, the distance becomes 1.01^{100} = 2.705 for simultaneous updates, stays near 1 (1.020) for alternating ones, and falls to 0.9901^{100} = 0.370 with extragradient. The code prints these three numbers.

Projects and assessment

The official assessment (master-mva.com) is based on individual mini-projects on object recognition, natural language processing and reinforcement learning: two according to Mode de validation, three according to Evaluation. The labs of the sessions start them and give feedback. The Université Paris-Saclay course detail describes the assessment as partial continuous assessment (“Evaluation Continue non Intégrale”). No weights and no written exam are displayed.

Solo subjects 1–3 below are shaped like these mini-projects, one per domain, each a paper study. Solo subject 4 and the group subjects are larger variants for further work. The written exam is only a self-assessment proposal, and the rubric is a proposal for the mini-projects.

Solo projects

1. [Solo · ≈ 20 h] Mini-project 1, object recognition: how transferable are the features of a CNN? A paper study of Yosinski, Clune, Bengio and Lipson, “How transferable are features in deep neural networks?” (NeurIPS 2014), at the scale of CIFAR-10 (Krizhevsky, 2009).
   • Question: which layers of a network trained on some classes help on other classes, and does fine-tuning recover what freezing loses?
   • Implement: split CIFAR-10 into two halves A and B of five classes, and train a small residual network (ch. 2) on each. Transfer its first blocks from A to B, frozen (AnB) or fine-tuned (AnB+), with the self-transfer BnB as control (ch. 5).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 2, 3, 5. Laptop or free GPU tier.
   • Success: the shape of Figure 2 of the paper over 3 seeds: the dip of BnB in the middle layers, the drop of AnB at the top, and AnB+ at or above the baseline.
2. [Solo · ≈ 20 h] Mini-project 2, natural language processing: regularising an LSTM language model. A paper study of Zaremba, Sutskever and Vinyals, “Recurrent neural network regularization” (arXiv, 2014).
   • Question: where should dropout go in an LSTM language model, and how much perplexity does it save?
   • Implement: a two-layer word-level LSTM language model (ch. 6) on the language-modelling version of the Penn Treebank (Marcus, Santorini and Marcinkiewicz, 1993), or on WikiText-2 (Merity and coauthors, 2017). Use truncated backpropagation through time and gradient clipping. Compare no dropout, dropout on the non-recurrent connections only (the paper’s choice), and dropout everywhere (ch. 3).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 3, 6. Free GPU tier: the paper’s medium model (two layers of 650 units, 39 epochs) trains in a few hours.
   • Success: validation and test perplexities of the three variants with the paper’s ordering, and the medium model within 10 % of the perplexity the paper reports for it.
3. [Solo · ≈ 20 h] Mini-project 3, reinforcement learning: does DQN overestimate, and does Double DQN fix it? A paper study of van Hasselt, Guez and Silver, “Deep reinforcement learning with double Q-learning” (AAAI 2016).
   • Question: how large is the bias of Proposition 9.5 in practice?
   • Implement: DQN with a replay buffer and a target network (Definition 9.4), then Double DQN. Use CartPole-v1 of Gymnasium, the maintained successor of OpenAI Gym (Brockman and coauthors, 2016), and two MinAtar games (Young and Tian, 2019). Compare the predicted values with the discounted returns of Monte Carlo rollouts of the greedy policy.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 3, 9. Runs on a laptop CPU, with at most 500 000 environment steps per MinAtar run.
   • Success: an average return of at least 475 on CartPole-v1 over 100 evaluation episodes. On each game, estimated and true values over training for both methods, over 5 seeds, showing whether Double DQN shrinks the gap, as in the paper.
4. [Solo · ≈ 15 h] Contrastive pre-training at small scale: which augmentations matter? A paper study of Chen, Kornblith, Norouzi and Hinton, “A simple framework for contrastive learning of visual representations” (ICML 2020).
   • Implement: SimCLR (ch. 8) with a ResNet-18 on CIFAR-10: InfoNCE with a temperature, a projection head, then a linear probe on frozen features (ch. 4).
   • Ablations: crops alone, colour distortion alone, or both; temperatures 0.1, 0.5 and 1; probing before or after the projection head.
   • Deliverables: code, a 4-page report. Chapters: ch. 4, 8. Free GPU tier: five pre-training runs of about 50 epochs each, with batch size 256, each under an hour.
   • Success: linear-probe accuracies against a random encoder and a supervised encoder of the same architecture, and the two findings of the paper reproduced: the composition of crops and colour distortion beats each alone, and features before the head probe better than after it.

Group projects

1. [Group of 3 · ≈ 30 h each] Object detection: two stages against one, and what the focal loss buys. A paper study of Lin, Goyal, Girshick, He and Dollár, “Focal loss for dense object detection” (ICCV 2017).
   • Data: PASCAL VOC 2007 (Everingham and coauthors, 2010), trainval for training and test for evaluation.
   • Implement: the VOC evaluation (IoU, NMS and AP of ch. 7) from scratch, checked against a reference implementation. Fine-tune the Faster R-CNN and RetinaNet models of torchvision from ImageNet-pre-trained backbones (ch. 5).
   • Ablations: the focusing parameter in {0, 0.5, 1, 2, 5}, the NMS threshold, the anchor scales.
   • Split: evaluation code and error analysis; the two-stage detector; the one-stage detector and the focal loss.
   • Deliverables: repository, an 8-page report, a 15-minute defence. Chapters: ch. 2, 5, 7. Free GPU tier, with images resized to a shorter side of 480 pixels and about 10 epochs per run.
   • Success: mAP at IoU 0.5 for both detectors, the group’s evaluation code within 0.5 points of the reference, and the direction of the paper’s ablation (focusing parameter 2 better than 0) reproduced or its failure explained.
2. [Group of 2–3 · ≈ 30 h each] GANs: measuring and curing mode collapse. A paper study of Gulrajani, Ahmed, Arjovsky, Dumoulin and Courville, “Improved training of Wasserstein GANs” (NeurIPS 2017).
   • Data: the 8-Gaussian mixture of ch. 5, then MNIST (LeCun, Bottou, Bengio and Haffner, 1998).
   • Implement: a DCGAN with the non-saturating loss (ch. 5), a WGAN with weight clipping, and WGAN-GP. For evaluation, train an MNIST classifier, then count the digit classes generated and compute a Fréchet distance on its features, as FID does with Inception features (Heusel and coauthors, 2017).
   • Split: the models; the evaluation; the training dynamics, with Exercises 11 and 20 as starting points.
   • Deliverables: repository, an 8-page report, a 15-minute defence. Chapters: ch. 3, 5. Laptop or free GPU tier.
   • Success: for each method over 3 seeds, the modes covered and the Fréchet distance against training time, and a clear statement of which methods collapse or oscillate, with figures of the 2D samples.
3. [Group of 3 · ≈ 30 h each] Domain-adversarial transfer from house numbers to handwritten digits. A paper study of Ganin and coauthors, “Domain-adversarial training of neural networks” (JMLR 2016).
   • Data: SVHN (Netzer and coauthors, 2011) as the labelled source, MNIST as the target, unlabelled during training.
   • Implement: a source-only CNN; fine-tuning on 10 target labels per class, as a reference; DANN with a gradient-reversal layer (ch. 5).
   • Analysis: the domain distance between source and target features, before and after adaptation, measured by a domain classifier as in the paper; the link with Proposition 5.6; t-SNE plots.
   • Split: baselines and data; DANN; analysis.
   • Deliverables: repository, an 8-page report, a 15-minute defence. Chapters: ch. 2, 5. Free GPU tier.
   • Success: a gain of at least 10 points of target accuracy over the source-only model, over 3 seeds, together with a smaller measured domain distance.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, no electronic device; four questions, 20 points.

1. Question (4 points). Explain why a vanilla recurrent network struggles to learn dependencies over long lags, what gradient clipping fixes and what it does not, and how the LSTM cell helps.

   Expected:

   • The gradient over a lag T - t is a product of Jacobians, bounded by \|W\|^{T-t} (Proposition 6.3): it vanishes when the norm is below 1 and can explode above (1.5 pts).
   • Clipping rescales a large gradient and keeps its direction, so it handles explosion; it cannot restore a vanished signal (1 pt).
   • Along the cell path the Jacobian is the product of the forget gates (Proposition 6.5); with gates near 1, for instance a forget bias of 1 or more, the signal crosses long lags (1.5 pts).

2. Question (5 points). A network has one hidden ReLU layer without biases and a scalar output z = w_2^\top \mathrm{ReLU}(W_1 x), with the loss \frac12 (z - y)^2 and x = (1, 2), \qquad W_1 = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}, \qquad w_2 = (1, 2), \qquad y = 1 . (a) Compute the forward pass and the loss. (b) Compute the gradients with respect to w_2 and W_1. (c) Take one gradient step of size 0.1 on w_2 only, and give the new loss.

   Expected:

   • (a) W_1 x = (-1, 2), the hidden layer is a = (0, 2), z = 4 and the loss is 4.5 (1.5 pts).
   • (b) The output error is z - y = 3, so the gradient in w_2 is 3a = (0, 6). Back through the ReLU, \delta_1 = (0, 6), and the gradient in W_1 is \delta_1 x^\top, with rows (0, 0) and (6, 12) (2.5 pts).
   • (c) w_2 becomes (1, 1.4), so z = 2.8 and the loss falls to 1.62 (1 pt).

3. Question (5 points). State and prove the perceptron convergence theorem. Apply it to data with R = 2 and margin \gamma = 0.1, and explain why it says nothing about XOR.

   Expected:

   • The statement of Theorem 1.3 (1 pt).
   • Each mistake raises the alignment with w^\star by at least \gamma and the squared norm by at most R^2; Cauchy–Schwarz gives k \gamma \leq \sqrt{k} R (2.5 pts).
   • At most (2 / 0.1)^2 = 400 mistakes (0.5 pt).
   • XOR is not linearly separable (Proposition 1.4), so no margin exists and the assumption fails (1 pt).

4. Question (6 points). (a) For a fixed generator, derive the optimal discriminator of the GAN game, and the expression of the generator’s criterion with the Jensen–Shannon divergence. (b) On two points, let p_{\text{data}} = (1/2, 1/2) and p_g = (3/4, 1/4); compute D^\star, C(G) and the divergence. (c) Explain why the generator is trained with the non-saturating loss, and what mode collapse is.

   Expected:

   • (a) Pointwise maximisation of the integrand of V gives D^\star = p_{\text{data}} / (p_{\text{data}} + p_g); substituting gives C(G) = -\log 4 + 2\, \mathrm{JSD} (Proposition 5.2, Theorem 5.3) (2.5 pts).
   • (b) D^\star = (0.4, 0.667), and the four terms of C(G) add up as follows, so the divergence is (-1.319 + 1.386) / 2 = 0.034 (1.5 pts): C(G) = \tfrac12 \log 0.4 + \tfrac12 \log \tfrac23 + \tfrac34 \log 0.6 + \tfrac14 \log \tfrac13 = -0.458 - 0.203 - 0.383 - 0.275 = -1.319 .
   • (c) When the discriminator rejects the samples confidently, the minimax loss saturates while the non-saturating loss keeps a unit gradient in the logit (Proposition 5.4); mode collapse sends many noise values to a few outputs that fool the current discriminator (2 pts).

Grading rubric (proposal)

References

Official references (master-mva.com course page).

• I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016 — the reading material of the course, free online at deeplearningbook.org.
• Software tools named by the course page: Python, Keras and PyTorch. A. Paszke et al., “PyTorch: an imperative style, high-performance deep learning library”, NeurIPS 2019; F. Chollet et al., Keras, software, 2015.
• Related conferences named by the course page: NIPS (now NeurIPS), ICML, ICLR, CVPR, ECCV and ICCV.
• Related course named by the course page: Deep Learning in Practice, semester 2.

Textbooks.

• C. M. Bishop, H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024.
• S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023.
• A. Zhang, Z. C. Lipton, M. Li, A. J. Smola, Dive into Deep Learning, Cambridge University Press, 2023.
• R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994.
• A. Griewank, A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd ed., SIAM, 2008.

Ch. 1 — history and cognitive science.

• W. S. McCulloch, W. Pitts, “A logical calculus of the ideas immanent in nervous activity”, Bulletin of Mathematical Biophysics 5, 1943; D. O. Hebb, The Organization of Behavior, Wiley, 1949.
• F. Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain”, Psychological Review 65, 1958; A. B. J. Novikoff, “On convergence proofs on perceptrons”, Symposium on the Mathematical Theory of Automata, Polytechnic Institute of Brooklyn, 1962; M. Minsky, S. Papert, Perceptrons, MIT Press, 1969.
• S. Linnainmaa, master’s thesis on the Taylor expansion of accumulated rounding errors (reverse-mode differentiation), University of Helsinki, 1970; P. J. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, PhD thesis, Harvard University, 1974.
• K. Fukushima, “Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position”, Biological Cybernetics 36, 1980.
• D. E. Rumelhart, G. E. Hinton, R. J. Williams, “Learning representations by back-propagating errors”, Nature 323, 1986; D. E. Rumelhart, J. L. McClelland and the PDP Research Group, Parallel Distributed Processing, MIT Press, 1986.
• Y. LeCun et al., “Backpropagation applied to handwritten zip code recognition”, Neural Computation 1, 1989; G. E. Hinton, S. Osindero, Y.-W. Teh, “A fast learning algorithm for deep belief nets”, Neural Computation 18, 2006.
• J. Deng et al., “ImageNet: a large-scale hierarchical image database”, CVPR 2009; A. Krizhevsky, I. Sutskever, G. E. Hinton, “ImageNet classification with deep convolutional neural networks”, NeurIPS 2012.
• J. A. Fodor, Z. W. Pylyshyn, “Connectionism and cognitive architecture: a critical analysis”, Cognition 28, 1988; B. M. Lake, T. D. Ullman, J. B. Tenenbaum, S. J. Gershman, “Building machines that learn and think like people”, Behavioral and Brain Sciences 40, 2017.
• D. H. Hubel, T. N. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex”, Journal of Physiology 160, 1962.
• D. L. K. Yamins et al., “Performance-optimized hierarchical models predict neural responses in higher visual cortex”, PNAS 111, 2014; D. L. K. Yamins, J. J. DiCarlo, “Using goal-driven deep learning models to understand sensory cortex”, Nature Neuroscience 19, 2016.
• W. Schultz, P. Dayan, P. R. Montague, “A neural substrate of prediction and reward”, Science 275, 1997.
• T. P. Lillicrap, D. Cownden, D. B. Tweed, C. J. Akerman, “Random synaptic feedback weights support error backpropagation for deep learning”, Nature Communications 7, 2016; T. P. Lillicrap, A. Santoro, L. Marris, C. J. Akerman, G. Hinton, “Backpropagation and the brain”, Nature Reviews Neuroscience 21, 2020.

Ch. 2–3 — feedforward networks, optimisation and regularisation.

• G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals and Systems 2, 1989; M. Leshno, V. Ya. Lin, A. Pinkus, S. Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function”, Neural Networks 6, 1993; M. Telgarsky, “Benefits of depth in neural networks”, COLT 2016.
• K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition”, CVPR 2016, and “Delving deep into rectifiers: surpassing human-level performance on ImageNet classification”, ICCV 2015.
• X. Glorot, Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks”, AISTATS 2010.
• D. P. Kingma, J. Ba, “Adam: a method for stochastic optimization”, ICLR 2015; I. Loshchilov, F. Hutter, “Decoupled weight decay regularization”, ICLR 2019.
• S. Ghadimi, G. Lan, “Stochastic first- and zeroth-order methods for nonconvex stochastic programming”, SIAM Journal on Optimization 23, 2013; L. Bottou, F. E. Curtis, J. Nocedal, “Optimization methods for large-scale machine learning”, SIAM Review 60, 2018 — background for Theorem 3.2.
• N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting”, JMLR 15, 2014.
• S. Ioffe, C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift”, ICML 2015; J. L. Ba, J. R. Kiros, G. E. Hinton, “Layer normalization”, arXiv:1607.06450, 2016.

Ch. 4 — representation learning and siamese networks.

• Y. Bengio, A. Courville, P. Vincent, “Representation learning: a review and new perspectives”, IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 2013.
• P. Baldi, K. Hornik, “Neural networks and principal component analysis: learning from examples without local minima”, Neural Networks 2, 1989; P. Vincent, H. Larochelle, Y. Bengio, P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders”, ICML 2008.
• J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, R. Shah, “Signature verification using a ‘siamese’ time delay neural network”, NIPS 1993; S. Chopra, R. Hadsell, Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification”, CVPR 2005; R. Hadsell, S. Chopra, Y. LeCun, “Dimensionality reduction by learning an invariant mapping”, CVPR 2006.
• F. Schroff, D. Kalenichenko, J. Philbin, “FaceNet: a unified embedding for face recognition and clustering”, CVPR 2015.
• G. Koch, R. Zemel, R. Salakhutdinov, “Siamese neural networks for one-shot image recognition”, ICML Deep Learning Workshop, 2015; J. Snell, K. Swersky, R. Zemel, “Prototypical networks for few-shot learning”, NeurIPS 2017; O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, D. Wierstra, “Matching networks for one shot learning”, NeurIPS 2016.
• B. M. Lake, R. Salakhutdinov, J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction”, Science 350, 2015 (Omniglot).

Ch. 5 — generative adversarial networks and transfer learning.

• I. Goodfellow et al., “Generative adversarial nets”, NeurIPS 2014; A. Radford, L. Metz, S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks”, ICLR 2016.
• M. Arjovsky, S. Chintala, L. Bottou, “Wasserstein generative adversarial networks”, ICML 2017; I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. Courville, “Improved training of Wasserstein GANs”, NeurIPS 2017.
• L. Mescheder, A. Geiger, S. Nowozin, “Which training methods for GANs do actually converge?”, ICML 2018; C. Daskalakis, A. Ilyas, V. Syrgkanis, H. Zeng, “Training GANs with optimism”, ICLR 2018.
• P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, “Image-to-image translation with conditional adversarial networks”, CVPR 2017; J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks”, ICCV 2017; T. Karras, S. Laine, T. Aila, “A style-based generator architecture for generative adversarial networks”, CVPR 2019.
• M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local Nash equilibrium”, NeurIPS 2017 (FID).
• J. Yosinski, J. Clune, Y. Bengio, H. Lipson, “How transferable are features in deep neural networks?”, NeurIPS 2014; A. S. Razavian, H. Azizpour, J. Sullivan, S. Carlsson, “CNN features off-the-shelf: an astounding baseline for recognition”, CVPR Workshops 2014; M. Oquab, L. Bottou, I. Laptev, J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks”, CVPR 2014; S. Kornblith, J. Shlens, Q. V. Le, “Do better ImageNet models transfer better?”, CVPR 2019.
• S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, J. W. Vaughan, “A theory of learning from different domains”, Machine Learning 79, 2010; Y. Ganin et al., “Domain-adversarial training of neural networks”, JMLR 17, 2016.

Ch. 6 — recurrent networks and language.

• J. L. Elman, “Finding structure in time”, Cognitive Science 14, 1990; Y. Bengio, P. Simard, P. Frasconi, “Learning long-term dependencies with gradient descent is difficult”, IEEE Transactions on Neural Networks 5, 1994; R. Pascanu, T. Mikolov, Y. Bengio, “On the difficulty of training recurrent neural networks”, ICML 2013.
• S. Hochreiter, J. Schmidhuber, “Long short-term memory”, Neural Computation 9, 1997; F. A. Gers, J. Schmidhuber, F. Cummins, “Learning to forget: continual prediction with LSTM”, Neural Computation 12, 2000; R. Jozefowicz, W. Zaremba, I. Sutskever, “An empirical exploration of recurrent network architectures”, ICML 2015.
• K. Cho et al., “Learning phrase representations using RNN encoder–decoder for statistical machine translation”, EMNLP 2014; I. Sutskever, O. Vinyals, Q. V. Le, “Sequence to sequence learning with neural networks”, NeurIPS 2014; D. Bahdanau, K. Cho, Y. Bengio, “Neural machine translation by jointly learning to align and translate”, ICLR 2015.
• T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, “Distributed representations of words and phrases and their compositionality”, NeurIPS 2013; J. Pennington, R. Socher, C. D. Manning, “GloVe: global vectors for word representation”, EMNLP 2014; M. E. Peters et al., “Deep contextualized word representations”, NAACL 2018.
• W. Zaremba, I. Sutskever, O. Vinyals, “Recurrent neural network regularization”, arXiv:1409.2329, 2014; M. P. Marcus, B. Santorini, M. A. Marcinkiewicz, “Building a large annotated corpus of English: the Penn Treebank”, Computational Linguistics 19, 1993; S. Merity, C. Xiong, J. Bradbury, R. Socher, “Pointer sentinel mixture models”, ICLR 2017 (WikiText).

Ch. 7 — object detection.

• M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge”, International Journal of Computer Vision 88, 2010; T.-Y. Lin et al., “Microsoft COCO: common objects in context”, ECCV 2014.
• R. Girshick, J. Donahue, T. Darrell, J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014; R. Girshick, “Fast R-CNN”, ICCV 2015; S. Ren, K. He, R. Girshick, J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks”, NeurIPS 2015.
• T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, “Feature pyramid networks for object detection”, CVPR 2017; T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, “Focal loss for dense object detection”, ICCV 2017.
• J. Redmon, S. Divvala, R. Girshick, A. Farhadi, “You only look once: unified, real-time object detection”, CVPR 2016; W. Liu et al., “SSD: single shot multibox detector”, ECCV 2016.
• N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, “End-to-end object detection with transformers”, ECCV 2020; H. W. Kuhn, “The Hungarian method for the assignment problem”, Naval Research Logistics Quarterly 2, 1955.

Ch. 8 — self-supervised methods.

• C. Doersch, A. Gupta, A. A. Efros, “Unsupervised visual representation learning by context prediction”, ICCV 2015; M. Noroozi, P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles”, ECCV 2016; S. Gidaris, P. Singh, N. Komodakis, “Unsupervised representation learning by predicting image rotations”, ICLR 2018; R. Zhang, P. Isola, A. A. Efros, “Colorful image colorization”, ECCV 2016.
• J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding”, NAACL 2019; K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, “Masked autoencoders are scalable vision learners”, CVPR 2022.
• A. van den Oord, Y. Li, O. Vinyals, “Representation learning with contrastive predictive coding”, arXiv:1807.03748, 2018; D. McAllester, K. Stratos, “Formal limitations on the measurement of mutual information”, AISTATS 2020; T. Wang, P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere”, ICML 2020.
• T. Chen, S. Kornblith, M. Norouzi, G. Hinton, “A simple framework for contrastive learning of visual representations”, ICML 2020; K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, “Momentum contrast for unsupervised visual representation learning”, CVPR 2020.
• J.-B. Grill et al., “Bootstrap your own latent: a new approach to self-supervised learning”, NeurIPS 2020; X. Chen, K. He, “Exploring simple siamese representation learning”, CVPR 2021; J. Zbontar, L. Jing, I. Misra, Y. LeCun, S. Deny, “Barlow Twins: self-supervised learning via redundancy reduction”, ICML 2021; A. Bardes, J. Ponce, Y. LeCun, “VICReg: variance-invariance-covariance regularization for self-supervised learning”, ICLR 2022.
• Y. Tian, X. Chen, S. Ganguli, “Understanding self-supervised learning dynamics without contrastive pairs”, ICML 2021; M. Caron et al., “Emerging properties in self-supervised vision transformers”, ICCV 2021.

Ch. 9 — deep reinforcement learning.

• C. J. C. H. Watkins, P. Dayan, “Q-learning”, Machine Learning 8, 1992; V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature 518, 2015; H. van Hasselt, A. Guez, D. Silver, “Deep reinforcement learning with double Q-learning”, AAAI 2016.
• R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning 8, 1992; R. S. Sutton, D. McAllester, S. Singh, Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation”, NIPS 1999 (proceedings 2000).
• V. Mnih et al., “Asynchronous methods for deep reinforcement learning”, ICML 2016; J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, “Proximal policy optimization algorithms”, arXiv:1707.06347, 2017; L. Ouyang et al., “Training language models to follow instructions with human feedback”, NeurIPS 2022.
• D. Silver et al., “Mastering the game of Go with deep neural networks and tree search”, Nature 529, 2016; D. Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play”, Science 362, 2018.
• A. G. Barto, R. S. Sutton, C. W. Anderson, “Neuronlike adaptive elements that can solve difficult learning control problems”, IEEE Transactions on Systems, Man, and Cybernetics 13, 1983 (cart-pole); G. Brockman et al., “OpenAI Gym”, arXiv:1606.01540, 2016; K. Young, T. Tian, “MinAtar: an Atari-inspired testbed for thorough and reproducible reinforcement learning experiments”, arXiv:1903.03176, 2019.

Ch. 10 — attention and transformers.

• A. Vaswani et al., “Attention is all you need”, NeurIPS 2017; A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, “Improving language understanding by generative pre-training”, OpenAI technical report, 2018; A. Dosovitskiy et al., “An image is worth 16x16 words: transformers for image recognition at scale”, ICLR 2021.
• C. Olsson et al., “In-context learning and induction heads”, Transformer Circuits Thread, 2022 (arXiv:2209.11895).

Ch. 11 — research directions.

• J. Kaplan et al., “Scaling laws for neural language models”, arXiv:2001.08361, 2020; J. Hoffmann et al., “Training compute-optimal large language models”, NeurIPS 2022.
• A. Radford et al., “Learning transferable visual models from natural language supervision”, ICML 2021 (CLIP); M. Oquab et al., “DINOv2: learning robust visual features without supervision”, Transactions on Machine Learning Research, 2024.
• J. Ho, A. Jain, P. Abbeel, “Denoising diffusion probabilistic models”, NeurIPS 2020; Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-based generative modeling through stochastic differential equations”, ICLR 2021; N. Carlini et al., “Extracting training data from diffusion models”, USENIX Security 2023.
• C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, “Understanding deep learning requires rethinking generalization”, ICLR 2017; M. Belkin, D. Hsu, S. Ma, S. Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off”, PNAS 116, 2019.
• C. Szegedy et al., “Intriguing properties of neural networks”, ICLR 2014; I. Goodfellow, J. Shlens, C. Szegedy, “Explaining and harnessing adversarial examples”, ICLR 2015; A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, “Towards deep learning models resistant to adversarial attacks”, ICLR 2018.
• M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture”, CVPR 2023.
• J. Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model”, Nature 588, 2020; S. Levine, A. Kumar, G. Tucker, J. Fu, “Offline reinforcement learning: tutorial, review, and perspectives on open problems”, arXiv:2005.01643, 2020; P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, “Deep reinforcement learning from human preferences”, NeurIPS 2017.
• V. Lepetit, F. Moreno-Noguer, P. Fua, “EPnP: an accurate O(n) solution to the PnP problem”, International Journal of Computer Vision 81, 2009; M. Rad, V. Lepetit, “BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth”, ICCV 2017; B. Mildenhall et al., “NeRF: representing scenes as neural radiance fields for view synthesis”, ECCV 2020.
• O. Ronneberger, P. Fischer, T. Brox, “U-Net: convolutional networks for biomedical image segmentation”, MICCAI 2015.

Data sets used in the projects.

• A. Krizhevsky, “Learning multiple layers of features from tiny images”, technical report, University of Toronto, 2009 (CIFAR-10); Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, “Gradient-based learning applied to document recognition”, Proceedings of the IEEE 86, 1998 (MNIST); Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, “Reading digits in natural images with unsupervised feature learning”, NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011 (SVHN).