Deep Learning

Contents

• Introduction and overview
• 0. Review: matrix differential calculus and numerical stability
• 1. From the neuron to the deep network
• 2. Backpropagation
• 3. Optimization in practice
• 4. Convolutional networks (CNN)
• 5. Sequences: RNN, LSTM
• 6. Attention and transformers
• 7. Transfer learning and fine-tuning
• 8. Training in practice
• 9. Generative models: AE, VAE, GAN
• 10. Research at DaSciM (Vazirgiannis, Lutzeyer, Zhu)
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who has already trained an MNIST-like MLP (or a toy CNN) and wants to understand why it converges — or blows up into NaNs — before opening Cybenko. Empirical first: a stuck loss, a vanishing gradient, 99% train / 70% test accuracy; only then do we name depth, activations, backprop.

From perceptrons to transformers: networks, backpropagation, CNNs, RNNs, attention, generative models. The program’s LLMs are transformers; without this module, architecture, training and fine-tuning stay opaque. Guiding thread: a network is a computation graph; attention is reread as a GNN in Period 2.

What you will be able to do

• Build a neural network from scratch: layers, activations, initialization, and understand what depth buys.
• Unroll backpropagation by hand on a small network, and code it in PyTorch without copying a tutorial.
• Diagnose a failing training run: gradient explosion/vanishing, learning rate, normalization, numerical stabilization.
• Choose an architecture according to the data: CNNs for images, RNN/LSTM for sequences, transformers for everything else.
• Explain the attention mechanism and the transformer architecture, up to the explicit link with GNNs.
• Distinguish the generative families: autoencoders, VAEs, GANs and, in self-study, diffusion models — their objectives, their losses, their uses.
• Adapt a pretrained model to a new task by transfer learning and fine-tuning, with the right layer freezing/tuning reflexes.
• Launch a realistic training run: GPU, mixed precision, experiment tracking, reproducibility.

Official objectives and syllabus (2026-2027 catalog)

Official objectives: the SynapseS sheet publishes a description and a detailed programme but no separate “objectives” block. The description places the course in the era of large data sets, where the challenge is to extract knowledge from them and build intelligent services.

It presents deep learning as the paradigm that learns rich representations directly from raw data, through convolutional networks, transformers and large pretrained models, with applications in computer vision, natural language processing, data generation, bioinformatics and learning on graphs, and it announces both the theoretical foundations and concrete large-scale applications. The official keywords are machine learning, deep learning, artificial intelligence, graph neural networks.

The detailed programme (“minor changes may occur during the course”) has nine blocks, from a general introduction to machine learning to reinforcement learning. The table below gives one row per topic of that programme. The first five blocks, the machine-learning half, are taught in the twin module Machine Learning (MDC_51006_EP, mandatory in the same period); the rows give the exact sections there.

¹ The official sheet writes “réseaux de neurones conventionnels” (“conventional neural networks”), to be read as “convolutional” in view of the listed applications. ² The French sheet writes “intégrations”, a literal translation of “embeddings”. Referring the machine-learning half to the twin module is an organizational choice of these notes, flagged here. In return, these materials go beyond the official programme on three points: the detailed treatment of attention and transformers (the programme names only “modern natural language processing”), transfer learning and fine-tuning (ch. 7), and training in practice (ch. 8: GPU, mixed precision, reproducibility).

Where this course fits in the program

Year 1 · Period 1 course (36 h, 4.5 ECTS) preselected by the Graph × LLM route. It is one of the two ways to satisfy the “Deep Learning once” rule; the other is the P2 variant, taken in Period 2. It comes right after the Machine Learning module, of which it is the natural continuation: the perceptron is reread there as a linear model, optimization as stochastic gradient descent.

The official 2026-2027 sheet displays no prerequisites (its tab is empty); the positioning below describes the foundations actually assumed. The two refreshers — Refresher in Statistics and Refresher in Computer Science — are assumed acquired, notably NumPy matrix computation for the labs.

Downstream, this module (or its P2 variant) is the gateway to the “large models” part of the program: Introduction to Text Mining and NLP and Large Language Models (M2) build directly on the “Attention and transformers” chapter; Multimodal Generative AI extends the generative models (VAE, GAN) and fine-tuning. The attention chapter is also reread as a GNN in the Graph Machine and Deep Learning for Generative AI module: a transformer processes a set of tokens as a complete graph, and this reading unifies the two sides of the program.

The “Training in practice” chapter finally prepares the large-scale projects of Period 2 and the internships: this is where research code becomes engineering code.

Course outline

• Review: matrix differential calculus and numerical stability: gradients of matrix functions and conditioning — the background that avoids 80% of training bugs.
• From the neuron to the deep network: perceptron, activations, initialization, expressivity of depths — and, in closing (section 1.5), the reinforcement learning paradigm required by the official programme, taught in the last session as the sheet lists it last.
• Backpropagation: the chain rule on a computation graph, unrolled then automated.
• Optimization in practice: SGD and variants (momentum, Adam), learning-rate scheduling, batch normalization.
• Convolutional networks (CNN): convolution, pooling, classical architectures — learned vision.
• Sequences: RNN, LSTM: processing the sequential before attention, and understanding their limits.
• Attention and transformers: the mechanism underlying LLMs, also read as message passing on a graph, and its use on images (ViT).
• Transfer learning and fine-tuning: reusing a pretrained model — the most frequent professional gesture.
• Training in practice: GPU, mixed precision, experiment tracking — from research code to reproducible code.
• Generative models: AE, VAE, GAN: representing and generating data, three strategies compared, then diffusion in one self-study section.
• Research at DaSciM (Vazirgiannis, Lutzeyer, Zhu): the research of Michalis Vazirgiannis, Johannes Lutzeyer and Ye Zhu, from transformers to graphs.

Session plan

36 h in class = 12 sessions of 3 h. The volume is the one of the SynapseS 2026–2027 sheet (36 h in person).

The sheet gives no session-by-session timetable, but its detailed programme lists the topics in order, and the split below follows that order: its machine-learning blocks are taught first, in the Machine Learning module of the same period; then come neural networks and the perceptron (sessions 2–3), convolutional and recurrent networks with their applications (sessions 7–8), modern NLP (session 9), unsupervised deep learning (sessions 11–12), and reinforcement learning last (session 12), as on the sheet.

Optimization (ch. 3) and transfer learning and training practice (ch. 7–8) are additions of these notes, placed where the architectures need them.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods (backpropagation, optimization, the architectures), 3 = advanced material (generative models and their full derivations); the level never goes down from one session to the next.

Self-study, outside the 36 h (about 5 h in total): ch. 9.4 Diffusion in one section, with a box on normalizing flows (≈ 2 h) · ch. 10 Research at DaSciM — Graph of Words, GNN expressivity, steering diffusion models (≈ 3 h).

How to study this module

• Take the review chapter seriously: numerical stability and matrix computation are presented there because they condition backpropagation, the pivotal chapter of the course.
• Unroll backpropagation by hand on a two-layer network before coding it: this is the exercise that separates those who understand from those who copy.
• Run all the notebooks: the course alternates derivation and experiment, and each chapter has its PyTorch lab whose reflexes (shapes, dtypes, device) serve until M2.
• After the attention chapter, redraw the transformer diagram from memory and identify where a “graph” would be: this reading will be asked of you again in the Graph ML module.
• Actually fine-tune a small pretrained model on your own data: the transfer learning chapter is a gesture, not a reading.
• If you have a gap in optimization or generalization, go back to the Machine Learning module: chapters 2 and 3 here are its direct continuation.

0. Review: matrix differential calculus and numerical stability

Deep learning is, 90% of it, differential calculus applied to functions with billions of parameters. This chapter consolidates the L3 tools we will need every week: differentiation of matrix functions, the central limit theorem (to understand why SGD works) and a bit of numerical stability (to understand why naive implementations produce NaNs).

0.1. Differentiation of matrix functions

A network’s parameters are matrices; one must know how to differentiate with respect to a matrix. The most convenient convention: the gradient of a function f : \mathbb{R}^{m \times n} \to \mathbb{R} is the matrix G such that f(X + H) = f(X) + \langle G, H\rangle + o(\|H\|) with the Frobenius inner product \langle G, H \rangle = \mathrm{tr}(G^\top H). Three identities suffice for this whole course.

0.2. CLT and mini-batch noise

Why optimize on mini-batches rather than on the whole dataset? Let D_n = (\xi_1, \dots, \xi_n) be n iid samples from a distribution P, and the empirical risk \hat R_n(\theta) = \frac1n \sum_{i=1}^n \ell(\theta; \xi_i). For a mini-batch B = (\xi_{i_1}, \dots, \xi_{i_b}) of b indices drawn uniformly and independently, the mini-batch gradient is \hat g(\theta) = \frac1b \sum_{j=1}^b \nabla_\theta \ell(\theta; \xi_{i_j}).

0.3. Numerical stability: softmax and the log-sum-exp trick

The softmax \hat y_i = e^{z_i} / \sum_j e^{z_j} appears in all classification losses and in attention. In floating-point precision (float32, max \approx 3.4 \times 10^{38}), the naive implementation overflows: e^{1000} = \infty, and \infty/\infty = \mathrm{NaN}. The universal trick is to subtract the maximum before exponentiating.

import torch

z = torch.tensor([1000., 1001., 999.])
# print(torch.softmax(z, 0))          # NaN with a naive implementation
print(torch.logsumexp(z, 0))          # 1001.4076: stable (PyTorch always applies it)
print(torch.softmax(z - z.max(), 0))  # stable form: [0.2447, 0.6652, 0.0900]

1. From the neuron to the deep network

Concrete scene: 2-layer MLP on MNIST-like data, ReLU, SGD. After 5 epochs, train loss ~0.08 and test ~0.35; you add an 8th layer and the loss freezes (or becomes NaN). This is not “deep learning is magic”: it is mis-calibrated initialization, activation and depth. This chapter names these objects — neuron, MLP, non-linearity — before the universal approximation theorem.

1.1. The perceptron: the first learning rule

The official syllabus opens its neural-network block with “perceptrons and backpropagation”. Rosenblatt’s perceptron (1958) is a single neuron with a sign activation, \hat y = \mathrm{sign}(\langle w, x\rangle), for labels y \in \{-1, +1\}; a bias is absorbed by appending a constant coordinate 1 to x. Its interest today is twofold: it is the simplest learning rule with a convergence proof, and its failure on non-separable data explains why hidden layers, and therefore backpropagation, are needed.

1.2. The universal approximation theorem

1.3. Initialization: derivation of Xavier and He

A bad initialization produces activations that explode or vanish from the forward pass onward. Analyze a linear layer a = Wh with h \in \mathbb{R}^{n_{\mathrm{in}}}, \qquad W \in \mathbb{R}^{n_{\mathrm{out}} \times n_{\mathrm{in}}}, iid centered weights of variance v_W, iid centered inputs of variance v_h, independent of the weights.

1.4. Activation functions, compared

The usual approximation of GELU, used in BERT/GPT, is 0.5x\big(1 + \tanh\big(\sqrt{2/\pi}\,(x + 0.044715\,x^3)\big)\big). Note the kinship with ReLU — GELU weights x by the probability \Phi(x) that x is “activated” according to a standard Gaussian: it is a soft gate, differentiable at 0.

1.5. Beyond supervised: reinforcement learning

Everything in this chapter assumes labeled pairs (x, y). The official syllabus of the course lists a third paradigm, reinforcement learning: an agent interacts with an environment through actions and observes no label, but a scalar reward. This section sets up the framework; the full treatment (algorithms, theory, autonomous agents) is in the Reinforcement Learning and Autonomous Agents module (Period 2).

2. Backpropagation

Training a network by gradient descent needs, at every step, the gradient of the loss with respect to every weight — millions of partial derivatives. Backpropagation computes all of them for the price of about two forward passes, by applying the chain rule backward on the computation graph; this chapter proves it, runs it by hand on a small example, and counts its memory cost.

2.1. The theorem and its proof

The same recursion, written for a batch, fits in a few lines of NumPy. The code below trains nothing: it computes the forward pass and the backward pass of a two-layer ReLU network with a softmax cross-entropy, then checks every partial derivative against a central finite difference in float64. Do this check once for every hand-written backward pass.

import numpy as np

rng = np.random.default_rng(0)
n, d_in, d_h, K = 16, 4, 32, 3                        # batch, input, hidden, classes
X, y = rng.normal(size=(n, d_in)), rng.integers(0, K, n)
p = {"W1": rng.normal(0, np.sqrt(2 / d_in), (d_h, d_in)), "b1": np.zeros(d_h),   # He init (1.3)
     "W2": rng.normal(0, np.sqrt(2 / d_h), (K, d_h)), "b2": np.zeros(K)}

def forward(p):
    A1 = X @ p["W1"].T + p["b1"]                      # a = W x + b, one example per row
    H1 = np.maximum(A1, 0)                            # ReLU
    Z = H1 @ p["W2"].T + p["b2"]                      # logits
    Z = Z - Z.max(1, keepdims=True)                   # stable softmax (prop. 0.3)
    P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
    return -np.log(P[np.arange(n), y]).mean(), (A1, H1, P)

def backward(p, cache):
    A1, H1, P = cache
    D2 = P.copy(); D2[np.arange(n), y] -= 1; D2 /= n  # delta at the logits: softmax - one-hot
    D1 = (D2 @ p["W2"]) * (A1 > 0)                    # (W2^T delta2) * sigma'(a1), theorem 2.1
    return {"W1": D1.T @ X, "b1": D1.sum(0), "W2": D2.T @ H1, "b2": D2.sum(0)}

loss, cache = forward(p)
g = backward(p, cache)

eps = 1e-6                                            # central differences, in float64
for name, W in p.items():
    num = np.zeros_like(W)
    for idx in np.ndindex(W.shape):
        old = W[idx]
        W[idx] = old + eps; lp = forward(p)[0]
        W[idx] = old - eps; lm = forward(p)[0]
        W[idx] = old
        num[idx] = (lp - lm) / (2 * eps)
    rel = np.linalg.norm(num - g[name]) / (np.linalg.norm(num) + np.linalg.norm(g[name]))
    print(name, f"relative error {rel:.0e}")         # below 1e-9: the backward pass is right

The error is measured on the whole tensor, because an entry-by-entry relative error blows up on entries whose gradient is nearly zero.

2.2. Cost: memory, compute, and how to reduce them

• Complexity. A single forward pass + one backward pass cost \mathcal{O}(|\theta|) operations (up to a constant factor): the gradient of all the weights for the price of about two evaluations — this is the decisive contribution of reverse mode (Linnainmaa, 1970; Rumelhart, Hinton & Williams, 1986) over forward-mode differentiation, which would require |\theta| passes.
• Memory. The backward pass needs the activations of every layer: memory grows like the batch size \times the number of activations, not like |\theta|. For a transformer, this is the limiting factor.
• Checkpointing (Griewank; Chen, 2016): store only one layer out of \sqrt L and recompute the missing intermediates during the backward pass — \mathcal{O}(\sqrt L) memory instead of \mathcal{O}(L), for ~33% extra compute. torch.utils.checkpoint.
• Mixed precision: store/compute in float16 or bfloat16 (2× memory and tensor throughput), while keeping float32 master copies of the weights and scaling up the loss (loss scaling) to prevent small gradients from canceling in fp16. bfloat16 has the same exponent range as fp32 — more robust.

import torch

torch.manual_seed(0)
x, y = torch.randn(8, dtype=torch.float64), torch.randn(2, dtype=torch.float64)
W1 = torch.randn(64, 8, dtype=torch.float64, requires_grad=True)   # autograd tracks the operations
W2 = torch.randn(2, 64, dtype=torch.float64, requires_grad=True)

def f(W1, W2): return 0.5 * ((y - W2 @ torch.relu(W1 @ x))**2).sum()
loss = f(W1, W2)
loss.backward()          # automatic backprop
print(W1.grad.shape, W2.grad.shape)           # same shapes as W1, W2

# Check one entry by central finite difference, in float64 (do it for every new custom gradient)
j = int(W2.grad[0].abs().argmax())            # an entry with a nonzero gradient
eps = 1e-6
with torch.no_grad():
    W2[0, j] += eps;     lp = f(W1, W2)
    W2[0, j] -= 2 * eps; lm = f(W1, W2)
    W2[0, j] += eps
print((lp - lm).item() / (2 * eps), W2.grad[0, j].item())   # the two agree to about 9 digits

In float32, a loss of order 10 to 100 carries a rounding error of about 10^{-6} to 10^{-5}, as large as the change that a perturbation of 10^{-6} produces: the two numbers then disagree. Use float64, or accept a rough check with \epsilon \approx 10^{-3} in float32.

3. Optimization in practice

Training minimizes the risk R(\theta) = \mathbb{E}_{(x,y)\sim P}\big[\ell(f_\theta(x), y)\big] through its empirical version \hat R_n, by stochastic gradient descent: at each step, the gradient is estimated on a mini-batch (proposition 0.2). This chapter analyzes why each ingredient of the modern recipe (momentum, Adam, warmup, normalization, regularization) is there.

3.1. Convergence of SGD: the quadratic case, fully proved

The generic model of non-convexity is hard; the quadratic case, for its part, is solved exactly and illuminates all the others. Let f(\theta) = \tfrac12 \theta^\top A \theta - b^\top \theta with A symmetric positive definite, eigenvalues \lambda_1 \ge \dots \ge \lambda_d > 0, \lambda_1 = L (Lipschitz constant of the gradient), \lambda_d = \mu, condition number \kappa = L/\mu.

The minimizer \theta^\star = A^{-1}b satisfies \nabla f(\theta^\star) = 0, and the (for now deterministic) descent reads \theta_{k+1} = \theta_k - \eta (A\theta_k - b).

3.2. Momentum: convergence rates

3.3. Adam, derived step by step

3.4. Step scheduling and warmup

• Warmup (linear from 0 to \eta over a few hundred/thousand steps): at the very start of training, the features are random and the first gradients have a large norm; moreover, the bias correction of \hat v_t (prop. 3.5) divides by 1 - \beta_2^t, which varies fast when t is small — the ratio \hat m/\sqrt{\hat v} is then very noisy. A small step at the start lets the statistics settle. Pre-norm architectures (chapter 6) strongly reduce this need.
• Decay: cosine \eta_t = \eta_{\min} + \tfrac12(\eta - \eta_{\min})(1 + \cos(\pi t/T)), or steps (\times 1/10). Quadratic motif (theorem 3.1): close to the minimum, too large a step makes the SGD noise orbit around the solution — decreasing \eta divides the oscillation radius.
• LR range test (Smith, 2015): grow \eta geometrically over one epoch and plot the loss; the optimal \eta sits just before the minimum of this curve (chapter 8).

3.5. BatchNorm and LayerNorm

3.6. Regularization: the complete tour

• Dropout (Srivastava et al., 2014): during training, each activation is multiplied by an iid mask M_j \sim \mathrm{Bernoulli}(1-p) then divided by 1-p (“inverted”); at inference, identity. Proof that the expectation is preserved: \mathbb{E}[M_j/(1-p)] = 1, hence \mathbb{E}[\tilde h_j] = h_j — no scaling is needed at test time. Ensemble interpretation: each mask defines a sub-network; training samples 2^{n} architectures sharing their weights, and inference approximately averages their consensus (the exact “mean network” approximation holds for a single final linear block; Gal & Ghahramani, 2016, give it a Bayesian reading, used as MC dropout in Privacy and Uncertainty Quantification, ch. 8, M2).

• Weight decay \tfrac{\lambda}{2}\|\theta\|^2: SGD + decay gives \theta \leftarrow (1 - \eta\lambda)\theta - \eta \hat g — multiplicative shrinkage at each step (implicit ridge). AdamW (Loshchilov & Hutter, ICLR 2019) applies the decay directly to the weights and not through Adam’s preconditioner: via \hat m/\sqrt{\hat v}, the decay would undergo a per-coordinate normalization that makes it uneven; decoupled, it becomes a homogeneous ridge again. It is the standard for transformers (typically \lambda \sim 10^{-2}, much larger than in the SGD era).

• Label smoothing \varepsilon \in [0.05, 0.2]: the target becomes q_k = (1 - \varepsilon)\,\mathbf{1}[k = y] + \varepsilon/K. The cross-entropy penalizes -\log p_k \to \infty as p_k \to 0.

  A hard target pushes the probability of the true class toward p^\star \to 1, so \log p^\star \to 0, which needs a logit gap that grows without bound (overconfidence). The smoothed target puts the optimum at p^\star = 1 - \varepsilon + \varepsilon/K < 1, a finite logit gap, and improves calibration.

• Data augmentation: inject the known invariances (rotations/crops in vision, synonyms in NLP) — the cheapest regularization in parameters; it acts on the data, not on the model.

• Early stopping: stop when the validation loss rises again — a regularization at zero compute cost, equivalent to an implicit penalization of the norm (cf. Machine Learning module, bias-variance).

4. Convolutional networks (CNN)

A fully connected layer on a 224 × 224 colour image needs about 150,000 weights per neuron, and it ignores that neighbouring pixels are related and that an object means the same thing wherever it appears. A convolutional layer solves both problems by sliding one small filter over the whole image. This chapter defines the convolution, computes output sizes and receptive fields, and explains why residual connections let these networks grow deep.

4.1. Discrete convolution: general definition and dimension arithmetic

4.2. Receptive field

4.3. Transposed conv (deconvolution)

The gradient of the convolution with respect to the input is itself a convolution, by the flipped kernel and with adjusted padding — called transposed convolution, and used as is to upsample (GANs, segmentation, autoencoders).

Case with no stride, no dilation: a “valid” conv (p=0) maps I \to O = I - K + 1; its transpose maps O \to I = O + K - 1 by inserting K-1 border zeros and convolving with the flipped kernel.

Proof: write \mathrm{conv} as a linear map y = Cx by blocks; then \nabla_x = C^\top \nabla_y, and C^\top is the matrix of the transposed conv.

With stride and padding, the general formula: I' = s\,(O - 1) - 2p + d(K-1) + \mathrm{op} + 1, where \mathrm{op} is an optional extra padding — the exact mirror image of theorem 4.2. Tap overlap when s > 1 creates “checkerboard” artifacts, fixed by upsample + conv.

4.4. Historical architectures, parameters on the counter

import torch.nn as nn

conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
# parameters: 3*3*3*32 + 32 = 896, independent of the H x W size
x = torch.randn(16, 3, 64, 64)        # batch of 16 RGB 64x64 images
print(conv(x).shape)                   # [16, 32, 64, 64]  (theorem 4.2 formula)

class ResBlock(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.conv1 = nn.Conv2d(c, c, 3, padding=1)
        self.conv2 = nn.Conv2d(c, c, 3, padding=1)
        self.norm1, self.norm2 = nn.BatchNorm2d(c), nn.BatchNorm2d(c)
    def forward(self, x):
        h = torch.relu(self.norm1(self.conv1(x)))
        return torch.relu(x + self.norm2(self.conv2(h)))   # skip connection

5. Sequences: RNN, LSTM

Text, audio and time series have a variable length and an order that matters, so the network must carry information from one step to the next. A recurrent network does this with a hidden state updated at every step. This chapter shows why its gradient vanishes or explodes over long sequences, and how the gates of the LSTM mitigate the problem.

5.1. RNN and backpropagation through time

5.2. LSTM: complete equations and the role of the gates

• f_t, forget gate: fraction of the old memory kept — it is what carries long memory; often initialized with b_f > 0 (positive forget bias) so that f \approx 1 at the start.
• i_t, input gate: how much of the candidate \tilde c_t is written; \tilde c_t: the proposed content.
• o_t, output gate: how much of the cell is exposed in h_t.
• Why it mitigates vanishing: the cell follows c_t = f_t \odot c_{t-1} + (\dots), an additive highway — the gradient \partial c_t/\partial c_{t-1} = \mathrm{diag}(f_t) + i_t-terms stays close to the identity when f_t \approx 1 and i_t modest, where the RNN systematically multiplied by W_h^\top D_t. The product of the Jacobians can stay ~1 over long horizons: long dependencies get through. (The problem is not solved: for very long dependencies, attention will fix it — chapter 6.)

5.3. Applications: matching the architecture to the task

The official syllabus closes its “Deep learning I” block with applications. The same backbones (a CNN for grids, an RNN for sequences) serve very different tasks; what changes is the output head, the loss and the evaluation.

6. Attention and transformers

A recurrent network reads a sequence one step at a time: training cannot be parallelized over positions, and information from far back must survive many multiplications. Attention lets every token look directly at every other token in a single matrix operation. This chapter defines it, explains the \sqrt{d_k} scaling, and assembles the transformer block on which every LLM is built.

6.1. Scaled dot-product: why \sqrt{d_k}

6.2. Block architecture: pre-norm, positions, RoPE

• Post-norm (Vaswani et al., original): y = \mathrm{LN}(x + \mathrm{Att}(x)) — the residual connection passes through the normalization. Pre-norm (GPT-2 and after): y = x + \mathrm{Att}(\mathrm{LN}(x)) — the identity path is intact (proposition 4.6): the gradient reaches the input without attenuation, which makes training stable at large depth and reduces the need for warmup; a final LayerNorm compensates for the absence of output normalization. The current standard.

• Absolute positional encodings. Attention alone ignores the order of the tokens (6.4), so the original transformer adds to the embedding x_m of the token at position m a vector p_m \in \mathbb{R}^d. The sinusoidal code of Vaswani et al. is p_{m,2i} = \sin(m\,\omega_i), \qquad p_{m,2i+1} = \cos(m\,\omega_i), \qquad \omega_i = 10000^{-2i/d}. Each pair of coordinates turns at its own frequency, so p_{m+k} is a fixed rotation of p_m, the same for every m: an offset is a linear function of the code.

• Learned positional encodings. A table P \in \mathbb{R}^{n_{\max} \times d} is trained with the rest of the model (BERT, GPT-2, ViT). It adapts to the data but has no vector for positions beyond n_{\max}.

• RoPE (rotary position embedding, Su et al., 2021): instead of adding a positional code, one rotates each pair of coordinates (q_{2i}, q_{2i+1}) by an angle m\,\theta_i depending on the position m of the token (and on i), with \theta_i = b^{-2i/d}.

  Relative character, in three lines (the full derivation is theorem 2.3 of the LLMs module): planar rotations commute and R(m)^\top R(n) = R(n - m), so \langle R(m)\, q,\; R(n)\, k \rangle = q^\top R(m)^\top R(n)\, k = q^\top R(n-m)\, k, the score depends only on the gap n - m: position is encoded within the interaction — this is the encoding of modern LLMs. The rotation is defined at every position, but a model generalizes poorly beyond its training context; extending the window rescales the angles (interpolation, YaRN), usually with a short fine-tuning (Advanced Topics in LLMs, ch. 3).

6.3. Costs and efficient attention

• Complexity: \mathcal{O}(n^2 d) time and \mathcal{O}(n^2) memory (QK^\top matrix of size n \times n per head) — the LLM bottleneck. The MLP is \mathcal{O}(n d^2): attention dominates as soon as n \gtrsim d.

• Where the time goes: the GPU memory hierarchy. A GPU has two memories that matter here. HBM is the large “GPU RAM” (40–80 GB on an A100, about 1.5–2 TB/s). SRAM is the small on-chip memory of each streaming multiprocessor (192 KB on each of the 108 of an A100, about 20 MB in all, roughly ten times faster).

  An A100 computes about 300 TFLOP/s in bf16, so an operation that does fewer than about 150 FLOPs per byte read from HBM waits for memory: it is memory-bound. Softmax, masking and dropout do a few FLOPs per byte, and standard attention writes the n \times n scores to HBM and reads them back, \Theta(n^2) traffic. This paragraph is the program’s reference for the cost of FlashAttention; the P2 variant, ch. 6 develops the roofline model.

• FlashAttention (Dao et al., 2022): exact, but IO-aware — it tiles the computation of \mathrm{softmax}(QK^\top)V into blocks loaded into SRAM (the GPU’s fast memory) and never writes the n \times n matrix to HBM, maintaining online log-sum-exp statistics (proposition 0.3 makes the softmax incremental: blocks are merged with m and the partial sum).

  Result: same FLOPs, \mathcal{O}(n) memory, and \Theta(n^2 d^2 / M) HBM accesses for an SRAM of size M instead of \Theta(n^2) (Dao et al., theorem 2), hence ×2 to ×4 speed in practice.

• Approximations: sparse, linear, sliding window (cf. LLMs module, M2).

6.4. A complete block in PyTorch

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    """A pre-norm transformer block: multi-head attention (causal by default) + MLP."""
    def __init__(self, d, n_heads, p_drop=0.1, causal=True):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.n_heads, self.d_head, self.causal = n_heads, d // n_heads, causal
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
        self.drop = nn.Dropout(p_drop)

    def forward(self, x):                          # x: (B, n, d)
        B, n, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).chunk(3, -1)
        # multi-head: (B, n, d) -> (B, n, h, d_head) -> (B, h, n, d_head)
        def split(t): return t.view(B, n, self.n_heads, self.d_head).transpose(1, 2)
        q, k, v = split(q), split(k), split(v)
        mask = (torch.tril(torch.ones(n, n, dtype=torch.bool, device=x.device))  # causal mask:
                if self.causal else None)                                     # True = may attend
        att = F.scaled_dot_product_attention(q, k, v,            # softmax(QK^T/sqrt(d_head))V
                                             attn_mask=mask, dropout_p=0.0)
        att = att.transpose(1, 2).reshape(B, n, d)               # merge the heads back
        x = x + self.drop(self.proj(att))                        # residual 1 (pre-norm)
        x = x + self.drop(self.mlp(self.ln2(x)))                 # residual 2
        return x

torch.manual_seed(0)
x = torch.randn(2, 16, 128)
model = nn.Sequential(*[Block(128, 4) for _ in range(4)])
print(model(x).shape)                              # [2, 16, 128]

# Without positions (and without the causal mask), self-attention is permutation-
# equivariant: permuting the tokens permutes the outputs. Positions are added to x (6.2).
max_n, d = 512, 128
pos = torch.arange(max_n).unsqueeze(1)                        # (max_n, 1)
freq = 10000 ** (-torch.arange(0, d, 2) / d)                  # (d/2,) frequencies omega_i
pe = torch.zeros(max_n, d)
pe[:, 0::2], pe[:, 1::2] = torch.sin(pos * freq), torch.cos(pos * freq)   # sinusoidal
learned = nn.Embedding(max_n, d)                              # or a learned table
n = x.shape[1]
print(model(x + pe[:n]).shape, model(x + learned(torch.arange(n))).shape)

• Position: without positions, self-attention is permutation-equivariant: permuting the tokens permutes the outputs in the same way (proved as proposition 4.4 of Advanced DL). Invariance only comes after a symmetric pooling, such as a mean over tokens. To use the order, one adds positional encodings (sinusoidal, learned, or RoPE, 6.2) — and on graphs: the analogue is the problem of positional/structural encodings, at the heart of chapter 10 and of the Advanced GNN module (M2);
• MLP: two layers with GELU, expansion ratio ~4 — this is where most of the parameters live (~2/3);
• Cross-attention (encoder–decoder models: the original transformer, T5): a second attention sub-block in the decoder takes its queries from the decoder states Y and its keys and values from the encoder output H, \mathrm{CrossAtt}(Y, H) = \mathrm{softmax}\Big(\frac{(YW_Q)(HW_K)^\top}{\sqrt{d_k}}\Big)\, HW_V, one output row per target token and no mask on the source. In code it is F.scaled_dot_product_attention(q_from_Y, k_from_H, v_from_H). The same operation injects a text prompt into an image diffusion model (9.4).
• Causal mask: forbid j > i for autoregression — exactly the factorization \mathbb{P}(x) = \prod_t \mathbb{P}(x_t \mid x_{<t}) of chapter 9 and of the causal language models of section 6.5.

6.5. Modern NLP: tokens, embeddings and pretraining objectives

The official syllabus lists “modern natural language processing” and “embeddings”. A transformer consumes vectors, not text: this section explains how a sentence becomes a sequence of vectors, and what a language model is trained to predict before any task-specific adaptation (chapter 7).

import torch, torch.nn.functional as F
vocab, n = 50_000, 4                            # |V| tokens, n positions
logits = torch.randn(n, vocab)                  # one row of scores per position
targets = torch.randint(0, vocab, (n,))         # the next tokens
loss = F.cross_entropy(logits, targets)          # average NLL over positions
print(loss.item(), torch.exp(loss).item())       # loss in nats, perplexity

6.6. Transformers beyond text: the vision transformer (ViT)

A transformer only needs a sequence of vectors. The vision transformer of Dosovitskiy et al. (2021) turns an image into such a sequence by cutting it into patches, and then uses the encoder of 6.4 unchanged, without the causal mask.

import torch, torch.nn as nn

B, C, H, W, P, d = 8, 3, 224, 224, 16, 768
x = torch.randn(B, C, H, W)
patchify = nn.Conv2d(C, d, kernel_size=P, stride=P)      # = one shared linear map per patch
tok = patchify(x).flatten(2).transpose(1, 2)               # (B, 196, 768)
cls = nn.Parameter(torch.zeros(1, 1, d))                   # [CLS] token
pos = nn.Parameter(torch.zeros(1, 1 + tok.shape[1], d))    # learned positions
z = torch.cat([cls.expand(B, -1, -1), tok], 1) + pos       # (B, 197, 768): ready for the blocks
print(z.shape)

7. Transfer learning and fine-tuning

Training “from scratch” has become the exception: the dominant practice consists in starting from a model pre-trained on immense corpora (ImageNet for vision, terabytes of text for LLMs) and adapting it. This chapter gives the framework and the vocabulary; the LLMs module (M2) applies it at scale.

7.1. The three adaptation regimes

• Frozen features / linear probing: freeze the whole backbone, train a linear classifier (or a small head) on the pre-trained representations. A valuable diagnostic: feature quality is measured by the performance of a simple softmax on top (exercise 12, step (d)).
• Full fine-tuning: retrain all the weights with a smaller step (typically 10 to 100× smaller than pre-training) — the risk of catastrophic forgetting (loss of pre-trained knowledge) is managed by a decaying step and few epochs.
• Parameter-efficient fine-tuning (PEFT): train only a small fraction of new parameters, the rest staying frozen — all the more relevant as models grow.

def set_regime(model, regime):          # a torchvision resnet18 whose model.fc = nn.Linear(512, 10)
    for p in model.parameters():
        p.requires_grad = (regime == "full")
    if regime == "partial":
        for p in model.layer4.parameters():
            p.requires_grad = True
    for p in model.fc.parameters():     # the new head is always trained
        p.requires_grad = True
    return sum(p.numel() for p in model.parameters() if p.requires_grad)   # 5130 / 8398858 / 11181642

7.2. Domain adaptation, in a nutshell

When the target has no labels (labeled source domain → unlabeled target domain), domain adaptation aligns the distributions: adversarial (a discriminator tries to guess the domain from the features: they must become non-discriminative), or via distribution distances (MMD, Wasserstein distances — a preview: they are defined in the Optimal Transport module and in Graph ML 9.6). The guiding idea: learn representations invariant to the domain but discriminative for the task.

7.3. LoRA: a mathematical sketch

import torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base                       # W0 frozen
        for p in self.base.parameters(): p.requires_grad = False
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)
# trainable params: r*(in+out) instead of in*out

8. Training in practice

A network that does not learn almost always has a mundane bug. This chapter is the diagnostic protocol we will apply in the labs, in the order in which it must be applied.

8.1. The training recipe, in order

1. Shape sanity check: one forward pass without errors, the dimensions of every activation verified (rule of ch. 0: gradients have the shape of their variable).
2. Overfit a single batch: 32 examples, train until the loss is \approx 0. If the model cannot do it, the bug is structural (data, loss, initialization, wild learning rate) — no point training longer. A model unable to memorize 32 examples is broken or too small; this is THE test that avoids weeks of silent error.
3. LR range test (Smith, 2015): \eta increasing geometrically from 10^{-7} to 10^{0}, plot the loss: plateau → descent → divergence. Take the \eta at the dip (or slightly before): it sets AdamW/SGD in one epoch.
4. Small full training run on ~10% of the data, with the train/val curves: check that the dynamics are healthy before scaling up.
5. Scaling + regularization: larger batch (or gradient accumulation), mixed precision, data augmentation, dropout, weight decay — in that order of payoff (ch. 3).
6. Checkpointing: save regularly (weights + optimizer state + current step + data generator seed), to resume exactly.

8.2. Monitoring: reading your curves

Tools: tensorboard/wandb; systematically log the loss, accuracy, gradient norm, weight norm, the fraction of zero activations (dying ReLU), and the activation histograms of a few layers.

8.3. GPU: the levers that matter

• Mixed precision (bf16 preferably): ~2× throughput, memory divided by ~2 — the first reflex (ch. 2).
• Batch and accumulation: saturate the GPU (measure utilization); if memory runs out, accumulate small batches; adjust the BatchNorm statistics accordingly.
• DataLoader: workers > 0, pin_memory, prefetch — a GPU waiting for data is a wasted GPU (CPU preprocessing is often the real bottleneck).
• torch.compile, fused kernels (FlashAttention, ch. 6), avoid CPU→GPU synchronizations inside the loop (the .item() / prints at each step).

The loop below puts the first and last of these levers together with experiment tracking. Autocast runs the matrix products in 16 bits and keeps reductions in float32; the gradient scaler is needed only for float16, whose narrow exponent range lets small gradients underflow (ch. 2.2). It runs as is on a CPU, in bfloat16.

import torch, torch.nn as nn
from torch.utils.tensorboard import SummaryWriter        # pip install tensorboard

device = "cuda" if torch.cuda.is_available() else "cpu"
model = nn.Sequential(nn.Linear(32, 256), nn.GELU(), nn.Linear(256, 10)).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)
use_fp16 = device == "cuda" and not torch.cuda.is_bf16_supported()  # old GPUs: fp16
dtype = torch.float16 if use_fp16 else torch.bfloat16
scaler = torch.amp.GradScaler(device, enabled=use_fp16)  # loss scaling, needed for fp16 only
writer = SummaryWriter("runs/exp1")                      # then: tensorboard --logdir runs

for step in range(100):
    x = torch.randn(64, 32, device=device)
    y = torch.randint(0, 10, (64,), device=device)
    with torch.autocast(device_type=device, dtype=dtype):  # matmuls in 16 bits, reductions in fp32
        loss = nn.functional.cross_entropy(model(x), y)
    opt.zero_grad(set_to_none=True)
    scaler.scale(loss).backward()          # multiply the loss so that small fp16 gradients survive
    scaler.unscale_(opt)                   # true gradients again, before clipping
    gnorm = nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(opt)                       # skips the step if an inf or NaN appeared
    scaler.update()
    if step % 10 == 0:                     # .item() waits for the GPU: log every k steps only
        writer.add_scalar("train/loss", loss.item(), step)
        writer.add_scalar("train/grad_norm", gnorm.item(), step)
writer.close()

8.4. Reproducibility

import torch, numpy as np, random

def seed_everything(seed=42):
    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

seed_everything(42)
torch.backends.cudnn.deterministic = True    # determinism (speed cost)
torch.backends.cudnn.benchmark = False       # disable non-deterministic autotuning

Set the seeds (python/numpy/torch/cuda), the data shuffling, and record the library versions; some CUDA kernels remain non-deterministic by nature (atomics); aim for reproducibility up to a tolerance rather than bit-for-bit, and document configurations and seeds in every experiment. Reproducibility is also a debugging tool: a random bug you know how to reproduce is half fixed.

9. Generative models: AE, VAE, GAN

The networks of the previous chapters predict a label y from an input x; a generative model learns the distribution of x itself, so that it can produce new images or sentences. This chapter compares three ways of doing so — autoencoders, variational autoencoders and generative adversarial networks — and derives the training objective of each.

9.1. Deterministic autoencoder and its pathology

An autoencoder compresses x into a code z = E_\phi(x) and reconstructs \hat x = D_\theta(z), trained on \|x - \hat x\|^2. Useful for representation learning though it is, it is not a generative model: the latent space has no imposed structure — sampling z at random gives aberrant reconstructions. The VAE corrects precisely this.

9.2. VAE: the ELBO fully derived

Sections 9.2 and 9.3 are the site’s reference derivations of the ELBO and of the GAN equilibrium: the Multimodal Generative AI module and others recall them and link here.

9.3. GAN: the minimax game and its equilibrium

9.4. Diffusion in one section (self-study)

A VAE samples in one decoder pass but blurs; a GAN is sharp but unstable. Diffusion models (Sohl-Dickstein et al., 2015; Ho, Jain & Abbeel, 2020) take a third route: a fixed process turns data into noise in many small Gaussian steps, and a network learns to undo one step at a time, with a plain regression as its loss.

This section gives the formulas that the rest of the program uses: discrete diffusion on graphs (Graph ML 9.5), Geometric Data ch. 7–8 and Advanced DL 10.4.

From the ELBO to a regression. The reverse model is a Gaussian chain started from p(x_T) = \mathcal{N}(0, I): p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\big(\mu_\theta(x_t, t),\ \sigma_t^2 I\big). Treating x_{1:T} as the latent variables of a VAE, theorem 9.2 gives \log p_\theta(x_0) \ge \mathrm{ELBO}, and the negative ELBO splits into one term per step: -\mathrm{ELBO} = L_T + \sum_{t=2}^{T} L_{t-1} + L_0, \qquad L_{t-1} = \mathbb{E}_q\, \mathrm{KL}\big(q(x_{t-1} \mid x_t, x_0)\,\big\|\,p_\theta(x_{t-1} \mid x_t)\big). L_T has no parameter, since the prior is fixed, and L_0 is a decoder term.

Ho et al. drop the weight of proposition 9.9: the samples get better, at the price of a looser bound.

Classifier-free guidance (Ho & Salimans, 2022). To condition on a prompt c (a class, a caption), train one network \varepsilon_\theta(x_t, t, c) and replace c by an empty token \varnothing on about 10% of the examples. At sampling time, extrapolate from the unconditional prediction toward the conditional one: \tilde\varepsilon_w = \varepsilon_\theta(x_t, \varnothing) + w\,\big(\varepsilon_\theta(x_t, c) - \varepsilon_\theta(x_t, \varnothing)\big). With w = 1 this is the plain conditional model; w > 1 over-guides, giving samples more faithful to the prompt and less diverse. By proposition 9.11, it samples with the score \nabla \log p_t(x) + w\, \nabla \log p_t(c \mid x). This is the site’s convention for the guidance weight (notation page); the original paper writes w' = w - 1.

The toy below trains \varepsilon_\theta on a mixture of eight Gaussians in the plane and samples with algorithm 9.10. It runs on a CPU in under a minute.

import math, torch, torch.nn as nn

torch.manual_seed(0)
def data(m):                                          # 8 Gaussians on a circle of radius 2
    k = torch.randint(0, 8, (m,)) * (math.pi / 4)
    return 2 * torch.stack([k.cos(), k.sin()], 1) + 0.1 * torch.randn(m, 2)

T = 100
beta = torch.linspace(1e-4, 0.2, T)                   # beta_1, ..., beta_T (index t - 1)
alpha, abar = 1 - beta, torch.cumprod(1 - beta, 0)    # abar_t = alpha_1 ... alpha_t
net = nn.Sequential(nn.Linear(2 + 16, 256), nn.SiLU(), nn.Linear(256, 256), nn.SiLU(),
                    nn.Linear(256, 2))
def eps_theta(x, t):                                  # the step t enters as sin/cos features
    f = t.float().unsqueeze(1) / T * torch.arange(1, 9) * math.pi
    return net(torch.cat([x, f.sin(), f.cos()], 1))
opt = torch.optim.Adam(net.parameters(), lr=1e-3)

for step in range(5000):                              # training: regress the noise
    x0, t = data(256), torch.randint(0, T, (256,))
    eps = torch.randn_like(x0)
    xt = abar[t, None].sqrt() * x0 + (1 - abar[t, None]).sqrt() * eps   # prop. 9.8
    loss = ((eps - eps_theta(xt, t)) ** 2).mean()     # L_simple
    opt.zero_grad(); loss.backward(); opt.step()

with torch.no_grad():                                 # ancestral sampling (algorithm 9.10)
    x = torch.randn(2000, 2)
    for t in reversed(range(T)):
        e = eps_theta(x, torch.full((2000,), t))
        x = (x - beta[t] / (1 - abar[t]).sqrt() * e) / alpha[t].sqrt()
        if t > 0:
            x = x + beta[t].sqrt() * torch.randn_like(x)   # sigma_t^2 = beta_t
ang = torch.arange(8) * (math.pi / 4)
dist = torch.cdist(x, 2 * torch.stack([ang.cos(), ang.sin()], 1)).min(1)
print(f"{(dist.values < 0.4).float().mean():.0%} of samples within 0.4 of a mode, "
      f"{dist.indices.unique().numel()} modes out of 8")   # about 90%, 8 modes

From the VAE decoder to text-to-image. Latent diffusion (Rombach et al., 2022) runs the whole process of this section in the latent space of a pretrained autoencoder trained with a light KL penalty, as in 9.2. A 512 \times 512 \times 3 image becomes a 64 \times 64 \times 4 latent, the denoiser works there, and the decoder maps the final latent back to pixels.

The prompt enters through the cross-attention of 6.4, with keys and values computed by a frozen text encoder (in Stable Diffusion, the public release of latent diffusion, the CLIP text encoder of Radford et al., 2021), and classifier-free guidance sets how strongly it is followed. The families of this chapter thus end up in one pipeline, built in full in the Multimodal Generative AI module.

10. Research at DaSciM (Vazirgiannis, Lutzeyer, Zhu)

This chapter connects the course to the current research of three researchers of the DaSciM group: Michalis Vazirgiannis (responsible of the course on the sheet), Johannes Lutzeyer (in its teaching team) and Ye Zhu (not listed on the 2026–2027 sheet). All three topics are rooted in the previous chapters. The arXiv references are in the bibliography.

10.1. Michalis Vazirgiannis: from the “Graph of Words” to textual GNNs

10.2. Johannes Lutzeyer: where are the parameters? Expressivity and position

10.3. Ye Zhu: steering diffusion models

Exercises

1. ★ Backprop on paper. Redo example 2.2 by hand (a scalar tanh network). Then write the backward pass of a two-layer ReLU network with the recursion of theorem 2.1, and check it numerically, as in the NumPy code of section 2.1, by central finite differences \frac{\mathcal{L}(\theta + \epsilon e_j) - \mathcal{L}(\theta - \epsilon e_j)}{2\epsilon} on a small network.

2. ★ Activation. Show that if \sigma is linear, an MLP of arbitrary depth represents exactly the affine maps. Deduce why linear cascades are useless.

3. ★ 1D attention by hand. For n = 3 tokens, d_k = 2, compute by hand QK^\top/\sqrt{d_k}, the row-wise softmax, and the result \mathrm{softmax}(\cdot)V. Implement it, compare, and check that the rows of the attention matrix sum to 1.

4. ★ Permutation-equivariance. Take the block of 6.4 with causal=False, in eval() mode and without positional encoding. Check experimentally that permuting the tokens permutes the output rows in the same way (equivariance), that the mean over the tokens is then invariant, and that adding a positional encoding breaks both properties. The proof is proposition 4.4 of Advanced DL.

5. ★ Quadratic SGD. Take A = \mathrm{diag}(4, 1), b = 0, \theta_0 = (1, 1).

   Write the sequence (\theta_k) for \eta = 1/2 then \eta = 1, and verify theorem 3.1 (per-mode contraction, bound 2/L).

6. ★ Convolution arithmetic. A 32 \times 32 input, three successive convs (5×5, stride 2, padding 2; 3×3, stride 1, padding 1; 3×3, stride 1, padding 1, dilation 2): intermediate output sizes and final receptive field (proposition 4.4).

7. ★ LoRA. Count the trainable parameters of a GPT-like model with 12 blocks (d = 768, attention + MLP) adapted by LoRA r = 8 on the attention projections only, and compare with full fine-tuning.

8. ★★ He initialization. Recover the factor 2 of He initialization: for h = \mathrm{ReLU}(a) with a symmetric zero-mean, compute \mathbb{E}[h^2] then the variance of the next layer.

9. ★★ Adam bias correction. Prove proposition 3.5 for v_t (second moment), then compute the update \hat m_1/\sqrt{\hat v_1} at the first step and explain why the warmup protects.

10. ★★ Gaussian KL. Derive the formula of proposition 9.4 in dimension 1, plot \mathrm{KL}(\mathcal{N}(\mu, \sigma^2) \| \mathcal{N}(0,1)) as a function of \mu and \sigma, and locate the minimum.

11. ★★ Gaussian GAN. For p_{\mathrm{data}} = \mathcal{N}(0, 1) and p_G = \mathcal{N}(\mu, \sigma^2), write the logit of D^\star (difference of the log-densities) and check that it is a trinomial in x. Where does the equilibrium lie?

12. ★★★ Project. In PyTorch: classify MNIST with (a) an MLP, (b) a small CNN, (c) a mini-transformer on patches (16 patches of 7×7, a 4 × 4 grid, as in the ViT of 6.6). Plot the train/val curves, compare the parameter counts and accuracy, comment with the bias-variance decomposition, then apply the protocol of chapter 8 (overfitting a batch, LR range test). (d) Train the CNN of (b) on the digits 0–4, freeze it, and train a linear classifier on its features for the digits 5–9 (linear probing, section 7.1); compare with the CNN trained from scratch on 5–9.

Projects and assessment

The SynapseS sheet gives the grade format (numeric out of 20, with a US letter grade), but “the section for the LLGA track is empty”: no assessment form, resit rule, pass threshold or ECTS is displayed for LLGA, and no breakdown of assessments is given for any degree. The subjects, the exam and the rubric below are therefore proposals of these notes, for instance half project and half written exam. They exercise objectives O1 (ML and DL foundations), O2 (generative modelling) and O3 (large language models), with O7 (hands-on practice) throughout.

Group projects 1 and 3 are the Deep Learning building blocks of two Year 1 end-of-year projects of the Graph × LLM route: a GraphRAG question-answering system (Y1.1) and the generation of molecules from text (Y1.4).

Solo projects

1. [Solo · ≈ 20 h] Backprop from scratch and an initialization study. Does a hand-written backward pass match autograd, and which initializations keep the signal alive through depth?
   • Implement: extend the NumPy MLP of 2.1 to any depth and to sigmoid, tanh, ReLU and GELU, and check it against PyTorch autograd and a float64 central-difference checker. Measure activation and gradient variances layer by layer in a 20-layer network for zero, Xavier and He initializations (1.3), and reproduce XOR (example 1.4).
   • Data: synthetic Gaussian inputs, XOR, and MNIST (LeCun et al., 1998).
   • Deliverables: code with tests, a 4-page report with one variance-versus-depth plot per activation, a 10-minute talk.
   • Chapters: ch. 0, 1, 2.
   • Success: a relative gradient error below 10^{-7} on every tensor. With width 256 and ReLU, He keeps the activation variance after 20 layers within a factor 2 of the input, while Xavier divides it by about 2^{20} \approx 10^6, as proposition 1.6 predicts.
2. [Solo · ≈ 25 h] Paper study: decoupled weight decay (Loshchilov & Hutter, ICLR 2019). Is Adam with an L2 penalty really worse than AdamW, and does the best weight decay depend less on the learning rate once decoupled?
   • Warm-up: check theorem 3.1 and proposition 3.3 on quadratics with \kappa \in \{10, 100, 1000\}. The number of iterations to reach an error of 10^{-6} should grow like \kappa for gradient descent and like \sqrt\kappa for tuned momentum.
   • Implement: train a ResNet-20 (He et al., 2016) on CIFAR-10 (Krizhevsky, 2009) with Adam + L2 and with AdamW, over a 4 × 4 grid of learning rates and weight decays, with warmup and a cosine schedule (3.4). A 30-epoch run takes a few minutes on a free GPU tier.
   • Paper: I. Loshchilov, F. Hutter, Decoupled Weight Decay Regularization, ICLR 2019 (its figures 1 and 2: test error over a grid of learning rates and weight decays).
   • Deliverables: code, the two heatmaps, a 5-page report, a 10-minute talk.
   • Chapters: ch. 3, 4, 8.
   • Success: log-log slopes of 1 and 1/2 on the quadratics, within 10%; heatmaps that reproduce the paper’s finding (a basin less tilted for AdamW), or a report that explains why not.

Group projects

1. [Group of 3 · ≈ 30 h each] Mini-GPT: tokenizer, pretraining and LoRA adaptation. What does each ingredient of a small decoder-only transformer buy, and how close does LoRA come to full fine-tuning?
   • Implement: a BPE tokenizer (definition 6.3); a pre-norm causal transformer with RoPE (6.2–6.4) of about 10 M parameters, pretrained on TinyStories (Eldan & Li, 2023); ablations of the \sqrt{d_k} scaling and of pre- versus post-norm; then adaptation to SST-2 sentiment classification (Socher et al., 2013) by linear probe, LoRA with r \in \{4, 8, 16\} and full fine-tuning (ch. 7).
   • Split: member A builds the tokenizer, the data pipeline and the perplexity evaluation; member B the model, the pretraining and the ablations; member C the adaptation and the attention maps drawn as graphs (6.3).
   • Deliverables: a repository, a 6-page report, a 15-minute talk.
   • Chapters: ch. 6, 7, 8; it leads to NLP, ch. 7 and Large Language Models, section 4.3.
   • Success: a validation perplexity far below that of a unigram model; LoRA with r = 8 within 2 points of full fine-tuning with under 2% of the weights trained, or an explanation of the gap.
2. [Group of 3 · ≈ 30 h each] Inductive biases on images: MLP, CNN, ViT and a graph network. How much data does each architecture need, and what does each one build in? This extends exercise 12 and bridges to Graph ML.
   • Implement: on CIFAR-10, an MLP, a small ResNet (ch. 4), a ViT with 4 × 4 patches (6.6), and a GNN on SLIC superpixel graphs (Achanta et al., 2012). Train each on 1k, 5k and 50k images with the protocol of ch. 8, with tracked and seeded runs (8.4).
   • Paper: the CIFAR10 superpixel benchmark of V. P. Dwivedi et al., Benchmarking Graph Neural Networks, JMLR 2023, whose GNN results the group reproduces for one model.
   • Split: member A the MLP and the CNN; member B the ViT; member C the superpixel graphs and the GNN (Graph ML, ch. 5); shared training and logging code.
   • Deliverables: a repository, a 6-page report, a 15-minute talk. Chapters: ch. 4, 6, 8, and Graph ML ch. 5.
   • Success: a table of parameters, accuracy and data efficiency for the four models over three seeds; the reproduced GNN within the range reported by Dwivedi et al. for its size; a paragraph that explains the ranking at 1k images by the inductive biases of 4.1 and 6.6.
3. [Group of 2 · ≈ 25 h each] VAE, GAN and diffusion: theory checks and failure modes. Do the generative families reach their theoretical optimum where it is known, and how do they fail?
   • Implement: a linear autoencoder compared with PCA (9.1); a VAE with the reparameterization trick (9.2); a GAN on the eight-Gaussian ring and on MNIST, with the Gaussian check of theorem 9.6 and exercise 11; the diffusion toy of 9.4 on the same ring.
   • Split: member A the autoencoders and the VAE (ELBO curves, posterior collapse); member B the GAN and diffusion (mode counts, samples, latent interpolations).
   • Data: synthetic rings and MNIST. Deliverables: code, a 5-page report, a 10-minute talk.
   • Chapters: ch. 9; it prepares the Multimodal Generative AI module.
   • Success: principal angles between the autoencoder subspace and the PCA subspace below 1°; the number of ring modes covered by the GAN and by diffusion over five seeds; a fitted logit of D^\star that matches the trinomial of exercise 11.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet allowed, no electronic device; four exercises, 20 points.

1. Question (5 points). Scaled dot-product attention. (a) Write single-head attention for n tokens of width d and give its cost. (b) If the components of a query and a key are i.i.d., centred, with variance 1, show that their dot product has variance d_k, and say what goes wrong without the factor 1/\sqrt{d_k}. (c) Without positions and without mask, show that self-attention is permutation-equivariant. How does one obtain an invariant output?

   Expected:

   • (a) The formula of definition 6.1, with time \mathcal{O}(n^2 d) and memory \mathcal{O}(n^2) (1 point).
   • (b) The proof of proposition 6.2; without the factor, the softmax saturates into a one-hot vector and its gradient vanishes (2 points).
   • (c) For a permutation matrix \Pi, the scores become \Pi QK^\top \Pi^\top, the row softmax commutes with this re-indexing, and the output becomes \Pi times the old one; a mean over the tokens is invariant (2 points).

2. Question (6 points). Take f(w_1, w_2) = \tfrac12\big(w_2\,\mathrm{ReLU}(w_1 x) - t\big)^2, \qquad x = 2,\ w_1 = 0.5,\ w_2 = -1,\ t = 1. Then: (a) compute the forward values. (b) Compute both partial derivatives by backpropagation. (c) Take one gradient step with \eta = 0.1 and give the new loss. (d) A 3 × 3 convolution from 3 to 16 channels, stride 2, padding 1, reads a 64 × 64 image: give the output size and the number of parameters.

   Expected:

   • (a) a = w_1 x = 1, h = 1, y = -1, loss 2 (1 point).
   • (b) \partial\mathcal{L}/\partial y = y - t = -2, so \partial\mathcal{L}/\partial w_2 = -2 \cdot h = -2 and \partial\mathcal{L}/\partial w_1 = (-2)(-1)(1)(2) = 4 (2 points).
   • (c) w_1 = 0.1 and w_2 = -0.8; the new loss is \tfrac12(1.16)^2 = 0.6728 (1.5 points).
   • (d) Theorem 4.2 gives \lfloor 63/2 \rfloor + 1 = 32, so a 32 \times 32 output, with 3 \cdot 9 \cdot 16 + 16 = 448 parameters (1.5 points).

3. Question (5 points). For a fixed generator of density p_G, (a) prove that the discriminator maximizing V(G, D) is D^\star = p_{\mathrm{data}}/(p_{\mathrm{data}} + p_G); (b) deduce V(G, D^\star) = -\log 4 + 2\,\mathrm{JSD}(p_{\mathrm{data}} \,\|\, p_G), and say for which G the minimum is reached.

   Expected: (a) pointwise maximization of a \log D + b \log(1 - D), concave, with zero derivative at D = a/(a+b) (2 points). (b) Substitute, introduce the mixture m = \tfrac12(p_{\mathrm{data}} + p_G) and recognize the two KLs toward m (2 points); the JSD is non-negative and vanishes only when p_G = p_{\mathrm{data}} (1 point).

4. Question (4 points). Gradient descent on f(\theta) = \tfrac12 \theta^\top A \theta with A = \mathrm{diag}(10, 1). (a) For which step sizes does it converge from every start? (b) Give the best constant step and its rate. (c) About how many iterations divide the error by 10^3?

   Expected: (a) 0 < \eta < 2/L = 0.2 (1 point). (b) \eta^\star = 2/(L + \mu) = 2/11 \approx 0.18, rate \rho = (\kappa - 1)/(\kappa + 1) = 9/11 \approx 0.82, by theorem 3.1 (2 points). (c) k \ge \ln 1000 / \ln(11/9) \approx 34.4, so 35 iterations (1 point).

Grading rubric (proposal)

References

Textbooks and courses

• I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016 — free online; chapters 6-8 for this module.
• A. Zhang, Z. Lipton, M. Li, A. Smola, Dive into Deep Learning, 2023 — d2l.ai: code-first interactive course (PyTorch), very close to the labs.
• A. Karpathy, Neural Networks: Zero to Hero — building a GPT from scratch. Its lecture Let’s build GPT: from scratch, in code, spelled out (2023) gives the reading “attention = communication between the nodes of a directed graph” that grounds the LLGA bridge of section 6.3.
• S. Raschka, Machine Learning with PyTorch and Scikit-Learn, Packt, 2022 — for practice.

Founding papers (course chronology)

• F. Rosenblatt, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Psychological Review 65(6), 1958; A. Novikoff, On Convergence Proofs on Perceptrons, Symposium on the Mathematical Theory of Automata, 1962 — section 1.1.
• P. Baldi, K. Hornik, Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima, Neural Networks 2(1), 1989 — the linear autoencoder of section 9.1.
• G. Cybenko, Approximation by Superpositions of a Sigmoidal Function, Mathematics of Control, Signals and Systems, 1989 — theorem 1.5; M. Leshno, V. Lin, A. Pinkus, S. Schocken, Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function, Neural Networks 6(6), 1993 — the non-polynomial case.
• X. Glorot, Y. Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, AISTATS 2010 — Xavier initialization.
• A. Krizhevsky, I. Sutskever, G. Hinton, ImageNet Classification with Deep CNNs, NeurIPS 2012 (AlexNet); K. Simonyan, A. Zisserman, Very Deep Convolutional Networks (VGG), ICLR 2015.
• D. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 — algorithm 3.4, derived step by step.
• K. He et al., Delving Deep into Rectifiers (He init), ICCV 2015; Deep Residual Learning for Image Recognition (ResNet), CVPR 2016.
• S. Ioffe, C. Szegedy, Batch Normalization, ICML 2015; J. Ba, J. Kiros, G. Hinton, Layer Normalization, 2016; S. Santurkar et al., How Does Batch Normalization Help Optimization?, NeurIPS 2018.
• R. Pascanu, T. Mikolov, Y. Bengio, On the Difficulty of Training Recurrent Neural Networks, ICML 2013 — theorem 5.2.
• N. Srivastava et al., Dropout, JMLR 15, 2014; I. Loshchilov, F. Hutter, Decoupled Weight Decay Regularization (AdamW), ICLR 2019.
• D. Hendrycks, K. Gimpel, Gaussian Error Linear Units (GELU), 2016.
• T. Mikolov et al., Distributed Representations of Words and Phrases and their Compositionality (word2vec), NeurIPS 2013; O. Levy, Y. Goldberg, Neural Word Embedding as Implicit Matrix Factorization, NeurIPS 2014 — section 6.5.
• O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015 — section 5.3.
• R. Sennrich, B. Haddow, A. Birch, Neural Machine Translation of Rare Words with Subword Units (BPE), ACL 2016 — section 6.5.
• A. Vaswani et al., Attention Is All You Need, NeurIPS 2017 — the paper to read line by line.
• J. Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019 — masked language modelling, section 6.5.
• J. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021; T. Dao et al., FlashAttention, NeurIPS 2022.
• A. Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT), ICLR 2021 — section 6.6.
• I. Tenney, D. Das, E. Pavlick, BERT Rediscovers the Classical NLP Pipeline, ACL 2019 — section 7.1.
• D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR 2014; I. Goodfellow et al., Generative Adversarial Nets, NeurIPS 2014.
• J. Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics, ICML 2015; J. Ho, A. Jain, P. Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS 2020; Y. Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021; J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, 2022 — section 9.4.
• L. Dinh, J. Sohl-Dickstein, S. Bengio, Density Estimation using Real NVP, ICLR 2017; R. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022; A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), ICML 2021 — section 9.4.
• E. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022; A. Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL-IJCNLP 2021.
• L. Smith, Cyclical Learning Rates for Training Neural Networks, WACV 2017 (LR range test); T. Chen et al., Training Deep Nets with Sublinear Memory Cost, 2016 (checkpointing).

Research at DaSciM (Vazirgiannis, Lutzeyer, Zhu)

• F. Rousseau, M. Vazirgiannis, Graph-of-Word and TW-IDF: New Approach to Ad Hoc IR, CIKM 2013 — the “Graph of Words” of chapter 10; F. Rousseau, E. Kiagias, M. Vazirgiannis, Main Core Retention on Graph-of-Words for Single-Document Keyword Extraction, ECIR 2015.
• G. Nikolentzos, A. Tixier, M. Vazirgiannis, Message Passing Attention Networks for Document Understanding, AAAI 2020; for comparison, Y. Zhang, X. Yu, Z. Cui, S. Wu, Z. Wen, L. Wang, Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks, ACL 2020.
• G. Michel, G. Nikolentzos, M. Vazirgiannis, Path Neural Networks: Expressive and Accurate Graph Neural Networks, ICML 2023.
• J. Lutzeyer, C. Wu, M. Vazirgiannis, Sparsifying the Update Step in Graph Neural Networks, 2021.
• Y. Abbahaddou, J. Lutzeyer, M. Vazirgiannis, Graph Neural Networks on Discriminative Graphs of Words, NeurIPS 2023 (extended version).
• Z. Cui, J. Lutzeyer et al., SHAKE-GNN: Scalable Hierarchical Kirchhoff-Forest Graph Neural Network, 2025.
• Y. Zhu et al., Boundary Guided Learning-Free Semantic Control with Diffusion Models, NeurIPS 2023; Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation, ICLR 2023; Diffusion in Diffusion, ICLR 2024; The Silent Assistant: NoiseQuery as Implicit Guidance, ICCV 2025.