Deep Learning

Contents

• Introduction and overview
• 0. Foundations: the field, the PyTorch pipeline, autodiff, optimization and the transformer block
• 1. Counting: FLOPs, memory, scaling laws
• 2. Parallelism: data, tensor, pipeline
• 3. Communication: all-reduce and friends
• 4. Memory efficiency: checkpointing, ZeRO, mixed precision
• 5. Efficient inference
• 6. Real systems: roofline, GPU, and the complete budget of a 7B LLM
• 7. Stability, robustness, reinforcement learning and generative models
• 8. Theoretical complexity of training
• 9. Research readings: lower bounds and Lipschitz regularity
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with the sheet’s prerequisites — basics of machine learning and Python — who wants to learn deep learning together with its costs: the FLOPs, memory and communication of a training run or of a served model. Empirical first: each topic starts from a failure seen in practice — an out-of-memory error on a 7B model caused by a poor sharding choice, a speedup of only 1.2× despite 8 GPUs (Amdahl’s law) — and only then names the tools that explain it: complexity bounds, data, tensor and pipeline parallelism, quantization.

The module covers the P1 foundations in cost-oriented form, then adds what the lower bounds allow and how real GPUs behave.

What you will be able to do

• Write, debug and monitor a complete PyTorch training pipeline for a network of several million parameters.
• Explain automatic differentiation, universal approximation and what is (and is not) guaranteed in non-convex training.
• Build a transformer block from the multilayer perceptron (attention, positions, pre-LN, causal mask), and choose and justify an architecture for images (CNN, with its equivariance), sequences (RNN, LSTM) and text generation (causal transformer).
• Bound the sensitivity of a network to input perturbations, and explain adversarial examples and adversarial training.
• State the policy gradient theorem and the principles of deep Q-learning, and count what a deep RL run costs in rollouts and memory.
• Derive the training objectives of VAEs, GANs and diffusion models, and implement a diffusion sampler.
• Count the FLOPs and memory of a model, and connect these counts to scaling laws.
• Explain the theoretical complexity bounds of training and what they rule out.
• Choose a parallelism strategy — data, tensor, pipeline — and justify it quantitatively.
• Analyze collective communications (all-reduce and friends) and their effective cost.
• Reduce the memory footprint of a training run with checkpointing and ZeRO.
• Make inference efficient and know at what performance price.
• Read a roofline and predict the behavior of a real system, from GPU to 7B LLM.

Official objectives and syllabus (2026-2027 catalog)

Officially, the APM_52183_EP course unit offers an in-depth presentation of the practical and theoretical aspects of deep learning: on the one hand the mastery of Python / PyTorch implementation (implementation, debugging, visualization), on the other the analysis of foundations (automatic differentiation, universality results, non-convex optimization, equivariant models).

The stated objective is twofold: give an overview of the deep learning landscape and of its applications, in traditional fields and in new ones, and make students able to train by themselves a deep network of several million parameters, with PyTorch.

Prerequisites stated on the sheet (for the engineering degree and the exchange programmes): basics in Machine Learning and coding in Python.

The sheet lists a nine-item programme, in the order below; every item, and every theoretical topic of the description, is mapped to the section that teaches it.

Where this course fits in the program

One of the six options of the Period 2 “Choose 1” group, and the second way to satisfy the track rule “Deep Learning required once, in P1 or P2”: a student who did not take Deep Learning in Period 1 takes this course instead, as the AI-systems route of the home page does. It therefore assumes no earlier deep learning course.

Official prerequisites (stated for the engineering degree and the exchange programmes): basics in Machine Learning and coding in Python. In this programme they are covered by Machine Learning (Period 1) and by chapters 2–6 of the off-curriculum Refresher on neural networks. Chapter 0 builds everything else, from the training pipeline to the transformer block, before the cost chapters.

The other options of the same group (Optimization for AI, Multimodal Generative AI, …) cannot be taken alongside this course: the page cites their notes only as optional reading. It runs in parallel with the mandatory Reinforcement Learning, Graph ML and NLP courses, to which sections 7.2 and 7.3 and chapter 0 link.

Downstream, the module directly prepares ML Ops and LLM Engineering Principles (M2), where these cost accounts become deployment decisions, and Advanced Topics in Large Language Models (M2), whose large-scale analyses build on the same scaling laws and budgets.

Course outline

• Foundations: the field, the PyTorch pipeline, autodiff, optimization and the transformer block — the tasks of deep learning, universal approximation, a complete training script with its debugging checks, automatic differentiation, optimization for deep networks with its rates, and the transformer block built from the multilayer perceptron.
• Counting: FLOPs, memory, scaling laws — the module’s basic discipline, quantify before asserting, applied to the architectures of the official syllabus: CNNs and equivariant models, recurrent networks, transformers for text generation.
• Parallelism: data, tensor, pipeline — the three main ways of distributing a training run, and their trade-offs.
• Communication: all-reduce and friends — the collectives that connect the workers, and their share of total time.
• Memory efficiency: checkpointing, ZeRO — fitting models and states that do not fit.
• Efficient inference — cutting the cost of the trained model at the moment it serves.
• Real systems: roofline, GPU, 7B LLM — confronting theoretical accounts with hardware, on a textbook case.
• Stability, robustness, reinforcement learning and generative models — Lipschitz bounds and adversarial examples, deep Q-learning and policy gradients, VAEs, GANs and diffusion.
• Theoretical complexity of training — the lower bounds that say what no trick will ever beat, once the systems side is known.
• Research readings: lower bounds and Lipschitz regularity — the papers of K. Scaman and co-authors behind chapter 8 and section 7.1 (not the works of the official team), read with the module’s tools.
• Exercises — counts, bounds and parallelism choices to defend.
• Projects and assessment — solo and group subjects that measure what the course counts, a written-exam proposal and a grading rubric.
• References — the systems and theory papers underlying each chapter.

Session plan

36 h in class = 12 sessions of 3 h. The volume is the one of the SynapseS 2026–2027 sheet (36 h in person).

The sheet lists a nine-item programme without dates; the plan below follows its order — items 1–3 in sessions 1–3, items 4–6 in sessions 3–5, items 7–9 in sessions 9–11 — and interleaves this module’s systems chapters where they rely on the material just taught. The split into sessions is our proposal.

Level 1 = foundations (pipeline, autodiff, building blocks, first counts), 2 = core architectures, systems and methods, 3 = proof-heavy theory; the level never goes down from one session to the next.

Self-study, outside the 36 h (about 3 h in total): ch. 9 Research readings — the papers behind the lower bounds, and the Lipschitz regularity of networks (≈ 3 h).

How to study this module

• Work through chapter 0 seriously: it is the course’s only foundation (no earlier deep learning course is assumed), and the cost vocabulary is introduced there.
• Quantify everything: for every architecture encountered elsewhere, practice estimating FLOPs and memory before looking up the answer.
• Never separate theory from hardware: a complexity bound is read facing a roofline.
• Recompute the systems chapter’s examples on paper, then check against GPU orders of magnitude.
• Keep the rates of chapter 0 (theorem 0.10, proposition 0.11) at hand: the \sqrt{\kappa} of the lower bounds of chapter 8 starts there.
• Do the exercises by estimating first, computing second: the order of magnitude first is the target skill.

0. Foundations: the field, the PyTorch pipeline, autodiff, optimization and the transformer block

This chapter lays the foundation that the rest of the module relies on, assuming only the sheet’s prerequisites: basics of machine learning and Python. It covers the first three themes of the official syllabus — the field and its tasks (with the universal approximation theorem), the training pipeline in PyTorch, and optimization for deep learning, with automatic differentiation in between — and ends by building the transformer block from the multilayer perceptron.

Readers who want backpropagation worked by hand on tiny networks first can use chapters 2–6 of the off-curriculum Refresher on neural networks.

The field: tasks, architectures and what a network can represent

Deep networks are the default tool for vision and language, and are spreading to the sciences. The official syllabus follows the main families of tasks; the table says where each one is treated here.

Why should a network be able to represent the function that solves such a task? The classical answer is the universal approximation theorem.

The training pipeline in PyTorch

Every training script, from a small classifier to a large language model, is built from the same five objects: a dataset (indexable, returning one example and its label), a data loader (batching, shuffling, parallel loading), a model (an nn.Module whose parameters are registered automatically), a loss, and an optimizer that updates the parameters from their gradients. The loop alternates a training phase (forward pass, loss, backward pass, update) and an evaluation phase without gradients.

The script below trains a ResNet-18 — about 11.2 million parameters, the order of size the official objectives ask students to master — on CIFAR-10.

import torch, torch.nn as nn
from torch.utils.data import DataLoader
import torchvision, torchvision.transforms as T

device = "cuda" if torch.cuda.is_available() else "cpu"
norm = T.Normalize((0.49, 0.48, 0.45), (0.25, 0.24, 0.26))
train = torchvision.datasets.CIFAR10("data", train=True, download=True,
    transform=T.Compose([T.RandomCrop(32, padding=4), T.RandomHorizontalFlip(), T.ToTensor(), norm]))
test = torchvision.datasets.CIFAR10("data", train=False, download=True,
    transform=T.Compose([T.ToTensor(), norm]))
train_dl = DataLoader(train, batch_size=128, shuffle=True, num_workers=4)
test_dl = DataLoader(test, batch_size=512)

model = torchvision.models.resnet18(num_classes=10).to(device)
print(sum(p.numel() for p in model.parameters()))        # 11,181,642
epochs = 30
opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=0.1, total_steps=epochs * len(train_dl))
loss_fn = nn.CrossEntropyLoss()

for epoch in range(epochs):
    model.train()
    for x, y in train_dl:
        x, y = x.to(device), y.to(device)
        loss = loss_fn(model(x), y)
        opt.zero_grad(set_to_none=True)
        loss.backward()                 # reverse-mode automatic differentiation
        opt.step(); sched.step()
    model.eval(); correct = 0
    with torch.no_grad():               # no graph is stored during evaluation
        for x, y in test_dl:
            correct += (model(x.to(device)).argmax(1).cpu() == y).sum().item()
    print(f"epoch {epoch}: last train loss {loss.item():.3f}, test acc {correct / len(test):.3f}")

Automatic differentiation: what loss.backward() computes

A network is a program made of elementary operations. Automatic differentiation computes the exact derivatives of such a program — not a finite-difference approximation, and not a symbolic formula — by applying the chain rule operation by operation.

Optimization for deep learning

Training minimizes the empirical risk F(\theta) = \frac1n\sum_i \ell(h_\theta(x_i), y_i), which is non-convex as soon as there is a hidden layer.

Two algorithms do almost all the work. The first is SGD with momentum, where g_k is the mini-batch gradient at step k: m_{k+1} = \beta m_k + g_k, \qquad \theta_{k+1} = \theta_k - \eta\, m_{k+1}. The second is Adam/AdamW (definition 0.9 below), which divides each coordinate of the step by a running estimate of the gradient’s magnitude. Three facts specific to deep networks decide whether they work: initialization, the learning rate, and clipping.

From MLP to transformer block

The cost chapters count convolutional networks and transformers. This section builds both from the multilayer perceptron, so that the module needs no earlier deep learning course.

Shapes follow the site convention. A linear layer is h = Wx + b with W \in \mathbb{R}^{d_\mathrm{out} \times d_\mathrm{in}} (the weight of nn.Linear). A sequence of n tokens of width d is a matrix X \in \mathbb{R}^{n \times d}, one token per row.

Convolution: output size and receptive field. A convolutional layer replaces the dense matrix by a small k \times k filter slid over the image; ch. 1 writes it out (definition 1.7) and counts its cost. With padding p and stride s, an input of side H gives an output of side H' = \Big\lfloor \frac{H + 2p - k}{s} \Big\rfloor + 1 .

For a 32-pixel input, a 3×3 filter and padding 1, the output keeps side 32 with stride 1 (the “same” padding of the ResNet-18 above) and halves to 16 with stride 2.

Each output of a stack of L such layers with stride 1 sees a patch of side 1 + L(k-1) of the input, its receptive field: it grows linearly with depth, and faster with strides or pooling.

Attention. In a convolution each position mixes a fixed neighbourhood; in attention each token decides, from the data, which other tokens to read.

Without the scaling, at d_k = 64 the scores have standard deviation 8. A softmax of such spread-out scores is nearly one-hot, and its Jacobian \mathrm{diag}(p) - pp^\top is then nearly zero: gradients stop flowing through the head. The scaling plays the role that He initialization (prop. 0.7) plays for linear layers.

Positions. Attention alone ignores order: permuting the rows of X permutes the rows of the output in the same way (permutation equivariance, the symmetry studied in ch. 1, definition 1.10). Order is injected by adding a position vector p_i to the embedding of token i:

• learned positions: one trained vector per position, as in GPT-2;
• sinusoidal positions (Vaswani et al. 2017): p_{i,2j} = \sin\big(i/10000^{2j/d}\big), \qquad p_{i,2j+1} = \cos\big(i/10000^{2j/d}\big), whose fixed frequencies make p_{i+\Delta} a rotation of p_i, hence an easy function of relative position.

Recent LLMs rotate the queries and keys instead of adding vectors (RoPE, see the Large Language Models course).

Tokens: byte-pair encoding in five lines (Sennrich et al. 2016).

1. Start from single bytes.
2. Count the adjacent pairs of symbols in a training corpus.
3. Merge the most frequent pair into a new symbol, and record the merge.
4. Repeat until the vocabulary reaches its target size (50\,257 tokens for GPT-2).
5. Encode new text by replaying the merges in order.

Frequent words become one token, rare words split into pieces, and no word is ever out of vocabulary. Variants and their trade-offs: Text Mining and NLP, ch. 1.

Residual connections make each sublayer a correction to the identity. The gradient reaches the first layers through the sums, without crossing every block’s Jacobian, so the product of prop. 1.13 (the vanishing gradient of recurrent networks) never forms.

Pre-LN (normalizing before each sublayer, from GPT-2 on) keeps that identity path clean; the original post-LN design needed a long warmup to train at all (Xiong et al. 2020).

The block below is a complete, runnable version; ch. 1 counts its parameters (12d^2 plus biases) and its FLOPs.

import torch, torch.nn as nn

class Block(nn.Module):                      # pre-LN transformer block, causal
    def __init__(self, d=768, h=12):
        super().__init__()
        self.h, self.ln1, self.ln2 = h, nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x):                     # x: (batch, n, d)
        b, n, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(b, n, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        mask = torch.ones(n, n, dtype=torch.bool, device=x.device).tril()
        att = (q @ k.transpose(-2, -1)) / (d // self.h) ** 0.5      # (b, h, n, n) scores
        att = att.masked_fill(~mask, float("-inf")).softmax(-1)
        x = x + self.proj((att @ v).transpose(1, 2).reshape(b, n, d))
        return x + self.mlp(self.ln2(x))

print(sum(p.numel() for p in Block().parameters()))   # 7,087,872 = 12 d^2 + 13 d

LoRA in five lines (Hu et al. 2022). Fine-tuning all the weights costs the full 16 bytes per parameter of ch. 1 (definition 1.4). LoRA freezes each pretrained matrix W_0, of size d_\mathrm{out} \times d_\mathrm{in}, and learns a low-rank correction: W = W_0 + BA, \qquad B \in \mathbb{R}^{d_\mathrm{out} \times r}, \quad A \in \mathbb{R}^{r \times d_\mathrm{in}}, \quad r \ll \min(d_\mathrm{in}, d_\mathrm{out}) .

B starts at zero, so training starts from the pretrained model. Only r(d_\mathrm{in} + d_\mathrm{out}) parameters per matrix receive gradients and Adam states. After training, BA is added into W_0, so inference costs nothing extra.

For d_\mathrm{in} = d_\mathrm{out} = 4096 and r = 8: 65\,536 trainable parameters instead of 16.8 M per matrix, 256 times fewer.

1. Counting: FLOPs, memory, scaling laws

A GPU budget is easily burned because nobody counted the FLOPs and the bytes before launching the job. This chapter teaches the module’s first skill, exact counting: predicting the compute and the memory of a training run from the architecture alone.

The point of counting by hand: An engineer who can estimate the FLOPs of a training run by hand saves GPU-days — and catches configuration errors before they explode.

def flops_transformer(N_params, D_tokens, fwd_ratio=2.0, bwd_ratio=4.0):
    return (fwd_ratio + bwd_ratio) * N_params * D_tokens

C = flops_transformer(6.7e9, 1e12)                  # LLaMA-7B ~ 6.7B params, 1T tokens
print(f"{C:.2e} FLOPs")                              # 4.02e22
peak, mfu = 312e12, 0.45                             # A100 bf16, 45% MFU
hours = C / (peak * mfu) / 3600
print(f"{hours:,.0f} GPU-hours")                     # ~80,000 -- see ch. 6 and Touvron et al. 2023

Image analysis: convolutional networks

Images are large (a 224 \times 224 colour image has 150\,528 values) and their content does not depend on where an object sits. Convolutional networks exploit both facts: each output looks only at a small neighbourhood (locality), and the same filter is applied at every position (weight sharing).

Equivariant models

Proposition 1.8 is one instance of a general design principle: when the task is unchanged by a group of transformations of the input, build that symmetry into the architecture instead of learning it from augmented data.

The layer x \mapsto ax + b(\mathbf{1}^\top x)\mathbf{1} processes each element (ax_i) and adds a summary of the whole set (b\sum_j x_j): it is the skeleton of DeepSets and of message passing on graphs. Built-in symmetry gives the right behaviour on every input, not only on those seen with augmentation, and it saves parameters and data.

Sequential data: recurrent networks

Time series, audio and sensor streams have a variable length and an order. A recurrent network reads them one step at a time and carries a state that summarizes the past; it can output a single prediction for the whole sequence (classification) or one prediction per step (regression, forecasting).

Text generation with transformers

A language model assigns a probability to every sequence of tokens and generates text by sampling from it one token at a time. The transformer of prop. 1.2 becomes a generator through two changes: a causal mask, and a decoding rule.

2. Parallelism: data, tensor, pipeline

This chapter answers a practical question: how to split one training run over many GPUs. There are three axes, often combined (the “3D parallelism” of large systems); for each, we give the principle, the decomposition of the time per step and the communication cost.

3. Communication: all-reduce and friends

This chapter studies the operation that connects the workers. The real bottleneck of distributed training is rarely compute: it is the exchange of gradients. One collective operation must be known in depth — the all-reduce — and two ways of implementing it, one that saves bandwidth and one that saves latency.

4. Memory efficiency: checkpointing, ZeRO, mixed precision

The 16 bytes per parameter of ch. 1 and the activations rarely fit on one GPU. This chapter presents three ways of trading a little computation or communication for a lot of memory: recomputing activations, partitioning the training states, and storing numbers on fewer bits.

5. Efficient inference

At scale (millions of requests), inference dominates the total cost: it deserves its own tools. Four levers, from the simplest to the deepest.

6. Real systems: roofline, GPU, and the complete budget of a 7B LLM

This chapter confronts the counts of the previous chapters with real hardware. It explains why a GPU rarely reaches its peak throughput, bounds the speedup of any parallelization, and ends with the complete budget of a 7B model.

7. Stability, robustness, reinforcement learning and generative models

The previous chapters trained networks for supervised tasks and counted what that costs. This chapter covers the last three themes of the official syllabus: how stable a trained network is under small changes of its input, how deep networks learn to act from rewards instead of labels, and how they learn to generate new data. Each section gives the founding objects and one result proved in full; the specialised modules of the programme go further.

7.1. Stability and robustness

Stability has two meanings. Training stability — signals that neither vanish nor explode through depth or time — was treated with initialization (prop. 0.7), the recurrent case (prop. 1.13), normalization, residual connections and clipping. Robustness asks whether the trained model keeps its prediction when the input is perturbed: by an adversary who chooses a tiny worst-case perturbation, or by a shift between training and deployment data.

The product bound is easy but often loose by orders of magnitude, and computing the exact Lipschitz constant of a deep network is NP-hard (Virmaux & Scaman 2018, ch. 9). Certified defences therefore constrain the norms during training (spectral normalization, orthogonal layers) so that the radius of (ii) becomes meaningful.

7.2. Deep reinforcement learning

In reinforcement learning no label says what the right output is: an agent acts, the environment answers with a new state and a reward, and the goal is to maximize the accumulated reward. Deep networks enter as function approximators for values or policies over large state spaces (pixels, board positions, text).

The mandatory Reinforcement Learning and Autonomous Agents course of the same period proves the founding results. This section states them, then reads them through the module’s question: what does deep RL cost?

What deep RL costs. Three quantities separate it from supervised training.

• Rollout throughput. Every policy-gradient step needs fresh trajectories from the current policy: H environment steps and H forward passes of the policy network per trajectory. For small policies the environment, usually on the CPU, sets the speed. Vectorized environments step E copies together, turning E matrix–vector products into one matrix product: the batching move of example 6.2.
• Variance per FLOP. An estimate averaged over m trajectories has a standard error proportional to 1/\sqrt m: halving the error costs four times the rollouts. A baseline that divides the variance by k saves a factor k of rollouts, hence of FLOPs, at equal precision (example 7.6, exercise 10).
• Replay memory. DQN keeps the last million transitions. An Atari state stacks four 84 × 84 grey frames, 28 KB in uint8; storing s and s' for every transition takes about 56 GB, storing each frame once about 7 GB — a buffer sized in bytes, like the KV cache of ch. 5.

7.3. Generative models

A generative model learns a distribution p_\theta close to the data distribution p_\mathrm{data}, in order to sample new images, sounds or molecules. Autoregressive models were met in ch. 1 (text generation); the three other families below differ by how they make training tractable.

Training only regresses noise; generating needs the reverse of each forward step. The reverse step is intractable in general, but it becomes a Gaussian once the clean data point is known.

Diffusion returns in the mandatory Graph Machine Learning course, whose section 9.5 makes it discrete to generate graphs. The notes of Multimodal Generative AI develop latent diffusion and text-to-image models; that course is a sibling of this one in the same “Choose 1” group, so it is read as notes, not taken alongside.

8. Theoretical complexity of training

The previous chapters made each training step faster with more GPUs; this chapter asks whether there is a limit to what parallelism can buy. The answer is yes, and it is provable: for distributed optimization over a network, lower bounds say how many computation and communication rounds any algorithm needs. This chapter is the most theoretical of the module, and its lower bound is the central result of the papers read in ch. 9.

9. Research readings: lower bounds and Lipschitz regularity

Who wrote these papers (2026-2027 catalog): the APM_52183_EP course unit “Apprentissage profond de la théorie à la pratique” is led by Boris Bonev (“Responsable” on the 2026–2027 sheet and on the track page), with P. Bianchi, C. Bonet and B. Bonev on the instructional engineering team.

The works presented below are those of Kevin Scaman (previously MSR–INRIA) and co-authors, the lower-bound results underlying chapter 8 and the Lipschitz analysis behind section 7.1: they are cited as classic references of the field, not as the works of the official team.

Three papers to know:

Exercises

1. ★ Counting. Prove the “per token” formula 24d^2 + 4nd of a transformer block (prop. 1.2), then verify theorem 1.3 on a GPT-2 small (N \approx 124M, 12 blocks, d = 768): compute C for D = 3B tokens and compare with a torch.profiler (or flops-profiler) profile within ±10%. Expected solution: 6ND \approx 6 \times 1.24\times10^{8} \times 3\times10^{9} \approx 2.2\times10^{18} \ \text{FLOPs}, about 4.4 A100-hours at 45\,\% MFU.

2. ★ Pipeline bubble. Re-prove theorem 2.3 (draw the 1F1B Gantt chart for p=4, m=8). Then: what minimal m for an efficiency ≥ 90% with p = 16 stages? Solution: m/(m+15) \geq 0.9 \Rightarrow m \geq 135.

3. ★ Inference memory wall. (a) Recover the KV cache formula (def. 5.1) for a model with grouped-query attention (GQA: g head groups instead of h) — what saving?

   (b) Compute the peak throughput of a 13B fp16 on an A100 (2 TB/s) and on a 1 TB/s GPU.

   (c) Show that, as far as the weights are concerned, int8 doubles this ceiling and int4 quadruples it, and explain why batching raises the aggregate throughput but not this per-sequence ceiling.

   Solution: (a) factor g/h; (b) ≈ 77 then 38 tok/s; (c) the weights go from 2N bytes to N (int8) and N/2 (int4): the ceiling is multiplied by 2 and 4, up to the KV cache, which stays in bf16.

4. ★ Roofline. A kernel reads X \in \mathbb{R}^{m \times n} and w \in \mathbb{R}^n and computes Xw (2mn FLOPs, (mn + n + m)\times4 bytes).

   (a) Intensity I(m) as m \to \infty (at fixed n=4096)?

   (b) On an A100 (312 TFLOP/s, 2 TB/s), for which m is the GEMV memory-bound?

   (c) Deduce why autoregressive decoding is memory-bound and what batching changes there (hint: I grows with batch size — but not the wall per token).

5. ★★ All-reduce. Prove theorem 3.2 on the reduce-scatter (induction on the steps).

   Then plot T_\mathrm{ring} and T_\mathrm{tree} (prop. 3.3) for S = 1 GB, \alpha = 2\,\mu s, b = 25 GB/s, P = 2 \dots 512 on a log scale. Crossover? Which mode for a bf16 LLM-13B gradient?

   Partial solution: the ring wins as soon as S/P \gg \alpha b.

6. ★★ Checkpointing. A model with 30 blocks, activations 1.2 GB per block, 15 GB of memory available for activations.

   (a) Verify that storing all the activations does not fit.

   (b) Apply theorem 4.1: how many segments, what memory, what time overhead?

   (c) Show that with a single level of checkpoints and k segments the memory is about (k + L/k)\,a, hence never below 2\sqrt{L}\,a; explain how recursive checkpointing goes below this floor at the price of more recomputation (Chen et al. 2016 reach O(\log L) memory for O(L \log L) computation).

   Solution: (a) 36 GB > 15 GB; (b) 5 or 6 segments, (5 + 6) \times 1.2 \approx 13.2 GB, about +\tfrac13 of time.

7. ★★ Complete budget (the capstone lab). Take up the table 6.4 with N = 13B, D = 300B tokens (a “Chinchilla-lite” budget: comment on the D/N ratio), on 64 H100 GPUs (≈ 990 TFLOP/s bf16, 40% MFU, 700 W). Compute: FLOPs, GPU-hours, calendar time, states memory (ZeRO-3 over 8 nodes?), energy, FR/world emissions. Conclude with three sentences of architecture recommendation.

   Partial answers: 2.3\times10^{22} FLOPs; ≈ 16,500 GPU·h; ≈ 11 days; 208 GB of states → ZeRO mandatory.

8. ★★ Receptive field and translation equivariance. (a) Show that a stack of L convolutions with kernel k and stride 1 has a receptive field of side 1 + L(k-1). Compare the weights of three stacked 3 × 3 layers with those of one 7 × 7 layer, with c channels in and out and no bias.

   (b) With circular padding, show that the layer of definition 1.7 commutes exactly with a cyclic shift (torch.roll), and check it numerically.

   (c) Show that a stride-2 convolution is equivariant to shifts by an even number of pixels — a shift by 2 of the input shifts the output by 1 — but not to a shift by 1. Check both numerically.

   Solution: (a) each layer widens the patch by k - 1, so three 3 × 3 layers see 7 × 7 pixels with 27c^2 weights instead of 49c^2. (b) the proof of prop. 1.8 holds verbatim with indices taken modulo the image size.

   3. The stride-2 output at (i, j) is the stride-1 output at (2i, 2j): an even shift 2t moves it by t; an odd shift falls between the samples, and no shift of the output reproduces it.

9. ★★ A DDPM sampler on 2-D data. (a) For Gaussian data x_0 \sim \mathcal{N}(0, s^2 I), show that the best noise predictor at step t (the minimizer of the loss of definition 7.10) is \varepsilon^\star(x_t, t) = \frac{\sqrt{1 - \bar\alpha_t}}{\bar\alpha_t s^2 + 1 - \bar\alpha_t}\,x_t .

   (b) Implement the ancestral sampler of definition 7.13 with \varepsilon^\star, 100 steps, \beta_t linear from 0.001 to 0.2 and \sigma_t^2 = \beta_t. From 100 000 noise draws, compare the standard deviation of the samples with s, for s = 0.5 and s = 2.

   (c) With the schedule of (b), train an MLP \varepsilon_\theta(x_t, t) (three hidden layers of 128 units, input (x_t, t/T_\mathrm{diff})) on scikit-learn’s make_moons (noise 0.05, standardized), then sample 2\,000 points and compare them with the data.

   Solution: (a) the minimizer is the conditional mean \mathbb{E}[\varepsilon \mid x_t]. The pair (\varepsilon, x_t) is Gaussian, with covariance \sqrt{1 - \bar\alpha_t} and \mathrm{Var}\,x_t = \bar\alpha_t s^2 + 1 - \bar\alpha_t.

   2. About 0.51 and 1.98 (0.47 and 1.94 with \sigma_t^2 = \tilde\beta_t): each reverse mean is exact for Gaussian data, its variance only approximately.

   3. 4 000 Adam steps with batches of 512 (seconds on a CPU) put about 99 % of the samples within 0.3 of a data point.

10. ★★ REINFORCE with a baseline: variance and cost. (a) In the bandit of example 7.6, write \sigma = \sigma(\theta) and show that the estimator with a constant baseline b has variance \sigma(1-\sigma)\big[(1-\sigma)(1-b)^2 + \sigma b^2 - \sigma(1-\sigma)\big].

    (b) Deduce the variance-optimal baseline. At \sigma = 0.9, compare no baseline, b = 0.9 (the mean reward J(\theta)) and the optimum, and convert each variance into the number of rollouts needed for a 10\,\% relative error.

    (c) Implement REINFORCE on Gymnasium’s CartPole-v1 with a policy 4 \to 64 \to 2, without baseline and with a learned value baseline b(s_t) (allowed by the RL course, ch. 5). With 8 vectorized environments, report environment steps per second, and the environment steps and policy FLOPs needed to reach an average return of 475, over 5 seeds.

    Solution: (a) the estimator equals (1-b)(1-\sigma) with probability \sigma and b\sigma with probability 1-\sigma; its mean is \sigma(1-\sigma).

    2. The optimum b^\star = 1 - \sigma makes the estimator constant, with variance 0. At \sigma = 0.9 the variance is 0.0009 without baseline (about 11 rollouts), 0.0576 with b = 0.9 (about 711 rollouts) and 0 with b^\star (one rollout). The mean reward is not the variance-optimal baseline: here it multiplies the cost by 64.

11. ★★★ Paper + measurement. Choose: Megatron (TP), ZeRO, or FlashAttention.

    (a) Summarize the mechanism in 10 lines with the module’s formulas.

    (b) Measure an effect on a small real setup (2 GPUs suffice): DP scaling 1→2→4 on a model that fits in memory (verify the \frac{2(P-1)}{P}\frac{S}{b} term of theorem 3.2 with torch.distributed and a barrier-timer).

    (c) Present in 10 minutes: key figure, one reproduced number, one limitation.

    Grid: accuracy of the quoted formulas, honesty of the measurement (error bars!), relevance of the chosen limitation.

Projects and assessment

Official assessment. The fiche above quotes the SynapseS 2026–2027 sheet: a numerical grade out of 20, “validation by project” stated for the engineering degree and the exchange programmes, a resit allowed (the better of the two grades is kept), and the course unit earned at 10/20. The modalities specific to the LLGA track are not displayed: that section of the sheet is empty. Everything below is our proposal, built to fit a validation by project: subjects, a written exam and a rubric.

The projects exercise the master objectives O1 (deep learning foundations), O3 (large language models, through their training and serving budgets) and O7 (hands-on practice: profiling, distributed training, serving), and O2 (generative modelling) through the diffusion sampler of section 7.3. Their measured numbers are meant to fill the compute-budget line of the charter of the end-of-year projects.

Solo projects

1. [Solo · ≈ 25 h] Count, then measure: 6ND, memory and the roofline on GPT-2 small. Question: how close do the counts of ch. 1 come to a profiler on a real model, and why do they miss where they miss?
   • Implement: a FLOP and memory counter from prop. 1.2, theorem 1.3 and definition 1.4; profile the forward and backward passes of the Hugging Face gpt2 checkpoint (124 M parameters, Radford et al. 2019) and its generation with torch.profiler; place the main kernels on the roofline of definition 6.1.
   • Data and hardware: WikiText-103 sequences of 1 024 tokens; one free Colab or Kaggle T4 GPU.
   • Deliverables: code, a 4-page report with a predicted-versus-measured table (FLOPs, activation memory, MFU, KV-cache growth), a 10-minute talk.
   • Chapters: ch. 1, 5, 6.
   • Success: FLOPs within ±10 %, KV cache within ±5 %, every larger gap explained.
2. [Solo · ≈ 25 h] Paper study: Lipschitz regularity and certified robustness (Virmaux & Scaman 2018). Question: how loose is the product of spectral norms of prop. 7.2, and what does a tighter Lipschitz estimate buy?
   • Implement: train the ResNet-18 of ch. 0 on CIFAR-10 three ways — standard, PGD adversarial training (Madry et al. 2018), spectral normalization (Miyato et al. 2018). For each, compute the product bound, a layer-wise power-iteration estimate in the spirit of the paper’s SeqLip, the certified radius of prop. 7.2 (ii) extended to ten classes through the margin, and the FGSM and PGD accuracy at budget 8/255.
   • Data and hardware: CIFAR-10, and CIFAR-10-C (Hendrycks & Dietterich 2019) for the distribution shift of section 7.1; one free T4 GPU, with reduced epochs.
   • Deliverables: code, a 4–6 page report that also counts the cost of each training with theorem 1.3, a 10-minute talk.
   • Chapters: ch. 0, 7.1, 9.
   • Success: a table of three trainings by five metrics with error bars over three seeds, and the gap between bound and estimate explained.
3. [Solo · ≈ 20 h] Variance per FLOP: REINFORCE at laptop scale. It connects this course to Reinforcement Learning and Autonomous Agents (theorem 5.1, ch. 6). Question: how much compute do a baseline, a batch of trajectories and vectorized environments save to reach a given return?
   • Implement: exercise 10 (c) extended — REINFORCE without baseline, with a learned value baseline, and the PPO of the RL course; vary the number of vectorized environments (1 to 64) and of trajectories per update.
   • Data and hardware: Gymnasium CartPole-v1 and LunarLander-v3; a laptop CPU.
   • Deliverables: code, a 4-page report, a 10-minute talk.
   • Chapters: ch. 1 (counting), 6 (batching), 7.2.
   • Success: environment steps and policy FLOPs to the threshold (475 on CartPole, 200 on LunarLander) with intervals over five seeds, and a measured gradient variance that follows the trend of example 7.6.

Group projects

1. [Group of 3 · ≈ 30 h each] From DDP to ZeRO-3: measuring the all-reduce term and Amdahl’s ceiling. Question: does the ring formula of theorem 3.2 predict the step time of real data-parallel training, and when does sharding pay?
   • Implement: train GPT-2 small (or the ResNet-18 of ch. 0) with DDP, ZeRO-1 (PyTorch’s ZeroRedundancyOptimizer), ZeRO-2 (FSDP’s SHARD_GRAD_OP or DeepSpeed) and FSDP full sharding (Zhao et al. 2023), on 1 and 2 GPUs (Kaggle’s free 2 × T4), and on 2 to 8 CPU processes with the gloo backend for the communication fit.
   • Measure: step time with error bars; the bandwidth term of theorem 3.2, fitted from all-reduce micro-benchmarks; the gain of overlap and of the bucket size; an Amdahl serial fraction (prop. 6.3); memory per GPU against definition 4.2.
   • Split: member A, DDP and the all-reduce benchmarks; member B, ZeRO-1, ZeRO-2 and FSDP memory; member C, the fits, Amdahl and the decision tree.
   • Deliverables: code, a 6-page report, the decision tree of ch. 2 annotated with the group’s own numbers, a 15-minute talk.
   • Chapters: ch. 2, 3, 4, 6; extends exercise 11.
   • Success: a fitted bandwidth within a factor 2 of the nominal link, and memory per GPU within 10 % of definition 4.2 plus the measured activations.
2. [Group of 3 · ≈ 30 h each] Serving a small LLM under a budget: quantization, speculative decoding, batching. It connects this course to Large Language Models (ch. 6) and ML Ops and LLM Engineering (ch. 3). Question: which combination of levers gives the best quality–latency–throughput–energy trade-off on one free GPU?
   • Implement: serve Qwen2.5-1.5B-Instruct with Hugging Face transformers, separating time to first token (prefill) from time per output token (decode); int8 and 4-bit weights (bitsandbytes: LLM.int8() of Dettmers et al. 2022, NF4 of Dettmers et al. 2023); speculative decoding with Qwen2.5-0.5B-Instruct as draft model; static batches of 1 to 32 requests.
   • Measure: the acceptance rate against example 5.3, and the four columns of example 5.4 — perplexity on WikiText-2 and accuracy on 500 HellaSwag questions, p50 and p99 latency, tokens per second, energy from nvidia-smi power readings.
   • Split: member A, quantization; member B, speculative decoding; member C, batching, the measurement harness and energy.
   • Deliverables: code, a 6-page report, a 15-minute talk built on the four-column table.
   • Chapters: ch. 5, 6.
   • Success: at least five configurations measured on all four columns, and the speedup predicted from the measured acceptance rate within 20 % of the observed one.
3. [Group of 4 · ≈ 30 h each] Paper study: a Chinchilla sweep at small scale (Hoffmann et al. 2022). Question: does the compute-optimal ratio of ch. 1, about 20 tokens per parameter, hold for tiny language models trained on simple text?
   • Implement: a family of causal transformers built from the block of ch. 0, from 1 to 60 M parameters, so that each budget’s optimum (about 3, 9 and 20 M at 20 tokens per parameter) is bracketed. At three budgets of about 10^{15}, 10^{16} and 5 \times 10^{16} FLOPs (counted as 6ND), train five or six models of different sizes each; fit the IsoFLOP minima and the law L(N, D) = E + A/N^\alpha + B/D^\beta.
   • Data and hardware: TinyStories (Eldan & Li 2023); one or two free T4 GPUs, about 4 \times 10^{17} FLOPs and 10 GPU-hours in all.
   • Split: member A, data and tokenizer; member B, model family and training loop; member C, the sweep and the fits; member D, evaluation and sampling.
   • Deliverables: code, a 6-page report with perplexities and samples under temperature and nucleus sampling (example 1.16), a 15-minute talk.
   • Chapters: ch. 0, 1, 6.
   • Success: the optimal size and ratio at each budget with bootstrap intervals, compared honestly with the Chinchilla ratio.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet and a non-programmable calculator allowed; four questions on 20 points — a course question, a computation and two short proofs.

1. Question (4 points). Course. Explain why generating tokens one at a time with batch size 1 is memory-bound, while the prefill of a long prompt is compute-bound. Give the arithmetic intensity of each phase in FLOPs per byte (bf16 weights), and say what batching, the KV cache and int8 weights change.

   Expected: decode multiplies each weight matrix by one vector, 2 FLOPs per 2-byte weight, so the intensity is about 1, far below the A100 ridge of 156 (example 6.2): tokens per second are capped by bandwidth over weight bytes (definition 5.1). Prefill reuses each weight for the n prompt tokens, an intensity of about n, compute-bound beyond about 150 tokens.

   Batching B sequences multiplies the decode intensity by B. The KV cache avoids recomputing the past but adds bytes read per token. Int8 halves the weight bytes and doubles the memory-bound ceiling.

2. Question (6 points). Computation. A dense transformer with 3 B parameters is trained on 60 B tokens on 32 A100 GPUs (312 TFLOP/s bf16 peak, 40 % MFU), with Adam in mixed precision. (a) Total FLOPs and the ratio of tokens to parameters. (b) GPU-hours and calendar days. (c) Training-state memory, in total and per GPU with ZeRO-3. (d) With a global batch of one million tokens and bf16 gradients: the bandwidth term of the ring all-reduce at 25 GB/s, against the compute time of one step.

   Expected: (a) 6ND = 1.08 \times 10^{21} FLOPs, 20 tokens per parameter (Chinchilla-optimal). (b) 1.08 \times 10^{21} / (1.25 \times 10^{14}) FLOP/s ≈ 8.7 million GPU-seconds ≈ 2 400 GPU-hours, about 75 h or 3.1 days on 32 GPUs. (c) 16 bytes per parameter: 48 GB in total, 1.5 GB per GPU.

   4. The gradient weighs 6 GB, so the bandwidth term is 2 \times \tfrac{31}{32} \times 6/25 \approx 0.47 s; one step computes 1.8 \times 10^{16} FLOPs in about 4.5 s. Communication is about 10 % of compute, and overlap (ch. 3) hides it.

3. Question (5 points). Short proof: checkpointing. A network of L layers keeps a bytes of activations per layer. (a) With one level of checkpoints, one every L/k layers, show that the peak activation memory of the backward pass is about (k + L/k)\,a, minimal at k = \sqrt L, where it equals 2\sqrt L\,a. (b) Show that the extra computation is one forward pass, about a third of a training step (theorem 1.3).

   Expected: (a) the k boundary activations stay stored, and the backward pass of a segment recomputes its L/k layers and holds them together. The sum k + L/k is minimal where its derivative 1 - L/k^2 vanishes.

   2. Each layer is recomputed once in the forward direction: 2ND extra FLOPs out of 6ND, that is +1/3.

4. Question (5 points). Short proof: diffusion. For the forward process of definition 7.10, show that q(x_{t-1} \mid x_t, x_0) is Gaussian and compute its variance. Then explain in two sentences why the sampler of definition 7.13 still needs a network, given this closed form.

   Expected: by Bayes and the Markov property, the posterior is proportional to a product of two Gaussians in x_{t-1}, of precisions \alpha_t/\beta_t and 1/(1 - \bar\alpha_{t-1}). The precisions add, and since \alpha_t(1 - \bar\alpha_{t-1}) + \beta_t equals 1 - \bar\alpha_t, \tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\,\beta_t .

   The mean depends on x_0, unknown when sampling; the network predicts the noise, hence x_0, from x_t (prop. 7.12, second form).

Grading rubric (proposal)

References

• K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, L. Massoulié, Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks, ICML 2017.
• K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, L. Massoulié, Optimal Convergence Rates for Convex Distributed Optimization in Networks, JMLR, 2019.
• A. Virmaux, K. Scaman, Lipschitz regularity of deep neural networks: analysis and efficient estimation, NeurIPS 2018.
• J. Kaplan et al., Scaling Laws for Neural Language Models, 2020.
• J. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), NeurIPS 2022.
• H. Touvron et al., LLaMA: Open and Efficient Foundation Language Models, 2023 — the source of the 82,432 GPU-hours of ch. 6.
• M. Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019 — tensor parallelism.
• Y. Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, NeurIPS 2019.
• S. Rajbhandari, J. Rasley, O. Ruwase, Y. He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC 2020.
• V. Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, 2022 — the activation memory of ch. 1 and 4.
• P. Patarasuk, X. Yuan, Bandwidth optimal all-reduce algorithms for clusters of workstations, Journal of Parallel and Distributed Computing, 2009 — the optimality of the ring (ch. 3).
• T. Chen et al., Training Deep Nets with Sublinear Memory Cost (checkpointing), 2016.
• T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022.
• P. Micikevicius et al., Mixed Precision Training, ICLR 2018.
• B. Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, CVPR 2018.
• S. Han, H. Mao, W. J. Dally, Deep Compression, ICLR 2016.
• G. Hinton, O. Vinyals, J. Dean, Distilling the Knowledge in a Neural Network, 2015.
• Y. Leviathan, M. Kalman, Y. Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023.
• Z. You et al., Large Batch Training of Convolutional Networks (LARS), 2017.
• Y. Nesterov, Introductory Lectures on Convex Optimization, Kluwer, 2004 — proposition 0.11 and the foundation of ch. 8.
• G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals and Systems, 1989; K. Hornik, “Approximation capabilities of multilayer feedforward networks”, Neural Networks, 1991; M. Leshno, V. Lin, A. Pinkus, S. Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function”, Neural Networks, 1993 — theorem 0.1.
• M. Telgarsky, Benefits of depth in neural networks, COLT 2016.
• A. Griewank, A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd ed., SIAM, 2008; W. Baur, V. Strassen, “The complexity of partial derivatives”, Theoretical Computer Science, 1983 — prop. 0.5.
• K. He, X. Zhang, S. Ren, J. Sun, Delving Deep into Rectifiers, ICCV 2015 — He initialization (prop. 0.7); Deep Residual Learning for Image Recognition, CVPR 2016 — ResNet.
• Y. Dauphin et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, NeurIPS 2014; R. Ge, F. Huang, C. Jin, Y. Yuan, Escaping From Saddle Points, COLT 2015.
• M. Zaheer et al., Deep Sets, NeurIPS 2017; T. Cohen, M. Welling, Group Equivariant Convolutional Networks, ICML 2016; B. Bonev et al., Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere, ICML 2023.
• S. Hochreiter, J. Schmidhuber, “Long Short-Term Memory”, Neural Computation, 1997.
• A. Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020 — nucleus sampling.
• I. Goodfellow, J. Shlens, C. Szegedy, Explaining and Harnessing Adversarial Examples, ICLR 2015; A. Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018.
• V. Mnih et al., “Human-level control through deep reinforcement learning”, Nature, 2015 — DQN; R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning, 1992 — REINFORCE.
• D. P. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR 2014; I. Goodfellow et al., Generative Adversarial Nets, NeurIPS 2014; J. Ho, A. Jain, P. Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS 2020.
• S. Ghadimi, G. Lan, Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming, SIAM Journal on Optimization, 2013 — theorem 0.10; Y. Nesterov, “A method for solving the convex programming problem with convergence rate O(1/k²)”, Soviet Mathematics Doklady, 1983 — proposition 0.11.
• D. P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, ICLR 2015; I. Loshchilov, F. Hutter, Decoupled Weight Decay Regularization, ICLR 2019 — definition 0.9.
• A. Vaswani et al., Attention Is All You Need, NeurIPS 2017; R. Xiong et al., On Layer Normalization in the Transformer Architecture, ICML 2020; R. Sennrich, B. Haddow, A. Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016 — BPE; E. J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 — the transformer block of ch. 0.
• A. Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021 — ViT (example 1.9).
• V. Sanh, L. Debut, J. Chaumond, T. Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019 — distillation figures of prop. 5.2.
• T. Miyato, T. Kataoka, M. Koyama, Y. Yoshida, Spectral Normalization for Generative Adversarial Networks, ICLR 2018; D. Hendrycks, T. Dietterich, Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019 — CIFAR-10-C (projects).
• R. Eldan, Y. Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, 2023; A. Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2), 2019 — projects.
• Y. Zhao et al., PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel, VLDB 2023; T. Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022; T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023 — projects.
• PyTorch Distributed (DDP/FSDP) and NCCL — the lab tools.