Advanced Deep Learning

Contents

• Introduction and overview
• 0. Refresher: transformers, optimization, normalization
• 1. Reproducible research: experimental protocol
• 2. Projects: read, explain, replicate
• 3. Efficient architectures: MobileNet and distillation
• 4. Vision Transformers: ViT, DeiT, Swin
• 5. Modern convnets: the ConvNeXt lesson
• 6. Mixture of Experts (MoE)
• 7. Self-supervision I: contrast (SimCLR and its lineage)
• 8. Self-supervision II: masking (MAE, BEiT, DINOv2)
• 9. Training at scale: recipes and scaling laws
• 10. Beyond recognition: neural fields, inference, sampling and physics
• 11. Course research notes (Kalogeiton, Lutzeyer, Zhu, Wang)
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who already runs a ViT or a ResNet and wants to read — and reproduce — the pivotal papers. Empirical first: each topic starts from a result that does not hold up, such as a fine-tuning gain from ImageNet to a business domain that turns into a loss when the batch size changes, or a contrastive pre-training that “cheats” through a leak in the augmentations; only then do we name the tools that explain it: experimental protocol, ablations, the architectures and objectives themselves.

The module is a reading seminar on ViT, ConvNeXt, self-supervision, mixtures of experts and large-scale training.

What you will be able to do

• Place the pivotal papers (ViT, DeiT, Swin, ConvNeXt, MoE, SimCLR, MAE, DINOv2, MobileNet) in their lineage and their legacies.
• Compare architecture families on measurable criteria rather than on an isolated benchmark.
• Explain the two major families of self-supervision — contrastive and masked — and their trade-offs in data and compute.
• Dissect an experimental protocol and distinguish the demonstrated contribution from the packaging.
• Use training recipes at scale and interpret scaling laws.
• Replicate an experiment from a paper and report on it reproducibly.
• Explain the founding objects of the official research themes — symmetry and equivariance, robustness, neural fields, analysis-by-synthesis, amortized inference, sampling, physics-informed learning.
• Present a research paper in writing and orally, in the seminar’s format.

Official objectives and syllabus (2026-2027 catalog)

According to the CSC_52087_EP — “Apprentissage profond avancé” sheet: the official objective is to initiate students into the advanced principles of deep learning — mathematical foundations, architecture design and practical applications.

The description stresses that the field moves at a very fast pace, with new techniques and architectures appearing constantly, and that future practitioners must keep up with the latest advances; the assessment combines homework, quizzes and team projects; the language is English.

The keywords are deep learning, transformers, computer vision and symmetry in deep learning. The detailed programme lists nine themes, with the initials of the instructor in charge; the table maps each of them, in the sheet’s order, to the section that teaches it.

The sheet names the themes only, so the content of each section is our reading of its title; the 2026–2027 instructors may treat a theme from another angle. Complement to the official programme — the seminar chapters on the experimental protocol (ch. 1), the replication project (ch. 2), ConvNeXt (ch. 5), mixture of experts (ch. 6) and training at scale (ch. 9) do not appear in the published themes; they are the method and the background that the reading of the themed papers relies on.

Where this course fits in the program

An option of period 2 of M1, outside the recommended track. Official prerequisites: not communicated (the “Prerequisites” tab of the SynapseS sheet is empty); in practice, the official description addresses students already initiated into deep learning. The foundation — backpropagation, CNNs, attention, training in practice — comes from the P1 Deep Learning module, whose necessary elements (transformers, optimization, normalization) are reactivated in chapter 0.

Downstream, this module directly prepares Advanced Topics in Large Language Models (M2), which extends the critical reading of papers on the LLM side, and above all the research track of the master’s program: the seminar format and the replication project are its methodical training. Chapter 10 (neural fields, analysis-by-synthesis, sampling, physics) connects to the Image Synthesis and Multimodal Generative AI modules of the same period.

Course outline

• Refresher: transformers, optimization, normalization — the P1 foundation brought back to the level of rigor that reading papers requires.
• Reproducible research: experimental protocol — seeds, datasets, honest measurements: the rules of experimental proof.
• Projects: read, explain, replicate — the final project: choose a paper, explain it, replicate one of its experiments.
• Efficient architectures: MobileNet and distillation — cutting costs for embedded targets and transferring knowledge from a large network to a small one.
• Vision Transformers: ViT, DeiT, Swin — the transition from convolutions to attention in vision, generation after generation, then the symmetries built into representations and robustness to perturbations.
• Modern convnets: the ConvNeXt lesson — what transformers taught convnets, and the modern rereading of CNNs.
• Mixture of Experts (MoE) — growing capacity by conditioning the computation per expert, without exploding the cost.
• Self-supervision I: contrastive (SimCLR and its lineage) — learning representations by bringing views of the same image closer together.
• Self-supervision II: masking (MAE, BEiT, DINOv2) — learning by reconstructing the invisible, up to the DINOv2 lineage.
• Training at scale: recipes and scaling laws — the practical rules of heavy training and the laws linking size, data and performance.
• Beyond recognition: neural fields, inference, sampling and physics — coordinate networks, analysis-by-synthesis through differentiable models, amortized inference, sampling as a design choice, physics-informed learning, and where to pursue the generative themes in the other modules.
• Course research notes (Kalogeiton, Lutzeyer, Zhu, Wang) — the papers of the module’s instructors, read with the same grid.
• Exercises — short applications to anchor each paper.
• Projects and assessment — solo and group project subjects, a written-exam proposal and a grading rubric.
• References — the papers and sources of each chapter.

Session plan

38.25 h in class = 9 sessions of 4.25 h. The volume is the one of the SynapseS 2026–2027 sheet (38.25 h: 18 h of lectures and 20.25 h of exercise sessions), which we read as nine weeks of a 2 h lecture followed by a 2.25 h exercise session.

The sheet’s detailed programme lists nine themes without dates; the plan follows their order — symmetry (session 3), robustness (4), self-supervision (5–6), neural fields and analysis-by-synthesis (7), amortized inference, sampling and physics (8) — with one exception: distillation, the seventh theme, is taught in session 2 with efficient architectures, because its accounting is used by chapters 4 to 6 and teaching it after the generative themes would make the difficulty go back down.

The seminar’s own chapters (protocol, ConvNeXt, mixture of experts, scale) are interleaved where they are needed, and the split into sessions is our proposal. Since the course is a reading seminar, the research notes of the instructors (ch. 11) are taught in class.

Level 1 = foundations (refresher, protocol, accounting), 2 = the pivotal papers, 3 = research themes and critique; the level never goes down from one session to the next.

Self-study, outside the 38.25 h: no chapter is left to self-study, since the seminar reads every chapter in class; the replication project is carried out between sessions.

How to study this module

• Read each paper before the corresponding chapter, even without understanding everything: the course is built for the second reading.
• Redo the results tables yourself: which data, which seeds, which measurements — that is where credibility is at stake.
• Choose your project paper early in the period, and keep a narrow question rather than a general summary.
• Keep a reading notebook: one question, one method, one proof, one limitation per paper.
• Return to chapter 0 whenever transformers or normalization become shaky — the whole seminar rests on them.
• Confront your readings with the other P2 modules: the same papers come back seen from another angle.

0. Refresher: transformers, optimization, normalization

Module format: a reading seminar. Each chapter centers on a pivotal paper that we read in full: motivation, method, experiment — and we critique the protocols (datasets, seeds, compute budgets) like a reviewer.

The four instructors each intervene on their own ground (chapter 11): vision and video (Kalogeiton), graphs (Lutzeyer), diffusion models (Zhu), multimodal and video generation (Wang). Chapter 1 (the experimental protocol) and chapter 9 (training at scale) equip the final project (chapter 2): replicating a published result within a given compute budget — the acid test of modern deep learning.

1. Reproducible research: experimental protocol

A reported gain is only worth the protocol that produced it. This chapter fixes the minimal protocol that every experiment of the seminar must follow, shows how much the random seed alone can move a result, and turns replication into a method.

1.1. Seeds matter (really)

Picard (2021) — “torch.manual_seed(3407) is all you need” — trains identical classifiers over 10,000 seeds: the accuracy gap between seeds reaches several points; the choice of seed can flip a ranking of methods.

Bouthillier et al. (2021) generalize: the variance due to seeds (init, data order, augmentations) must be estimated and reported; comparing two means without variability is comparing noise. Tools: Student’s t confidence intervals over n runs; paired tests when the seeds are coupled; correction for multiple testing (see the Refresher in Statistics module for tests, the Machine Learning module for model selection).

1.2. Replication: the module’s method

1. pick a number: one row, one table of the pivotal paper;
2. scale down: same task, model ×0.2 (µP, ch. 9, helps transpose the hyperparameters); a budget declarable in GPU-hours;
3. instrument: experiment tracking (versioned configs + seeds + curves — W&B/tensorboard; determinism of the loaders; no leakage: the test set is looked at only at the end);
4. report: paper vs replication table ± standard deviation; analyze the gap (data? recipe? hardware?) — that is where one learns.

2. Projects: read, explain, replicate

The official sheet announces “a combination of homework, quizzes and team projects”, with details to come. We propose to read the team part as a paper presentation and a replication project; this chapter describes both, lists typical replication topics, and states what must be delivered.

2.1. Reading seminar (30 % of the proposed grade)

Each pair takes charge of one pivotal paper among: ViT, DeiT, Swin, ConvNeXt, Switch, SimCLR, MAE, MobileNetV2, EfficientNet, Chinchilla, µP — and presents it in 20 minutes + 10 of discussion: motivation, contribution, decisive experiment, protocol flaws. Grading grid: understanding (5), rendering of the experiments (5), critical eye (5), answering questions (5).

2.2. Replication project (60 % of the proposed grade)

One number, one replication, one budget. Typical topics:

• mini ViT on CIFAR-10/100: 4×4 patches, 6 blocks — replicate the accuracy vs dataset size curve (10k → 50k); discussion: inductive bias (ch. 4);
• SimCLR vs supervised training: same ResNet-18, same data; linear evaluation; augmentation ablation (ch. 7);
• mini MAE: 75% masking on unlabeled STL-10; reconstruct and transfer (ch. 8);
• mini MoE: 8 experts top-2 on a character transformer; plot f_i (balancing; ch. 6) with and without the auxiliary loss;
• mini scaling: fit L(N, D) = E + A/N^\alpha + B/D^\beta over a grid of MLPs/transformers; compare the predicted optimum with theorem 9.2 (ch. 9);
• µP on an MLP: verify that the optimal lr becomes independent of width under the parametrization (ch. 9);
• distillation: ResNet-18 teacher → MobileNetV2 student on a dataset of your choice; accuracy vs latency curves (ch. 3).

2.3. Deliverables

1. repository: code (deterministic, fixed seeds), versioned config, training/evaluation scripts;
2. report (6–8 pages): question, protocol (def. 1.1), paper/replication table with n seeds ± standard deviation, analysis of the gaps, budget (GPU-hours);
3. defense: 15 minutes, including “the figure of the paper you wish you had seen” — your best addition to the original protocol.

3. Efficient architectures: MobileNet and distillation

Accuracy is not the only constraint: a model must also fit a latency, memory or energy budget. This chapter gives the exact accounting of the convolution that made mobile networks possible, and the distillation technique that transfers what a large network knows to a small one.

3.1. Depthwise separable — the exact accounting

• MobileNetV1 (Howard, 2017): stacks separable + RELU blocks; two compression hyperparameters: width multiplier \alpha (channels × \alpha) and resolution multiplier \rho (input × \rho) — cost \approx \alpha^2 \rho^2;
• MobileNetV2 (Sandler, 2018): block with inverted bottleneck (expand d \to 6d pointwise → depthwise → project 6d \to d') and linear bottleneck (no nonlinearity after the projection — ReLU would destroy information in the low-dimensional space); the shortcuts connect the bottlenecks — this is the block ConvNeXt will reuse (ch. 5) and EfficientNet will stack;
• EfficientNet (Tan & Le, 2019): compound scaling — depth, width and resolution grow together: \text{depth} = \alpha^{\phi}, \text{width} = \beta^{\phi}, \text{resolution} = \gamma^{\phi} with \alpha \beta^2 \gamma^2 \approx 2 (FLOPs × ~2 per step of \phi) — the architecture as a parameterized family, not an isolated model; a forerunner of the scaling laws of chapter 9.

3.2. Knowledge distillation

Applications: compression (student = MobileNet, ch. 3.1 — the standard production chain); self-distillation (DeiT, ch. 4; Born-Again Networks); and label-free distillation on massive unlabeled data (Noisy Student, DINOv2). Bridges: quantization/pruning on the deployment side is covered in the Computer Vision (P2) module, chapter 8; distilled LLMs in the M2 Large Language Models module.

3.3. The “efficiency” reading grid

4. Vision Transformers: ViT, DeiT, Swin

This chapter asks how attention, designed for text, can process images, and at what cost. It reads three papers in sequence: ViT, which treats image patches as tokens; DeiT, which shows that the training recipe matters as much as the architecture; and Swin, which brings locality back. It ends with the first two themes of the official programme: the symmetries that architectures build into their representations (4.4) and robustness to perturbations and distribution shifts (4.5).

4.1. ViT — “An Image is Worth 16x16 Words”

The pivotal paper (Dosovitskiy et al., ICLR 2021) transposes the transformer to images with disarming radicality: cut the image into patches and treat them as tokens.

• Patch embedding: the image x \in \mathbb{R}^{3 \times H \times W} is cut into N = HW/P^2 patches of P \times P (typically 16×16); each patch is linearly projected into a vector of dimension d — implemented as a convolution with P \times P kernel and stride P;
• special tokens: a prepended \texttt{[class]} token (its final state serves for classification) and learned additive positional embeddings (ViT has no a priori knowledge of the 2D structure: everything must be learned);
• classification: MLP + softmax on \texttt{[class]}.

import torch, torch.nn as nn

class ViT(nn.Module):
    def __init__(self, img=224, patch=16, d=768, L=12, heads=12, n_classes=1000):
        super().__init__()
        self.n = (img // patch) ** 2                     # 196 patches
        self.patch = nn.Conv2d(3, d, kernel_size=patch, stride=patch)
        self.cls = nn.Parameter(torch.zeros(1, 1, d))
        self.pos = nn.Parameter(torch.zeros(1, self.n + 1, d))
        layer = nn.TransformerEncoderLayer(
            d, heads, dim_feedforward=4*d, batch_first=True, norm_first=True)
        self.encoder = nn.TransformerEncoder(layer, L)
        self.head = nn.Linear(d, n_classes)

    def forward(self, x):
        p = self.patch(x).flatten(2).transpose(1, 2)     # (B, n, d)
        z = torch.cat([self.cls.expand(p.size(0), -1, -1), p], dim=1)
        z = self.encoder(z + self.pos)
        return self.head(z[:, 0])                        # [class] token

4.2. DeiT — training matters as much as the architecture

Touvron et al. (2021) obtain competitive ViTs on ImageNet-1k alone (1.3 M images): strong augmentation (RandAugment, mixup, cutmix), regularization (stochastic depth, tuned weight decay), the AdamW optimizer, and token distillation. A second token \texttt{[dist]} is supervised by the outputs of a convnet teacher (RegNet); the student thereby learns the inductive bias that the convnet provides for free.

Moral: comparing architectures with unequal training recipes is meaningless — we will return to this with ConvNeXt (ch. 5) and the experimental protocol (ch. 1).

4.3. Swin — local attention via shifted windows

To curb the quadratic growth of attention with the number of patches, the Swin Transformer (Liu et al., ICCV 2021) makes three choices:

1. attention computed within local windows of M \times M patches (cost linear in the number of windows);
2. each subsequent layer shifts the windows by M/2 — cross-window connections arise from the shift;
3. progressive patch merging like a convolutional encoder — Swin serves as a dense backbone (detection, segmentation) where ViT was 1 token = 1 class.

ViT (global, data-dependent, quadratic) and Swin (hierarchical, local, linear) embody the two poles that all subsequent architectures negotiate.

4.4. Symmetry in deep learning representations

The three architectures of this chapter differ above all in the symmetries they build in. A convolution commutes with translations; attention, as shown below, commutes with permutations of its tokens, which is why ViT must learn the geometry of the image from its positional embeddings. Symmetry also lives on the weight side: many different parameter vectors compute the same function, and any comparison of learned representations must account for it.

4.5. Robustness in deep learning

An accuracy measured on an i.i.d. test split says nothing about two failures that matter in deployment: small worst-case perturbations of the input, and natural shifts of the input distribution. Robustness claims about architectures — “transformers are more robust than CNNs” — are a favourite place for the protocol flaws of chapter 1.

5. Modern convnets: the ConvNeXt lesson

After a decade of “transformers win everywhere”, the ConvNeXt paper (Liu et al., CVPR 2022) asks the reverse question: what are modernized convnets worth under an equal training recipe? Method: start from a ResNet and align it step by step with the ViT choices — each step is ablated and measured:

Paper’s conclusion: the “modernized” convnet (ConvNeXt-T, 82.0) exceeds Swin-T (81.3) at comparable FLOPs and throughput. The methodological lesson is stronger than the result: training (line 1) and scale details matter more than the architecture family — a direct application of the protocol of chapter 1, and a spectacular application of the “one change at a time” protocol.

6. Mixture of Experts (MoE)

A dense network pays for all its parameters on every token. This chapter studies the layer that breaks this link: many expert sub-networks, of which a learned router activates only a few per token, so that capacity can grow without the compute per token growing with it.

6.1. Balancing the load

The canonical risk: the gate learns to send everything to the same experts (routing collapse) — the other experts receive no more gradient and die. The historical remedy of Shazeer et al. (2017): noise in the routing + an auxiliary loss. Switch Transformer (Fedus et al., 2022) formalizes the load balancing loss:

Here T is the number of tokens in the batch, P_i the average probability assigned to expert i by the gate (differentiable), and f_i the fraction of the routing slots actually sent to it (non-differentiable).

Each token fills k slots, so dividing by k makes the f_i sum to 1; with Switch’s top-1 routing it is the plain token fraction.

Under uniform routing, f_i = P_i = 1/n_E, so \sum_i f_i P_i = 1/n_E and the loss equals \alpha_{\mathrm{aux}} whatever the number of experts. The gradient flows only through the P_i, each weighted by f_i: it lowers the gate probability of the experts that already receive more than their share of tokens, which pushes the routing towards balance.

Added to this are the capacity per expert, k \times (\text{tokens}/n_E) \times a capacity factor — beyond it, tokens “overflow” (passed on to the rest of the network) — and per-expert parallelization (expert parallelism: each device holds experts).

The M2 Large Language Models module states the same layer and the same loss with the same symbols, and its exercise 9 asks for the minimum of the loss: balance is the minimum when f_i = P_i, but not in general (exercise 7 (b) below); this chapter keeps the statements and reads them on vision and character-level models.

import torch, torch.nn as nn, torch.nn.functional as F

class MoE(nn.Module):
    def __init__(self, d=1024, r=4, n_E=8, k=2, alpha_aux=0.01):
        super().__init__()                     # r = expansion factor: hidden width r*d
        self.experts = nn.ModuleList(
            [nn.Sequential(nn.Linear(d, r * d), nn.GELU(), nn.Linear(r * d, d))
             for _ in range(n_E)])
        self.gate = nn.Linear(d, n_E)
        self.k, self.n_E, self.alpha_aux = k, n_E, alpha_aux

    def forward(self, x):                      # x: (T, d), the T tokens of the batch
        logits = self.gate(x)                  # (T, n_E)
        probs = logits.softmax(-1)
        topk_w, topk_i = probs.topk(self.k, dim=-1)
        topk_w = topk_w / topk_w.sum(-1, keepdim=True)
        out = torch.zeros_like(x)
        for e in range(self.n_E):
            mask = (topk_i == e)               # tokens assigned to expert e
            sel = mask.any(-1)
            if sel.any():
                out[sel] += topk_w[sel].masked_fill(~mask[sel], 0).sum(-1, keepdim=True) \
                            * self.experts[e](x[sel])
        f = torch.zeros(self.n_E, device=x.device)   # routed fraction (non-diff.)
        f.scatter_add_(0, topk_i.flatten(),
                       torch.ones(topk_i.numel(), device=x.device))
        f = f / (self.k * x.size(0))           # k slots per token, so sum(f) = 1
        aux = self.alpha_aux * self.n_E * (f * probs.mean(0)).sum()
        return out, aux                           # (output, auxiliary loss)

6.2. Panorama and readings

• Shazeer et al. 2017 (“Outrageously Large Neural Networks”): MoE over LSTM, noisy top-k, proof that n_E can grow without exploding the compute;
• GShard / Switch: translation at 600B parameters, then the top-1 single expert and clean balancing — the era of transformer MoE;
• Mixtral 8x7B (2024): an open LLM, 8 experts top-2, about 47B parameters in total of which about 13B are active per token (attention layers are shared, only the MLPs are replicated) — MoE arrives in everyday models; analysis in the M2 Large Language Models module (inference);
• beyond: finely learned routings (residual capacity, shared experts, MoE on alternating layers). Reading question: what do the experts actually specialize in? (Interpretability studies often show syntactic more than topical specializations.)

7. Self-supervision I: contrast (SimCLR and its lineage)

Labels are expensive; images are not. This chapter shows how to pre-train a network without any label, by asking it to recognize two transformed views of the same image among many others.

7.1. SimCLR — the complete recipe

• two views: compositional stochastic augmentations (crop + flip + color + blur); batch of N images → 2N views; two views of the same image = positive pair; all other pairs = negatives;

• encoder f(\cdot) (ResNet) + projection head g(\cdot) (2-layer MLP): the loss is applied on the output of g, but one transfers before the projection — the projection leaves the instance information in a dedicated subspace and protects the encoder’s features;

• NT-Xent loss (normalized temperature-scaled cross-entropy):

  Key result — NT-Xent loss \ell_{i,j} = -\log \frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)} {\sum_{k \neq i} \exp(\mathrm{sim}(z_i, z_k)/\tau)}, \qquad \mathrm{sim}(u, v) = \frac{u^\top v}{\|u\| \|v\|}.

  The loss is averaged over both directions (i,j) and (j,i); \tau (0.1–0.5) controls the hardness of the negatives.

  A useful sanity check: at initialization the normalized embeddings are nearly orthogonal, all similarities are close to 0, and the softmax is nearly uniform over the 2N - 1 candidates. With a batch of N = 256 images the initial loss is therefore close to \log 511 \approx 6.2. A first logged value far from it signals a bug in the masking of the diagonal or in the pairing of the views.

7.2. MoCo and BYOL — two decisive variations

• MoCo (He et al., 2020): contrast requires large batches of negatives; MoCo replaces them with a queue of features from previous batches (“free” negatives) and a momentum encoder (exponential average of the weights) so that the negatives in the queue stay consistent — contrast without a big batch;
• BYOL (Grill et al., 2020): no negatives at all — the online network predicts the output of the target (momentum-updated); one expects a trivial collapse (a constant = perfect solution); it does not happen, thanks to the asymmetric prediction head and the absence of gradient computation on the target side — a famous counter-example to “negatives are mandatory” and an excellent discussion topic (what happens with lower-dimensional projections? see the VICReg literature, which adds variance and covariance terms);
• CLIP (Radford et al., 2021): image-text contrast on 400 M (image, caption) pairs — the same mathematical object as SimCLR, the positive pair being an image and its caption; recalled below.

CLIP in one paragraph. Two encoders map a batch of N (image, caption) pairs to unit vectors, u_i for the images and v_i for the captions, and the N \times N similarity matrix S_{ij} = u_i^\top v_j / \tau is read as two classification problems. Each image must pick its own caption among the N (softmax over row i), and each caption its own image (softmax over column j); the loss averages the two cross-entropies, and the temperature is learned. The negatives come from the other modality only. Zero-shot classification then scores an image against captions such as “a photo of a dog”. The Multimodal GenAI module (ch. 5) goes further; nothing here depends on it.

def nt_xent(z1, z2, tau=0.5):
    """z1, z2: (N, d) head outputs for the two views; normalized below."""
    import torch
    import torch.nn.functional as F
    z1, z2 = F.normalize(z1, dim=-1), F.normalize(z2, dim=-1)
    z = torch.cat([z1, z2], 0)                      # (2N, d)
    sim = z @ z.t() / tau                           # (2N, 2N)
    N = z1.size(0)
    mask = ~torch.eye(2 * N, dtype=torch.bool, device=z.device)
    sim_i = sim.masked_fill(~mask, float("-inf"))
    target = (torch.arange(2 * N, device=z.device) + N) % (2 * N)   # positive of i
    return F.cross_entropy(sim_i, target)           # softmax over the 2N - 1 other views

8. Self-supervision II: masking (MAE, BEiT, DINOv2)

Contrastive learning needs carefully designed augmentations and many negatives. This chapter presents the other route to label-free pre-training, borrowed from BERT: hide most of the image and train the network to fill in what is missing.

8.1. Masked Autoencoders (MAE)

He et al. (CVPR 2022): mask 75% of the patches and reconstruct the missing pixels. Two asymmetries make the whole design:

• asymmetric encoder: only the 25% of visible patches enter the ViT — the training cost is reduced accordingly: the token-wise parts (projections, MLP) cost 4 times less and the attention products 16 times less (and this is what makes high masking effective: the task remains difficult yet cheap);
• lightweight decoder: the masked tokens (learned embeddings + positions) are re-inserted after the encoder; a small decoder (often narrow and shallow) reconstructs each missing patch; the decoder is discarded at transfer time.

8.2. BEiT and DINOv2

• BEiT (Bao et al., ICLR 2022): replace the pixels with discrete visual tokens — a pre-trained VQ-VAE quantizes the patches into codes; the objective becomes a classification of the masked patches toward the codes (like BERT’s MLM — see the NLP module); in practice more stable than pixel regression;
• DINOv2 (Oquab et al., 2023): the contrastive + masking duo, trained on a curated corpus LVD-142M (automated selection: similarity with reference corpora, deduplication); distillation between a teacher (momentum-updated, centered — no labels) and multi-resolution students (definition 8.3) produces features that are ready to use (linear) for classification, depth, segmentation — one of the best current open backbones, and a cheap frozen backbone for the replication projects of ch. 2.

Centering alone would push the target towards the uniform distribution, and sharpening alone (T_{\mathrm{te}} \approx 0.04 against T_{\mathrm{st}} = 0.1) towards a one-hot one; together they avoid both collapses, two forms of the constant encoder of proposition 7.3, without any negative. The views are two large crops, seen by both networks, and several small crops seen by the student only (multi-crop).

DINOv2 adds three ingredients to this loss: an iBOT term, the same cross-entropy applied to the patch tokens that are masked in the student’s input (the masking of section 8.1, inside a distillation); a KoLeo regularizer that spreads the features of a batch over the sphere; and Sinkhorn–Knopp centering of the teacher in place of the moving average.

9. Training at scale: recipes and scaling laws

Large training runs cannot be tuned by trial and error. This chapter collects the recipe that makes them stable, the empirical laws that predict their loss from the model size and the data, and a parametrization that lets hyperparameters found on a small model transfer to a large one.

9.1. The modern recipe

• learning rate: linear warm-up (a few thousand steps — stabilizes Adam and the norms; indispensable for large batches) then cosine decay; decoupled weight decay (AdamW);
• linear scaling rule (Goyal et al., 2017): batch × k ⇒ lr × k (valid as long as the batch stays “under the horizon”; beyond, diminishing gains — hence LARS/LAMB which normalize the gradient per layer);
• regularization at scale: label smoothing — the target \mathrm{CE} is (1-\varepsilon) on the correct class and \varepsilon/(K-1) elsewhere; stochastic depth; mixup/cutmix;
• mixed precision: bf16 (or fp16 + loss scaling) — beware of reductions (sums in fp32) and sensitive functions; as chapter 1 stresses, a result obtained on one hardware configuration does not always transpose exactly to another;
• minimal distributed setup: data parallel (DDP); beyond that, state/optimizer sharding (FSDP/ZeRO); the Emerging Subjects in ML module covers distributed SGD in depth.

9.2. Scaling laws — Kaplan then Chinchilla

Kaplan’s own joint law has another form, L(N, D) = \Big[\big(N_c/N\big)^{\alpha_N/\alpha_D} + D_c/D\Big]^{\alpha_D}, and led to the opposite advice: at fixed compute, grow the model much faster than the data (N^\star \propto C^{0.73}). Chinchilla’s refit, with a learning-rate schedule matched to the length of each run, moved the allocation to roughly equal parts — theorem 9.2.

The general derivation is theorem 3.1 of the M2 Large Language Models module, with the same symbols; this module keeps the symmetric sketch and the replication angle (ch. 2.2).

9.3. MuP — transferring hyperparameters (sketch)

The practical problem: the scaling law applies to losses, but every model size requires its own hyperparameters (lr above all) — and one cannot grid-search a 70B model.

µP (muParametrization) (Yang & Hu — Tensor Programs IV–V): a rule for parametrizing the initializations and the lr by width (hidden layers; multipliers in d^{-1/2}, d^{-1}…) under which the training behavior becomes independent of width. One can then tune the hyperparameters on a small model and transfer them with zero trials to the large one (muTransfer) — spectacular savings on very large runs. We demonstrate the idea on an MLP of variable width (exercise 11) and refer to the paper for the full program.

The rule for Adam, the optimizer of section 9.1, fits in one table (Yang et al., table 3). Here n_{\mathrm{in}} is the fan-in of the weight matrix, and SP is the standard parametrization:

The factors are read relative to a base width at which \eta was tuned. Under SP, the updates of the hidden and output layers grow with the width, so the best \eta drifts as the model widens; under µP every layer’s update stays of order one, so the best \eta found on a narrow proxy stays close to the best on the wide model. In transformers, µP also replaces the attention scaling 1/\sqrt{d_k} by 1/d_k.

10. Beyond recognition: neural fields, inference, sampling and physics

The previous chapters trained networks to recognize. The second half of the official programme turns to networks that represent signals and scenes, that are inverted to explain observations, that sample, and that meet physics. This chapter gives the founding objects of each theme, with one worked example each; the instructors’ own research on them is in chapter 11, and section 10.6 points to the modules where they are developed.

10.1. Neural fields

A plain MLP on raw coordinates learns the low frequencies of a signal first and fits fine detail very slowly (the spectral bias of networks). Two remedies are standard: map the coordinates through sinusoids before the MLP (Fourier features, Tancik et al. 2020; the positional encoding of NeRF, Mildenhall et al. 2020), or use sine activations throughout (SIREN, Sitzmann et al. 2020).

10.2. Differentiability in deep learning and analysis-by-synthesis

Analysis-by-synthesis explains an observation y by searching for the parameters z of a generative process R (a renderer, a simulator, a decoder) that reproduce it: \min_z \|R(z) - y\|^2 + \lambda\,\Omega(z). When R is differentiable, this is solved by gradient descent through R: fitting a NeRF to photographs (ch. 10.1) is analysis-by-synthesis with z = \theta; recovering the pose, shape or lighting of an object by differentiable rendering is another instance. The difficulty is that many useful operations are not differentiable, or have zero gradient almost everywhere.

10.3. Amortized inference

Analysis-by-synthesis solves one optimization problem per observation. When the same kind of inference is needed for millions of observations, it pays to learn the map from observation to answer once, and then answer with a single forward pass: this is amortized inference.

10.4. Rethinking the role of sampling in modern deep learning

Sampling enters deep learning at three levels, and each has become a design choice rather than a default: which training data are sampled (mini-batches, hard negatives in contrastive learning, curated corpora such as LVD-142M in ch. 8.2); how a generative model is sampled (temperature, guidance, number of diffusion steps); and how many samples are drawn and selected at test time. The first level has a clean optimization answer.

10.5. Deep learning and physics

Physics and deep learning meet in both directions. Networks are used to solve physical problems: emulators trained on simulations (weather models such as GraphCast or FourCastNet produce global forecasts in seconds), and physics-informed networks that are trained to satisfy an equation. And physics supplies ideas to deep learning: diffusion models come from non-equilibrium thermodynamics (Sohl-Dickstein et al. 2015) and sample with Langevin-type dynamics; conservation laws and symmetries become equivariances (section 4.4).

10.6. Cross-references to the other modules

The detail of the generative and multimodal themes lives in the corresponding modules:

• CLIP and vision-language alignment: image-text NT-Xent (ch. 7), zero-shot classification — Multimodal GenAI module, chapter 5;
• diffusion: the core formulas are in Deep Learning P1, section 9.4; score matching and LDM — Multimodal GenAI module, chapters 3–4; discrete and cross-conditional diffusion is a research area of the team (Ye Zhu, ch. 11);
• text-to-image and multimodal evaluation (FID, CLIPScore) — Multimodal GenAI module, chapters 6–8;
• video generation and generative cinema: temporal segmentation, consistency, camera — the work of Xi Wang (ch. 11);
• MoE in LLMs and LLM quantization — M2 Large Language Models module, ch. 2 and section 6.4; LLM distillation — section 4.3 of the same module;
• GNNs and graphs — Graph ML module, chapters 5–7; the work of Johannes Lutzeyer (ch. 11) is connected to it.

Cross-cutting question of the module: where does the border between “advanced vision” and “multimodal” lie? Working answer: as soon as the loss puts two modalities (or two views!) in correspondence, we are in contrast/alignment; as soon as it generates, we are in the generative; the backbones (ch. 3–8) are shared by all.

11. Course research notes (Kalogeiton, Lutzeyer, Zhu, Wang)

The seminar closes with an encounter with local research: each instructor presents a slice of their work, in direct continuation of the chapters. Representative publications (all verified):

11.1. Vicky Kalogeiton — vision, video, action

• V. Kalogeiton, P. Weinzaepfel, V. Ferrari, C. Schmid, Action Tubelet Detector for Spatio-Temporal Action Localization (ACT-detector), ICCV 2017 — detecting tubelets: series of boxes linked in time, forerunner of end-to-end spatio-temporal action detection;
• V. Kalogeiton, P. Weinzaepfel, V. Ferrari, C. Schmid, Joint Learning of Object and Action Detectors, ICCV 2017 — unifying in a single network the detection of objects (ch. 1 of the Computer Vision (P2) module) and of actions;
• R. Ghermi, X. Wang, V. Kalogeiton, I. Laptev, Long Story Short: Story-level Video Understanding from 20K Short Films (SF20K), 2024 — 20k films, long-term questioning; “story” understanding beyond the short clip;
• Vicky Kalogeiton’s work on video, guidance and distillation is also read in the research notes of Multimodal Generative AI (CSC_52002_EP, period 2, ch. 9), a course whose 2026–2027 sheet names Xi Wang as head and does not list her.

11.2. Johannes Lutzeyer — graph representation learning

• G. Michel, G. Nikolentzos, J. Lutzeyer, M. Vazirgiannis, Path Neural Networks: Expressive and Accurate Graph Neural Networks, ICML 2023 — learning salient paths for message passing (see the Graph ML module, ch. 5);
• M. Chatzianastasis, J. Lutzeyer, G. Dasoulas, M. Vazirgiannis, Graph Ordering Attention Networks, AAAI 2023 — the order of neighbors as first-class information in attention;
• Node Feature Kernels Increase Graph Convolutional Network Performance (HAL) — improving GNNs on attribute-poor graphs; themes: GNN expressivity, interpretability, applications (transportation networks: predicting the effects of road policies on Paris). Johannes Lutzeyer co-directs year M1 of the master’s program and co-teaches the Graph ML module.

11.3. Ye Zhu — diffusion models, semantic control

• Y. Zhu, Y. Wu, K. Olszewski, J. Ren, S. Tulyakov, Y. Yan, Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation (CDCD), NeurIPS 2022 — discrete diffusion + contrastive loss for cross-modal music↔︎image generation;
• Y. Zhu, Y. Wu, Z. Deng, O. Russakovsky, Y. Yan, Boundary Guided Learning-Free Semantic Control with Diffusion Models (BoundaryDiffusion), NeurIPS 2023 — controlling the semantics of frozen generators via boundaries in latent space, without retraining;
• “Denoising Diffusion Probabilistic Models to Predict the ENSO” (2023, arXiv) — diffusion in the service of climate forecasting (El Niño): a bridge between data science × climate science. Research themes: discrete diffusion, generator control, scientific applications.

11.4. Xi Wang — video generation and generative cinema

• R. Ghermi, X. Wang, V. Kalogeiton, I. Laptev, Long Story Short (SF20K), 2024 — co-author; benchmarks and instruction-tuning for long-term video understanding;
• Xi Wang (VISTA/LIX post-doc, now assistant professor at l’X) works on cinematic video generation: camera, editing, narrative consistency — extending text-to-video models (see the Multimodal GenAI module, ch. 8) with a filmmaking grammar; profile and publications.

Exercises

1. ★ Patch embedding. For a 224×224 image, patches 16, d = 768: how many tokens, what size for the positional embedding, how many parameters in the projection; compare with patches 14 (ViT-B/14) — attention vs projection FLOPs.
2. ★ Complexities. Prove the “attention” row of proposition 4.1 (count every matrix product); for n = 196, d = 768, which term dominates; from which n does attention exceed the MLP?
3. ★ MoE. With n_E = 16, k = 2, d = 1024, expansion r = 4 (hidden width 4096), 24 layers: total parameters, active parameters, FLOPs/token; what fraction of the model can an 80 GB device serve in bf16? (params ≈ 2 GB per billion in bf16.)
4. ★ Separable. Derive the cost of a MobileNetV2 block (6× pointwise expansion, 3×3 depthwise, pointwise projection to d' channels) as a function of the input width d, of d' and of the map size. For (d, d') = (24, 96): (a) what share of the cost goes to each of the three layers? (b) The depthwise + projection pair replaces a standard 3×3 convolution on the expanded channels: show that the ratio is 1/d' + 1/9 (proposition 3.1 holds for any input width) and that it gives the ~8 factor here.
5. ★ Seeds. With the numbers of example 1.2, how many seeds would be needed for the 0.5% gap to become significant at 95%? (Statistical power; take \sigma \approx 0.55.)
6. ★★ Recipe vs architecture. ConvNeXt gains +2.7 points from the training recipe alone; if a future paper compares two architectures with different recipes, by how much might its “gain” be tainted? Propose a fair protocol (def. 1.1).
7. ★★ Balancing. Use the loss of section 6.1.
   • 1. Compute \sum_i f_i P_i when the gate probabilities and the routing are both uniform.
   • 2. Build an example with n_E = 3 experts, top-1 routing and two tokens where \sum_i f_i P_i < 1/n_E: the auxiliary loss is not minimized by balance in general.
   • 3. Explain why it nevertheless pushes towards balance in practice (look at its gradient with respect to the gate probabilities, f_i being treated as a constant).
   • 4. Why does the factor n_E in \mathcal{L}_{\mathrm{aux}} make its value at uniform routing independent of the number of experts, and why must f_i be divided by k for this to hold with top-2 routing?
8. ★★ NT-Xent. Implement the loss of ch. 7 and verify: (a) N = 1 makes the loss degenerate (no negatives — explain the code’s behavior); (b) the gradient with respect to z_i pushes toward z_j and away from the most similar negatives; (c) the effect of \tau \in \{0.1, 0.5, 2\} on the gradient norm.
9. ★★ MAE. On natural images, why does normalizing each patch by its own mean/variance change the task; estimate the loss of a constant decoder that predicts, for every masked patch, the mean of the visible patches (the bound to beat).
10. ★★ General Chinchilla. Redo the proof of theorem 9.2 with \alpha \neq \beta and show N^\star \propto C^{\beta/(\alpha+\beta)}, \qquad D^\star \propto C^{\alpha/(\alpha+\beta)}; apply to \alpha = 0.34, \beta = 0.28 (exponents \approx 0.45 and 0.55) and compare with the symmetric case.
11. ★★ µP sketch. On an MLP of width d with N(0, \sigma^2) initialization, compute the variance of the activations as a function of \sigma^2 and d (one layer); which dependence on d keeps the variance constant; how does this foreshadow hyperparameter transfer?
12. ★★★ Paper critique. Take the main table of DeiT: identify the baselines, seeds, budgets; what additional experiment would you ask the authors for; write a one-page “review” (structure: summary, strengths, weaknesses, questions).

Projects and assessment

Official assessment. The fiche above quotes the SynapseS 2026–2027 sheet: “a combination of homework, quizzes and team projects”, with “more details later”, and a numerical grade out of 20. The assessment section of the sheet for the LLGA track is not displayed (empty). Everything below is our proposal. The subjects follow the format of chapter 2 — one number, one replication, one budget, the deliverables of section 2.3 (shorter for solo subjects) — and the written exam can serve as the quiz part.

The projects exercise the master objectives O1 (deep learning foundations: architectures, objectives, scaling), O7 (hands-on practice under the replication protocol of definition 1.1) and O8 (trustworthy AI, through the robustness of section 4.5), with O3 (large language models) through the MoE and scaling subjects.

No end-of-year capstone uses this elective as a graded work package, but two group subjects double as ablations there: the self-supervised image tower of group project 2 for Y2.3, and the distilled edge model of group project 3 for the efficient-inference part of Y1.5.

Solo projects

1. [Solo · ≈ 25 h] A mini-MoE character language model and its routing. It connects this course to Large Language Models (ch. 2) and builds on the block of Deep Learning P1, ch. 6. Question: does load balancing prevent routing collapse, and does a sparse model beat a dense one at equal active compute?
   • Implement: a 6-layer character transformer whose MLPs are replaced by the MoE of section 6.1 (n_E = 8, top-1 and top-2), trained with and without the auxiliary loss; a dense baseline with the same FLOPs per token (proposition 6.2).
   • Data and hardware: enwik8 (the first 100 MB of English Wikipedia, Hutter Prize; the standard 90/5/5 split), Tiny Shakespeare for debugging; one free Colab or Kaggle T4 GPU.
   • Deliverables: code, a 4–6 page report with the curves of f_i over training and a probe of what each expert receives (letters, digits, markup, spaces), a 10-minute talk.
   • Chapters: ch. 1, 6, 9.
   • Success: bits per character with three seeds ± standard deviation for each model; the auxiliary loss at about \alpha_{\mathrm{aux}} under balanced routing, a max/min ratio of the f_i below 2 with the loss, and a documented collapse without it.
2. [Solo · ≈ 25 h] Paper study: hyperparameter transfer with µP (Yang, Hu et al. 2021, “Tensor Programs V”). Question: on a small model, does the best learning rate stay put as the width grows under µP, and drift under the standard parametrization?
   • Implement: an MLP (and, as a stretch, a 4-layer transformer) at widths 128 to 4 096, parametrized with the table of section 9.3 by hand or with Microsoft’s mup package; sweep Adam’s learning rate over powers of 2 at every width, in both parametrizations.
   • Data and hardware: CIFAR-10 (Krizhevsky 2009) for the MLP, WikiText-2 (Merity et al. 2017) for the transformer; a laptop or one free T4.
   • Deliverables: code, a 4–6 page report with loss-versus-learning-rate curves per width (the format of the paper’s figures 1 and 3), a 10-minute talk.
   • Chapters: ch. 1, 9 (exercise 11).
   • Success: three seeds per point; the optimal learning rate located with an interval at each width; the drift under SP and the stability under µP measured in octaves, or an honest explanation if they do not appear.
3. [Solo · ≈ 20 h] Paper study: Fourier features and spectral bias (Tancik et al. 2020). Question: how does the scale of a Fourier feature mapping set the trade-off between blur and noise when a field interpolates an image?
   • Implement: the session 7 lab extended — a 4-layer ReLU MLP from (x, y) to RGB, with no mapping, the NeRF encoding of example 10.2, and Gaussian features at scales 1, 10 and 100; a SIREN baseline (Sitzmann et al. 2020).
   • Data and hardware: 512 × 512 crops of DIV2K images (Agustsson & Timofte 2017), trained on every other pixel and tested on the rest, as in the paper; a laptop.
   • Deliverables: code, a 4-page report, a 10-minute talk.
   • Chapters: ch. 1, 10.1.
   • Success: test PSNR with three seeds per mapping; a curve of PSNR against the scale with an interior optimum; the gain over no mapping compared with the paper’s table (about 6 dB on natural images).

Group projects

1. [Group of 3 · ≈ 30 h each] Recipe or architecture? A fair small-budget ResNet, ConvNeXt and ViT study. Question: once the training recipe is the same, how much of the gap between the three families — in accuracy and in robustness — is left?
   • Implement: a ResNet-18, a small ConvNeXt and a ViT with 4 × 4 patches, at about 11 M parameters each, trained on CIFAR-100 under one shared modern recipe (AdamW, RandAugment, mixup, label smoothing) and under their native recipes; replay steps 1–3 of the ConvNeXt table (ch. 5) at this scale.
   • Robustness, to test the claims of Bai et al. (2021): AutoAttack at \epsilon = 1/255 on the standard-trained models (at 8/255 they all fall to about 0 %); single-step adversarial training at \epsilon = 8/255 (FGSM with a random start, Wong, Rice & Kolter 2020) of the ResNet and the ViT under the shared recipe, evaluated with PGD-10 and AutoAttack on 1 000 test images; mean accuracy on CIFAR-100-C (Hendrycks & Dietterich 2019).
   • Split: member A, ResNet and the shared recipe; member B, ViT; member C, ConvNeXt steps and the robustness harness.
   • Data and hardware: CIFAR-100 and CIFAR-100-C; about 60 T4-hours in all, within the free Kaggle quota of the three members.
   • Deliverables: code, a 6–8 page report (section 2.3), a 15-minute talk.
   • Chapters: ch. 1, 4, 4.5, 5.
   • Success: five seeds per configuration with Welch t-tests; the recipe effect and the architecture effect reported separately; every robustness claim backed by AutoAttack.
2. [Group of 3 · ≈ 30 h each] A self-supervised backbone shoot-out on STL-10: SimCLR, MAE and DINO. Question: at equal pre-training compute and without labels, which objective gives the best features, and for which evaluation?
   • Implement: SimCLR on a ResNet-18 (ch. 7), MAE on a ViT-Tiny (ch. 8.1), DINO on the same ViT-Tiny (definition 8.3), all at the same GPU-hour budget; a shared evaluation harness — linear probe, k-NN with k = 20, fine-tuning on the 5 000 labelled images.
   • Ablations: SimCLR without colour jitter; MAE at masking ratios of 50, 75 and 90 %; DINO without centering. Check the initial NT-Xent loss against \log(2N - 1) and show the collapse of proposition 7.3 by removing the negatives.
   • Split: one member per method; the harness is shared.
   • Data and hardware: STL-10 (Coates, Ng & Lee 2011): 100 000 unlabelled and 5 000 labelled 96 × 96 images; one free T4 per member.
   • Deliverables: code, a 6–8 page report, a 15-minute talk.
   • Chapters: ch. 1, 7, 8.
   • Success: compute matched within 10 %; three seeds per evaluation; each collapse diagnosed by the standard deviation of the normalized embeddings falling to zero.
3. [Group of 3 · ≈ 30 h each] Distil and certify: an edge-ready student with robustness guarantees. It connects this course to Computer Vision Applications (ch. 8, deployment). Question: what does distillation cost in accuracy, latency and robustness, and how loose is a Lipschitz certificate?
   • Implement: a ResNet-18 teacher and a MobileNetV2 student trained with the loss of definition 3.2 for T \in \{1, 4, 10\} and \lambda \in \{0.5, 0.9\}; export to ONNX and measure CPU latency (median and 99th percentile); PGD and AutoAttack accuracy; certified radii from proposition 4.7, with L bounded by the product of per-layer spectral norms (power iteration).
   • Split: member A, teacher and distillation sweep; member B, export and latency; member C, attacks and certificates.
   • Data and hardware: CIFAR-10; one free T4 and a laptop CPU for latency.
   • Deliverables: code, a 6–8 page report with an accuracy–latency plot, a 15-minute talk.
   • Chapters: ch. 1, 3, 4.5.
   • Success: three seeds per configuration; the student within 3 points of the teacher at a measured speed-up; certified and empirical robust accuracy reported side by side.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet and a non-programmable calculator allowed; four questions on 20 points — a course question, a computation and two short proofs.

1. Question (5 points). Course. (a) Explain why a Siamese objective that only pulls two views together can be minimized by a useless encoder. (b) Name the mechanism by which SimCLR, BYOL and DINO each avoid this. (c) Why does MAE mask 75 % of the patches where BERT masks 15 % of the tokens, and what does the high ratio save in compute?

   Expected: (a) proposition 7.3: the constant encoder gives similarity 1 for every pair, a global optimum. (b) SimCLR: the negatives in the denominator; BYOL: the asymmetric predictor, the stop-gradient and the momentum target; DINO: centering plus sharpening of the teacher, with a momentum teacher.

   3. Images are spatially redundant, so 15 % is solved by interpolation; at 75 % the task becomes semantic. The encoder sees a quarter of the tokens: token-wise costs fall by 4 and attention by 16.

2. Question (6 points). Computation. A ViT-S/16 (d = 384, 12 blocks, MLP expansion 4) reads 224 \times 224 images. (a) Number of tokens, parameters per block and in total, forward FLOPs per image. (b) Every MLP becomes a MoE with n_E = 8, top-2: total parameters, active parameters per token. (c) The value of \mathcal{L}_{\mathrm{aux}} under uniform routing with \alpha_{\mathrm{aux}} = 0.01.

   Expected: (a) n = 197; 12d^2 \approx 1.77M per block; with the patch projection (0.29M), about 21.5M in total.

   The forward pass costs 2 \times 21.5\text{M} \times 197 \approx 8.5 GFLOPs, plus attention 12 \times 4n^2 d \approx 0.7 GFLOPs, so about 9.2 GFLOPs (example 4.2).

   2. Per block, attention 4d^2 \approx 0.59M plus 8 \times 8d^2 \approx 9.4M: about 121M in total. Active: 0.59 + 2 \times 1.18 \approx 2.95M per block, about 36M per token.

   3. f_i = P_i = 1/8, so the loss is 0.01 \times 8 \times 8 \times 1/64 = 0.01.

3. Question (5 points). Short proof: compute-optimal allocation. Minimize A N^{-\alpha} + B D^{-\beta} under 6ND = C with \alpha \neq \beta. Show that N^\star \propto C^{\beta/(\alpha+\beta)} and D^\star \propto C^{\alpha/(\alpha+\beta)}. With (\alpha, \beta) = (0.34, 0.28), by how much does the number of tokens per parameter grow when the budget is multiplied by 10?

   Expected: substitute D = C/(6N); setting the derivative to zero gives N^{\alpha+\beta} = \frac{\alpha A}{\beta B}\,(C/6)^{\beta}, hence the exponent \beta/(\alpha+\beta) \approx 0.45, and D^\star = C/(6N^\star) has exponent \alpha/(\alpha+\beta) \approx 0.55. The ratio D^\star/N^\star grows as C^{0.097}: a factor 10^{0.097} \approx 1.25.

4. Question (4 points). Short proof: certificate. Prove proposition 4.7. Then take a classifier on 32 \times 32 \times 3 images with margin m = 3 and a bound L = 10: does the certificate cover every \ell_\infty perturbation of one grey level, \epsilon = 1/255?

   Expected: the proof of proposition 4.7 — each logit difference is \sqrt 2 L-Lipschitz, so the margin cannot change sign before \|\delta\|_2 = m/(\sqrt 2 L).

   Here the radius is 3/(10\sqrt 2) \approx 0.212, while the \ell_\infty ball of radius 1/255 reaches \ell_2 norm \sqrt{3072}/255 \approx 0.217: just not covered.

Grading rubric (proposal)

References

• A. Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT), ICLR 2021.
• H. Touvron et al., Training data-efficient image transformers & distillation through attention (DeiT), ICML 2021.
• Z. Liu et al., Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021; A ConvNet for the 2020s (ConvNeXt), CVPR 2022.
• N. Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, ICLR 2017.
• W. Fedus, B. Zoph, N. Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR 2022.
• A. Q. Jiang et al., Mixtral of Experts, 2024.
• T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A Simple Framework for Contrastive Learning of Visual Representations (SimCLR), ICML 2020.
• K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum Contrast for Unsupervised Visual Representation Learning (MoCo), CVPR 2020.
• J.-B. Grill et al., Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (BYOL), NeurIPS 2020.
• T. Wang, P. Isola, “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere”, ICML 2020.
• K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, Masked Autoencoders Are Scalable Vision Learners (MAE), CVPR 2022.
• H. Bao, L. Dong, F. Wei, BEiT: BERT Pre-Training of Image Transformers, ICLR 2022.
• M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, A. Joulin, Emerging Properties in Self-Supervised Vision Transformers (DINO), ICCV 2021.
• M. Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, 2023.
• A. Howard et al., MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, 2017; M. Sandler et al., MobileNetV2: Inverted Residuals and Linear Bottlenecks, CVPR 2018.
• M. Tan, Q. Le, EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, ICML 2019.
• G. Hinton, O. Vinyals, J. Dean, Distilling the Knowledge in a Neural Network, 2015.
• P. Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017.
• J. Kaplan et al., Scaling Laws for Neural Language Models, 2020; J. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), NeurIPS 2022.
• G. Yang, E. J. Hu et al., Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (µP/muTransfer), NeurIPS 2021.
• D. Picard, torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision, NeurIPS 2021 (workshop) — an author from l’X; X. Bouthillier et al., “Accounting for Variance in Machine Learning Benchmarks”, MLSys 2021.
• A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), ICML 2021 — detailed in the Multimodal GenAI module, ch. 5.
• Symmetry and robustness (sections 4.4–4.5): T. Cohen, M. Welling, Group Equivariant Convolutional Networks, ICML 2016; S. Kornblith, M. Norouzi, H. Lee, G. Hinton, Similarity of Neural Network Representations Revisited (CKA), ICML 2019; S. Ainsworth, J. Hayase, S. Srinivasa, Git Re-Basin: Merging Models modulo Permutation Symmetries, ICLR 2023; I. Goodfellow, J. Shlens, C. Szegedy, Explaining and Harnessing Adversarial Examples, ICLR 2015; A. Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018; F. Croce, M. Hein, Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks (AutoAttack), ICML 2020; D. Hendrycks, T. Dietterich, Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019; Y. Bai, J. Mei, A. Yuille, C. Xie, Are Transformers More Robust Than CNNs?, NeurIPS 2021; E. Wong, L. Rice, J. Z. Kolter, Fast is better than free: Revisiting adversarial training, ICLR 2020.
• Neural fields and analysis-by-synthesis (sections 10.1–10.2): B. Mildenhall et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, ECCV 2020; M. Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020; V. Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions (SIREN), NeurIPS 2020; S. Liu, T. Li, W. Chen, H. Li, Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning, ICCV 2019.
• Amortized inference and sampling (sections 10.3–10.4): C. Cremer, X. Li, D. Duvenaud, Inference Suboptimality in Variational Autoencoders, ICML 2018; Y. Kim et al., Semi-Amortized Variational Autoencoders, ICML 2018; A. Katharopoulos, F. Fleuret, Not All Samples Are Created Equal: Deep Learning with Importance Sampling, ICML 2018; J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, 2022; J. Song, C. Meng, S. Ermon, Denoising Diffusion Implicit Models, ICLR 2021.
• Deep learning and physics (section 10.5): M. Raissi, P. Perdikaris, G. E. Karniadakis, “Physics-informed neural networks”, Journal of Computational Physics 378, 2019; J. Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics, ICML 2015.
• Research group: V. Kalogeiton et al., ACT-detector and Joint Learning of Object and Action Detectors, ICCV 2017; R. Ghermi, X. Wang, V. Kalogeiton, I. Laptev, SF20K, 2024; G. Michel, G. Nikolentzos, J. Lutzeyer, M. Vazirgiannis, Path Neural Networks, ICML 2023; M. Chatzianastasis, J. Lutzeyer et al., Graph Ordering Attention Networks, AAAI 2023; Y. Zhu et al., Discrete Contrastive Diffusion (CDCD), NeurIPS 2022; BoundaryDiffusion, NeurIPS 2023.
• Lab tools: PyTorch, transformers, open_clip, torch.compile — and the experiment tracking of your choice (wandb/tensorboard).