Multimodal Generative AI

Contents

• Introduction and overview
• 0. Review: probabilities, networks, ELBO
• 1. VAEs and latent spaces
• 2. GANs: adversary, equilibrium, Wasserstein
• 3. Diffusion: DDPM and score matching
• 4. Guidance, DDIM, flows and latent diffusion
• 5. CLIP and vision-language alignment
• 6. Text-to-image: the complete pipeline
• 7. Evaluation, costs, ethics
• 8. Beyond images: video, sound, robot actions, 3D, multimodal LLMs
• 9. Course research notes (Kalogeiton et al.)
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who knows a CNN / autoencoder and wants to generate or align text×image. Empirical first: a VAE whose reconstructions blur (KL weight \beta_{\mathrm{KL}} too high); a GAN that mode-collapses after 20 k steps; a CLIP whose retrieval drops as soon as the domain leaves the web — then we name ELBO, diffusion, guidance.

VAE, GAN, diffusion, CLIP and text-to-image. Bridge to neural rendering and generative control in M2.

What you will be able to do

• Derive and train a VAE: ELBO, reparameterization, latent spaces and their properties.
• Explain adversarial GAN training, its equilibria and instabilities, and the contribution of the Wasserstein variant.
• Describe generation by diffusion: DDPM and score matching, and the link between the two viewpoints.
• Use guidance (notably classifier-free), accelerated sampling (DDIM) and latent diffusion.
• Relate diffusion, normalizing flows and flow matching through the change of variables and the probability-flow ODE, and train a flow-matching model.
• Understand CLIP and the alignment of vision-language spaces that makes text-to-image possible.
• Assemble the complete text-to-image pipeline and identify the role of each component.
• Explain how video, audio and robot actions are generated with the same latent-diffusion and token-modeling recipes (video DiT, codec tokens, diffusion policies, VLA models).
• Situate the further extensions of generation: 3D, multimodal LLMs and world models.
• Evaluate generative models (notably with FID), estimate their computational costs and discuss their ethical issues.

Official objectives and syllabus (2026-2027 catalog)

The official 2026–2027 sheet (French title “IA générative multimodale”, scheduled session “Multimodal Generative AI”, period X-3A-P2, from 04/01/2027 to 19/03/2027, taught in English) carries a two-paragraph description in English.

Generative AI lets machines interpret and process data through several modalities and sensors — vision, audio, touch — and its multimodal development is presented as both a major scientific advance and a paradigm shift for industries from healthcare to automotive, with tools such as Dall-E 3, Midjourney and GPT-Vision building on large vision models, and autonomous-driving companies expanding. The class announces that it goes beyond CNNs and transformers to the advanced topics of multimodal generative AI: “from image to video generation, sound, robot actions”.

Nothing else is displayed: no teaching volume, no prerequisites, no assessment modality, no programme.

Every element of the description now has a section in these notes. Our extensions, beyond the official description: the refresher chapter 0 (Gaussians, networks, and recalls of the ELBO proved in Deep Learning) and chapter 9 (research notes on video generation, guidance and distillation, self-study).

Where this course fits in the program

Module of period 2 of M1 (teaching volume not displayed on the 2026–2027 sheet; 36 h and 4.5 ECTS per the LLGA curriculum wiki), of which Xi Wang is the official head (“responsable” on the 2026–2027 sheet). It is one of the Period 2 “Choose 1” electives, and the Graph × LLM route of the home page preselects it. Official prerequisites: not displayed on the sheet. This is where the master’s program shifts from learning representations to multimodal generation.

• Prerequisites — Deep Learning (P1) for architectures and training. Its P2 variant belongs to the same “Choose 1” group as this course, so every student here took the P1 course, whose chapter 9 proves the ELBO and the GAN equilibrium; chapters 0–2 recall these results instead of deriving them again. The Signal Processing module is useful for the signal and spectrogram processing mobilized by the audio and video extensions.
• Downstream modules — it directly grounds Control in Generative AI (M2), which teaches how to steer these generative models; it prepares Advanced Topics in Large Language Models (M2) for multimodal LLMs, and Speech Technology for Conversational AI (M2) for speech generation.

Course outline

• Review: probabilities, networks, ELBO — the probabilistic and neural foundation, recalled from Deep Learning before the generative models.
• VAEs and latent spaces — encoding and decoding with a distribution: the first generative building block.
• GANs: adversary, equilibrium, Wasserstein — learning through a game between generator and discriminator.
• Diffusion: DDPM and score matching — denoising progressively: the paradigm that dominates generation.
• Guidance, DDIM, flows and latent diffusion — controlling generation, speeding it up, reading it as a flow, and making it computationally affordable.
• CLIP and vision-language alignment — the common space where text and image meet.
• Text-to-image: the complete pipeline — assembling tokenizer, latent space, diffusion and decoder.
• Evaluation, costs, ethics — measuring quality (FID among others), weighing costs, questioning uses.
• Beyond images: video, sound, robot actions, 3D, multimodal LLMs — the modalities named by the official description, then the current frontiers.
• Course research notes (Kalogeiton et al.) — a tour of the research work related to the course (self-study).
• Exercises and References — systematic training and sources to go further.

Session plan

36 h in class = 12 sessions of 3 h. The SynapseS 2026–2027 sheet of this course displays no teaching volume (neither in-person hours nor a lecture/tutorial split, and its description is shorter than in earlier versions), but the LLGA curriculum page of the programme wiki (msct.dix.polytechnique.fr/llga, last modified on 9 September 2026 by the programme director) states that “all subsequent M1 courses are 36h and will credit 4.5 ECTS”.

No session-by-session programme is published either, so the split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core models and their derivations, 3 = current frontiers; the level never goes down from one session to the next.

The ELBO, the reparameterization trick, the Gaussian KL and the GAN equilibrium are proved in Deep Learning (P1), which every student of this course took. Chapters 0–2 now take two sessions instead of four. Of the two sessions this frees, one goes to flows and flow matching (session 6), the other to more time for robot actions and the project (sessions 11–12).

Self-study, outside the 36 h (about 3 h in total): ch. 9 Course research notes — action tubelets, guidance-weight schedulers, video generation and one-step distillation (≈ 3 h).

How to study this module

• Start with the review section and make sure you can derive the ELBO again (Deep Learning, section 9.2): VAEs, and later latent diffusion, collapse without it.
• Train a small VAE on a simple dataset before tackling GANs: the contrast between the two ways of learning to generate is very instructive.
• For each model, write down in black and white the loss function and the sampling algorithm: in generative modeling, these two lines summarize everything.
• Follow the thread of latent spaces: VAEs, then latent diffusion, then CLIP form a progression, not three independent chapters.
• The Wasserstein distance, the Wasserstein-GAN and the FID illuminate one another: chapter 2 states the two transport results it needs (Kantorovich–Rubinstein duality, the Gaussian W_2) with proof sketches. Machine learning with Optimal Transport and Algorithmic Fairness sits in the same “Choose 1” group, so you cannot take it as well; read its chapters 2 and 4 for the proofs, and if you chose OT instead of this course, these notes are your reading for the generative side.
• Redo the exercises of each section before moving on to the next, and keep the chapter on evaluation, costs and ethics as a critical reading grid for everything else.

0. Review: probabilities, networks, ELBO

This module builds the entire edifice of multimodal generative models on four bricks: Gaussians (the only probabilistic object diffusion will need), the evidence lower bound (ELBO), the reparameterization trick (making sampling differentiable) and encoder-decoder architectures (CNN, U-Net, attention).

Every student of this course took Deep Learning in Period 1, whose chapter 9 proves the ELBO, the reparameterization trick and the Gaussian KL. This chapter recalls them in the form used later, and proves only what is new here: the sum of Gaussians that makes diffusion computable.

Networks. A CNN applies the same filter locally; its receptive field grows with depth, trading resolution for context. The U-Net couples an encoder (downsampling, global semantics) and a decoder (upsampling, local precision) through skip connections that short-circuit the bottleneck: it is the reference denoising architecture for diffusion.

Cross-attention injects the text: \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V, \qquad Q = H_{\text{image}}W_Q, \quad K = E_{\text{text}}W_K, \quad V = E_{\text{text}}W_V, where the rows of H_{\text{image}} are the spatial activations of the denoiser, the rows of E_{\text{text}} the tokens of the prompt, and d_k the width of a query: every image position “queries” the text (cf. Deep Learning module, ch. 6).

1. VAEs and latent spaces

Concrete scene: a team generates product images from short briefs. With a guidance weight that is too high, the images are saturated and leave the distribution of real photos; too low, the prompt is ignored. Such failures can only be diagnosed by someone who knows which objective each generator optimizes. This chapter starts with the simplest one, the VAE, which learns to compress an image into a few latent numbers and to decode it back.

Hierarchy and sampling. In a HiVAE, the ELBO becomes \mathbb{E}_{q(z_1, z_2\mid x)}\big[\log p_\theta(x \mid z_1) + \log p(z_1 \mid z_2) + \log p(z_2) - \log q_\phi(z_1 \mid x, z_2) - \log q_\phi(z_2 \mid x)\big], each level compressing the previous one (the “low” scales govern composition, the “high” ones the details — exactly the levels of a U-Net).

To generate: sample z \sim \mathcal{N}(0, I) (or the levels of the learned prior, as in VQ-VAE-2), then decode in a single network pass; latent traversals (interpolating z between two images) are the visual test of the continuity of the learned space.

import torch

class MiniVAE(torch.nn.Module):
    """Minimal convolutional VAE for MNIST-like data (28x28)."""
    def __init__(self, d=8):
        super().__init__()
        self.enc = torch.nn.Sequential(            # x: (B,1,28,28)
            torch.nn.Conv2d(1, 32, 4, 2, 1), torch.nn.ReLU(),   # 14x14
            torch.nn.Conv2d(32, 64, 4, 2, 1), torch.nn.ReLU(),  # 7x7
            torch.nn.Flatten(), torch.nn.Linear(64*7*7, 2*d))
        self.dec = torch.nn.Sequential(
            torch.nn.Linear(d, 64*7*7), torch.nn.Unflatten(1, (64, 7, 7)),
            torch.nn.ConvTranspose2d(64, 32, 4, 2, 1), torch.nn.ReLU(),
            torch.nn.ConvTranspose2d(32, 1, 4, 2, 1), torch.nn.Sigmoid())
    def forward(self, x):
        h = self.enc(x); mu, logvar = h.chunk(2, dim=1)
        z = mu + torch.exp(0.5*logvar) * torch.randn_like(mu)   # reparametrization
        return self.dec(z), mu, logvar

def vae_loss(x, xh, mu, logvar, beta_kl=1.0):
    rec = torch.nn.functional.binary_cross_entropy(
        xh, x, reduction="sum") / x.size(0)                      # reconstruction
    kl  = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / x.size(0)  # prop. 0.4
    return rec + beta_kl * kl                                   # -ELBO when beta_kl = 1

2. GANs: adversary, equilibrium, Wasserstein

VAE samples are blurry because the decoder is trained to be right on average. A GAN drops the likelihood altogether: a generator learns to produce samples that a second network, the discriminator, cannot tell apart from real data. This chapter analyzes that game, its equilibrium, its failure modes, and the Wasserstein variant that repairs its gradients.

3. Diffusion: DDPM and score matching

Diffusion models generate by reversing a destruction: noise is added to an image step by step until nothing is left, and a network learns to undo one step at a time. Each step is a small Gaussian problem, which makes training a stable regression — the reason why diffusion replaced GANs at the top of image generation.

Theorem 3.3 is about the conditional score, given the clean image x_0. Sampling needs the score \nabla\log p_t(x_t) of the noised data distribution p_t itself, which no formula gives. The next theorem shows that regressing on the conditional score is enough.

import torch

def forward_diffusion(x0, t, alphas_bar, eps=None):
    """x_t = sqrt(a_bar_t) x0 + sqrt(1 - a_bar_t) * eps   (in one step!)"""
    if eps is None: eps = torch.randn_like(x0)
    return alphas_bar[t].sqrt()[:, None, None, None] * x0 \
         + (1 - alphas_bar[t]).sqrt()[:, None, None, None] * eps, eps

# Training (typical DDPM loop):
# x0 = batch of images; t ~ U{1..T}
xt, eps = forward_diffusion(x0, t, alphas_bar)
loss = torch.nn.functional.mse_loss(unet(xt, t), eps)   # predict the noise
loss.backward()

# Sampling (T steps, formula of model 3.2):
x = torch.randn(n, c, h, w)                    # x_T ~ N(0, I)
for t in reversed(range(T)):
    eps = unet(x, t)
    mu = (x - beta[t]/(1 - alphas_bar[t]).sqrt() * eps) / alpha[t].sqrt()
    x = mu + beta[t].sqrt() * torch.randn_like(x) if t > 0 else mu

4. Guidance, DDIM, flows and latent diffusion

A trained diffusion model samples from the whole data distribution, slowly, in pixel space. This chapter adds the ingredients that turn it into a product: guidance, to follow a condition such as a prompt (4.1); faster samplers, to need 20 steps instead of 1000 (4.2); and latent diffusion, to work in a compressed space (4.4). In between, section 4.3 reads diffusion as a flow and derives flow matching, its simpler successor.

4.1. Guidance

4.2. Faster sampling: DDIM and the probability-flow ODE

Fast solvers. DPM-Solver handles the probability-flow ODE by separating the linear part (exactly integrable, since the drift f is linear in x) from the non-linear part (the score): “exponential integrator” schemes at 10–15 steps. Distillation (consistency models, LCM, adversarial distillation — cf. ch. 9, Di[M]o) targets 1 to 4 steps, and quantization (fp16, fp8) compresses the compute — a panorama revisited in M2 ML Ops.

4.3. Flows: from change of variables to flow matching

A diffusion model turns noise into data through many small stochastic steps. A flow does it with an invertible map, or with an ordinary differential equation. This section builds flows from the change-of-variables formula, shows that the probability-flow ODE of 4.2 is one, and derives flow matching.

Flow matching is the training rule behind Stable Diffusion 3, the action expert of π0 (section 8.3) and flow-based speech synthesis (Speech Technology, ch. 7); the change of variables is also the starting point of the graph flows of Graph Generative Models, ch. 4.

Continuous flows. Let a velocity field move points by \dot x = v(x, t). The law p_t of x_t then obeys the continuity equation, and along a trajectory the log-density changes by the divergence of v (Chen et al., 2018): \partial_t p_t + \nabla\cdot(p_t\,v) = 0, \qquad \frac{d}{dt}\log p_t(x_t) = -\nabla\cdot v(x_t, t). This is proposition 4.3 for the infinitesimal map x \mapsto x + v\,dt, whose log-determinant is \nabla\cdot v\,dt to first order.

The probability-flow ODE of 4.2 is such a flow, with v = f - \tfrac12 g^2\nabla\log p_t, and deterministic DDIM is one discretization of it. A trained diffusion model is therefore a continuous flow, learned by score matching instead of maximum likelihood. Flow matching keeps the flow and drops the diffusion: it chooses the path and regresses its velocity directly.

# Flow matching on two moons (definition 4.5, theorem 4.6): CPU, about a minute
import torch
from sklearn.datasets import make_moons

data = torch.tensor(make_moons(4096, noise=0.05)[0], dtype=torch.float32)
v = torch.nn.Sequential(torch.nn.Linear(3, 128), torch.nn.SiLU(),
                        torch.nn.Linear(128, 128), torch.nn.SiLU(),
                        torch.nn.Linear(128, 2))              # v_theta(x_t, t)
opt = torch.optim.Adam(v.parameters(), 1e-3)
for step in range(5000):
    x1 = data[torch.randint(len(data), (256,))]               # data, t = 1
    x0 = torch.randn_like(x1)                                 # noise, t = 0
    t = torch.rand(256, 1)
    xt = (1 - t) * x0 + t * x1                                # straight path
    loss = ((v(torch.cat([xt, t], 1)) - (x1 - x0))**2).mean() # CFM loss
    opt.zero_grad(); loss.backward(); opt.step()

@torch.no_grad()
def sample(n, steps=20):                                      # Euler on dx/dt = v
    x = torch.randn(n, 2)
    for k in range(steps):
        t = torch.full((n, 1), k / steps)
        x = x + v(torch.cat([x, t], 1)) / steps
    return x

In our run, the mean distance from a sample to the nearest of the 4096 training points was 0.13 with 2 Euler steps, 0.09 with 5 and 0.08 with 20: a handful of steps already lands on the moons.

4.4. Latent diffusion, ControlNet and DiT

• Latent diffusion (Rombach et al., 2022): an autoencoder (VQ or KL, ch. 1) compressing the image by a factor 4–8 per side, then diffusion in the latent; the denoiser (U-Net or DiT) receives the prompt via cross-attention (ch. 0, Networks) at several resolution levels; CLIP text encoder: this is the Stable Diffusion architecture;
• ControlNet (Zhang et al., 2023): conditioning by edges/depth/pose. One copies the blocks of the frozen U-Net encoder (“locked” weights \theta_{\mathrm{lock}}), trains a clone \theta_c fed by the conditioning map c, and combines them as y_c = \mathcal{F}(x;\theta_{\mathrm{lock}}) + \mathcal{Z}\big(\mathcal{F}\big(x + \mathcal{Z}(c;\theta_{z1});\,\theta_c\big);\,\theta_{z2}\big), where the zero-convolutions \mathcal{Z} (1×1 convolutions initialized to zero) guarantee that at step 0 both correction terms vanish and y_c = \mathcal{F}(x;\theta_{\mathrm{lock}}): ControlNet starts as the unchanged model and learns only the correction — the same “identity start” trick as zero residuals;
• Inpainting / img2img: noise the starting image up to a step t_0 (img2img: t_0 tunes fidelity to the start; strong = freedom), then denoise while imposing at every step x_t = m\odot x_t^{\mathrm{generated}} + (1-m)\odot x_t^{\mathrm{noised}} where m is the mask (the RePaint mixing rule; variants relax it at the end of the chain).

# Minimal DiT: a block modulated by time (adaLN-Zero)
class DiTBlock(torch.nn.Module):
    def __init__(self, d, n_heads):
        super().__init__()
        self.norm1 = torch.nn.LayerNorm(d, elementwise_affine=False)
        self.attn  = torch.nn.MultiheadAttention(d, n_heads, batch_first=True)
        self.norm2 = torch.nn.LayerNorm(d, elementwise_affine=False)
        self.mlp   = torch.nn.Sequential(torch.nn.Linear(d, 4*d), torch.nn.GELU(),
                                         torch.nn.Linear(4*d, d))
        self.mod = torch.nn.Sequential(torch.nn.SiLU(), torch.nn.Linear(d, 6*d))
        torch.nn.init.zeros_(self.mod[-1].weight)   # adaLN-Zero: scales, shifts
        torch.nn.init.zeros_(self.mod[-1].bias)     # and gates all start at 0
    def forward(self, h, cond):                     # h: (B, N, d) patches, cond: (B, d)
        g1, b1, a1, g2, b2, a2 = self.mod(cond).unsqueeze(1).chunk(6, dim=-1)
        u = self.norm1(h) * (1 + g1) + b1           # 1 + g: LN(h) survives at init
        h = h + a1 * self.attn(u, u, u, need_weights=False)[0]
        return h + a2 * self.mlp(self.norm2(h) * (1 + g2) + b2)

5. CLIP and vision-language alignment

To steer an image generator with a sentence, text and images must be comparable: a caption and the picture it describes should map to nearby vectors. CLIP learns such a shared space from 400 million captioned images found on the web; this chapter derives its contrastive loss and shows what the shared space makes possible.

The ViT in one paragraph (recap of Deep Learning, 6.6). A Vision Transformer cuts an H \times W image into P \times P patches, flattens each patch and maps it linearly to a d-dimensional token, adds a position embedding and one learned [CLS] token, then runs standard transformer blocks (self-attention + MLP) over the HW/P^2 + 1 tokens. The projected output of the [CLS] token is the image embedding v.

For CLIP’s ViT-B/32 at 224 \times 224 pixels, that is 7 \times 7 = 49 patch tokens plus one.

from PIL import Image
from transformers import CLIPProcessor, CLIPModel
import torch

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc  = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

img = Image.open("graphe.png")
texts = ["a graph drawing", "a cat", "a molecule", "a city map"]
inputs = proc(text=texts, images=img, return_tensors="pt", padding=True)
with torch.no_grad():
    out = model(**inputs)
probs = out.logits_per_image.softmax(dim=1)   # zero-shot classification
print(dict(zip(texts, probs[0].tolist())))

# InfoNCE loss "by hand" (batch of aligned pairs):
# logits = v_norm @ t_norm.T / tau ; targets = arange(N)
# loss = 0.5 * (CE(logits, tgt) + CE(logits.T, tgt))

6. Text-to-image: the complete pipeline

This chapter assembles the components of the previous chapters — tokenizer, text encoder, latent diffusion with guidance, decoder — into the pipeline of a modern text-to-image system, and explains the controls that users actually turn: prompts, negative prompts and seeds.

• Prompt engineering, quantitatively: the prompt acts as a direction in conditioning space: adding “photorealistic, 85 mm” shifts the embeddings and hence every CFG step. Measurable effects: position (the first tokens weigh more), saturation (beyond ~50 effective tokens, diminishing returns), syntactic weighting “(word:1.3)” rescales the contribution of that token to the conditioning at inference time;
• Negative prompt: the generalized version of CFG with two conditions — \varnothing is replaced by c_-: \tilde\epsilon = \epsilon_\theta(x_t,t,c_-) + w\big(\epsilon_\theta(x_t,t,c_+) - \epsilon_\theta(x_t,t,c_-)\big), the negative direction is repelled (thm. 4.1): “blurry, watermark, extra fingers”;
• Seeds: the seed fixes x_T (and the solver’s randomness); same seed + same prompt = same image under deterministic DDIM; one sweeps the seeds to pick the best realization before refining the prompt — the actual workflow of practitioners;
• Inference cost: (number of steps) × (denoiser cost) + decoding; latency governs the industrial choice (fast solvers, 1-step distillation when diversity matters less).

The pipeline in about forty lines. The Diffusers library (Hugging Face) ships the whole chain as one call, but writing the loop by hand shows each step of Architecture 6.1. The code below loads the four components of Stable Diffusion v1.4, encodes the prompt and the negative prompt, and runs DDIM with classifier-free guidance in latent space before decoding. On a free cloud GPU in half precision it takes a few seconds per image; on a CPU, a few minutes.

import torch
from diffusers import AutoencoderKL, UNet2DConditionModel, DDIMScheduler
from transformers import CLIPTokenizer, CLIPTextModel

repo = "CompVis/stable-diffusion-v1-4"
dev = "cuda" if torch.cuda.is_available() else "cpu"
dt = torch.float16 if dev == "cuda" else torch.float32
tok = CLIPTokenizer.from_pretrained(repo, subfolder="tokenizer")            # step 1
txt = CLIPTextModel.from_pretrained(repo, subfolder="text_encoder",
                                    torch_dtype=dt).to(dev)                 # step 2
unet = UNet2DConditionModel.from_pretrained(repo, subfolder="unet",
                                            torch_dtype=dt).to(dev)         # step 3
vae = AutoencoderKL.from_pretrained(repo, subfolder="vae",
                                    torch_dtype=dt).to(dev)                 # step 4
sched = DDIMScheduler.from_pretrained(repo, subfolder="scheduler")          # eta = 0

@torch.no_grad()
def encode(prompt):                                  # (1, 77, 768) text tokens
    ids = tok([prompt], padding="max_length", max_length=tok.model_max_length,
              truncation=True, return_tensors="pt").input_ids.to(dev)
    return txt(ids)[0]

@torch.no_grad()
def generate(prompt, negative="", w=7.5, steps=30, seed=0, size=512):
    cond, uncond = encode(prompt), encode(negative)  # c_+ and c_- (empty = no condition)
    g = torch.Generator().manual_seed(seed)          # the seed fixes x_T
    z = torch.randn((1, unet.config.in_channels, size // 8, size // 8), generator=g)
    z = (z * sched.init_noise_sigma).to(dev, dt)
    sched.set_timesteps(steps)                       # DDIM sub-sequence (ch. 4.2)
    for t in sched.timesteps:
        zz = sched.scale_model_input(torch.cat([z, z]), t)
        e_u, e_c = unet(zz, t, encoder_hidden_states=torch.cat([uncond, cond])).sample.chunk(2)
        eps = e_u + w * (e_c - e_u)                  # CFG, theorem 4.1
        z = sched.step(eps, t, z).prev_sample
    x = vae.decode(z / vae.config.scaling_factor).sample      # latent -> pixels
    x = ((x.clamp(-1, 1) + 1) * 127.5)[0].permute(1, 2, 0)
    return x.round().to(torch.uint8).cpu().numpy()            # (512, 512, 3) image

# from PIL import Image
# Image.fromarray(generate("a watercolor of a molecule", negative="blurry")).save("out.png")

Each knob of this section is one argument: w is the guidance weight, negative the negative prompt, seed the initial noise, steps the length of the DDIM sub-sequence. Exercise 14 starts from this loop.

7. Evaluation, costs, ethics

A generator produces a distribution rather than a single answer, so its quality cannot be read off one sample, and its value depends as much on its cost and its uses as on its images. This chapter gives the metrics and their pitfalls, then the costs and the ethical questions that any deployment raises.

• Costs: diffusion training = thousands of GPU-hours; inference = N denoiser steps + decoding; the quality/latency trade-off governs the industrial choice (fast solvers, distillation, quantization);
• Ethics — a balanced note:
  • Deepfakes: detectors (including detectors using diffusion itself), provenance;
  • Watermarking and provenance: C2PA signed metadata (an industrial coalition: camera → editor → publication), invisible watermarks robust to re-compression — none is infallible alone, defense in depth;
  • Copyright: training on web data — ongoing legal debates (opt-out, dataset licenses, filtering); on the generation side: similarity with training works must be monitored;
  • Bias: corpus biases (representation, stereotypes) reappear amplified at generation — output audits, data documentation;
  • European AI Act: a risk-tiered approach; synthetic generators are targeted there via transparency (marking of generated content, model documentation) — in connection with the M2 ethics seminar.

8. Beyond images: video, sound, robot actions, 3D, multimodal LLMs

Images were the testing ground; the same recipes — a compressed latent space, a denoiser or a transformer, a contrastive bridge to text — now extend to time (video), to sound, to the actions of a robot, to space (3D) and to language models that read images. The official description names exactly this progression, “from image to video generation, sound, robot actions”, and the sensors it lists (vision, audio, touch); sections 8.1–8.3 treat the three generated modalities, section 8.4 the remaining frontiers.

8.1. Video generation

8.2. Sound generation

8.3. Robot actions and tactile sensing

When a generative model drives a robot, the generated modality is the action: given observations o_t — camera images, joint positions, touch — produce the next commands a_t \in \mathbb{R}^d (end-effector displacement, gripper opening). Demonstrations by a human operator provide the data (imitation learning, RL module, section 6.5).

8.4. Multimodal LLMs, 3D and world models

• Captioning / VQA: image encoder (ViT) + text decoder (transformer); trained on pairs; graphical VQA (“what is the shortest path?”) — visual reasoning;
• Multimodal LLMs: how to make an LLM “see”?
  • Linear projector (LLaVA): h_v = W\,p_i + b on each ViT patch — the image “words” are projected into the LLM’s embedding space: simple, effective, lots of tokens;
  • Q-Former (BLIP-2): 32 learned query vectors that cross-attend to the image features and produce 32 embeddings — a compact “visual summary”; freezes the LLM while pretraining the bridge;
  • Two-stage training protocol: (1) alignment pretraining on immense (image, caption) corpora — learning the correspondence; (2) multimodal instruction tuning — conversations, reasoning, answering;
  • Audio: the codec tokens of section 8.2 are injected in the same way — same pattern, another modality;
• 3D: NeRF (sketch): an MLP (x, d) \mapsto (\sigma, c) (density, color) is optimized by differentiable volume rendering: C(r) = \int_{t_n}^{t_f} T(t)\,\sigma\big(r(t)\big)\,c\big(r(t), d\big)\,dt, \qquad T(t) = \exp\!\Big(-\!\int_{t_n}^{t}\sigma\big(r(s)\big)ds\Big), that is, “the opacity accumulated along the ray weights the colors” (the box below computes it). Recent 3D generative models prefer diffusion over point clouds and meshes — discretization, operators, correspondence: this is the syllabus of the M2 Geometric Data module, downstream;
• World models: action-conditioned generation of interactive videos — the video models of section 8.1 conditioned on the actions of section 8.3, used as learned simulators (link with the model-based RL of the RL module, section 6.4; Genie-like systems);
• Generative structured data: text-to-graph (generating the schema of a network), graph-to-text (a report from a knowledge graph) — the exact intersection of the Graph ML and NLP modules;
• Multimodal agents: perceiving (screenshot) and acting (click, code) — GUI agents combine vision + LLM + tools (RL module, ch. 7).

9. Course research notes (Kalogeiton et al.)

The 2026–2027 sheet names Xi Wang as head of the course; Vicky Kalogeiton, whose publications are presented below, is not listed on it, and this chapter is kept as research context for the course’s themes — video, guidance, distillation. Vicky Kalogeiton (CVLab — École Polytechnique / ECL; formerly Inria Grenoble, LEAR team) has worked since her PhD on a coherent program: localizing and understanding objects and actions in videos, and more recently, generating them. The three publications below exactly mark out the syllabus of this course — with video as the common thread.

Exercises

1. ★ Composition of noises. Show by induction that q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar\alpha_t} x_0, (1-\bar\alpha_t) I); why does this closed form make training efficient?
2. ★ KL of Gaussians. Prove proposition 0.4 in the scalar case by computing \mathbb{E}_q[\log q - \log p]; check that the KL is zero iff \mu_1 = \mu_2, \sigma_1 = \sigma_2 and strictly positive otherwise.
3. ★ Mode collapse. Simulate a 2D Gaussian→ring GAN; observe the collapse; compare with a diffusion model trained on the same set (diversity via LPIPS).
4. ★ CFG. Show that w = 1 recovers the conditional prediction and w = 0 the unconditional one; generate a grid w \in \{0, 1, 5, 10\} and comment on the trade-off.
5. ★ CLIP zero-shot. Compare the zero-shot accuracy of CLIP with the accuracy of a ResNet trained on 10 custom classes, as a function of the training size; where do the curves cross?
6. ★ Epsilon or score? Using x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon, show that \epsilon = -\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p(x_t \mid x_0).
7. ★ Truncation. For a StyleGAN whose w approximately follows a centered Gaussian, show that the rule w' = \bar w + \psi(w - \bar w), \psi \in [0,1], reduces the variance of the generated style and comment on the quality/diversity trade-off as \psi \to 0.
8. ★★ DDPM ELBO. Decompose the log-likelihood \log p_\theta(x_0) into per-step terms and show that the noise MSE dominates (the KLs between Gaussians are closed-form).
9. ★★ FID and Wasserstein. Recall the formula for W_2^2 between Gaussians (proposition 2.6) and show that the FID is an estimate of it (features; sampling bias).
10. ★★ Noise vs mean. Parameterize the denoiser as predicting x_0 (\hat x_0 = x_\theta(x_t, t)); express the corresponding loss and explain when this parameterization is preferable. Hint: recovering \hat x_0 from a predicted noise multiplies the noise error by \sqrt{(1-\bar\alpha_t)/\bar\alpha_t}, which blows up at large t (low signal-to-noise ratio); that is where predicting x_0, or the velocity, pays off.
11. ★★ InfoNCE temperature. In the contrastive loss of ch. 5, show the effect of \tau \to 0 (peaked distribution, gradients concentrated on the hardest negatives) and of \tau \to \infty (uniform gradients, slowed learning).
12. ★★ Video and audio budgets. (a) Redo Example 8.2 for a 10-second clip at 24 fps and 768 \times 768, with the same compression and patch sizes; give N, the full-attention pair count and the factorized one. (b) For a codec at 50 frames per second with 4 codebooks of 2048 entries, compute the bitrate, the number of tokens of a 30-second music clip and the number of decoding steps with and without the delay pattern of Example 8.4.
13. ★★ Diffusion policy on a toy task. Generate 2D demonstrations that avoid an obstacle on the left or on the right (Example 8.7); train (a) an MLP regressor with squared error and (b) a small conditional diffusion model over 8-step action chunks (Definition 8.6); report the collision rate of each and plot sampled trajectories.
14. ★★★ Project. Complete pipeline: light fine-tuning of a latent diffusion model on a small thematic set (LoRA on cross-attention), with parameterizable CFG; evaluate FID/CLIPScore before/after and analyze the effect of the guidance weight w — also testing variable schedules w(t) (cf. ch. 9).

import torch
G = torch.nn.Sequential(torch.nn.Linear(2, 64), torch.nn.ReLU(),
                        torch.nn.Linear(64, 64), torch.nn.ReLU(),
                        torch.nn.Linear(64, 2))          # noise -> 2D point
D = torch.nn.Sequential(torch.nn.Linear(2, 64), torch.nn.ReLU(),
                        torch.nn.Linear(64, 1))          # point -> logit

def sample_ring(n, r=1.0, noise=.05):                    # true law: a ring
    a = torch.rand(n) * 2 * torch.pi
    return torch.stack([r*torch.cos(a), r*torch.sin(a)], 1) + noise*torch.randn(n, 2)

optG, optD = torch.optim.Adam(G.parameters(), 2e-4), torch.optim.Adam(D.parameters(), 2e-4)
bce = torch.nn.functional.binary_cross_entropy_with_logits
for step in range(5000):
    x = sample_ring(256); z = torch.randn(256, 2)
    lossD = bce(D(x), torch.ones(256, 1)) + bce(D(G(z)), torch.zeros(256, 1))
    optD.zero_grad(); lossD.backward(); optD.step()
    z = torch.randn(256, 2)
    lossG = bce(D(G(z)), torch.ones(256, 1))             # non-saturated form
    optG.zero_grad(); lossG.backward(); optG.step()
    if step % 1000 == 0: print(step, float(lossD), float(lossG))
# Observe: G concentrates on an arc of the circle (mode collapse);
# measure diversity: average distance between generated pairs.

Projects and assessment

The official assessment is not displayed: the fiche above reads “Not displayed: the 2026–2027 sheet gives no assessment modality, and its LLGA section shows no resit rule or pass threshold”. The subjects below are proposals of these notes, not official rules: a solo or group project can serve as the continuous assessment, and the exam proposal as a final written exam if the teaching team holds one.

The projects exercise mainly O2 (generative modelling theory: diffusion, guidance, flows) and O6 (multimodal generation of images, sound and actions), O5 through the text–graph CLIP subject, O7 (hands-on work with real models and data) and O8 through the model cards (costs, bias, provenance).

Solo projects

1. [Solo · ≈ 20 h] Paper study: guidance-weight schedules under the microscope (Wang, Dufour, Andreou, Cani, Fernández Abrevaya, Picard and Kalogeiton, TMLR 2024). Does a guidance weight w(t) that varies along sampling really beat a constant weight, at equal compute?
   • Models and data: the class-conditional DiT-XL/2 at 256 px (Peebles and Xie, 2023; facebook/DiT-XL-2-256 in Diffusers) on 50 ImageNet classes, with the 2,500 ImageNet validation images of these classes as reference; then a text-to-image check with the Stable Diffusion loop of ch. 6 on 200 COCO 2017 validation captions.
   • Work: constant, linearly increasing, linearly decreasing and bell-shaped schedules at two mean weights, plus a negative-prompt variant (ch. 4.1, ch. 6); 20 DDIM steps (ch. 4.2); FID and KID, CLIPScore and mean pairwise LPIPS at equal N, backbone stated (ch. 7).
   • Success criterion: a fidelity–diversity Pareto plot with bootstrap error bars, and a clear yes or no on the paper’s main claim (ch. 9).
   • Deliverables: code, a 4–6 page report, a 10-minute talk; chapters 4, 6, 7 and 9.
2. [Solo · ≈ 20 h] Diffusion policy against regression on multimodal demonstrations. Exercise 13 grown into a small simulator: when does a generative policy beat a regressor, and at what latency?
   • Data: bimodal demonstrations from your own 2D navigation simulator (an obstacle passed on the left or on the right), then the low-dimensional Push-T dataset released with Chi et al. (2023).
   • Work: a squared-error regressor and a conditional diffusion policy over action chunks of 8–16 steps (Definition 8.6), sampled with 5–10 DDIM steps in receding horizon (ch. 4.2, 8.3); as an option, a flow-matching head (4.3), as in π0.
   • Success criterion: success and collision rates over 100 episodes, mode coverage (share of left and right passes), and control latency per step.
   • Deliverables: code, report, talk; chapters 3, 4 and 8. It connects to imitation learning in Reinforcement Learning, 6.5.
3. [Solo · ≈ 15 h] Flow matching against DDPM at equal compute. Is a straight path cheaper to sample than a diffusion?
   • Data: two moons and a ring of 8 Gaussians (synthetic), then MNIST at 28 × 28.
   • Work: the flow-matching code of 4.3 and a DDPM with the same network and the same number of gradient steps (ch. 3); samplers with 1, 2, 5, 10, 20 and 50 steps (Euler for the flow, DDIM for the diffusion); a Monte Carlo estimate of the loss floor C of theorem 4.6, compared with the final training loss.
   • Success criterion: quality against the number of steps, with 3 seeds: W_2 to held-out points in 2D (POT library), FID on the features of a small MNIST classifier.
   • Deliverables: code, report, talk; chapters 3 and 4.

Group projects

1. [Group of 3 · ≈ 30 h each] LoRA fine-tuning of a latent diffusion model, with a guidance analysis (exercise 14). How far does a LoRA of a few megabytes move a text-to-image model toward a theme, and how does the best guidance weight move with it?
   • Model and data: Stable Diffusion v1.4 (ch. 6); a small captioned thematic set, such as the Naruto BLIP-captions set of the Diffusers LoRA tutorial (lambdalabs/naruto-blip-captions) or a set you caption yourself with BLIP; licence checked (the Naruto set states none, so its samples stay out of public deliverables).
   • Work: LoRA of rank 4–16 on the attention projections of the denoiser, held-out captions; a guidance grid over w and the schedules w(t) of ch. 9; negative prompts.
   • Split: data, captions and training; sampling grid and guidance analysis; evaluation and a model card (GPU-hours, bias audit on prompts that depict people, provenance and watermark, ch. 7).
   • Success criterion: a before/after table at equal N with the backbone stated (FID or KID against held-out images, CLIPScore, LPIPS diversity), and the weight w that maximizes CLIPScore at fixed diversity, before and after LoRA.
   • Deliverables: code, an 8–10 page joint report with the model card, a 20-minute talk; chapters 4–7 and 9.
2. [Group of 3 · ≈ 30 h each] Text–graph CLIP and a text-guided graph diffusion for molecules. Can a contrastive text–molecule space retrieve molecules from their descriptions, then steer a graph generator toward them?
   • Data: ChEBI-20 (Edwards, Zhai and Ji, 2021), about 33,000 PubChem molecules with one natural-language description each, official split; molecular graphs from the SMILES strings with RDKit.
   • Work: a GIN encoder over molecular graphs (Graph ML, ch. 5) and a frozen pretrained text encoder with a trained projection (Text Mining & NLP, ch. 7), aligned by symmetric InfoNCE (ch. 5). Then a small discrete graph diffusion, DiGress-style (Graph ML, ch. 9), on the molecules of at most 20 heavy atoms, conditioned on the text embedding with condition dropout and classifier-free guidance (ch. 4.1).
   • Split: graph encoder and data pipeline; text side, contrastive training and retrieval; conditional graph diffusion and its evaluation.
   • Success criterion: text-to-molecule retrieval on the test split (MRR, Hits@1 and Hits@10, set against the numbers of Text2Mol); validity, uniqueness and novelty of generated molecules (RDKit), and their alignment with the prompt in the contrastive space as w grows.
   • Deliverables: code, joint report, talk; chapters 3–5 and 7. It is the core of the Y1.4 project, which adds a reward and RL fine-tuning (Reinforcement Learning, ch. 5).
3. [Group of 3 · ≈ 30 h each] From codec tokens to text-to-sound. How far does a small codec-token transformer with the delay pattern go, set against a pretrained latent audio diffusion model?
   • Data: Clotho v2.1 (Drossos, Lipping and Virtanen, 2020), several thousand Freesound clips of 15–30 s with five captions each, under Creative Commons licences; clips cut to 10 s.
   • Work: tokenize with the pretrained 24 kHz EnCodec (Défossez et al., 2022), keeping 4 codebooks (8.2); train a small decoder-only transformer with the delay pattern of MusicGen (Example 8.4), conditioned on frozen CLAP text embeddings with condition dropout and classifier-free guidance (ch. 4.1, ch. 5). Baseline: the pretrained AudioLDM (Liu et al., 2023) through Diffusers.
   • Split: tokenization, delay pattern and data; transformer, conditioning and sampling; evaluation and listening test.
   • Success criterion: Fréchet Audio Distance (Kilgour et al., 2019) and CLAP score on held-out captions, a blind preference test with at least 10 listeners, and the real-time factor of generation.
   • Deliverables: code, joint report, talk with audio samples; chapters 4, 5, 7 and 8. It leads to Speech Technology, ch. 6–7.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, a simple calculator allowed; four questions.

1. Question (5 points). Explain classifier-free guidance: how the model is trained, the sampling rule, the distribution it targets, the effect of the weight w, its cost per step, and how a negative prompt fits in.
   • Expected: condition dropout (about 10 %) trains one network for both the conditional and the unconditional noise prediction.
   • The rule of theorem 4.1, with w = 0 unconditional, w = 1 conditional and w > 1 an extrapolation.
   • By Bayes, the guided prediction is the score of p(x_t \mid c)^w\,p(x_t)^{1-w}: higher fidelity, lower diversity, saturation at large w.
   • Two network evaluations per step; a negative prompt replaces the empty condition by c_-, pushing samples away from it.
2. Question (6 points). A DDPM with T = 3 steps has the schedule \beta = (0.1, 0.2, 0.3).
   • 1. Compute \bar\alpha_1, \bar\alpha_2, \bar\alpha_3 and the law of x_3 given x_0.
   • 2. For x_0 = 2 and \epsilon = -1, compute x_3.
   • 3. Compute \tilde\beta_3 and the mean of model 3.2 at step 3 when the network predicts \epsilon exactly; check it against \tilde\mu_3(x_3, x_0).
   • Expected: (a) \bar\alpha = 0.9,\ 0.72,\ 0.504, so x_3 \mid x_0 \sim \mathcal{N}(0.710\,x_0,\ 0.496).
   • 2. x_3 = 0.710 \times 2 - 0.704 = 0.716.
   • 3. \tilde\beta_3 = 0.3 \times 0.28/0.496 \approx 0.169. The mean is (0.716 + 0.3/0.704)/\sqrt{0.7} \approx 1.364.
   • Check with Step 2 of the proof of model 3.2: \tilde\mu_3 = \frac{\sqrt{0.72}\times 0.3}{0.496}\times 2 + \frac{\sqrt{0.7}\times 0.28}{0.496}\times 0.716 \approx 1.026 + 0.338 = 1.364.
3. Question (5 points). Flow matching with the straight path of definition 4.5.
   • 1. Prove that the function v minimizing \mathbb{E}\|v(x_t, t) - (x_1 - x_0)\|^2 is u_t(x) = \mathbb{E}[x_1 - x_0 \mid x_t = x], and that the excess loss of any other v is \mathbb{E}\|v - u_t\|^2.
   • 2. With x_0 \sim \mathcal{N}(0, I), show that u_t(x_t) = (x_t - \mathbb{E}[x_0 \mid x_t])/t, and deduce that a flow-matching model accepts classifier-free guidance.
   • Expected: (a) write Y = x_1 - x_0; the residual Y - \mathbb{E}[Y \mid x_t] is orthogonal to every function of x_t, which splits the loss into \mathbb{E}\|v - u_t\|^2 plus a constant (theorem 4.6).
   • 2. Take conditional expectations in x_1 = (x_t - (1-t)\,x_0)/t and subtract \mathbb{E}[x_0 \mid x_t].
   • The map from the noise prediction to the velocity is affine with the same coefficients for every condition, so mixing velocities with the CFG rule equals mixing noise predictions.
4. Question (4 points). In the plane, let P_\theta be the law of (\theta, Z) with Z uniform on [0, 1]: a vertical segment at abscissa \theta.
   • 1. Compute \mathrm{JS}(P_0 \| P_\theta), \mathrm{KL}(P_\theta \| P_0) and W_1(P_0, P_\theta).
   • 2. Say what they imply for training a GAN and a WGAN (construction 2.4).
   • Expected: for \theta \neq 0 the supports are disjoint, so \mathrm{JS} = \log 2 and the KL is infinite; all three vanish at \theta = 0.
   • W_1 = |\theta|: the coupling that shifts each point by \theta costs |\theta|, and the 1-Lipschitz function f(x, y) = -\mathrm{sign}(\theta)\,x reaches this value in the dual of construction 2.4: \mathbb{E}_{P_0}[f] = 0 and \mathbb{E}_{P_\theta}[f] = -|\theta|.
   • JS is constant away from 0, so the GAN generator gets no gradient; W_1 has gradient \mathrm{sign}(\theta), the useful signal the WGAN critic provides (Arjovsky et al., 2017).

Grading rubric (proposal)

References

• D. Kingma & M. Welling, Auto-Encoding Variational Bayes, ICLR 2014 — the VAE; I. Higgins et al., β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, ICLR 2017 (OpenReview).
• A. van den Oord et al., Neural Discrete Representation Learning (VQ-VAE), NeurIPS 2017; A. Razavi et al., Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019.
• I. Goodfellow et al., GAN, NIPS 2014; M. Arjovsky et al., Wasserstein GAN, ICML 2017; I. Gulrajani et al., Improved Training of Wasserstein GANs (gradient penalty), NeurIPS 2017.
• T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, Improved Techniques for Training GANs (minibatch discrimination, Inception Score), NeurIPS 2016.
• T. Karras et al., A Style-Based Generator Architecture (StyleGAN), CVPR 2019; StyleGAN2, CVPR 2020; StyleGAN3, NeurIPS 2021.
• P. Isola et al., Image-to-Image Translation with Conditional Adversarial Networks (Pix2Pix), CVPR 2017; J.-Y. Zhu et al., CycleGAN, ICCV 2017.
• J. Ho, A. Jain, P. Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS 2020; A. Nichol & P. Dhariwal, Improved DDPM (cosine schedule), ICML 2021.
• J. Song et al., DDIM, ICLR 2021; Y. Song et al., Score-Based Generative Modeling through SDEs, ICLR 2021; C. Lu et al., DPM-Solver, NeurIPS 2022.
• Score matching and Langevin (ch. 3): P. Vincent, A Connection Between Score Matching and Denoising Autoencoders, Neural Computation 23(7), 2011; Y. Song & S. Ermon, Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019.
• Flows and flow matching (section 4.3): L. Dinh, J. Sohl-Dickstein, S. Bengio, Density Estimation using Real NVP, ICLR 2017; R. T. Q. Chen et al., Neural Ordinary Differential Equations, NeurIPS 2018; Y. Lipman et al., Flow Matching for Generative Modeling, ICLR 2023; X. Liu, C. Gong, Q. Liu, Flow Straight and Fast: Rectified Flow, ICLR 2023; P. Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Stable Diffusion 3), ICML 2024.
• R. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022 — Stable Diffusion; W. Peebles & S. Xie, DiT, ICCV 2023; L. Zhang et al., ControlNet, ICCV 2023.
• A. Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), ICML 2021; A. van den Oord et al., Representation Learning with Contrastive Predictive Coding (InfoNCE), 2018; R. Girdhar et al., ImageBind, CVPR 2023.
• J. Ho & T. Salimans, Classifier-Free Diffusion Guidance, 2022; B. Mildenhall et al., NeRF, ECCV 2020; J. Li et al., BLIP-2 (Q-Former), ICML 2023; H. Liu et al., LLaVA (linear projector), NeurIPS 2023.
• Video (section 8.1): J. Ho et al., Video Diffusion Models, NeurIPS 2022; A. Blattmann et al., Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (Video LDM), CVPR 2023; T. Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric & Challenges (FVD), 2018.
• Sound (section 8.2): N. Zeghidour et al., SoundStream, 2021; A. Défossez et al., High Fidelity Neural Audio Compression (EnCodec), 2022; Z. Borsos et al., AudioLM, 2023; J. Copet et al., Simple and Controllable Music Generation (MusicGen), NeurIPS 2023; H. Liu et al., AudioLDM, ICML 2023; J. Kong et al., HiFi-GAN, NeurIPS 2020.
• Robot actions and driving (section 8.3): C. Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023; A. Brohan et al., RT-2: Vision-Language-Action Models, CoRL 2023; M. Kim et al., OpenVLA, 2024; K. Black et al., π0: A Vision-Language-Action Flow Model for General Robot Control, 2024; A. Hu et al., GAIA-1: A Generative World Model for Autonomous Driving, 2023.
• Publications by V. Kalogeiton used in ch. 9: ACT-detector (ICCV 2017); Joint Learning of Object and Action Detectors (ICCV 2017); Analysis of Classifier-Free Guidance Weight Schedulers (TMLR 2024); Don’t drop your samples! (CVPR 2024); AKiRa (CVPR 2025); Di[M]o (ICCV 2025).
• A. Kotelnikov et al., TabDDPM, ICML 2023 / C. Vignac et al., DiGress, ICLR 2023 — diffusion on tabular/graph data (LLGA bridge).
• R. Zhang et al., LPIPS, CVPR 2018; J. Hessel et al., CLIPScore, EMNLP 2021.
• Text–graph alignment (ch. 5): C. Edwards, C. Zhai, H. Ji, Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries (ChEBI-20), EMNLP 2021; S. Liu et al., Multi-modal Molecule Structure–Text Model for Text-based Retrieval and Editing (MoleculeSTM), Nature Machine Intelligence 2023.
• Projects (data and metrics): T.-Y. Lin et al., Microsoft COCO: Common Objects in Context, ECCV 2014; K. Drossos, S. Lipping, T. Virtanen, Clotho: An Audio Captioning Dataset, ICASSP 2020; K. Kilgour et al., Fréchet Audio Distance, Interspeech 2019; Y. Wu et al., Large-Scale Contrastive Language-Audio Pretraining (LAION-CLAP), ICASSP 2023.
• HuggingFace Diffusers — the lab tool.