Large Language Models

Contents

• Introduction and overview
• 0. Background: from word2vec to transformers
• 1. Tokenization
• 2. The architecture in depth
• 3. Pre-training
• 4. Post-training: prompting, SFT and PEFT
• 5. RLHF, DPO, GRPO
• 6. Running an LLM: inference
• 7. RAG in depth
• 8. Agents, multimodality and perspectives
• 9. Evaluation and safety of LLMs
• 10. Research (Shang)
• Exercises / Labs
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who has already called a chat API or fine-tuned a small model and wants to know why the token cost explodes, why a RAG system hallucinates, or why DPO beats RLHF on a given budget.

Empirical first: we start from what goes wrong in practice — a perplexity that drops while the model stays useless, a KV-cache that runs out of GPU memory, a leaderboard score that does not survive production — and only then name the tools (tokenizer, scaling laws, alignment) that explain it.

The module covers the full LLM pipeline, from tokenization to safety, as theory and labs. It is a pillar of the M2: most other M2 modules build on it. The official volume is 24 h; these notes go deeper in places and flag what they add to the official description.

What you will be able to do

• Explain the complete pipeline for building an LLM, from tokenization to post-training.
• Adapt a model cheaply — prompting and in-context learning, LoRA/QLoRA, distillation — and know when each is enough.
• Implement alignment methods — SFT, RLHF, DPO, GRPO — and discuss their trade-offs.
• Run an LLM in inference and master its costs and limits.
• Build an in-depth RAG, beyond the basic tutorial.
• Explore agents, multimodality and the perspectives they open.
• Evaluate the performance and safety of an LLM with rigorous protocols.
• Find your way in the literature and Shang’s research work.

Official objectives and syllabus

Official description (master’s wiki): the SynapseS 2026-2027 sheet of this course unit is not yet published (the S3/S4 tracks of the catalog are currently empty); the description confirmed below is that of the wiki, which specifies the volume (24 h), credits (2 ECTS) and instructor.

An immersion into LLMs, mixing essential theory and hands-on work to develop both practical skills and conceptual understanding — with a view to the LLM development and deployment professions.

The syllabus starts with a brief panorama of historical NLP techniques, then addresses the transformer architecture (attention mechanism, tokenization) — the heart of modern LLMs —, the pre-training objectives (masked/denoising and causal language modeling, foundations of BERT, GPT and T5), the post-training techniques that refine a pre-trained model — instruction tuning (SFT), reinforcement learning from human feedback (PPO/DPO) and from verifiable rewards (GRPO) — and, finally, applications and future directions: RAG, agents, multimodality, alternative architectures.

The following table maps these official themes to the sections of the material.

Beyond this official description, the material adds three deepenings: inference in practice (ch. 6), LLM evaluation and safety (ch. 9) and the reading of Shang’s research (ch. 10).

Where this course fits in the program

An M2 module (period 1), 24 h for 2 ECTS, M2 badge. Three M1 courses prepare it: Introduction to Text Mining and NLP for text processing, Deep Learning for the transformer architecture, and Reinforcement Learning and Autonomous Agents whose PPO sheds light on the RLHF and GRPO of the alignment chapter.

Downstream, it is the gateway to Advanced Topics in Large Language Models, Explainability, Security, Privacy of LLMs, ML Ops and LLM Engineering Principles and Speech Technology for Conversational AI.

Course outline

• Background: from word2vec to transformers — the trajectory leading from word embeddings to the architecture at the center of the module.
• Tokenization — how text becomes a sequence of tokens, and why this choice matters so much.
• The architecture in depth — the transformer dissected: attention, blocks, variants and implementation details.
• Pre-training — objectives, data and costs of the model’s founding stage.
• Post-training: prompting, SFT and PEFT — going from the raw language model to the usable assistant: in-context learning first, then supervised fine-tuning, LoRA and distillation.
• RLHF, DPO, GRPO — reinforcement learning methods and their alternatives for aligning models.
• Running an LLM: inference — serving a model: constraints, optimizations and costs.
• RAG in depth — plugging models into external knowledge, seriously.
• Agents, multimodality and perspectives — LLMs that act, see and listen.
• Evaluation and safety of LLMs — measuring what models really do, and containing their risks.
• Research (Shang) — the papers of Guokan Shang, read with the module’s grid (self-study).
• Exercises / Labs — hands-on sessions that follow each stage of the pipeline.
• Projects and assessment — solo and group project subjects, a written-exam proposal and a grading rubric.
• References — the papers and sources of each chapter.

Session plan

24 h in class = 8 sessions of 3 h. The volume is the one the master’s wiki gives for this course unit (24 h, 2 ECTS; the SynapseS 2026–2027 sheet is not yet published); no session-by-session programme is published, so the split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced theory; the level never goes down from one session to the next.

Self-study, outside the 24 h (about 6.5 h in total): ch. 8.2b Vision-language models, worked, with Ex. 6 (≈ 2 h) · ch. 9.7 Privacy and bias of LLMs (≈ 1.5 h) · ch. 10 Research (Shang) — FrugalScore, curriculum pre-training, low-resource languages (≈ 3 h).

How to study this module

• Do the labs alongside the chapters: the LLM pipeline is understood hands-on.
• Keep the full pipeline in mind — tokenization, pre-training, alignment, inference — and place each new development in it.
• Review the M1 reinforcement learning module before the RLHF chapter: PPO is assumed known there.
• Read the key papers as you go: each chapter has one or two that sum it up.
• For each method, ask what it costs in compute and data: that is often what decides in practice.
• Test evaluation concepts on your own model outputs: measuring is harder than it looks.

0. Background: from word2vec to transformers

This chapter places the module in its history: it recalls what M1 already established and the sequence of ideas that led to today’s LLMs, so that each later chapter has a clear starting point.

The trajectory: n-grams → static embeddings → recurrent networks (LSTM) → attention (Bahdanau 2014) → transformer (Vaswani 2017) → BERT/GPT (2018-2019) → scaling (GPT-3, 2020) → alignment (InstructGPT 2022, RLHF) → general-purpose, multimodal, reasoning and agentic LLMs (2023-2026).

This module covers every technical stage of this stack, from the tokenizer (ch. 1) to safety (ch. 9), proving results that M1 left as claims.

1. Tokenization

Before a model sees any text, a tokenizer cuts it into units from a fixed vocabulary; this chapter explains how that vocabulary is built and why the choice weighs on cost and quality.

A concrete scene: send the same prompt in French and in English to a Llama-style tokenizer. The French version typically costs 15–25% more tokens, so your context budget shrinks, and a rare word splits into six subwords — the model “understands” less and bills more.

This is why BPE, WordPiece and Unigram come before the architecture.

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
okapi = tiktoken.encoding_for_model("gpt-3.5-turbo")   # cl100k vocabulary

fr = "Les graphes pour les LLMs : une révolution pour la recherche d'information."
de = "Donaudampfschifffahrtsgesellschaftskapitän"
for s in (fr, de, "1234 + 5678 = ?"):
    print(len(okapi.encode(s)), "vs", len(enc.encode(s)), "->", s[:40])

# "safe" tokenization: special tokens are NOT decoded by default
b = enc.encode("end <|endoftext|> next", allowed_special=set())
print("ids :", b, "| decode round-trip ok :", enc.decode(b))

2. The architecture in depth

A language model must turn a sequence of token embeddings into a distribution over the next token, at a cost that stays manageable for long sequences. This chapter dissects the decoder-only transformer that does this in every modern LLM, and the design choices (normalization, positions, attention heads, experts) that make it trainable and cheap to run.

• Parameters: where are they? embedding (|\mathcal{V}| \times d), attention (4d^2/block), MLP (8d^2/block: two matrices with a 4× expansion, or three SwiGLU matrices with hidden size \approx 8d/3; Llama 3 widens it to 3.5d, about 10.5d^2) — the rule \#\text{params} \approx 12\, n_{\text{layers}}\, d^2 and the training memory cost (parameters + Adam + activations + gradients: ~20 bytes/parameter in mixed precision; cf. ch. 3 for actual compute);
• LLGA bridge: attention is message passing on a token graph (Graph ML module ch. 5, Advanced GNN); graph transformers inherit from it.

3. Pre-training

Pre-training gives the model its knowledge of language by training it on trillions of tokens of raw text. This chapter answers three questions: which objective to optimize, which data to feed, and how to split a fixed compute budget between model size and data size.

3.1. The data pipeline: quality, filtering, deduplication

• Curation: web → rule-based filtering (length, code/punctuation ratio, languages), then model-based filtering (quality classifiers trained on reference pages — e.g. distilled judges); controlled mixture (web, code, books, articles, mathematics, synthetic data generated then filtered);
• MinHash-LSH deduplication (proved in NLP ch. 8.3): k-gram shingles → MinHash signatures (a min-hash collision has probability equal to the Jaccard similarity) → LSH bands/ranks with \mathbb{P}_{\text{detect}}(J)=1-(1-J^r)^b → remove near-duplicates down to a threshold. At scale: tens of TB of text, approximately linear clustering; measured effect: less memorization, better benchmark (deduplicating is regularizing — LLMs learn duplicates “twice”);
• Mixtures and curriculum: per-domain proportions (annealing: raise quality/lower web at the end of training); easy→hard ordering (by perplexity) improves compute efficiency — demonstrated at scale by Shang (ch. 10, “Beyond Random Sampling”);
• Multilingual: balancing per language (otherwise the tokenizer and the model drift toward English — cf. ch. 1 and 10).

3.2. Scaling laws: decomposition and the Chinchilla optimum

4. Post-training: prompting, SFT and PEFT

A pre-trained model only continues text; it does not follow instructions or hold a conversation. This chapter goes from the cheapest way to adapt it to the most invasive one.

It starts with prompting, which changes no weight (4.0), then supervised fine-tuning (4.1) and the reason a preference stage is still needed (4.2, continued in chapter 5). It ends with LoRA, QLoRA and distillation, which make fine-tuning affordable (4.3).

4.0. Prompting and in-context learning

Before changing any weight, one can change the input. A well-prompted model is the zero-cost baseline that every fine-tuning must beat.

• Order and format sensitivity: with the same demonstrations, some orders give near state-of-the-art accuracy and others near chance (Lu et al., 2022). Models favour the most frequent and the last label of the prompt; contextual calibration rescales by the probabilities given to a content-free input such as “N/A” (Zhao et al., 2021).
• What the demonstrations teach: the format, the label space and the input distribution matter more than whether the demonstration labels are correct (Min et al., 2022).
• Chain-of-thought (Wei et al., 2022): demonstrations that show intermediate steps, or simply “Let’s think step by step” (Kojima et al., 2022), raise accuracy on multi-step problems for large models. Sampling several chains and voting is self-consistency (RLVR box of ch. 5); training the model to produce good chains with verifiable rewards is GRPO (ch. 5).
• Practice: a system prompt for role and constraints, delimiters around untrusted content (9.4), an explicit output format, 3–8 diverse demonstrations, and the protocol of 9.6 to choose between prompts (Ex. 4).

Reading: ICL locates a task already seen in pre-training rather than learning a new one — hence the weak effect of wrong labels, and the failures on tasks absent from pre-training.

4.1. Supervised fine-tuning

# SFT with loss masking (HF TRL pseudo-code)
from trl import SFTTrainer

def format(example):                    # chat template + masked labels
    prompt  = f"<|user|>\n{example['question']}\n<|assistant|>\n"
    completion = example["answer"]
    return {"prompt": prompt, "completion": completion}
# - the loss is computed ONLY on the completion tokens (TRL SFTTrainer
#   does the masking automatically; in raw PyTorch: labels = -100 on the prompt)

trainer = SFTTrainer(model="meta-llama/Llama-3.1-8B",
                     train_dataset=dataset.map(format), ...)
trainer.train()

# pitfalls seen in the lab: template != inference template -> performance drop;
# padding token confused with eos; overly long answers -> silent truncation

4.2. Why alignment beyond SFT

4.3. Parameter-efficient fine-tuning and distillation

Full fine-tuning of an 8B model with Adam needs about 16–20 bytes per parameter (ch. 2), well over 100 GB. Two families avoid this cost: train a few new parameters on top of a frozen model (LoRA, QLoRA), or train a smaller model to imitate a large one (distillation).

Example — Llama-3-8B with r=16 on W_Q, W_K, W_V, W_O of its 32 layers (W_K and W_V map 4096 → 1024 dimensions under GQA): 32 \times 16 \times \big(2\cdot 8192 + 2\cdot 5120\big) \approx 13.6\text{M parameters} \approx 0.17\,\%\ \text{of the model}.

• QLoRA (Dettmers et al., 2023): store the frozen base in 4 bits (NF4, a data type matched to normally distributed weights), quantize the quantization constants as well, and page the optimizer states to CPU memory on spikes. Gradients flow through the frozen 4-bit weights into bf16 adapters. An 8B base then takes about 5 GB, and a 65B model can be fine-tuned on one 48 GB GPU.
• Limits: tasks far from pre-training (a new language, long continued pre-training) need a higher rank or full fine-tuning. LoRA learns less than full fine-tuning, but it also forgets less of what the base model knew (Biderman et al., 2024).

• Token-level distillation applies \mathcal{L}_{\text{KD}} at every position of the sequence; teacher and student must share the tokenizer.
• Sequence-level distillation (Kim & Rush, 2016) fine-tunes the student on texts generated by the teacher: this is the “data distilled from a large model” of 4.1, and the recipe of the small DeepSeek-R1 distilled models. It works across tokenizers.

5. RLHF, DPO, GRPO

This chapter continues chapter 4: once the model imitates good answers, we want it to prefer better answers to worse ones. It derives the three methods that do this — RLHF, which learns a reward and optimizes it with PPO; DPO, which skips the reward model; and GRPO, which uses verifiable rewards and no critic.

Notation: \pi_\theta is the policy being trained, \pi_{\mathrm{ref}} the frozen reference (the SFT model), y_w \succ y_l a preferred and a dispreferred response to the same prompt x, and \beta the KL coefficient.

The loss fits in a few lines of PyTorch. The only subtle part is the sequence log-probability: sum the token log-probabilities of the response only, with the prompt masked exactly as in SFT (4.1).

import torch
import torch.nn.functional as F

def sequence_logprob(model, input_ids, attention_mask, labels):
    """log pi(y|x): sum of token log-probs over the response (labels = -100 on the prompt)."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    labels = labels[:, 1:]                                # token t is predicted at t-1
    mask = labels != -100
    logp = torch.log_softmax(logits.float(), dim=-1)
    tok = torch.gather(logp, 2, labels.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (tok * mask).sum(-1)                           # shape (batch,)

def dpo_loss(policy, ref, batch, beta=0.1):
    """batch["chosen"] / batch["rejected"]: (input_ids, attention_mask, labels) for y_w / y_l."""
    lp_w = sequence_logprob(policy, *batch["chosen"])
    lp_l = sequence_logprob(policy, *batch["rejected"])
    with torch.no_grad():                                 # pi_ref is frozen
        ref_w = sequence_logprob(ref, *batch["chosen"])
        ref_l = sequence_logprob(ref, *batch["rejected"])
    margin = beta * ((lp_w - ref_w) - (lp_l - ref_l))     # implicit reward margin r_hat
    loss = -F.logsigmoid(margin).mean()
    accuracy = (margin > 0).float().mean()                # share of pairs ranked correctly
    return loss, accuracy.detach(), margin.detach()

# sanity check: at step 0, policy == ref, so margin = 0 and loss = log 2 ≈ 0.693

6. Running an LLM: inference

A trained model still has to produce text, one token at a time, under memory and latency constraints. This chapter explains how tokens are chosen (decoding), where the memory goes (the KV-cache), and the three main ways to make serving faster or cheaper without changing the output much: speculative decoding, quantization and smarter batching.

6.1. Decoding: formulas and effects

6.2. KV-cache: the memory arithmetic

6.3. Speculative decoding: the correction proved

6.4. Quantization: principles and errors

• Post-training int8: nearly lossless (batch calibration, per-channel scales); int4: the standard compromise;
• GPTQ (Frantar et al., 2022): layer by layer, minimize \|Wx-\hat W x\|_2^2 on a calibration set — column-by-column quantization with error compensation on the remaining columns (via the inverse of the Hessian \mathbf{H}=2\mathbf{X}\mathbf{X}^\top);
• AWQ (Lin et al., 2023): protect the “salient” channels (high activations) with per-channel scaling before quantization — the error is shifted toward weakly active channels;
• Typical errors: int8 ≈ +0.0-0.1 PPL; well-calibrated int4 ≈ +0.1-0.5 PPL (often imperceptible); below 4 bits, degradation accelerates (2-3-bit quantization = research); in production, always re-validate on your benchmarks, not PPL alone.

6.5. Serving: continuous batching, PagedAttention, costs

• Continuous batching (vLLM, Orca): schedule at the decoding step — finished sequences leave, new ones enter the same batch; throughput multiplied by ~10-20× vs static batching;

• PagedAttention (Kwon et al., 2023): the KV-cache split into pages (like virtual memory) — no more fragmentation; sharing of common prefixes (system prompt); ~2-4× memory gain → larger batches;

• Costs: two regimes — API (on the order of $/M tokens, steadily decreasing) vs on-prem (amortized GPU-hour: an H100 at ~3$/h serving ~2-5k tokens/s on an int4-quantized 8B → a few $/M tokens under heavy load).

  The real cost of an agent = repeated input tokens × history length (ch. 8); the “ML Ops” angle is deepened in the dedicated M2 module.

# KV-cache arithmetic (prop. 2.4 / 6.2): Llama-3-8B
L, kv_heads, d_head, bytes = 32, 8, 128, 2
per_tok = 2 * L * kv_heads * d_head * bytes      # bytes / token
for ctx in (8_000, 100_000, 1_000_000):
    print(f"context {ctx//1000:4d}k -> {per_tok*ctx/2**30:6.1f} GiB of KV-cache")

# speculative ("assisted") decoding in HF transformers: small = proposes, large = verifies
# out = target.generate(**inputs, assistant_model=draft, do_sample=True)
# result: outputs IDENTICAL in distribution to the large model alone (thm 6.3)

7. RAG in depth

A model only knows what was in its training data, up to its cutoff date, and cannot cite sources. Retrieval-augmented generation fixes this by fetching relevant documents at query time and placing them in the context; this chapter goes through each stage of such a pipeline and how to evaluate it.

It assumes the minimal pipeline of NLP ch. 7.4 — chunk, embed, retrieve, prompt — and goes deeper at each stage.

7.1. Chunking

• Fixed with overlap: 256-1024 tokens, stride 10-20%; structural splitting (sections, paragraphs) > blind splitting (cut sentences destroy retrieval);
• Semantic chunking: cut where the embedding similarity between successive windows drops (topic change);
• Multi-granularity: index by passages, but return the parent section to the LLM (small-to-big); for structured documents, keep titles/metadata in the chunk (pre-retrieval filters).

7.2. Retrieval: BM25, dense, hybrid — with numbers

7.3. RAG evaluation

7.4. GraphRAG and variants

• HyDE (Gao et al., 2022): generate a hypothetical answer to the question and search with its embedding — the imaginary answer is closer to the documents than the question;
• Structured queries: text → graph query (graph linearization, see the publications in ch. 10) or SQL (text-to-SQL): the LLM writes the query, the database answers — maximal auditability.

Retrieval over a knowledge graph. Vector RAG retrieves passages that look like the question; it fails when the answer needs a chain of relations, as in “drugs that target a gene associated with disease X”. GraphRAG retrieves a subgraph of a knowledge graph of (head, relation, tail) triples, built as in NLP ch. 6.4 and Graph ML ch. 10.1.

1. Seed nodes: link the question’s entities to graph nodes, by string matching or dense retrieval over node descriptions.
2. Subgraph:
   • k-hop: every node within k hops of the seeds — about \bar d^{\,k} nodes per seed for an average degree \bar d, often too many beyond k=2.
   • PCST (G-Retriever, He et al., 2024): nodes and edges get a prize that grows with their similarity to the question, edges a cost c; keep the connected subgraph S that maximizes \sum_{v\in S}\mathrm{prize}(v)+\sum_{e\in S}\mathrm{prize}(e)-c\,|E(S)|, a prize-collecting Steiner tree with fast approximate solvers. The result is small, connected and relevant.
3. Linearization: one triple per line, such as “(aspirin, treats, headache)”, plus short node descriptions, with the instruction to cite node ids.

Token cost. Vector RAG sends about 5 × 300 = 1,500 tokens. At about 10 tokens per triple, the 2-hop neighbourhood of an entity of degree 30 (about 900 edges) costs about 9,000 tokens, and a PCST subgraph of 30 edges about 300: pruning keeps GraphRAG affordable.

Global questions. Microsoft’s GraphRAG (Edge et al., 2024) also splits the graph into communities (Leiden algorithm) and has an LLM summarize each one, to answer questions about a whole corpus. The cost moves to indexing: one LLM call per chunk and per community.

Benchmark. STaRK (Wu et al., 2024) mixes textual and relational queries. Its STaRK-Prime split runs over the biomedical graph PrimeKG, with one text document per node, scored by Hit@1, Hit@5, Recall@20 and MRR; it is the benchmark of the final lab (ch. 8). The worked version, with a named corpus and baselines, is Graph ML ch. 10.4.

# minimal RAG (pseudocode with sentence-transformers + FAISS)
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

enc = SentenceTransformer("all-MiniLM-L6-v2")
idx = faiss.IndexFlatIP(384); idx.add(enc.encode(chunks, normalize_embeddings=True))

def answer(question, k=4):
    D, I = idx.search(enc.encode([question], normalize_embeddings=True), k)
    context = "\n".join(chunks[i] for i in I[0])
    prompt = f"Answer using the following context:\n{context}\n\nQuestion: {question}"
    return llm.chat(prompt)   # + citations = retrieved sources

# lab extension: cross-encoder reranking on the top-20, then
# RAGAS evaluation (faithfulness / answer relevance) on 30 questions.

8. Agents, multimodality and perspectives

So far the model only reads text and writes text. This chapter extends it in two directions: agents, which let it call tools and act in a loop, and multimodal models, which let images, audio and documents enter its context; it ends with the directions research is taking.

8.1. The ReAct loop, with a real trace

Question: which M1 module covers HMMs?

Thought: I need to search the master's curriculum.
Action: search["M1 curriculum NLP HMM"]
Observation: ... CSC_52082_EP "Introduction to Text Mining and NLP",
             ch. 6 "Sequences: NER, HMM, CRF" ...
Thought: the answer is in the list of chapters.
Action: finish["The NLP module (CSC_52082_EP), chapter 6"]

# modern function calling: JSON typed by the chat engine
tools = [{"type": "function", "name": "search",
          "parameters": {"query": "str", "k": "int"}}]
# the model emits {"name": "search", "arguments": {"query": "...", "k": 5}}
# -> the runtime executes, reinjects the result, and the loop continues

8.2. Multimodality: how an image enters a context

• Vision: a ViT encoder cuts the image into 16×16 patches → one token per patch (Deep Learning ch. 6.6 for ViT); a projector (MLP or Q-Former) aligns these tokens onto the LLM’s embedding space → the image becomes “text” inside the context (LLaVA scheme).

  The encoder is often pre-trained contrastively image-text — the common latent space of CLIP, worked in 8.2b (and in depth in the Multimodal GenAI module ch. 5);

• Audio: mel spectrograms → encoder (Whisper-style) → tokens; the “transcription + style” output feeds the LLM (voice assistant = encoder → LLM → synthesis pipeline);

• Video: sampled images (1-4/s) + temporal pooling — very high token cost: the sampling step choice is a compression choice;

• Documents: pages = images + interleaved OCR text (the number 1 enterprise use case — contracts, reports).

8.2b. Vision-language models, worked (self-study)

This self-study section (about 2 h, with Ex. 6) turns the vision bullet of 8.2 into a worked recipe: how an image becomes tokens, how CLIP aligns images and text, and how LLaVA plugs a vision encoder into an LLM. It is the vision-language recap that the Geometric Data module (ch. 8) points to.

• ViT recap (Deep Learning ch. 6.6): an H\times W image is cut into P\times P patches, each flattened and linearly projected, plus a position embedding; a transformer encoder processes the HW/P^2 patch tokens. ViT-L/14 at 336 px gives (336/14)^2 = 576 tokens.

Zero-shot classification follows directly: embed one prompt per class, “a photo of a {class}”, and predict the class whose text embedding is closest to the image embedding. No label of the target dataset is used.

• LLaVA (Liu et al., 2023): a frozen CLIP ViT-L/14 encodes the image; a projector W (linear in LLaVA, a two-layer MLP in LLaVA-1.5) maps its patch features to the LLM’s embedding space; the resulting visual tokens are placed in the prompt before the text tokens.
  • Stage 1, feature alignment: freeze the vision encoder and the LLM and train only W on about 600k image–caption pairs (filtered CC3M), the instruction being “describe the image”.
  • Stage 2, visual instruction tuning: keep the encoder frozen and train W and the LLM on about 158k multimodal conversations, with the SFT loss mask of 4.1: the loss covers the assistant’s answer only, never the image tokens or the user’s question.
• Token budget: 576 visual tokens for one 336-px image in LLaVA-1.5, as many as a page of text. High-resolution variants tile the image and multiply this count, which is why video (8.2) keeps few frames.

# zero-shot CIFAR-10 with open_clip (Ex. 6) — pip install open_clip_torch torchvision
import torch, open_clip
from torchvision.datasets import CIFAR10

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k")
tokenizer = open_clip.get_tokenizer("ViT-B-32")
model.eval()

data = CIFAR10(root="data", train=False, download=True, transform=preprocess)
loader = torch.utils.data.DataLoader(data, batch_size=256)
prompts = [f"a photo of a {c}" for c in data.classes]

with torch.no_grad():
    v = model.encode_text(tokenizer(prompts))
    v = v / v.norm(dim=-1, keepdim=True)              # one text embedding per class
    correct = 0
    for x, y in loader:
        u = model.encode_image(x)
        u = u / u.norm(dim=-1, keepdim=True)          # image embeddings
        pred = (u @ v.T).argmax(dim=-1)               # nearest class prompt (cosine)
        correct += (pred == y).sum().item()
print(f"zero-shot accuracy: {correct / len(data):.3f}")

8.3. Perspectives

• Alternative architectures: attention+SSM hybrids (ch. 2), xLSTM, latent transformers and diffusion language models (cf. ch. 10); the transformer does not have a monopoly;
• Small Language Models & distillation: the quality/cost/latency trade-off governs enterprise deployment (distillation in ch. 4.3, serving costs in ch. 6);
• Open research: reasoning (ch. 5), long-term memory, LLMs on graphs (linearization vs structured embeddings — ch. 10 and the Graph ML/Advanced GNN modules), evaluation (ch. 9).

9. Evaluation and safety of LLMs

A score on a benchmark is only useful if it measures what we think it measures, and a deployed model must resist misuse. This chapter covers how benchmarks are built and skewed, the biases of LLM judges, hallucinations, attacks on LLM systems, a statistical test to detect machine-generated text, and how to compare two models honestly; a self-study section on privacy and bias closes it.

9.1. Benchmarks: construction and contamination

9.2. LLM-as-judge: convenient, but biased

• Positional bias: the judge prefers the first (or second) answer depending on order — mitigation: evaluate in both orders and average;
• Verbosity / style bias: long, well-formatted answers are overrated — mitigation: rubrics (explicit criteria), constrained format;
• Self-preference: a judge favors answers from its own model family — mitigation: judge ensembles, judges trained for calibration (references: MT-Bench, G-Eval);
• Calibration: a judge score is not a probability — verify on a set where ground truth exists (fraction of judgments matching humans); cheap distilled metrics (FrugalScore, ch. 10) were born exactly from this need.

9.3. Hallucinations

9.4. Attacks: jailbreaks and prompt injections

This section is the guaranteed source on jailbreaks and prompt injections for the modules that use them later — Control of Generative AI, Security and Robustness, MLOps and Advanced Topics in LLMs; the Explainability, Security and Privacy module (ch. 5–6) goes deeper.

• Direct jailbreaks: role-play (“DAN”), hypothetical framing, encoding (base64, unicode), multi-turn crescendo — circumventing safety instructions;

• Indirect injections: the retrieved content (web page, PDF, tool result) contains malicious instructions — the number 1 attack on RAG/agent systems (ch. 7-8);

• Defense in depth: input/output classifiers (Llama Guard type), spotlighting (delimiting and marking untrusted content), least privilege on tools (the retriever does not write to the database!), prompt/data separation (ch. 1 — special tokens: the same hygiene), human in the loop for irreversible actions.

  No defense is complete — one reduces the attack surface.

9.5. Watermarking

9.6. Comparing two models honestly

A two-point gain on a 1,000-question benchmark may be noise. Before claiming that model B beats model A, measure the uncertainty due to the items, the prompt and the seed; this is the LLM form of Machine Learning ch. 4.9.

• Pair the items. Run A and B on the same n items and record per-item outcomes a_i, b_i (correct or not, or a score). Compute the standard error on the paired differences b_i-a_i: it is much smaller than the unpaired one when both models fail on the same hard items.
• Bootstrap over items. Resample the n items with replacement 10^4 times, recompute the difference each time, and report the 2.5 % and 97.5 % percentiles as a 95 % confidence interval. This works for any metric: F1, judge win rate, RAGAS faithfulness.
• Paired test. For right/wrong outcomes, McNemar’s test uses only the discordant items: n_{10} (A right, B wrong) and n_{01} (A wrong, B right). Under H₀ (equal accuracy), each discordant item favours B with probability 1/2, and \chi^2 = \frac{(n_{01}-n_{10})^2}{n_{01}+n_{10}} \;\approx\; \chi^2_1 . For other metrics, a paired permutation test randomly swaps a_i and b_i within each pair.
• Worked example. On 1,000 items, A scores 70.0 % and B 72.0 %, with n_{01}=60 and n_{10}=40: \chi^2 = \frac{(60-40)^2}{100} = 4.0, \qquad p\approx0.046 (0.057 with the exact binomial version). An unpaired two-proportion test on the same accuracies gives z\approx0.99, p\approx0.32: ignoring the pairing hides a real, if small, difference.
• Prompts and seeds. Scores move by several points with the prompt template and the sampling seed (4.0). Report the mean and the standard deviation over at least 3 templates and 3 seeds next to the item interval, fix the protocol (scoring mode, 9.1) before looking at the results, and correct for multiple comparisons (Holm) when many benchmarks are compared.

9.7. Privacy and bias of LLMs (self-study)

An LLM can leak pieces of its training data and treat groups of people unequally. This self-study section (about 1.5 h) gives the core measurements; the Explainability, Security and Privacy module (ch. 7–8) goes deeper.

• Extraction recipe (Carlini et al., 2021): generate many samples from short prefixes; rank them by a memorization score, such as the ratio of the log-perplexity to the zlib entropy (compressed length in bits); check the top candidates against the training data. On GPT-2 this exposed hundreds of verbatim training sequences, including names, phone numbers and URLs.
• Membership inference as a test. Decide H₀: x was not in the training set, against H₁: it was. By the Neyman–Pearson lemma the best test thresholds a likelihood ratio; in practice the target’s log-likelihood is calibrated by a reference model that did not see x: \Lambda(x) = \log p_\theta(x) - \log p_{\text{ref}}(x) \;>\; t . Report the true-positive rate at a low false-positive rate (0.1–1 %), not the AUC (Carlini et al., 2022). On large models pre-trained for about one epoch such attacks are close to chance (Duan et al., 2024); on small fine-tuning sets seen several times, they are not.
• Mitigations. Deduplicate (3.1): a sequence present 10 times in the training set is generated about 1,000 times more often than one present once (Kandpal et al., 2022). Fine-tune with DP-SGD (clipping C, noise multiplier \sigma; ESP ch. 8), which suits LoRA because few parameters are trained (Yu et al., 2022). Remove personal data before training; never put secrets in a system prompt.
• Bias evaluation, BBQ-style (Parrish et al., 2022): questions about two people from different social groups, in an ambiguous context (the correct answer is “unknown”) and a disambiguated one. Among the answers other than “unknown”, n_{\text{biased}} follow the stereotype; the bias score s = 2\,\frac{n_{\text{biased}}}{n_{\text{non-unknown}}} - 1 \in [-1, 1] is multiplied by (1-\text{accuracy}) in ambiguous contexts. Report it per category (age, gender, nationality…) with a bootstrap interval (9.6).

10. Research (Shang)

Guokan Shang (lead research scientist, MBZUAI; PhD at the LIX laboratory, École Polytechnique, 2021, under the supervision of M. Vazirgiannis — the LLGA loop) works on the evaluation of generative models, efficient pre-training, multilingual LLMs and dialogue summarization. Three research axes are presented in detail, then the rest of his publications, each positioned relative to the chapters.

Exercises / Labs

1. ★ Multilingual tokenization. Compare BPE vs Unigram vs SentencePiece on a multilingual corpus: compression (characters/token), coverage of numbers; measure the French “extra cost” over English for the same text.

   Proof. Guidance (expected results): aligned professional translations (fr/en/de) → tokens/word; typically fr ≈ 1.15-1.25× en; German with compounds even more — until frequent compounds become tokens; check that o200k (GPT-4o) narrows the gap vs cl100k. Conclude on the real cost (context, API price) of the same application depending on the language. ∎

2. ★ KV-cache arithmetic. For Llama-3-8B (32 layers, 8 KV heads, 128 dim) and a 128k-token context: compute the cache memory in fp16, then in int8, then with MQA; what maximum batch fits on an 80 GB GPU (int4 weights ≈ 5 GB)?

   Proof. Guidance: 128 KiB/token (prop. 2.4) → 128k tokens = 16 GiB; int8 → 8 GiB; MQA (1 KV head) → 2 GiB in fp16, 1 GiB in int8. The available budget is ≈ 80 − 5 = 75 GiB, so the batch goes from ~4 sequences (GQA, fp16) to ~9 (GQA, int8), ~37 (MQA, fp16) and ~75 (MQA, int8): GQA/MQA and cache quantization determine the served throughput. ∎

3. ★ Guided SFT with LoRA. Fine-tune a small instructed model (1-3B) on 500 (question, answer) pairs from your domain with LoRA (r=16 on W_Q, W_K, W_V, W_O): strict template, masked labels (code of ch. 4.1), 2-3 epochs. Count the trainable parameters; measure the effect of a wrong template at test time (a spectacular drop to witness once and for all).

   Proof. Guidance: for Llama-3.2-1B (d=2048, 16 layers, 8 KV heads of dimension 64, so W_K and W_V map 2048 → 512), Definition 4.3 gives 16 \times 16 \times \big(2\cdot 4096 + 2\cdot 2560\big) \approx 3.4\text{M} trainable parameters, about 0.3 % of the model; the PEFT library’s print_trainable_parameters() should agree. Track the loss on completion tokens only. With the inference template different from the training one, the answers lose the learned format and often continue the user’s turn. ∎

4. ★ Few-shot sensitivity. On the SST-2 validation set (GLUE), with a 0.5-1.5B base model, measure the accuracy of k-shot prompting for k\in\{0,1,4,8\}, with 5 random orders of the demonstrations for each k. Then apply contextual calibration (4.0) and compare.

   Proof. Guidance: score the two label words by their next-token probability rather than parsing free text. Expect the spread between orders to be large — several points, sometimes tens — and the predictions to lean toward one label. Calibration (divide the label probabilities by those obtained with the content-free input “N/A”, then renormalize) usually raises the mean and shrinks the spread. Report mean ± range per k (9.6). ∎

5. ★ Watermark. With \gamma=0.5 and T=400 tokens, compute z for a text carrying 230 green tokens; threshold at 5% (one-sided); how many tokens are needed to detect a +5% green bias?

   Proof. Guidance: z=(230-200)/10 = 3 → one-sided p ≈ 0.13 %: rejected (marked).

   For a green fraction \gamma+\varepsilon, the expectation of z is \varepsilon\sqrt{T/\gamma(1-\gamma)}. It reaches the one-sided 5 % threshold 1.645 when T = \Big(\frac{1.645}{\varepsilon}\Big)^2\gamma(1-\gamma) \approx 271 \text{ tokens for } \varepsilon=0.05, which detects the bias only half the time. For 80 % power, replace 1.645 by 1.645+0.84: T\approx 620 tokens — hence the difficulty on short texts. ∎

6. ★ CLIP zero-shot and projector size (self-study, 8.2b). (a) Count the parameters of LLaVA-1.5’s projector, a two-layer MLP from the 1024-dimensional ViT-L/14 features to the 4096-dimensional embeddings of a 7B LLM (biases included), and compare with the LLM. (b) Run the open_clip code of 8.2b on CIFAR-10, then replace the single prompt by the average embedding of 5 templates (“a photo of a {}”, “a blurry photo of a {}”, …).

   Proof. Guidance: (a) the projector has 1024\cdot4096 + 4096 + 4096\cdot4096 + 4096 = 20{,}979{,}712 \approx 21\text{M} parameters, about 0.3 % of the 7B LLM — stage 1 of LLaVA trains only these.

   (b) Expect roughly 90 % or more for ViT-B-32; the prompt ensemble typically adds about a point. Inspect the confusion matrix (cat/dog, automobile/truck). ∎

7. ★★ RoPE. Prove the relative property of thm 2.3 (R(a)^\top R(b) = R(b-a)), then check numerically that the attention score is invariant under a common translation of positions; implement context interpolation m\to m/L and measure perplexity beyond the training length.

   Proof. Guidance: 2D rotations = complex numbers — commutativity and e^{-ia}e^{ib}=e^{i(b-a)}; translation invariance follows from the theorem; in practice naive extrapolation degrades PPL, interpolation largely preserves it (this is the basis of extended contexts 32k→128k without full retraining). ∎

8. ★★ Scaling. From published data (LLaMA 1/2/3, Chinchilla), verify the law \mathcal{L} \approx A N^{-\alpha} + B D^{-\beta} + E by regression; situate each model relative to the compute optimum (thm 3.1).

9. ★★ MoE. Assume f_i=P_i for every expert (the router’s mean probabilities match the actual loads). Derive the minimum of the auxiliary loss \alpha_{\text{aux}} n_E\sum_i f_iP_i under \sum_i P_i=1 and show that it corresponds to uniform routing. Then show that without the assumption f_i=P_i the minimum is not uniform in general (compare Advanced Deep Learning, exercise 7 (b)); simulate the collapse (biased initialization of the router) without then with the loss.

   Proof. Guidance: with f_i=P_i the loss is \alpha_{\text{aux}} n_E\sum_i P_i^2. Cauchy–Schwarz gives 1=\Big(\sum_i P_i\Big)^2\le n_E\sum_i P_i^2, with equality iff P_i=1/n_E: uniform routing is the minimum, where the loss equals \alpha_{\text{aux}}, not 0.

   Without the assumption, take n_E=2, top-1 routing and four tokens: three with gate (\tfrac12+\varepsilon,\tfrac12-\varepsilon), sent to expert 1, and one with gate close to (0,1), sent to expert 2. Then f=(\tfrac34,\tfrac14) and P\approx(\tfrac38,\tfrac58), so \mathcal{L}_{\text{aux}}\approx 2\,\alpha_{\text{aux}}\Big(\tfrac34\cdot\tfrac38+\tfrac14\cdot\tfrac58\Big)=\tfrac78\,\alpha_{\text{aux}}<\alpha_{\text{aux}}. Tokens sent to expert 1 by a narrow margin keep P_1 low while f_1 is high, which lowers the loss below its value at balance.

   In the simulation, without the loss the winning expert attracts tokens (reinforced feedback) and specialization degenerates. ∎

10. ★★ DPO on paper. Derive the closed form \pi^\star of the KL-penalized problem (Gibbs variational principle), then the DPO loss by re-injecting into Bradley-Terry; check the gradient: it increases the log-ratio margin of the preferred responses y_w over the dispreferred ones y_l. Then check numerically that the PyTorch loss of ch. 5 equals \log 2 when \pi_\theta=\pi_{\mathrm{ref}}.

    Proof. Guidance: follow the proof of thm 5.2 step by step — the key argument is that \mathcal{J}=-\beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}e^{r/\beta}/Z)+\beta\log Z; then Z(x) disappears in the difference r_w-r_l.

    The gradient pushes \log\pi(y_w)-\log\pi_{\mathrm{ref}}(y_w) up and the y_l term down, modulated by \sigma(-\hat r) (hard pairs first).

    At \pi_\theta=\pi_{\mathrm{ref}}, \hat r=0 and the loss is -\log\sigma(0)=\log 2. ∎

11. ★★ Toy GRPO. On a verifiable task (binary arithmetic), train a mini-model with GRPO (group of 8 responses); plot accuracy and response length — observe the emergence of longer “reasoning”; compare gradient variance with/without normalization by the group standard deviation, and with the leave-one-out baseline of thm 5.3.

12. ★★ Speculative, on paper. Write out completely the proof of thm 6.3 in the case \gamma=2, then estimate the expected latency gain if q=p on 80% of the mass but diverges on the rest.

    Proof. Guidance: accept token 1 with probability \alpha=\sum_y\min(q,p)\approx0.8, then the second conditionally. With \gamma draft tokens and i.i.d. acceptances, the expected number of tokens produced per target forward is (Leviathan et al., 2023) \frac{1-\alpha^{\gamma+1}}{1-\alpha} = \frac{1-0.8^3}{0.2} \approx 2.44 \quad (\gamma=2), at most \gamma+1=3; the limit 1/(1-\alpha)=5 needs \gamma\to\infty. With a draft costing a fraction c of a target forward, the speedup is 2.44/(1+\gamma c), about 2.2× for c=0.05. ∎

13. ★★ RAG vs context length. Query an LLM with 4k/32k/128k of context over a corpus: does the “lost in the middle” effect appear? Compare with RAG; discuss the trade-offs (cost, faithfulness, indexing overhead).

14. ★★ Agent reliability. An agent is run n=10 times on each task of a benchmark.

    • (a) Show that \binom{c}{k}/\binom{n}{k} is an unbiased estimator of p^k for a task with success probability p and c successes.
    • (b) For a task with c=8, compute the estimates of pass^1, pass^2, pass^3 and compare with the plug-in \hat p^k.
    • (c) One run costs 60k input and 4k output tokens at 3 $ and 15 $ per million: what is the cost per solved task?

    Proof. Guidance: (a) \binom{c}{k}/\binom{n}{k} is the average, over all k-subsets of the n runs, of the product of their success indicators; each product has expectation p^k by independence.

    (b) 0.8, 28/45\approx0.62, 56/120\approx0.47, against the plug-in 0.8, 0.64, 0.51, which is biased upward.

    (c) 0.18+0.06=0.24 $ per run, so 0.24/0.8=0.30 $ per solved task. ∎

15. ★★ Comparing two models. On the 1,319 test problems of GSM8K, model A scores 61.0 % and model B 63.5 %, with n_{01}=95 problems solved by B only and n_{10}=62 by A only. (a) Run McNemar’s test. (b) Compute a paired bootstrap 95 % interval for the difference from per-item outcomes you simulate with these counts. (c) Explain why an unpaired test is the wrong tool. (d) Say what you would add before publishing the comparison.

    Proof. Guidance: (a) \chi^2=(95-62)^2/157\approx6.9, p\approx0.008 (exact binomial: 0.0104): B is better at the 5 % level; at the 1 % level only the χ² approximation says so, and the exact test falls just short. (b) The difference is 2.5 points and the interval runs from about 0.7 to 4.3 points, excluding 0. (c) The unpaired test ignores that both models share the hard problems, which inflates the standard error. (d) Repeat over 3 prompt templates and 3 seeds, and fix the answer-extraction rule in advance (9.1, 9.6). ∎

16. ★★★ Agent project. Extend the final lab of ch. 8 with an agent (8.1) that chooses, question by question, between vector retrieval, the graph retriever, or both.

    • Data: STaRK-Prime, 500 queries of the validation split, plus 50 relational questions you write and check yourself.
    • Systems: vector RAG, GraphRAG, the router, and an oracle that picks the better of the two retrievers for each query (an upper bound for the router).
    • Metrics: Hit@5 and MRR; answer faithfulness and citation validity; tokens, latency and cost per query; the router’s agreement with the oracle.
    • Deliverables: a repository with a one-command evaluation, a 4-page report with per-query-type paired intervals (9.6), a 10-minute talk.
    • Rubric (20 points): reproducible baselines 5; router design and ablation 4; evaluation with paired intervals 5; cost analysis 3; error analysis of 20 failures 3.

Projects and assessment

Official assessment: not displayed. The master’s wiki gives this course unit’s volume (24 h), credits and instructor, but no assessment modality, and its SynapseS 2026–2027 sheet is not yet published. The proposals below are ours: a project (solo or in a group) plus a written exam, weighted as the instructor decides. They exercise O3 (large language models, the core of the course), O5 (graphs × LLMs), O7 (hands-on and industry practice) and O8 (trustworthy AI), and they prepare the Year-2 transverse project.

Solo projects

1. [Solo · ≈ 20 h] The tokenizer tax: a multilingual and low-resource cost audit. How much more does the same content cost in tokens across languages and scripts, and does extending the vocabulary pay off?
   • Implement: characters per token and tokens per sentence for cl100k, o200k, the Llama-3 tokenizer and a SentencePiece Unigram model; then add 2,000–5,000 BPE merges learned on dialect text to a small open model, initialize each new embedding as the mean of its old sub-tokens, and fine-tune with LoRA (4.3), keeping the embedding and output rows trainable.
   • Data: FLORES-200 (NLLB Team, 2022), aligned sentences in English, French, German, Moroccan Arabic (ary_Arab) and Egyptian Arabic (arz_Arab); the Moroccan Arabic Wikipedia dump for fine-tuning.
   • Deliverables: code, a 4-page report, a 10-minute talk.
   • Chapters: ch. 1, 4.3, 10 (Atlas-Chat, Nile-Chat).
   • Success: a cost table with bootstrap intervals (9.6), and a measured answer on whether the extension shortens dialect sequences without hurting English. Compare models in bits per character, not per-token perplexity, since the tokenizations differ.
2. [Solo · ≈ 20 h] Serving an 8B model on a budget. What is the cheapest configuration that serves an 8B chat model at a target latency?
   • Implement: serve Llama-3.1-8B-Instruct with vLLM in AWQ and GPTQ int4 (fp16 if the GPU allows), with and without speculative decoding (draft: Llama-3.2-1B-Instruct, same tokenizer), at several batch sizes; measure throughput, P50/P95 latency and peak GPU memory.
   • Data: 500 ShareGPT prompts, as in vLLM’s benchmark scripts; the MT-Bench questions for a quality check. A 16 GB GPU (free Colab or Kaggle T4) suffices for the int4 runs.
   • Deliverables: benchmark scripts, a 4-page report with a cost curve in $ per million tokens, a 10-minute talk.
   • Chapters: ch. 2 (Prop. 2.4), ch. 6.
   • Success: measured KV-cache memory within 10 % of Proposition 2.4; the measured speculative speedup set against \frac{1-\alpha^{\gamma+1}}{1-\alpha} with the measured acceptance rate; the int4 quality drop reported with an interval.
3. [Solo · ≈ 25 h] Paper study: DPO versus PPO on controlled sentiment (Rafailov et al., 2023). Does DPO reach a better reward–KL trade-off than PPO, as the IMDb experiment of the DPO paper reports?
   • Implement: SFT of GPT-2 on IMDb reviews; preference pairs of sampled continuations ranked by a sentiment classifier; then DPO at four values of \beta, PPO (TRL) and best-of-n. Plot the reward against \mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}}) on held-out prefixes.
   • Data: IMDb (Maas et al., 2011); the reward classifier siebert/sentiment-roberta-large-english used in the paper.
   • Deliverables: code, a 4-page report stating which claims were and were not reproduced, a 10-minute talk.
   • Chapters: ch. 4.1, 5.
   • Success: reward–KL frontiers with 3 seeds per method, and a clear yes or no on the ordering of DPO and PPO at matched KL.

Group projects

1. [Group of 3 · ≈ 30 h each] From nanoGPT to an aligned assistant. How far does each training stage move a small model, and at what cost in KL drift and response length?
   • Implement: (A) pre-train nanoGPT models of three sizes and fit L(N,D) against the compute optimum (3.2); (B) LoRA-SFT of Qwen2.5-0.5B on GSM8K solutions with masked labels, then DPO on pairs of correct and incorrect sampled solutions; (C) GRPO with an exact-answer reward, with and without std normalization and with the leave-one-out baseline (5.3).
   • Data: TinyStories (Eldan & Li, 2023) for (A); GSM8K (Cobbe et al., 2021) for (B) and (C).
   • Split: one member per stage, one shared evaluation harness.
   • Deliverables: a repository, a 6-page report, a 15-minute talk.
   • Chapters: ch. 3, 4, 5, 9.6; PPO and REINFORCE from Reinforcement Learning ch. 5.
   • Success: scaling exponents with intervals; GSM8K test accuracy of SFT, DPO and GRPO compared by McNemar tests; KL and length curves.
2. [Group of 4 · ≈ 30 h each] GraphRAG versus vector RAG, with an agentic router. For which questions does a knowledge graph beat a vector index, and can an agent tell them apart?
   • Implement: Ex. 16 at full scale — hybrid retriever with reranking (7.2), PCST GraphRAG (7.4), ReAct router (8.1). The graph side follows Graph ML ch. 10.
   • Data: STaRK-Prime (Wu et al., 2024), all validation queries; STaRK-MAG (academic papers) as a transfer test.
   • Split: retrieval, graph, agent, and evaluation with the attacks — one member each.
   • Deliverables: a repository with a one-command evaluation, a 6-page report, a live demo.
   • Chapters: ch. 7, 8, 9.2, 9.4, 9.6.
   • Success: Hit@5 and MRR per query type with paired intervals; a router that beats the best single retriever and is compared with the oracle; citation validity ≥ 95 %; cost per 1,000 queries.
3. [Group of 3 · ≈ 25 h each] Judging the judges, attacking the assistant. How far can an LLM judge be trusted, and how easily is a RAG assistant hijacked?
   • Implement: (A) position, verbosity and self-preference bias of two open judges against human votes (9.2); (B) an 8–13-gram contamination check of GSM8K and MMLU against a public pre-training corpus through the infini-gram API (Liu et al., 2024); (C) indirect prompt injections against a RAG agent before and after spotlighting and least-privilege tools (9.4), and watermark detection before and after paraphrase (9.5).
   • Data: the MT-Bench human judgments (Zheng et al., 2023); BIPIA (Yi et al., 2023) for injections; the watermarking code of Kirchenbauer et al. (2023).
   • Deliverables: a repository, a 6-page report, a 15-minute talk with a live attack.
   • Chapters: ch. 9; Explainability, Security and Privacy of LLMs ch. 5–6 for the defences.
   • Success: each judge bias measured with a paired test (9.6); attack success rates before and after each defence, with intervals; watermark z-scores before and after paraphrase.

Written exam (proposal)

Format: 2 hours; one handwritten A4 sheet allowed; no electronic device; 4 questions — a course question, a computation, a short proof and a statistics question.

1. Question (5 points). Compare RLHF (with PPO), DPO and GRPO: for each, say what is learned, what is sampled during training, which models sit in memory, and one typical failure.

   Expected:

   • RLHF learns a reward model, then optimizes the policy online with PPO; four models (policy, frozen reference, reward model, critic); failures: reward hacking, length drift, KL explosion.
   • DPO learns the policy directly from offline pairs, with no sampling; two models (policy, reference); failures: overfitting the preference set, likelihoods of both responses falling, no exploration.
   • GRPO samples G responses per prompt and uses a group baseline, with no critic; policy and reference plus a reward function; failures: prompt weighting biased by the std normalization, long wrong answers, the cost of G samples.

2. Question (6 points). A model has 40 layers and 8 KV heads of dimension 128, with a bf16 cache; its weights take 16 GiB of an 80 GiB GPU. (a) KV-cache bytes per token and per 32,768-token sequence (2 pts). (b) The largest number of such sequences served at once (1 pt). (c) With speculative decoding, \gamma=4 and acceptance rate \alpha=0.7, the expected number of tokens per target forward (1.5 pts). (d) LoRA with r=8 on W_Q and W_V of a 32-layer model with d=4096 and full multi-head attention: the trainable parameters (1.5 pts).

   Expected: (a) 2 · 40 · 8 · 128 · 2 = 163,840 bytes = 160 KiB per token, so 5 GiB per sequence. (b) (80 − 16)/5 = 12.8, so 12 sequences. (c) (1-0.7^5)/0.3\approx2.77 tokens, out of at most 5. (d) 32 × 2 × 8 × (4096 + 4096) = 4,194,304 ≈ 4.2M.

3. Question (5 points). Let \pi_{\mathrm{ref}} be a reference policy, r a reward and \beta>0.

   • (a) Show that the objective \mathbb{E}_\pi[r]-\beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}) is maximized by \pi^\star\propto\pi_{\mathrm{ref}}\,e^{r/\beta} (3 pts).
   • (b) Deduce that the Bradley–Terry likelihood of a preference y_w\succ y_l does not involve the normalizer Z(x), and state the DPO loss (2 pts).

   Expected: (a) expand \mathrm{KL}(\pi\|\pi^\star) to get the identity of step 1 of Theorem 5.2: \mathbb{E}_\pi[r]-\beta\,\mathrm{KL}(\pi\|\pi_{\mathrm{ref}}) = \beta\log Z-\beta\,\mathrm{KL}(\pi\|\pi^\star); the KL is non-negative and zero only at \pi=\pi^\star.

   (b) Invert: r=\beta\log\frac{\pi^\star}{\pi_{\mathrm{ref}}}+\beta\log Z(x); the term \beta\log Z(x) cancels in \sigma(r_w-r_l), and replacing \pi^\star by \pi_\theta gives the DPO loss (key result of ch. 5).

4. Question (4 points). On 500 items, model A scores 80.0 % and model B 84.0 %; 15 items are solved by A only and 35 by B only. (a) Is B significantly better at the 1 % level (2 pts)? (b) A colleague runs an unpaired two-proportion test and finds p\approx0.10. Who is right, and why (2 pts)?

   Expected: (a) McNemar: \chi^2=(35-15)^2/50=8.0, p\approx0.005<0.01: yes. (b) The paired test: both models answer the same items, so the variance of the difference must be computed on paired outcomes. The unpaired test ignores that the models share their hard items, overestimates the standard error and loses power.

Grading rubric (proposal)

References

• Foundations: A. Vaswani et al., Attention Is All You Need, NeurIPS 2017 — to know by heart; A. Radford et al., GPT-2 (2019); T. Brown et al., GPT-3 (Language Models are Few-Shot Learners, NeurIPS 2020); OpenAI, GPT-4 Technical Report, 2023; Meta, The Llama 3 Herd of Models, 2024 (Llama 2, 2023).
• Architecture: J. Su et al., RoFormer (RoPE), 2021; N. Shazeer, Fast Transformer Decoding (MQA), 2019; J. Ainslie et al., GQA, EMNLP 2023; W. Fedus et al., Switch Transformers, JMLR 2022; T. Dao et al., FlashAttention, NeurIPS 2022; A. Gu, T. Dao, Mamba, 2023 (SSD/Mamba-2: arXiv:2405.21060).
• Scaling and pre-training: J. Kaplan et al., Scaling Laws, 2020; J. Hoffmann et al., Chinchilla, 2022; T. Besiroglu et al., Chinchilla Scaling: A Replication Attempt, 2024; Y. Zhang et al., Beyond Random Sampling (curriculum), EACL 2026.
• Prompting and in-context learning: T. Z. Zhao et al., Calibrate Before Use, ICML 2021; Y. Lu et al., Fantastically Ordered Prompts, ACL 2022; S. Min et al., Rethinking the Role of Demonstrations, EMNLP 2022; S. M. Xie et al., An Explanation of In-context Learning as Implicit Bayesian Inference, ICLR 2022; J. Wei et al., Chain-of-Thought Prompting, NeurIPS 2022; T. Kojima et al., Large Language Models are Zero-Shot Reasoners, NeurIPS 2022.
• Parameter-efficient fine-tuning and distillation: E. J. Hu et al., LoRA, ICLR 2022; T. Dettmers et al., QLoRA, NeurIPS 2023; D. Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024; G. Hinton, O. Vinyals, J. Dean, Distilling the Knowledge in a Neural Network, 2015; Y. Kim, A. M. Rush, Sequence-Level Knowledge Distillation, EMNLP 2016.
• Post-training: L. Ouyang et al., InstructGPT/RLHF, NeurIPS 2022; R. Rafailov et al., DPO, NeurIPS 2023; DeepSeek-AI, DeepSeekMath (GRPO), 2024; DeepSeek-V3, 2024 (modern MoE); X. Wang et al., Self-Consistency, ICLR 2023; A. Ahmadian et al., Back to Basics: REINFORCE-style optimization (RLOO), ACL 2024; Z. Liu et al., Understanding R1-Zero-Like Training (Dr. GRPO), 2025.
• Inference: Y. Leviathan et al., Speculative Decoding, ICML 2023; W. Kwon et al., vLLM/PagedAttention, SOSP 2023; E. Frantar et al., GPTQ, ICLR 2023; J. Lin et al., AWQ, MLSys 2024.
• RAG and agents: P. Lewis et al., Retrieval-Augmented Generation, NeurIPS 2020; S. Es et al., RAGAS, EACL 2024 (demo); L. Gao et al., HyDE, ACL 2023; N. Liu et al., Lost in the Middle, TACL 2024; S. Yao et al., ReAct, ICLR 2023; Tree of Thoughts, NeurIPS 2023; S. Yao et al., τ-bench, 2024 (pass^k); M. Chen et al., Evaluating Large Language Models Trained on Code, 2021 (pass@k).
• Vector indexes and graph retrieval: Y. A. Malkov, D. A. Yashunin, HNSW, IEEE TPAMI 2018; H. Jégou, M. Douze, C. Schmid, Product Quantization for Nearest Neighbor Search, IEEE TPAMI 2011; J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with GPUs (FAISS), IEEE Trans. Big Data 2019; X. He et al., G-Retriever, NeurIPS 2024; D. Edge et al., From Local to Global: A Graph RAG Approach, 2024 (cf. Graph ML module ch. 10); S. Wu et al., STaRK, NeurIPS 2024 Datasets and Benchmarks.
• Vision-language models: A. Dosovitskiy et al., An Image is Worth 16x16 Words (ViT), ICLR 2021; A. Radford et al., CLIP, ICML 2021; H. Liu et al., Visual Instruction Tuning (LLaVA), NeurIPS 2023; H. Liu et al., Improved Baselines with Visual Instruction Tuning (LLaVA-1.5), CVPR 2024.
• Evaluation and safety: D. Hendrycks et al., MMLU, ICLR 2021; J. Kirchenbauer et al., A Watermark for LLMs, ICML 2023; On the Reliability of Watermarks, ICLR 2024; L. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023; T. G. Dietterich, Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms, Neural Computation, 1998 (McNemar); N. Carlini et al., The Secret Sharer, USENIX Security 2019; Extracting Training Data from Large Language Models, USENIX Security 2021; Membership Inference Attacks From First Principles, IEEE S&P 2022; M. Duan et al., Do Membership Inference Attacks Work on Large Language Models?, COLM 2024; N. Kandpal, E. Wallace, C. Raffel, Deduplicating Training Data Mitigates Privacy Risks, ICML 2022; D. Yu et al., Differentially Private Fine-tuning of Language Models, ICLR 2022; A. Parrish et al., BBQ, Findings of ACL 2022; and (M1 NLP module ch. 9): Buscaldi’s works — memorization and hallucinations (arXiv:2409.18858; MUCH).
• Research (Shang) (ch. 10): M. Kamal Eddine, G. Shang et al., FrugalScore, ACL 2022; Y. Zhang, G. Shang et al., Beyond Random Sampling, EACL 2026; G. Shang et al., Atlas-Chat, 2024; Nile-Chat, ArabicNLP 2025; meeting summarization: ACL 2018; AACL 2020; COLING 2020; TACL 2023 survey; FREDSum; Claire; reliability: Broken Telephone, ACL 2025; linguistic diversity, TACL 2025; frugal RLVR, 2025; graph linearization, 2024.
• Project data and tools: NLLB Team, No Language Left Behind (FLORES-200), 2022; R. Eldan, Y. Li, TinyStories, 2023; K. Cobbe et al., GSM8K, 2021; A. L. Maas et al., Learning Word Vectors for Sentiment Analysis (IMDb), ACL 2011; J. Liu et al., Infini-gram, COLM 2024; J. Yi et al., BIPIA, 2023; K. Hines et al., Spotlighting, 2024.
• Lab companions: S. Raschka, Build a Large Language Model (From Scratch), Manning, 2024; Karpathy, Zero to Hero (nanoGPT); HuggingFace; vLLM; TRL (SFT/DPO/GRPO).