Control in Generative AI

Contents

• Introduction and overview
• 0. Reminders: generating, conditioning, decoding
• 1. Why control: safety, fidelity, intellectual property
• 2. Taxonomy: four families of control
• 3. Constrained decoding: grammars, automata, masks
• 4. Steering: controlling in activation space
• 5. RLHF: control in behavior space
• 6. Watermarking: proving the origin of a text
• 7. Controlling diffusions: CFG, ControlNet, inversion
• 8. Controlling graph generators
• 9. Evaluating control: fidelity versus diversity
• 10. Attacks on control and robustness
• Exercises / Labs
• Projects and assessment
• References

Introduction and overview

Audience: a generative-AI engineer who already knows how to generate and now has to steer the output (format, safety, provenance). Empirical first: each chapter starts from a concrete failure — a broken JSON despite the prompt, a watermark that no longer shows after a paraphrase — and only then names the technique that fixes it (constrained decoding, steering, RLHF/DPO, watermarking).

The module is organized around four families of control, which together turn a generator that merely produces plausible outputs into a reliable tool.

What you will be able to do

• Argue for the need for control (output safety, fidelity to an instruction, intellectual property) and choose the right family of response;
• Place any technique in the four-family taxonomy (conditioning, guidance, hard constraints, post-processing);
• Implement grammar- or automaton-constrained decoding, with the masking algorithm detailed step by step;
• Install and use steering vectors to orient a generation in activation space;
• Describe RLHF as a control in behavior space, from the reward model to policy optimization;
• Verify the presence of a textual watermark with the z-test, and discuss its limits;
• Steer a diffusion via classifier-free guidance, ControlNet, or inversion;
• Transpose control to graph generators, and measure the fidelity/diversity trade-off — including under attack.

Official objectives and syllabus

Official syllabus: course description not published — this module appears neither in the S3 track nor in the S4 track of the 2026-2027 SynapseS catalog (both currently empty), and the master’s wiki indicates that the detailed M2 curriculum is still under development: description, teaching hours and instructor are therefore “not published”. The themes below are those carried by the module in the master’s brochure, as restated by these course materials, mapped to the sections that cover them.

The exact split between brochure content and the materials’ own extensions can only be settled once the official course description is published; in any case, chapter 0 (reminders: generating, conditioning, decoding) is part of the common structure of the repository’s course materials.

Where this course fits in the program

M2 module offered in period 1 and preselected by the Graph × LLM route of the home page: it synthesizes building blocks seen elsewhere in the master rather than introducing a new heavy formalism.

Prerequisites (Year 1): classifier-free guidance and diffusion models from Multimodal Generative AI (ch. 7 builds directly on them); the KL-regularised policy lemma of Reinforcement Learning ch. 6 (recalled in ch. 5); graph generation from Graph ML ch. 9 (reread in ch. 8). Large Language Models runs in the same period, so ch. 0, 3 and 5 recall what they use from it (decoding rules, function calling, RLHF and DPO) instead of assuming it.

Downstream modules: the textual watermarking of ch. 6 is taken up and extended on the attestation and anonymization side by Explainability, Security, Privacy of LLMs (ch. 9), which also covers safety-side attacks; Graph Generative Models (period 2) develops in depth the graph generators whose control ch. 8 introduces; Speech Technology for Conversational AI mobilizes the same control reflexes in continuous settings (expressivity, latency, guardrails, audio watermarking in its ch. 9) in the voice conversational loop.

Course outline

• Reminders: generating, conditioning, decoding — refreshes sampling, conditioning, and decoding from the prerequisite modules.
• Why control: safety, fidelity, intellectual property — the three industrial and legal motivations that make control unavoidable.
• Taxonomy: four families of control — the reading grid that structures the rest of the course.
• Constrained decoding: grammars, automata, masks — guaranteeing syntactically valid outputs at decoding time, mask by mask.
• Steering: controlling in activation space — orienting the model during the forward pass, without retraining.
• RLHF: control in behavior space — what preference alignment can and cannot impose.
• Watermarking: proving the origin of a text — statistically marking a generation and proving it with a z-test.
• Controlling diffusions: CFG, ControlNet, inversion — from classifier-free guidance to editing by inversion.
• Controlling graph generators — imposing constraints and properties where the generated object is discrete and combinatorial.
• Evaluating control: fidelity versus diversity — measuring what every constraint costs the generated distribution.
• Attacks on control and robustness — circumventing grammars, steering, and watermarks, and defending these mechanisms.
• Exercises / Labs — guided implementations: masking automaton, minimal steering, watermark and its test.
• Projects and assessment — proposed solo and group projects, a written-exam proposal and a grading rubric.
• References — the founding papers of each family of control.

Session plan

24 h in class (assumed) = 8 sessions of 3 h. The fiche publishes no teaching hours: 24 h is the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS) and is assumed here; the session split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced material (attacks, impossibility results, open problems); the level never goes down from one session to the next.

Self-study, outside the 24 h: no chapter — every chapter is covered in class; the labs (Ex. 1–3 and 6) each need about 2 h of personal work after the session that introduces them.

How to study this module

• Learn the taxonomy of ch. 2 by heart and practice classifying every “controlled generative AI” paper or product you come across: it is the module’s durable analysis tool.
• For constrained decoding, simulate the masking algorithm by hand on a mini-automaton: the difficulty is in the implementation, not the idea.
• Do the labs in order: steering, watermark, and diffusion are better understood once the decoding foundation is mastered.
• For each technique, note a “cost” line (compute, quality, diversity) and a “guarantee” line: this module is a course in trade-offs.
• Read ch. 10 last but reread it twice: attacking control is the best way to understand why it holds.

0. Reminders: generating, conditioning, decoding

Every control technique of this module modifies one of three things: the way a model generates, the signal it is conditioned on, or the decoding rule that turns its scores into an output. This chapter lists what the prerequisite modules taught about each, so that the later chapters can refer to it.

Decoding is the first place where control happens, often without being named. Every rule below turns the logits into a law, and most of them do it by a mask followed by a renormalization.

1. Why control: safety, fidelity, intellectual property

Consider an agent that “follows” its prompt in every respect except the output grammar that the business application expects. This chapter explains why such failures are unavoidable without dedicated mechanisms, and motivates each family of control by the failure it addresses.

A generator is a law over an immense space. Asking it politely (a prompt) only constrains this law statistically: no theorem guarantees the exact format, the safety of the content or the traceability of the origin.

2. Taxonomy: four families of control

Many techniques claim to “control” a generator; this chapter sorts them into four families according to where they intervene and what they guarantee, so that every later chapter can be placed on the same grid.

3. Constrained decoding: grammars, automata, masks

The technical heart of the module for text: guaranteeing that an output authorized token by token belongs to a formal language — regular expression, context-free grammar (JSON, SQL), or logical constraint. The founding idea (PICARD, then Outlines and XGrammar): decoding is a walk in an automaton, and the LLM is only allowed to follow valid transitions.

3.1. From regex to automaton

3.2. The masking algorithm, in detail

# pedagogical skeleton: automaton-based constrained decoding (cf. lab 1)
import torch

class DfaMask:
    """dfa.walk(q, s): state reached by reading the string s from q, or None
    (dead prefix); the DFA is already pruned (prop. 3.2).
    token_str: {token_id: decoded string}, special tokens (EOS included) left out."""

    def __init__(self, dfa, token_str, vocab_size, eos_id):
        self.accepting, self.eos_id = dfa.accepting, eos_id
        self.next = {}                         # q -> {token_id: q2}
        self.allowed = {}                      # q -> boolean tensor, precomputed
        for q in dfa.states:                   # O(|Q| x |vocab| x |v|), once
            self.next[q] = {}
            ok = torch.zeros(vocab_size, dtype=torch.bool)
            for tid, s in token_str.items():
                q2 = dfa.walk(q, s)            # read character by character
                if q2 is not None:
                    self.next[q][tid] = q2
                    ok[tid] = True
            self.allowed[q] = ok

    def mask(self, logits, q):
        out = logits.masked_fill(~self.allowed[q], float("-inf"))  # -inf = forbidden
        if q in self.accepting:                # mask first, then restore EOS
            out[self.eos_id] = logits[self.eos_id]
        return out

# generation loop (the DFA is the state of the control, the LLM provides the logits)
#   q = dfa.q0
#   while True:
#       logits = model(y).logits[0, -1, :]
#       v = int(torch.multinomial(torch.softmax(dm.mask(logits, q), -1), 1))
#       if v == dm.eos_id:  break                        # only possible when q in F
#       q = dm.next[q][v]
#       y = torch.cat([y, torch.tensor([[v]])], dim=1)   # invariant: prefix of L(A)

\{"city": "[^"\\]*", "unit": "(C|F)"\}

4. Steering: controlling in activation space

Constrained decoding acts on the outputs. Steering acts inside the network: if concepts (“positivity”, “gender”, “refusal”) are encoded by directions of the activation space, then moving them should move the behavior. This is the most fascinating and least guaranteed control of the module.

5. RLHF: control in behavior space

RLHF was presented as alignment in Large Language Models ch. 5, taught in the same period; here we reread it as a control mechanism. Its closed form is a lemma of Year 1, recalled below; the chapter then adds the reward model, DPO and one GRPO step, and shows that RLHF is — mathematically — the twin of classifier-free guidance in ch. 7. This double reading is the module’s synthesis point.

The closed form needs a reward. In RLHF it is learned from human comparisons between two answers.

PPO and GRPO reach the same optimum by sampling instead of regression. One step of GRPO, by hand, shows the mechanism.

6. Watermarking: proving the origin of a text

Provenance constraint (def. 1.1 iii): modify the generation so that outputs carry a secret statistical mark, verifiable without wiggle room. The green/red scheme of Kirchenbauer et al. (ICML 2023) is the most taught because its test can be proved at the M2 level; its canonical statement on this site is Large Language Models, Theorem 9.3, with the same symbols. This chapter recalls the test, then spends its time on what the LLM course does not cover: power and sample size, distortion-free schemes, and the bridge with audio watermarking (Speech Technology, ch. 9) and with ch. 9 of the Explainability module.

Why 0.7–0.8 and not more? If the logits were i.i.d. and independent of the green list, the law of large numbers over a large vocabulary would give p_\delta \to \frac{\gamma e^\delta}{\gamma e^\delta + 1 - \gamma} \approx 0.88 \qquad (\gamma = 0.5,\ \delta = 2). Real models are often nearly certain of the next token, and a bump of \delta cannot move such a low-entropy step: the watermark only writes where the model hesitates.

The green/red scheme distorts the law: it shifts mass toward greens. A second scheme, due to Aaronson (2022), marks the text without changing the law of any single step.

Two caveats follow from the proof. A step where the model is certain (p_v = 1) emits its token whatever r, so it contributes an unmarked \mathrm{Exp}(1) term: as for green/red, the signal lives in the entropy. And for a fixed key and a fixed prompt, generation is deterministic: “distortion-free” holds over the keys, not across repeated samples of the same prompt — a diversity cost that ch. 9 can measure.

7. Controlling diffusions: CFG, ControlNet, inversion

Control of diffusion models was taught in Year 1 (Multimodal GenAI, ch. 4): CFG with its proof, and the ControlNet formula. Here we recall both in a few lines, state precisely what the CFG identity does and does not guarantee, prove why ControlNet’s zero-convolutions do not block learning, and spend the rest of the session on editing real images by inversion.

7.1. CFG: recall, and what the identity does not say

• At w = 1: pure conditional model;
• w > 1: over-guidance (fidelity \uparrow, diversity \downarrow, color saturation — the classic visual symptom);
• w < 1: interpolation toward the unconditional.

7.2. ControlNet: spatial control, without breaking the pre-training

7.3. Editing a real image: inversion and null-text

To edit a real image with a text-to-image model, one first needs the noise that the model would turn into this exact image. DDIM is deterministic, so it can be run backwards; the difficulty is that guidance breaks the reversal.

Only the unconditional branch depends on c_\varnothing^{(t)}, so the gradient flows through one of the two network passes of each guided step. Editing (Prompt-to-Prompt: substituting the cross-attention maps; InstructPix2Pix: learning editing on synthetic data) then operates along this faithful trajectory, by changing the prompt while keeping the optimized null embeddings.

8. Controlling graph generators

Synthesis chapter. Graph generators were introduced in Year 1 (Graph ML, ch. 9, sections 9.3–9.6); Graph Generative Models develops them in depth, but only in period 2, after this course. So this chapter is self-contained: half a page per model (8.1), then their controls read through the grid of ch. 2 (8.2), to show that the theory of control there is identical to that of text and images.

8.1. Three generators, in half a page each

8.2. Their controls, on the grid of ch. 2

Two further controls complete the grid. The first, extending an imposed scaffold, is in the DiGress paper itself (its Sec. 5 and Appendix E): after every reverse step, the scaffold’s nodes and edges are reset to their given values. This is the mixing rule of RePaint (Lugmayr et al., CVPR 2022) for image inpainting (Multimodal GenAI, 4.4), with no retraining, except that the paper mixes in the clean scaffold where RePaint mixes in a copy noised to the current step.

It guarantees the scaffold, a hard constraint on a subgraph, but in general not the exact conditional law of the rest of the graph: the paper’s Fig. 8 reports occasional long-range inconsistencies and suggests a resampling like RePaint’s as a possible remedy.

The second, text-to-graph generation, conditions on an LLM’s encoding of a description (Graph ML 9.7, Graph Generative Models ch. 7).

9. Evaluating control: fidelity versus diversity

Controlling without measuring is believing. This chapter sets the protocol: two axes (fidelity, diversity), one curve (the strength sweep), one decomposition (precision/recall), and a bound showing that the trade-off is structural, not an engineering defect.

10. Attacks on control and robustness

Control is a promise; this chapter tests the promise. Four surfaces, in the module’s order: the mask, steering/RLHF, the watermark, spatial control.

Exercises / Labs

1. ★ Lab: constrained decoding on regex (the module’s lab). (a) Build the DFA of ^\d{2}/\d{2}/\d{4}$ by hand (states, co-accessible pruning), table \mathrm{Adm}(q) on a mini-vocabulary including the multi-character tokens "12/", "19"; (b) implement DfaMask (code of ch. 3) with a small GPT’s tokenizer; (c) plug in as a HuggingFace LogitsProcessor and generate 1,000 dates; (d) measure the pass rate with/without mask and the time overhead compared to free generation.

   Proof. Guidance (expected results): without constraint, the pass rate of a small model on the exact requested date plateaus (2026-08-26, 26/08/2026 formats mixed).

   With mask: 100% — including false but syntactically valid dates (31/02/2026): semantics requires one more layer (bounds validation), the conclusion of ch. 10 experienced in the lab.

   Overhead: ~0% (the mask is negligible next to the transformer pass); the real cost is the precomputation of the index, paid once.

   Bonus: compare the histogram of the masked months with that of rejection sampling (proposition 2.4). They differ as soon as the mask removes different amounts of mass on different paths: the bias of proposition 2.2, measured. ∎

2. ★ The z-test in full. With \gamma = 0.5 and \delta = 2:

   • 1. simulate a million human texts of 200 tokens (vectorized binomial draws of s_G) and empirically verify \mathbb{P}(z > 4) \approx 3 \times 10^{-5} — with only ten thousand texts you would expect 0 or 1 exceedance;
   • 2. estimate p_\delta empirically by simulating the biased softmax at temperature 1 over a vocabulary of 50,000 tokens with i.i.d. Gaussian logits, of standard deviation 1 and then 2.

   Then: (c) plot the power of the test as a function of T \in [20, 500]; (d) derive the minimal T for power 0.9 at the threshold z = 4 (compare with example 6.3); (e) repeat with \gamma = 0.25 and conclude on the tuning.

   Proof. Guidance:

   • (a) the distribution of z sticks to \mathcal N(0,1) from T \geq 50 (De Moivre-Laplace); the exact binomial tail at T = 200 is about 2.5 \times 10^{-5}, slightly below the normal value;
   • (b) p_\delta \approx 0.88 for both values of \sigma: with i.i.d. logits independent of the green list, the law of large numbers gives \gamma e^\delta / (\gamma e^\delta + 1 - \gamma) whatever the variance. The values 0.7–0.8 of ch. 6 come from real models, whose low-entropy steps resist the bump;
   • (c) sigmoidal power in \sqrt{T};
   • (d) T \approx 40 with the simulated p_\delta \approx 0.88; about 100 with a real model’s p_\delta \approx 0.75;
   • (e) \gamma = 0.25 divides the variance under H0 (\times 0.75) and concentrates the signal — slightly better at fixed \delta, but lower text quality: the triple trade-off of ch. 6 is verified numerically. ∎

3. ★ Steering step by step. On GPT-2 small, replay the two experiments of example 4.2 with their own vectors, for l \in \{2, 5, 8, 10\} and \alpha \in \{-8, -4, -2, 0, 2, 4, 8\}: (i) the weddings vector, and the frequency of wedding words; (ii) the Love–Hate vector at layer l, and the sentiment of the continuations (an off-the-shelf sentiment classifier). For both, measure the perplexity and the intra-batch diversity (self-BLEU).

   Proof. Guidance: the topic and sentiment effects are clear and quasi-linear in \alpha at small \alpha (prop. 4.3); perplexity moves little at moderate \alpha then derails.

   Diversity drops as |\alpha| grows — the same fidelity/diversity curve as CFG (ch. 9), again. Discuss: why does steering “succeed” on broad latent concepts and fail to impose a precise format (answer: full support — ch. 2). ∎

4. ★★ Theoretical fidelity/diversity frontier. Prove the entropy bound 9.2 (Gibbs), then generalize example 9.4: arbitrary initial law, score c, plot (\mathrm{Fid}(w), H(w)) for w \in [0, 10] and verify that the curve converges to \big(\max c,\ H(p \mid \arg\max c)\big) — the corner (\max c, 0) when the maximizer of c is unique (in example 9.4 it is not, hence the limit 0.683 nats). What happens to the curve if c is noisy?

   Proof. Guidance: normalize p_w \propto p\, e^{w c} (exponential family) and write A(w) = \log \sum_y p(y) e^{w c(y)}, so that A'(w) = \mathbb{E}_w[c] and \frac{d}{dw}\mathbb{E}_w[f] = \mathrm{Cov}_w(f, c).

   Differentiating H(w) = -\mathbb{E}_w[\log p] - w\,\mathbb{E}_w[c] + A(w) gives dH/dw = -\mathrm{Cov}_w(\log p, c) - w\,\mathrm{Var}_w(c); for a uniform initial law the first term vanishes and H decreases for w > 0, otherwise it can rise briefly when c favors outputs that p makes rare.

   With noise, a plateau appears before the collapse: the analyst chooses the elbow. ∎

5. ★★ Zero-convolution: the gradient under the magnifier. For the ControlNet branch of proposition 7.2 at initialization: (a) show that the branch output is exactly zero; (b) compute \partial y / \partial W_2 and show it is nonzero; (c) show that \partial y / \partial \Theta_c = 0 at initialization and explain why this is not a deadlock; (d) compare with random initialization (branching F_c without Z): simulate 10 descent steps on a toy task and compare the initial losses.

   Proof. Guidance:

   • (a) Z_2 = W_2 h = 0;
   • (b) \partial y/\partial W_2 = h_c \neq 0: since Z_1(c) = 0, the copy receives x and returns h_c = F(x;\Theta), the pretrained activations;
   • (c) chain rule through W_2 = 0 (the gradients toward \Theta_c and Z_1 both carry the factor W_2), unlocked as soon as W_2’s first update happens;
   • (d) the variant without zero-conv starts from a high loss and destroys the frozen model’s output — the experiment reproduces the motivation figure of the ControlNet paper. ∎

6. ★★★ Attacking your own watermark (bonus). After lab 2: paraphrase the marked texts (via an LLM, instruction “rewrite with the same ideas”), measure the drop in z as a function of the rate of replaced tokens, and estimate the rate needed to fall below z = 2. Conclude on the sentence “a watermark is a marker, not a proof”.

   Proof. Guidance: each change that turns one green token red lowers s_G by one, hence z by 1/\sqrt{T\gamma(1-\gamma)} (prop. 10.2).

   A token replaced at random was green with probability p_\delta and is green with probability \gamma afterwards, so it lowers \mathbb E[z] by about (p_\delta - \gamma)/\sqrt{T\gamma(1-\gamma)} at its own position — and by as much again at the next position, whose green list it reseeds.

   Going from \mathbb E[z] = 7 to 2 means leaving only 2/7 of the positions marked. With the hash of the previous token, a position stays marked only if neither it nor its predecessor was replaced, with probability (1-\rho)^2 for a replacement rate \rho: hence \rho \approx 1 - \sqrt{2/7} \approx 0.47, nearly half of the tokens — the text that survives the attack is so transformed that content traceability (plagiarism) becomes the issue again, not that of the machine. Loop closed with ch. 10. ∎

Projects and assessment

The official assessment, from the fiche: not displayed — the 2026–2027 SynapseS catalogue has published no course description for this module (its S3 and S4 tracks are empty), so no modality, weight or exam is given. Everything below is therefore a proposal: one project, solo or in a group, assessed on a report and a talk, and a written exam for a final or resit session.

The projects exercise objectives O2 (generative modelling theory: the bias of masking, CFG and exponential tilting), O3 (LLMs: constrained decoding, steering, RLHF and DPO, watermarking) and O8 (trustworthy AI: provenance, attacks, robustness budgets), with O7 through the labs and O4–O5 through the graph generators of ch. 8. Exercise 1 (the automaton mask as a LogitsProcessor) is reused by the Year 2 transverse projects Y2.2, Y2.4 and Y2.5 of the projects page, and the second group project below feeds Y2.4.

Solo projects

1. [Solo · ≈ 25 h] Paper study: “A Watermark for Large Language Models” (Kirchenbauer, Geiping, Wen, Katz, Miers & Goldstein, ICML 2023). Does the z-test keep its power on a small open model, how many edits erase the mark, and what does the mark cost in text quality?
   • Implement: the green/red scheme (definition 6.1) and the unigram variant (Zhao et al., 2023) as LogitsProcessors; the z-test (theorem 6.2); three attacks — synonym swaps, homoglyphs, paraphrase by a small instruction model.
   • Data: prompts from the “realnewslike” split of C4, as in the paper; OPT-1.3B or a smaller open model on a free GPU; the human continuations of the same documents as the null sample.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 6, ch. 9, ch. 10.
   • Success: power curves against T, \gamma and \delta with intervals; no false positive at z > 4 on at least 1,000 human texts; the drop of z against the edit rate compared with the prediction of exercise 6; the perplexity cost measured under a larger model.
2. [Solo · ≈ 25 h] Prompt, steering or DPO: three ways to impose a formal register. At an equal quality budget, which control buys the most formality for the least loss of diversity?
   • Implement: prompting (conditioning); contrastive activation addition (Panickssery et al., ACL 2024, published under the name Rimsky) with a vector from formal/informal sentence pairs (ch. 4); a small DPO fine-tune with LoRA (proposition 5.3) on pairs ranked by a formality classifier; a sweep of each strength — prompt variants, \alpha, 1/\beta.
   • Data: news articles of CNN/DailyMail to summarize; the sentence-level formality ratings of Pavlick & Tetreault (TACL 2016) for the contrast pairs and the classifier; Qwen2.5-0.5B-Instruct.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 4, ch. 5, ch. 9.
   • Success: the three fidelity/diversity curves of proposition 9.3 on the same axes, with bootstrap intervals over 3 seeds; ROUGE against the reference summaries reported as the quality budget; the elbow of each method named and justified.
3. [Solo · ≈ 25 h] Paper study: “Null-text Inversion for Editing Real Images using Guided Diffusion Models” (Mokady, Hertz, Aberman, Pritch & Cohen-Or, CVPR 2023). How much of the drift of DDIM inversion under guidance does null-text optimization remove, and how many inner steps does it need?
   • Implement: DDIM inversion with w = 1 and regeneration at w = 7.5, to exhibit the drift (proposition 7.3); null-text optimization (definition 7.4), with the gradient through the unconditional branch only; one Prompt-to-Prompt word swap.
   • Data: Stable Diffusion v1.4, the model of the paper, through the diffusers library, in half precision on a free 16 GB GPU; 20 captioned images of COCO 2017 validation.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 7, ch. 9.
   • Success: PSNR and LPIPS of the reconstruction for 0, 1, 5 and 10 inner steps, with intervals over the 20 images, against plain inversion at w = 7.5; three edits shown with their failure cases.

Group projects

1. [Group of 3 · ≈ 30 h each] How biased is masked decoding? On languages small enough to enumerate, how far is the masked law from the true conditional, and what does it cost to correct it?
   • Implement: the mask of ch. 3 as a LogitsProcessor (exercise 1); the exact law p_\theta(\cdot \mid S) by scoring every element of S; the masked law from the per-step masses Z_t (proposition 2.2); rejection (proposition 2.4); sequential Monte Carlo with particles weighted by \prod_t Z_t (Lew et al., 2023); grammar-aligned decoding (Park, Wang, Berg-Kirkpatrick, Polikarpova & D’Antoni, NeurIPS 2024) as the paper study.
   • Data: three finite languages — the 366 dates of 2024 written DD/MM/YYYY, a list of 200 country names, the calls of a two-field JSON schema with enumerated values (example 3.5); GPT-2 and Qwen2.5-0.5B on a laptop CPU.
   • Split: one member on the automata and the exact enumeration; one on rejection, SMC and grammar-aligned decoding; one on the evaluation (KL, validity, diversity of ch. 9).
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 2, ch. 3, ch. 9. The same question for valence masks continues in Graph Generative Models, ch. 3.
   • Success: the exact divergence between the conditional and the masked law for each language and model; every measured ratio inside the band of proposition 2.2 for the observed \varepsilon; the divergence of SMC against the number of particles, at a counted number of forward passes.
2. [Group of 3 · ≈ 30 h each] Secure structured tool calls: a grammar-constrained agent under prompt injection. A mask guarantees valid calls; does it reduce, leave unchanged or hide the success of injections that stay syntactically valid?
   • Implement: an agent whose tool calls are constrained by their JSON schemas (example 3.5, with Outlines, XGrammar or the mask of exercise 1); indirect injections that keep every call valid (theorem 10.1); a probe of grammar complexity (compile time and states against nesting and enumerations); three defenses — semantic validation of arguments, least-privilege tools, spotlighting of untrusted content.
   • Data: the AgentDojo benchmark (Debenedetti et al., NeurIPS 2024 Datasets and Benchmarks), its banking and workspace suites; a small open instruction model with tool calling.
   • Split: the constrained tool layer; the red team (injections and complexity probe); the defenses and the evaluation.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 3, ch. 10. Builds on Large Language Models, 9.4 and its agents of ch. 8; feeds the dispatcher project Y2.4.
   • Success: utility and attack success rate without the mask, with it, then with each defense, with intervals over the tasks; the share of successful attacks whose calls were valid; a measured blow-up of states or compile time, and the limit that stops it.
3. [Group of 4 · ≈ 30 h each] One control curve, three modalities. Does one protocol — strength sweep, fidelity/diversity curve, robustness budget — describe guidance on images, text and molecules, and where does the analogy break?
   • Implement: a CFG sweep on a small class-conditional diffusion model trained with condition dropout (ch. 7); a steering sweep on GPT-2 (ch. 4); a conditional SMILES language model trained with a property-bin token dropped 10% of the time, guided in logit space as in example 7.5 (ch. 8); the protocol of proposition 9.3 and the budget of definition 10.3, under one attack per modality.
   • Data: Fashion-MNIST, its classes as conditions; the prompts of example 4.2; ZINC250k, with RDKit for QED and validity.
   • Split: one member per modality; one on the shared harness (fidelity, diversity, precision and recall, attacks).
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 4, ch. 7, ch. 8, ch. 9, ch. 10. Connects Multimodal GenAI, ch. 4 and Graph Generative Models, ch. 7.
   • Success: the three curves on normalized axes with intervals over 3 seeds; the elbow of each; at least one place where the single-step prediction of exercise 4 fails, explained.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet of notes allowed, no electronic device; four questions, 20 points.

1. Question (4 points). For each mechanism, give its family (definition 2.1), its guarantee and its main cost: (a) a JSON-schema mask on tool calls; (b) CFG with w = 5; (c) resampling until a validator accepts; (d) a green/red watermark. Then say when you choose a hard constraint rather than guidance. Expected:

   • 1. hard constraint: validity by construction, one pass plus a mask, and a law renormalized step by step, hence biased (proposition 2.2);
   • 2. guidance: statistical fidelity, two network passes per step, diversity lost as w grows (ch. 9);
   • 3. post-processing: the exact conditional law, at an expected 1/p_\theta(S \mid x) draws (proposition 2.4);
   • 4. guidance used for provenance: a statistical test, a quality cost that grows with \delta.
   • A hard constraint when a failure is catastrophic (a parser, a valence), guidance when it is gradual (tone, style).

2. Question (6 points). A green/red watermark uses \gamma = 0.25.

   • 1. A text of 400 tokens contains 150 green tokens: compute z and conclude at the threshold z = 4.
   • 2. How many green tokens must an attacker exchange for red ones to bring z below 4? Assume that each exchange lowers s_G by exactly one.
   • 3. With the exponential-minimum scheme instead, another text of 400 tokens has score 480. Conclude, first with a normal approximation of the null law, then with its exact tail, about 8 \times 10^{-5}.

   Expected:

   • 1. the standard deviation is \sqrt{400 \cdot 0.25 \cdot 0.75} = \sqrt{75} \approx 8.66, so z = 50 / 8.66 \approx 5.77 > 4: reject H0 (one-sided p-value \approx 4 \times 10^{-9}).
   • 2. each exchange lowers z by 1/8.66 \approx 0.115; z < 4 requires s_G < 134.6, so s_G \leq 134: 16 exchanges.
   • 3. under H0 the score is \mathrm{Gamma}(400, 1), of mean 400 and standard deviation 20, so the normal approximation gives z = 4, just at the threshold. The exact tail is larger than the normal one (3 \times 10^{-5}): at that level, do not accuse.

3. Question (5 points). Masking at each step the tokens that make a set S unreachable.

   • 1. Show on an example that it does not sample the conditional law p_\theta(\cdot \mid x, S).
   • 2. Prove that if every output of S has at most T steps and the mask never removes more than a mass \varepsilon, the masked law is within a factor (1-\varepsilon)^{\pm T} of the conditional one.

   Expected:

   • 1. the two-token example 2.3: the masked law gives aa the probability 0.5, the conditional law 0.05/0.55 \approx 0.09.
   • 2. on S the masked law is the model’s law divided by \prod_t Z_t, so the ratio is p_\theta(S \mid x) / \prod_t Z_t.
   • Each Z_t lies in [1-\varepsilon, 1], so the product over at most T steps lies in [(1-\varepsilon)^T, 1], and p_\theta(S \mid x) is an average of such products under the masked law: the ratio lies in the band.

4. Question (5 points). Let the numbers r_v, one per token of the vocabulary, be i.i.d. uniform on [0, 1], and let p be a law on the vocabulary. Prove that y = \arg\max_v r_v^{1/p_v} has law p. Then give the null law of the detection score of proposition 6.4 over T tokens.

   Expected:

   • E_v = -\log r_v are i.i.d. exponential of rate 1, and maximizing r_v^{1/p_v} is minimizing E_v / p_v, exponential of rate p_v.
   • The minimum of independent exponentials is reached at v with probability p_v / \sum_u p_u = p_v: no distortion at each step.
   • Under H0 the text ignores the key, each r_{y_t} is uniform, each term is \mathrm{Exp}(1), and the sum is \mathrm{Gamma}(T, 1): mean T, variance T.

Grading rubric (proposal)

References

• Constrained decoding: B. Willard, R. Louf, Efficient Guided Generation for LLMs, 2023 (library Outlines); Y. Dong et al., XGrammar: Flexible and Efficient Structured Generation Engine for LLMs, ICLR 2025 (code); T. Scholak, N. Schucher, D. Bahdanau, PICARD, EMNLP 2021; on the bias of masking (proposition 2.2): A. K. Lew, T. Zhi-Xuan, G. Grand, V. K. Mansinghka, Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs, 2023; K. Park, J. Wang, T. Berg-Kirkpatrick, N. Polikarpova, L. D’Antoni, Grammar-Aligned Decoding, NeurIPS 2024.
• Steering: A. Turner et al., Activation Addition: Steering Language Models Without Optimization, 2023; A. Zou et al., Representation Engineering: A Top-Down Approach to AI Transparency, 2023; K. Li et al., Inference-Time Intervention, NeurIPS 2023; G. Ilharco et al., Editing Models with Task Arithmetic, ICLR 2023.
• RLHF and alignment: L. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), NeurIPS 2022; R. Rafailov et al., Direct Preference Optimization, NeurIPS 2023; Z. Shao et al. (DeepSeek-AI), DeepSeekMath (GRPO), 2024 — the practical details are in the LLMs module (M2) ch. 5; the closed form is the lemma of Reinforcement Learning, ch. 6.
• Watermarking: J. Kirchenbauer et al., A Watermark for Large Language Models, ICML 2023; J. Kirchenbauer et al., On the Reliability of Watermarks for LLMs, ICLR 2024; X. Zhao et al., Provable Robust Watermarking for AI-Generated Text (unigram), 2023; R. Kuditipudi et al., Robust Distortion-free Watermarks for Language Models, TMLR 2024; S. Dathathri et al., Scalable watermarking for identifying large language model outputs (SynthID-Text), Nature 634, 2024; S. Aaronson, Watermarking of large language models (exponential-minimum scheme), talk and blog, 2022; S. Sadasivan et al., Can AI-Generated Text be Reliably Detected?, 2023; M. Christ, S. Gunn, R. Zamir, Undetectable Watermarks for Language Models, 2024 — extended by the Explainability, Security, Privacy module ch. 9.
• Diffusion: guidance and control: J. Ho, T. Salimans, Classifier-Free Diffusion Guidance, 2022; M. Chidambaram et al., What does guidance do? A fine-grained analysis in a simple setting, 2024; A. Bradley, P. Nakkiran, Classifier-Free Guidance is a Predictor-Corrector, 2024; L. Zhang, A. Rao, M. Agrawala, Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet), ICCV 2023 (code); R. Mokady et al., Null-text Inversion for Editing Real Images, CVPR 2023; A. Hertz et al., Prompt-to-Prompt Image Editing with Cross-Attention Control, ICLR 2023; C. Meng et al., SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR 2022; T. Brooks et al., InstructPix2Pix, CVPR 2023 — foundations in the Multimodal GenAI module ch. 3-4.
• Controlled graphs: C. Shi et al., GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation, ICLR 2020; B. Samanta et al., NeVAE: A Deep Generative Model for Molecular Graphs, AAAI 2019; C. Vignac et al., DiGress: Discrete Denoising Diffusion for Graph Generation, ICLR 2023 (scaffold extension: Sec. 5 and Appendix E); A. Lugmayr et al., RePaint: Inpainting using Denoising Diffusion Probabilistic Models, CVPR 2022; J. You, B. Liu, R. Ying, V. Pande, J. Leskovec, Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation (GCPN), NeurIPS 2018; W. Jin, R. Barzilay, T. Jaakkola, Junction Tree Variational Autoencoder for Molecular Graph Generation, ICML 2018; J. You et al., GraphRNN, ICML 2018; Y. Li et al., Learning Deep Generative Models of Graphs (DeepGMG), 2018 — introduced in Graph ML, ch. 9 and developed in the Graph Generative Models module (M2).
• Evaluation: T. Kynkäänniemi et al., Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019; FID and evaluation of generative models: Multimodal GenAI module ch. 7; distributional evaluation (MMD, FCD): Graph Generative Models module ch. 6.
• Regulatory framework: Regulation (EU) 2024/1689 — AI Act (art. 50, transparency of generated content); OWASP Top 10 for LLM Applications (injections).