Explainability, Security, Privacy of LLMs

Contents

• Introduction and overview
• 0. Recap: the black box and its three questions
• 1. Internal interpretability: probing and control
• 2. Reading the layers: logit lens, attention and its limits
• 3. Superposition and sparse autoencoders
• 4. Explaining an output: SHAP, LIME, gradients
• 5. Security: taxonomy of attacks
• 6. Defenses in depth
• 7. Privacy I: memorization and extraction
• 8. Privacy II: membership inference and DP
• 9. Anonymization and watermarking
• 10. Evaluating safety: red teaming, AI Act
• Exercises / Labs
• Projects and assessment
• References

Introduction and overview

Audience: an engineer running an LLM in production who must answer three questions: why did the model produce this output, how can it be broken, and what does it leak?

Empirical first: each part starts from a concrete failure — a linear “explanatory” probe that does not generalize, a jailbreak that passes the filter, a membership inference on customer data — and only then names the tools (probing, sparse autoencoders, SHAP, defenses, differential privacy).

The module works on three fronts of the same black box: interpretability, security and privacy.

What you will be able to do

• Design probing experiments and causality tests to know what a layer actually encodes;
• Read the layers of a transformer with the logit lens, and honestly discuss the limits of attention weights;
• Explain the superposition phenomenon and use sparse autoencoders to extract interpretable features;
• Explain an individual output by SHAP, LIME or gradients, knowing the pitfalls of each method;
• Map the attacks on an LLM: jailbreaks, prompt injections, poisoning, extraction;
• Design a defense in depth (filtering, alignment, isolation) without believing in the single defense;
• Audit a model’s memorization, conduct a membership inference attack and apply DP;
• Anonymize, watermark, and put an LLM through a red teaming exercise aligned with the AI Act.

Official objectives and syllabus

Official course description not published (S3/S4 catalog empty): as of the start of the 2026 academic year, the “LLGA 2 – Semestre 3” and “LLGA 2 – Semestre 4” tracks of the 2026-2027 SynapseS catalog still list no courses, and the master’s wiki indicates that the detailed M2 program is “under development” (period 2: “to be completed soon”).

The program below therefore reflects the master’s brochure (period 2), with an indicative volume of ~24 h; these course materials are calibrated to that volume, i.e. about two hours of work per chapter, labs included.

The following table maps each brochure theme to the course sections; the materials’ own additions are marked “going beyond the syllabus”.

Where this course fits in the program

An M2 module offered in period 2 and preselected by both routes of the home page (Graph × LLM and AI systems): it is the convergence point of the master’s three “trust” foundations, applied to the central object of the LLGA.

Prerequisites: Large Language Models (mandatory, period 1), above all its ch. 9 — jailbreaks and injections (9.4), the watermark test (Theorem 9.3) and the privacy self-study 9.7 — together with the transformer and post-training chapters, without which one does not know what to probe.

For the DP of ch. 8: Privacy and Uncertainty Quantification ch. 1–4 (period 1), where the results used here are proved; Emerging ML ch. 5 (Year 1, optional) is a first contact. Ch. 8 opens with a one-page DP summary, so a reader who took neither can follow.

Co-requisites (optional, same period): Fundamentals of Security and Robustness for AI (attack taxonomy, membership inference) and ML Ops and LLM Engineering Principles (CI, monitoring, sandboxing). Neither is needed: ch. 0 states the threat model on its own, and the ML Ops pointers are optional reading.

Downstream: the module equips responsible deployment — the guarantees, audit and documentation that the AI Act requires for high-risk models — and feeds both the end-of-studies internship (industries audit their LLMs) and research pursuits on model safety (alignment, interpretability).

Course outline

• Recap: the black box and its three questions — reposition the attention architecture and training of LLMs before opening them.
• Internal interpretability: probing and control — probe the representations, then prove their role by intervening on them.
• Reading the layers: logit lens, attention and its limits — project the intermediate layers onto the vocabulary, and do not over-interpret attention.
• Superposition and sparse autoencoders — why a model encodes more concepts than dimensions, and how to separate them.
• Explaining an output: SHAP, LIME, gradients — attribute a prediction to input tokens, with the guarantees and illusions of each method.
• Security: taxonomy of attacks — jailbreaks, prompt injections, poisoning, extraction: the adversarial menagerie of LLMs.
• Defenses in depth — filtering, alignment guardrails, isolation: layering imperfect defenses that work together.
• Privacy I: memorization and extraction — measure what an LLM gives back verbatim from its corpus, and how to extract it.
• Privacy II: membership inference and DP — infer the presence of an individual in the data, and answer with differential privacy.
• Anonymization and watermarking — clean the corpora and prove the origin of a generation.
• Evaluating safety: red teaming, AI Act — organize the attack on one’s own system and document regulatory compliance.
• Exercises / Labs — probing, attacking and defending a small LLM, membership inference attack.
• Projects and assessment — solo and group subjects, a written-exam proposal and a grading rubric.
• References — the interpretability, security and privacy papers that set the state of the art.

Session plan

24 h in class (announced) = 8 sessions of 3 h. The SynapseS 2026–2027 sheet publishes no volume for this module; 24 h is the indicative volume quoted with the brochure programme in the fiche above, and also the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS); the session split below is our proposal for covering these notes in that time.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods; the level never goes down from one session to the next. Each of the three parts (interpretability, security, privacy) is taught at the core level once the first session has set the vocabulary.

Self-study, outside the 24 h: no chapter — every chapter is covered in class; the central lab (Ex. 8, with its GraphRAG variant) and the red-team protocol (Ex. 11) each need about 3 h of personal work, and Ex. 12 (b) about 2 h.

How to study this module

• Open a small open-weight model yourself from the first week (small transformer, accessible layers): all the interpretability chapters are understood by handling, not by reading.
• For each explanation method, first look for the counter-example where it deceives: methodical distrust is the skill of the module.
• Alternate roles in pairs during the labs: attacker then defender — it is red teaming training, not a classic tutorial.
• Keep a “safety” logbook of an LLM you use: every hallucination, leak or jailbreak observed becomes an example for ch. 10.
• Read the AI Act text cited in reference with the risk categories in mind: the regulatory vocabulary will become your professional vocabulary.

0. Recap: the black box and its three questions

Before opening, attacking or protecting a trained LLM, one must agree on what the object is and on what an attacker is assumed to be able to do. This chapter recalls the prerequisites, states the three questions of the module and defines the threat model used throughout.

This module asks three questions about one and the same object — the trained LLM — which correspond to three research communities:

• Explainability / interpretability (ch. 1-4): what does the model compute, and why this output? Internal interpretability (looking at the weights and activations) vs post-hoc (explaining a given output);
• Security (ch. 5-6): what can an attacker do — break the alignment (jailbreak), hijack applications (injection), corrupt the training (poisoning), steal the model?
• Privacy (ch. 7-9): what leaks — training data given back (memorization/extraction), membership in the corpus (membership inference), provenance of contents (watermarking)?

The capabilities have two coordinates. The first is the access level:

• white box: weights, gradients and activations are known (an open-weight model) — the setting of the gradient attacks of ch. 5 and of the interpretability chapters;
• grey box: the architecture, or the log-probabilities returned by an API, are known, but not the weights;
• black box: only the text answers of an API or of an application are seen.

The second is the phase in which the attacker acts, which gives the four families of attacks used in the rest of the module:

• evasion, at inference: an input crafted to make the deployed model misbehave — jailbreaks and prompt injections (ch. 5);
• poisoning, at training: examples planted in the training or fine-tuning data to install a behavior or a backdoor (5.4);
• extraction: copying the model through its API (5.4), or recovering its training data (ch. 7);
• membership inference: deciding whether a given example was in the training set (ch. 8).

1. Internal interpretability: probing and control

A linear probe reads “sentiment” from layer 6 with 95% accuracy, and a slide concludes that the model “uses” sentiment there. This chapter shows how to probe the representations of a model and, above all, what a probe does and does not prove.

import torch
from transformers import GPT2Model, GPT2TokenizerFast
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token            # GPT-2 has no pad token
model = GPT2Model.from_pretrained("gpt2").eval()     # frozen

@torch.no_grad()
def states(texts, layer=6):                          # residual stream activations
    batch = tokenizer(texts, return_tensors="pt", padding=True)   # right padding
    out = model(**batch, output_hidden_states=True)
    last = batch["attention_mask"].sum(1) - 1        # last real token, not a pad
    h = out.hidden_states[layer]                     # (n, T, d)
    return h[torch.arange(h.shape[0]), last].numpy()

X = states(phrases)                                  # (n, d=768)
Xtr, Xte, ytr, yte = train_test_split(X, pos_tags, test_size=.2)
probe = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
print("probe accuracy:", probe.score(Xte, yte))          # ~0.9 on POS in layer 6-8

1.1. What probes have revealed

Alain & Bengio (2016) launched the method on image classifiers: the linear separability of the classes increases monotonically with depth. Probes of transformer language models later found another profile: linguistic properties peak at intermediate layers and fade toward the output, whose layers specialize in predicting the next token (Liu et al., NAACL 2019; Tenney, Das & Pavlick, ACL 2019, for the order in which properties appear).

Probes have also mapped: early syntactic positions, named entities, world-fact relations (“the Eiffel Tower is in Paris”) manipulated by subspace edits, linearly decodable language and sentiment… The emerging hypothesis: the residual stream is a communication channel where each head writes to and reads from relatively linear subspaces.

1.2. The trap: information ≠ use

2. Reading the layers: logit lens, attention and its limits

Probes test one property at a time; one would also like to watch the prediction form as it travels through the layers, and to know whether attention weights say which tokens mattered. This chapter gives a tool for the first question and a warning for the second.

2.1. Logit lens and tuned lens

The logit lens is approximate — the intermediate representations are not in the basis of the final decoder. The tuned lens (Belrose et al., 2023) learns, for each layer, an affine transformation that “reframes” the intermediate prediction toward the final distribution; the trajectories become sharp and quantifiable (Jensen-Shannon distance between layers, anticipation of future tokens). Concrete uses: locating where a hallucination forms, where a factual answer is decided, where a model that contradicts its context is cheating.

2.2. Attention: useful tool, misleading explanation

3. Superposition and sparse autoencoders

If each neuron stood for one concept, interpreting a network would amount to labelling its neurons; in practice a single neuron responds to unrelated concepts. This chapter explains why, and how sparse autoencoders recover interpretable directions from the activations.

3.1. The problem: more concepts than dimensions

3.2. The answer: overcomplete dictionaries with SAEs

Bricken et al. (Towards Monosemanticity, 2023) obtain interpretable and manipulable features on a one-layer model.

Templeton et al. (Scaling Monosemanticity, 2024) scale up: SAEs trained on the residual stream of Claude 3 Sonnet, dictionaries on the order of a million to tens of millions of latents, abstract features (the Golden Gate Bridge, code security, irony…), and the demonstration that forcing a feature (steering: amplifying its activation) modifies the model’s behavior in a targeted way — the correlation → intervention loop of ch. 1 closes.

# Training of a toy SAE on the activations of a layer (free-form schema)
import torch
d, m, lmbda = 768, 8192, 1e-3
Wenc = torch.nn.Parameter(torch.randn(m, d) / d**0.5)
benc = torch.nn.Parameter(torch.zeros(m))
Wdec = torch.nn.Parameter(torch.randn(d, m) / m**0.5)
bdec = torch.nn.Parameter(torch.zeros(d))

def sae_loss(h):                       # h: batch of activations (frozen)
    z = torch.relu(h @ Wenc.T + benc)  # sparse code
    rec = z @ Wdec.T + bdec
    return ((h - rec)**2).mean() + lmbda * z.abs().mean(), z

# After training: sort the columns by activation frequency, read the
# top-k examples per feature -> human annotation -> feature dashboard

4. Explaining an output: SHAP, LIME, gradients

Chapters 1–3 look inside the model; a user or an auditor more often asks why the model gave this answer to this prompt. This chapter attributes an output to the input tokens, with the guarantees and the pitfalls of each method.

4.1. Perturbation methods: LIME and KernelSHAP

Perturbation methods treat the model as a black box: they switch parts of the input off and watch the output move. Machine Learning 6.4 gave the tabular versions: permutation importance for the global question, and TreeSHAP, which computes Shapley values exactly by walking down the trees. Neither transfers to an LLM: shuffling a column means nothing for a prompt, and there are no trees to walk.

Write z' \in \{0,1\}^M for a coalition of the M interpretable features (tokens, words or passages), h_x(z') for the input in which the features with z'_i = 0 are replaced by a baseline, and F for the scalar output explained.

4.2. The king method for LLMs: integrated gradients

4.3. Attributions for an LLM: uses and pitfalls

• Use 1 — debugging a RAG: on “why does the model cite this passage?”, per-token IG over the context shows which sentences carry the answer — and detects “ghost” citations (the model answers against the majority attribution → hallucination suspect);
• Use 2 — analyzing a refusal: which tokens trigger the safety refusal? (natural interface with ch. 6);
• Pitfall 1 — arbitrary baseline: changing x' changes the attributions; it must be justified (neutral token? empty sentence?) and reported;
• Pitfall 2 — discreteness: text is discrete, gradients live in the continuous space of embeddings — tokens off the path’s distribution can distort the integral (hence the discrete-gradient variants);
• Pitfall 3 — attribution is not the reason: it quantifies a local sensitivity of the model, not a mental process; the rule of ch. 1 (close with an intervention) still applies.

# Captum: IG on the probability of the generated token, prompt tokens on the x-axis
from captum.attr import IntegratedGradients
ig = IntegratedGradients(lambda emb: score_of_output_token(model, emb))

attributions, delta = ig.attribute(
    input_embeddings,                       # (1, n_tokens, d)
    baselines=pad_embeddings(input_embeddings),
    target=output_token_id, n_steps=50,
    internal_batch_size=8, return_convergence_delta=True)

for tok, a in zip(prompt_tokens, attributions.sum(-1).squeeze()):
    print(f"{tok!r:20s} {a:+.3f}")          # reading: which token pushes the output

5. Security: taxonomy of attacks

An LLM application can be attacked through its prompt, through the documents it reads, through its training data or through its API. This chapter maps these attacks with the threat model of chapter 0, then details the two that matter most in practice: jailbreaks and prompt injections.

5.1. The attack matrix

5.2. Jailbreaks: breaking the alignment

Alignment defenses (refusals learned by RLHF/DPO — LLMs module ch. 5) are statistical behaviors, not logical barriers. Wei et al. (2023) bring them down by analyzing the competing objectives: competence (follow any instruction) vs safety (refuse some) — any formulation that maximizes the competence objective (role-play, fictional context, question nested inside code, under-represented languages) can disable the refusal without any “forbidden word” appearing.

The classic families: personas (DAN), encode/decode (base64, morse…), multi-turn (gradually building up the context), “end of conversation” via special tokens (cf. LLMs module ch. 1).

5.3. Prompt injections: the Trojan horse of applications

5.4. Poisoning, stealing, side channels

• Fine-tuning poisoning (Wan et al., 2023): about a hundred booby-trapped examples in an instruction set suffice to install trigger-based backdoors (a chosen trigger phrase → a systematically wrong polarity, in the original experiment); legacy of BadNets (Gu et al., 2017). Ops: the provenance of training data is a security control (cf. ML Ops module ch. 9);
• Model extraction (Tramèr et al., 2016): by equations (solving the parameters from the returned probabilities) or by distillation (re-training a clone from queries) — APIs that expose the top-k logprobs facilitate the first; watermarking (ch. 9) is the forensic answer;
• Side channels: token-by-token streaming (inter-token time) and the length of responses leak information about the internal logits; timing attacks and keystroke inference show that an assistant “writes” while you type. A surface typically forgotten by threat models.

6. Defenses in depth

Chapter 5 showed that no filter can recognize every attack. This chapter therefore designs the system so that attacks are rarer and, above all, so that a successful attack can do little damage.

7. Privacy I: memorization and extraction

A language model trained on private documents may give some of them back word for word. This chapter defines memorization in operational terms, measures it with planted canaries, and shows how an attacker extracts memorized text.

7.1. Measuring: the canary and exposure

7.2. Attacking: the GPT-2 extraction

# Mini-extraction: recover the canaries planted in a small LM (exercise 2)
import zlib, torch

@torch.no_grad()
def log_ppl(model, tok, text):          # mean token negative log-likelihood
    ids = tok(text, return_tensors="pt").input_ids
    return model(ids, labels=ids).loss.item()

def zlib_bits(text):                    # zlib entropy: compressed length in bits
    return 8 * len(zlib.compress(text.encode()))

@torch.no_grad()
def extract(model, tok, prompt, n=500, temp=1.2):
    ids = tok(prompt, return_tensors="pt").input_ids
    outs = model.generate(ids, do_sample=True, temperature=temp, max_new_tokens=48,
                          num_return_sequences=n, pad_token_id=tok.eos_token_id)
    cands = tok.batch_decode(outs, skip_special_tokens=True)
    # Carlini et al. (2021) "zlib" score: low = likely for the model,
    # yet not trivially compressible (repetitive boilerplate is pushed down)
    ranked = sorted(cands, key=lambda s: log_ppl(model, tok, s) / zlib_bits(s))
    return ranked[:20]                  # -> check the intersection with the canaries

8. Privacy II: membership inference and DP

Even when nothing is given back verbatim, an attacker may learn whether a given person’s record was in the training set. This chapter measures that risk with membership inference attacks and answers it with differential privacy.

8.1. Membership inference: “was this point in the corpus?”

For an LM the signal is the loss: members have abnormally low perplexity. Calibrating the target’s log-likelihood by a reference model that did not see x (LLM 9.7) removes the part of a low perplexity that only says “x is an easy text”.

8.2. The guarantee: differential privacy applied to LLMs

9. Anonymization and watermarking

Two last privacy and provenance tools act on the data rather than on the training: anonymizing the corpus before it is used, and marking the generated text so that its origin can be proved later. This chapter shows why the first is harder than it looks and how the second is tested.

9.1. Anonymization: harder than it looks

The classic pipeline: entity detection (NER — NLP module ch. 6) → masking/replacement (pseudonymization) → re-training. Three traps:

(i) quasi-identifiers (postal code + date + occupation) re-identify without any named “entity” — absolute anonymization does not exist, one reduces a risk;

(ii) synthetic data generated by an LLM is not safe by construction: the generator also memorizes (ch. 7);

(iii) blind replacement breaks utility (names carry syntactic and semantic information).

Regulatory position: “definitive” anonymization takes the data out of the GDPR; pseudonymization does not — and the AI Act (ch. 10) requires reports on the training data, not their disappearance.

9.2. Watermarking: signing the generation

The independence of the green indicators under H_0 is a heuristic — the tokens of a natural text are correlated — so the calibration is checked on human text (Ex. 10). What the LLM module does not cover is the power of the test.

10. Evaluating safety: red teaming, AI Act

The previous chapters give attacks and defenses one by one; a deployed system must be evaluated as a whole and documented for the regulator. This chapter organizes the attack on one’s own system and maps the result onto the AI Act.

10.1. Red teaming: attacking one’s own system, methodically

Two pivotal results: Ganguli et al. (2022) describe the Anthropic program — human red team from the design stage, targeted automated attacks, finding novel “jailbreaks”; Perez et al. (2022) have the model attacked by another model (RL: the attacker is rewarded when the target produces forbidden content) — scaling: larger attackers find more attacks, transferable. Red teaming becomes a training loop: failures feed the preference data to harden the alignment (ch. 6), which is re-red-teamed — the cycle never stops, it stabilizes.

10.2. The regulatory framework: the AI Act

The Regulation (EU) 2024/1689 (“AI Act”) organizes AI by levels of risk: unacceptable (prohibited: social scoring, subliminal manipulation…), high risk (heavy requirements: risk management, data, documentation, human oversight), limited risk (transparency), minimal. Key points for an LLM:

• General-purpose AI models (GPAI): technical transparency obligations (documentation, copyright compliance policy, summary of the training data); for models presenting a systemic risk (capability threshold): evaluations, mitigation, incident reporting;
• Content transparency: interaction with an AI to be disclosed; generated or manipulated contents (“deepfakes”) to be marked machine-readably — watermarking (ch. 9) becomes a compliance tool;
• Timeline: entry into force on 1 August 2024; prohibitions from 2 February 2025; general-purpose AI models from 2 August 2025; transparency duties (art. 50) from 2 August 2026, with until 2 December 2026 for providers of generative systems already on the market; high-risk requirements from 2 December 2027 (Annex III uses) and 2 August 2028 (Annex I products), after the Digital Omnibus on AI, Regulation (EU) 2026/1744 (dates and details on the ethics seminar page);
• Complementary: NIST AI RMF (govern / map / measure / manage) — the operational grid; ISO/IEC 42001 (management); GDPR for data processing; sectoral codes of conduct.

Exercises / Labs

1. ★ Probes and selectivity (ch. 1). Train linear probes (language, sentiment, POS category) on every layer of a GPT-2; plot the accuracy per layer; add the control tasks of Hewitt & Liang and report the selectivity; conclusion on “is the information linearly encoded?”.
2. ★ Canaries and extraction (ch. 7). Place canaries (random phone numbers) in a fine-tuning corpus with k = 1, 10, 100 repetitions; measure the exposure (Metric 7.2); launch the extraction attack ranked by the perplexity/zlib ratio (Experiment 7.3); repeat after deduplication — conclude.
3. ★ Attributions (ch. 4). Compute the IG of a RAG answer with respect to its context tokens; verify the completeness (thm. 4.6) numerically (sum of the attributions vs logit gap); compare two baselines (pad vs empty sentence) and comment on the stability.
4. ★ Regulatory framework (ch. 10). Classify three scenarios (internal HR assistant, public health chatbot, journalistic content generation) in the AI Act risk pyramid; list for each the concrete obligations and the ML Ops artifact that attests to them (registry? eval? monitoring?) — prop. 10.2 in practice.
5. ★★ Home-grown tuned lens (ch. 2). Implement the logit lens on GPT-2 (final LayerNorm + unembedding at every layer); train the affine transformations of the tuned lens per layer (regression toward the final distribution); visualize the trajectory of the top token on 50 examples — identify the “late switches”.
6. ★★ Attention under control (ch. 2). Reproduce the experiment of Jain & Wallace on a small model: show that an adversarial (permuted) attention distribution preserves the prediction; then show with patching that an identified induction head is causally necessary for pattern completion. Two conclusions, one method.
7. ★★ SAE on a toy model (ch. 3). Train the toy autoencoder with 5 features in 2 dimensions (variable rarities); observe the phase transitions; then train an SAE (m = 8d) on the activations of a layer of a small LM; annotate the 20 most active features and attempt a steering (amplify a feature during generation).
8. ★★ Indirect injection on a toy RAG (ch. 5-6 — central lab). Build the “master’s documents” RAG; slip in a booby-trapped page (instruction hidden in white-on-white); demonstrate the exfiltration to a simulated “mail tool”; harden by layers (delimiters, injection classifier, permissions, human confirmation); report the ASR before/after for paraphrased attacks (adaptive!).
   • GraphRAG variant: index the same documents as a knowledge graph with entity and community summaries (LLM 7.4, Graph ML ch. 10). Plant the injection in one entity description or in a forged triple, and measure how far it spreads — how many community summaries and answers it reaches — against the chunk index. Which defense layer of ch. 6 still applies when the poisoned text has been rewritten by the summarizer?
9. ★★ MIA on fine-tuning (ch. 8). Fine-tune a small LM on a restricted corpus (members) with a disjoint corpus (non-members); plot the ROC of the MIA by perplexity; repeat with DP-SGD (Opacus) at \varepsilon \in \{3, 8\}; plot utility vs attack advantage — the trade-off curve of Result 8.3.
10. ★★ End-to-end watermark (ch. 9). Implement the green-list scheme (hashing of the previous token, \gamma = 0.5, \delta = 2) on an open LM; verify the calibration of the z-test on unmarked texts (Q-Q plot); measure the power at T \in \{50, 200, 500\} and compare it with Proposition 9.2 (at this \delta, use the exact green probability of its proof, not the first-order one); attack with light then heavy paraphrase; discuss the detectability/quality trade-off (perplexity before/after).
11. ★★★ Red team (ch. 10). Write a red teaming protocol for the application of exercise 8: policy, attack taxonomy, 30 attacks (including 10 adaptive), severity grid; execute, report ASR per family + blast radius; turn the 5 most effective attacks into CI tests.
12. ★★★ KernelSHAP from scratch, and which attribution to trust (ch. 4). (a) On the three-token table of Example 4.4, solve the weighted least-squares problem of Proposition 4.3 over the 8 coalitions (give the two infinite weights a value of 10^6) and check that it returns (11/30, 14/30, 5/30); rerun LIME with \sigma \in \{0.25, 0.5, 1, 2\} and plot the coefficients. (b) On 50 RAG prompts, attribute the answer to the context passages with LIME, sampled KernelSHAP (256 coalitions) and IG: agreement of the rankings, stability over 5 seeds, deletion curves. Which method would you show an auditor, and why?

Projects and assessment

The 2026–2027 SynapseS sheet publishes no description of this module, so its assessment is not displayed (see the fiche above: the S3/S4 tracks of the catalog are still empty). The subjects, the exam and the rubric below are proposals of these notes, not official rules.

The projects exercise mainly O8 (trustworthy and responsible AI: interpretability, attacks, privacy, regulation), with O3 (the LLM internals that probes and attacks rely on) and O5 for the GraphRAG subject. Ex. 11 (the red-team protocol) sets the format of the attack suite in every Year 2 trust work package, and Ex. 8 (the injection lab, with its GraphRAG variant) is reused by all the Year 2 projects.

Solo projects

1. [Solo · ≈ 20 h] A feature atlas of GPT-2 small, checked by intervention. Which sparse-autoencoder features of a middle layer survive a causal test, and how do the dictionary size and the sparsity penalty change the answer?
   • Work: train SAEs (Definition 3.2) with m \in \{8d, 16d, 32d\} and a sweep of \lambda on the residual stream of layers 4 and 8. Measure the reconstruction error, the loss increase when the SAE output replaces the activation, the dead latents and the shrinkage (ch. 3 critical note).
   • Then: label 30 features from their top examples, compare each with a linear probe and its control task (Problem 1.2), and test it by ablation, activation patching (Definition 1.3) and steering (3.2).
   • Data: GPT-2 small (Radford et al., 2019) run over about 10 M tokens of OpenWebText (Gokaslan & Cohen, 2019), activations streamed rather than stored.
   • Success criterion: the reconstruction–sparsity curves for each m; at least 10 labelled features whose ablation or steering moves a measured behaviour more than a random direction of the same norm, over 3 seeds.
   • Deliverables: code, a 4–6 page report with a feature dashboard, a 10-minute talk; chapters 1, 2 and 3.
2. [Solo · ≈ 20 h] Do attributions find the passages a RAG answer rests on? Attention weights, gradient × input, IG with two baselines, LIME and sampled KernelSHAP, compared at passage level.
   • Work: score each method by the rank of the two gold supporting paragraphs among the ten of each question; by deletion and insertion curves (remove the top-attributed passages, re-score the answer); and, for IG, by the completeness gap of Theorem 4.6 against the number of steps. Then test whether a low attribution to the passage the model cites predicts a wrong citation.
   • Data: HotpotQA in its distractor setting (Yang et al., EMNLP 2018), 300 questions, with an open instruct model of 0.5–1.5 B parameters such as Qwen2.5-0.5B-Instruct.
   • Success criterion: precision at 2 on the gold paragraphs per method, with bootstrap intervals; an IG completeness gap below 1 % of the logit gap at the chosen number of steps; a stated verdict on attention weights (2.2).
   • Deliverables: code, report, talk; chapters 2 and 4, Ex. 3 and 12. It extends the RAG evaluation of LLM ch. 7.
3. [Solo · ≈ 25 h] Paper study: Quantifying Memorization Across Neural Language Models (Carlini, Ippolito, Jagielski, Lee, Tramèr & Zhang, ICLR 2023). Do the paper’s log-linear laws — memorization grows with model size, with duplication and with the prompt length (Proposition 7.4 states the first two) — hold on a model family whose training data and order are public?
   • Work: prompt the Pythia models from 70 M to 1.4 B parameters (Biderman et al., ICML 2023) with prefixes of 50 to 500 tokens taken from their training sequences, and count the exact 50-token continuations under greedy decoding. The duplication law needs the number of copies of each sequence; it is approached here by comparing the models trained on the standard and on the deduplicated Pile at equal size.
   • Extension: rank unconditional samples by the perplexity/zlib ratio of Experiment 7.3 and check the top ones against the sampled training data.
   • Data: a few thousand sequences of The Pile (Gao et al., 2020) read from the training order released with Pythia.
   • Success criterion: the size and prompt-length curves with intervals, set against the paper’s figures; the deduplication effect quantified; a one-page critique of the definition (“extractable” is not “harmful”).
   • Deliverables: code, report, talk; chapters 7 and 8.

Group projects

1. [Group of 3 · ≈ 30 h each] Hardening a GraphRAG assistant against indirect prompt injection. How much does each defense layer of ch. 6 lower the adaptive attack success rate, and does a knowledge graph spread an injection further than a chunk index?
   • Work: build the “master’s documents” assistant of Ex. 8 twice, over chunks and over a knowledge graph with entity and community summaries (LLM 7.4, Graph ML ch. 10), with a simulated mail tool. Write an attack suite — direct, indirect, encoded, paraphrased, and adaptive attacks aimed at each defense — then add the layers of the ch. 6 table one by one.
   • Data: the public pages of these notes as the corpus, and the injection tasks of AgentDojo (Debenedetti et al., NeurIPS 2024 Datasets and Benchmarks) as attack templates.
   • Split: the two retrieval pipelines and the graph; the attack suite and the adaptive attacks; the defense layers, the CI suite and the latency budget.
   • Success criterion: per layer, the ASR per attack family with 95 % intervals, the blast radius, the false-positive rate on 200 legitimate questions and the added latency (Proposition 6.2); the spread of one planted injection in each index.
   • Deliverables: code, a CI regression suite, a red-team report in the format of Ex. 11, a 20-minute talk; chapters 0, 5, 6 and 10. It is the trust work package of a Year 2 project such as Y2.1, with the CI gates of MLOps ch. 2.
2. [Group of 3 · ≈ 30 h each] Privacy audit of a private fine-tune: memorization, membership inference and DP. Which mitigation buys the most privacy per point of utility on a small corporate-style fine-tune?
   • Work: fine-tune GPT-2 small or Pythia-160M on synthetic customer e-mails with canaries repeated k \in \{1, 10, 100\} times. Measure the exposure (Metric 7.2), run the extraction of Experiment 7.3, and draw the MIA ROC for a loss attack and a reference-calibrated attack (LLM 9.7).
   • Mitigations: deduplication, PII scrubbing with Microsoft Presidio, and DP-SGD on LoRA adapters with Opacus at \varepsilon \in \{1, 3, 8\} (Result 8.3), compared on one privacy–utility plot.
   • Data: e-mails generated from templates with the Faker library (the generator is released), mixed into WikiText-103 (Merity et al., ICLR 2017) so that personal data is rare.
   • Split: the corpus, the canaries and the exposure; extraction and MIA; mitigations, accountant and utility.
   • Success criterion: held-out perplexity against exposure and against the TPR at 0.1 % FPR for every mitigation, over 3 seeds; the measured MIA advantage set against the DP bound of exam question 4; a one-page recommendation for a data-protection committee.
   • Deliverables: code, joint report, talk; chapters 7, 8 and 9. It uses Privacy and UQ ch. 4 (DP-SGD accounting) and Security & Robustness Def. 8.3 (LiRA).
3. [Group of 2 · ≈ 25 h each] A watermark bench for machine-readable marking. Is the green-list z-test calibrated on human text, and what does each removal attack cost the attacker in quality?
   • Work: implement the green-list scheme (Ex. 10) and the unigram variant with a fixed green list (Zhao et al., ICLR 2024). Check the z-test with a Q-Q plot on human text, measure the power at T \in \{50, 200, 500\} against Proposition 9.2, and attack with token edits, round-trip translation and paraphrase by an open instruct model.
   • Data: prompts and human continuations from the news-like split of C4 (Raffel et al., JMLR 2020), as in Kirchenbauer et al.; OPT-1.3B (Zhang et al., 2022) as the generator.
   • Split: the two schemes and their detectors; the attacks and the quality metrics.
   • Success criterion: the empirical false-positive rate at thresholds 2, 3 and 4 against the Gaussian prediction; a detectability–perplexity curve per attack and scheme; a one-page note on what the scheme can attest under the AI Act marking duty (ch. 10).
   • Deliverables: code, joint report, talk; chapters 9 and 10. It continues LLM Theorem 9.3 and the watermarking chapter of Control in Generative AI.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, a simple calculator allowed; four exercises, 20 points.

1. Question (5 points). A team reports two results: (i) a linear probe reads sentiment from layer 6 of its model with 95 % accuracy, “so the model uses sentiment at layer 6”; (ii) a new input filter lowers the attack success rate from 90 % to 0 % on 50 public jailbreaks, “so the assistant is secure”. Review each claim: what it shows, what it does not show, and which experiment would settle it.
   • Expected: (i) a probe shows that the information is decodable, not that it is used (Problem 1.2). Report the selectivity against a control task with a low-capacity probe, then intervene — amnesic projection or activation patching (Definition 1.3) — and check that the output changes.
   • 2. Known attacks measure only the non-adaptive ASR (Proposition 6.2). Attack the published filter with paraphrase, encoding and optimization, report the worst case, the false-positive rate on legitimate requests and the latency. With 0 successes out of 50, the 95 % upper bound on the true ASR is still about 3/50 = 6 % (rule of three, 10.1).
2. Question (5 points). A green-list watermark uses \gamma = 0.5, and a text of T = 200 tokens contains 130 green tokens. (a) Compute z and decide at the threshold z > 4. (b) On marked text from a low-entropy domain, the green probability is only 0.6: give the expected score at the same length and the approximate power. (c) Which length brings the expected score to 4, and about which length gives a power of 95 %?
   • Expected: (a) z = (130 - 100)/\sqrt{50} = 30/7.07 \approx 4.24 > 4: the text is flagged, with a one-sided p-value of about 10^{-5}.
   • 2. The expected score is \mathbb{E}[z] = \frac{0.1\,T}{\sqrt{0.25\,T}} = 0.2\sqrt{T} \approx 2.83 . Under H_1 the standard deviation of z is \sqrt{0.24/0.25} \approx 0.98, so the power is about \mathbb{P}(Z > 1.19) \approx 0.12.
   • 3. 0.2\sqrt{T} = 4 gives T = 400, where the power is only 0.5. A power of 0.95 needs 0.2\sqrt{T} \approx 4 + 1.645 \times 0.98 \approx 5.61, so T \approx 790 tokens.
3. Question (5 points). (a) Prove the completeness of integrated gradients (Theorem 4.6) for a differentiable F. (b) Let F(t) = 1 - \mathrm{ReLU}(1 - t) on \mathbb{R}, with input x = 2 and baseline x' = 0. Compute the gradient × input attribution and the IG attribution, and explain the difference.
   • Expected: (a) set g(\alpha) = F(x' + \alpha(x - x')). The chain rule gives g'(\alpha) = \sum_i (x_i - x'_i)\,\partial_i F\bigl(x' + \alpha(x - x')\bigr), the integrand of \sum_i \mathrm{IG}_i, and the fundamental theorem of calculus gives g(1) - g(0) = F(x) - F(x').
   • 2. F(t) = \min(t, 1), so F'(2) = 0 and gradient × input gives 0, although F(2) - F(0) = 1. IG gives 2\int_0^1 F'(2\alpha)\,d\alpha = 2 \times \tfrac12 = 1, the whole gap: the gradient saturates at the input, the path integral does not (the kink at t = 1 is a single point of the path).
4. Question (5 points). A membership-inference test decides “member” when the trained model falls in an event S; the training algorithm is (\varepsilon, \delta)-DP. (a) Show the two inequalities below. (b) Deduce the bound on the advantage. (c) Evaluate it for \delta = 0 at \varepsilon = 1 and \varepsilon = 8, and comment in the light of Result 8.3. \mathrm{TPR} \leq e^{\varepsilon}\,\mathrm{FPR} + \delta, \qquad 1 - \mathrm{FPR} \leq e^{\varepsilon}\,(1 - \mathrm{TPR}) + \delta, \qquad \mathrm{Adv} \leq \frac{e^{\varepsilon} - 1 + 2\delta}{e^{\varepsilon} + 1}.
   • Expected: (a) let D' be the training set with the target and D the set without it, a neighbouring pair. The decision is a post-processing of the model, so the TPR is the probability that M(D') falls in S and the FPR the probability that M(D) does. DP applied to this event gives the first inequality; applied to its complement, with the roles of the two sets swapped, the second.
   • 2. Maximize \mathrm{TPR} - \mathrm{FPR} over the polygon cut by the two constraints: the maximum is at the vertex where both are tight, \mathrm{FPR} = (1 - \delta)/(e^{\varepsilon} + 1) and \mathrm{TPR} = e^{\varepsilon}\,\mathrm{FPR} + \delta, which gives the bound.
   • 3. With \delta = 0 the bound is \tanh(\varepsilon/2): 0.46 at \varepsilon = 1 and 0.9993 at \varepsilon = 8, almost vacuous. The guarantee at the budgets of Result 8.3 is a worst case; measured attacks are far weaker, which is why a DP fine-tune is also audited empirically (Ex. 9).

Grading rubric (proposal)

References

• Interpretability. G. Alain & Y. Bengio, Understanding Intermediate Layers Using Linear Classifier Probes, 2016; J. Hewitt & P. Liang, Designing and Interpreting Probes with Control Tasks, EMNLP 2019; Y. Belinkov, Probing Classifiers: Promises, Shortcomings, and Advances, Computational Linguistics 2022; nostalgebraist, “Interpreting GPT: the logit lens”, blog post, 2019; N. Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, 2023; S. Jain & B. Wallace, Attention is not Explanation, NAACL 2019; S. Wiegreffe & Y. Pinter, Attention is not not Explanation, EMNLP-IJCNLP 2019; C. Olsson et al., In-context Learning and Induction Heads, Transformer Circuits 2022; N. F. Liu, M. Gardner, Y. Belinkov, M. Peters, N. A. Smith, Linguistic Knowledge and Transferability of Contextual Representations, NAACL 2019; I. Tenney, D. Das, E. Pavlick, BERT Rediscovers the Classical NLP Pipeline, ACL 2019; Y. Elazar, S. Ravfogel, A. Jacovi, Y. Goldberg, Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals, TACL 2021; K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and Editing Factual Associations in GPT, NeurIPS 2022; K. Wang, A. Variengien, A. Conmy, B. Shlegeris, J. Steinhardt, Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023.
• Superposition / SAE. N. Elhage, T. Hume, C. Olsson et al., Toy Models of Superposition, Transformer Circuits 2022; T. Bricken et al., Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, 2023; T. Templeton et al., Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, 2024.
• Post-hoc explainability. S. Lundberg & S.-I. Lee, A Unified Approach to Interpreting Model Predictions (SHAP), NeurIPS 2017; M. Ribeiro, S. Singh, C. Guestrin, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier (LIME), KDD 2016; M. Sundararajan, A. Taly, Q. Yan, Axiomatic Attribution for Deep Networks (Integrated Gradients), ICML 2017.
• Security. A. Zou, Z. Wang, N. Carlini, et al., Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG), 2023; A. Wei, N. Haghtalab, J. Steinhardt, Jailbroken: How Does LLM Safety Training Fail?, NeurIPS 2023; K. Greshake, S. Abdelnabi, S. Mishra, et al., Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, ACM AISec 2023; F. Perez & I. Ribeiro, Ignore Previous Prompt: Attack Techniques For Language Models, 2022; A. Wan, E. Wallace, S. Shen, D. Klein, Poisoning Language Models During Instruction Tuning, ICML 2023; T. Gu, B. Dolan-Gavitt, S. Garg, BadNets, 2017; F. Tramèr, F. Zhang, A. Juels, et al., Stealing Machine Learning Models via Prediction APIs, USENIX Security 2016.
• Privacy. N. Carlini, C. Liu, Ú. Erlingsson, et al., The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks, USENIX Security 2019; N. Carlini, F. Tramèr, E. Wallace, et al., Extracting Training Data from Large Language Models, USENIX Security 2021; N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, C. Zhang, Quantifying Memorization Across Neural Language Models, ICLR 2023; R. Shokri, M. Stronati, C. Song, V. Shmatikov, Membership Inference Attacks Against Machine Learning Models, IEEE S&P 2017; M. Abadi, A. Chu, I. Goodfellow, et al., Deep Learning with Differential Privacy, CCS 2016; D. Yu, S. Naik, A. Backurs, et al., Differentially Private Fine-tuning of Language Models, ICLR 2022; X. Li, F. Tramèr, P. Liang, T. Hashimoto, Large Language Models Can Be Strong Differentially Private Learners, ICLR 2022; N. Carlini, S. Chien, M. Nasr, et al., Membership Inference Attacks From First Principles (LiRA), IEEE S&P 2022; L. Sweeney, Simple Demographics Often Identify People Uniquely, Carnegie Mellon University, Data Privacy Working Paper 3, 2000; L. Sweeney, k-anonymity: a model for protecting privacy, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10(5), 2002; A. Machanavajjhala, D. Kifer, J. Gehrke, M. Venkitasubramaniam, ℓ-diversity: Privacy beyond k-anonymity, ACM TKDD 2007; C. Dwork & A. Roth, The Algorithmic Foundations of Differential Privacy, Foundations and Trends in TCS 2014 (proved in the Privacy and UQ module ch. 1–4).
• Watermarking. J. Kirchenbauer, J. Geiping, Y. Wen, et al., A Watermark for Large Language Models, ICML 2023; M. Christ, S. Gunn, O. Zamir, Undetectable Watermarks for Language Models, COLT 2024; X. Zhao, P. Ananth, L. Li, Y.-X. Wang, Provable Robust Watermarking for AI-Generated Text, ICLR 2024; R. Kuditipudi, J. Thickstun, T. Hashimoto, P. Liang, Robust Distortion-free Watermarks for Language Models, TMLR 2024.
• Red teaming / compliance. E. Perez, S. Huang, F. Song, et al., Red Teaming Language Models with Language Models, 2022; D. Ganguli, L. Lovitt, J. Kernion, et al., Red Teaming Language Models to Reduce Harms, 2022; Regulation (EU) 2024/1689 — AI Act; NIST AI Risk Management Framework 1.0, 2023; OWASP Top 10 for LLM Applications.
• Projects: data and models. Z. Yang et al., HotpotQA, EMNLP 2018; S. Biderman et al., Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, ICML 2023; L. Gao et al., The Pile, 2020; E. Debenedetti et al., AgentDojo, NeurIPS 2024 Datasets and Benchmarks; S. Merity et al., Pointer Sentinel Mixture Models (WikiText-103), ICLR 2017; C. Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (C4), JMLR 2020; S. Zhang et al., OPT: Open Pre-trained Transformer Language Models, 2022; A. Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2), 2019; A. Gokaslan & V. Cohen, OpenWebText Corpus, 2019.
• Lab tools: Captum (attributions), Opacus (DP-SGD), mechanistic interpretability glossary (TransformerLens for patching).