Fundamentals of Security and Robustness for AI

Contents

• Introduction and overview
• 0. Recap: the foundation and the vocabulary
• 1. Threat model and taxonomy of attacks
• 2. Adversarial examples: the phenomenon and its distances
• 3. Optimization-based attacks: FGSM, PGD, CW, AutoAttack
• 4. Defenses: adversarial training
• 5. Certification: randomized smoothing and formal verification
• 6. Trojans and backdoors
• 7. Data poisoning
• 8. Model stealing and membership inference
• 9. Beyond images: LLMs and graphs
• 10. Evaluating and deploying
• Exercises / Labs
• Projects and assessment
• References

Introduction and overview

Audience: an engineer shipping a model who must anticipate attacks on it. Empirical first: the module starts from concrete failures — a classifier whose clean accuracy of 94% falls to 11% under a PGD attack with budget \varepsilon_{\mathrm{atk}} = 8/255, an ad-hoc defense that collapses under an adaptive attack — and only then names the concepts (threat model, adversarial training, certification).

It covers the defensive foundations of machine-learning security: adversarial examples, poisoning, extraction and membership inference, and standardized evaluation (RobustBench).

What you will be able to do

• Write the threat model of a system: what the adversary knows, can modify, and seeks to obtain;
• Build adversarial examples by optimization (FGSM, PGD, CW) and understand what AutoAttack guarantees;
• Measure the adversarial phenomenon with the right distances and separate the problem from plain classification errors;
• Train a robust model (adversarial training) and quantify its extra cost in compute and accuracy;
• Certify robustness by randomized smoothing or formal verification, and state what the certificate covers;
• Detect and analyze trojans, backdoors and data poisoning;
• Situate intellectual-property attacks: model stealing, membership inference;
• Transpose the approach to LLMs and graphs, and read a robustness benchmark (RobustBench) with a critical eye.

Official objectives and syllabus

Official course description not published (S3/S4 catalog empty): as of the start of the 2026 academic year, the “LLGA 2 – Semestre 3” and “LLGA 2 – Semestre 4” tracks of the 2026-2027 SynapseS catalog still list no courses, and the master’s wiki indicates that the detailed M2 program is “under development” (period 2: “to be completed soon”).

The program below therefore reflects the master’s brochure (period 2), with an indicative volume of ~24 h; these course materials are calibrated to that volume, i.e. about two hours of work per chapter, labs included. The following table maps each brochure theme to the course sections; the materials’ own additions are marked “going beyond the syllabus”.

Where this course fits in the program

An M2 module offered in period 2 and preselected by both routes of the home page: it is the “security” foundation on which the specialized modules at the end of the master open.

Prerequisites:

• Deep Learning or its P2 variant (gradient training, convolutions — the optimization attacks are grafted onto it);
• Graph ML ch. 5 (message passing and the GCN layer of Derivation 5.2), the model attacked in ch. 9;
• Large Language Models 9.4 (jailbreaks, prompt injections), recalled in the primer that opens ch. 9.

Useful if taken: Advanced Graph Neural Networks ch. 7, which develops Nettack and the graph defenses; the federated learning part of Emerging ML, where Byzantine clients prepare the poisoning question and differential privacy is first met (ch. 5). Differential privacy is recalled in one page in ch. 5 (Definition 5.4), with the proofs in Privacy and Uncertainty ch. 1–4.

Co-requisite and downstream modules: Explainability, Security, Privacy of LLMs runs in the same period and is an optional co-requisite, not a prerequisite: it takes up this taxonomy on LLMs (jailbreaks, injections, defenses in depth, ch. 5–6), and neither module requires the other. The threat-model reflexes also serve the operational guardrails of the ML Ops and LLM Engineering Principles module, and the trust work package of the end-of-year project Y2.5.

Course outline

• Recap: the foundation and the vocabulary — attacker, surface, guarantee: the lexicon without which nothing can be discussed.
• Threat model and taxonomy of attacks — classify attacks by knowledge (white/black/gray box) and by goal.
• Adversarial examples: the phenomenon and its distances — the experimental phenomenon, and why the choice of distance changes everything.
• Optimization-based attacks: FGSM, PGD, CW, AutoAttack — the derived and proven attacks, up to standardized evaluation.
• Defenses: adversarial training — the reference defense, its minimax formulation and its real cost.
• Certification: randomized smoothing and formal verification — going from “it resists tests” to “it is proven”, and under what conditions.
• Trojans and backdoors — models that behave well until the attacker’s chosen trigger.
• Data poisoning — corrupting learning upstream, when only the data is touched.
• Model stealing and membership inference — the intellectual-property and confidentiality attacks, a bridge to the LLM privacy module.
• Beyond images: LLMs and graphs — adversarial prompts, structure attacks: the foundation transported to the master’s objects.
• Evaluating and deploying — rigorous protocols (RobustBench) and defensive deployment decisions.
• Exercises / Labs — implement FGSM/PGD, adversarial training and a graph attack (Nettack-style, Exercise 10).
• Projects and assessment — solo and group project subjects, a written-exam proposal and a grading rubric.
• References — the founding papers of each family of attacks and defenses.

Session plan

24 h in class (announced) = 8 sessions of 3 h. The SynapseS 2026–2027 sheet publishes no volume for this module; 24 h is the indicative volume quoted with the brochure programme in the fiche above, and also the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS); the session split below is our proposal for covering these notes in that time. Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods and their main theorems; the level never goes down from one session to the next.

Self-study, outside the 24 h: no chapter — every chapter is covered in class; the labs (Ex. 1–8) each need about 2 h of personal work after their session, the final synthesis (Ex. 9) about 4 h and the graph attack lab (Ex. 10) about 3 h.

How to study this module

• Attack before defending: implement PGD from the very first week — one understands a defense only after failing to break it.
• For each attack, write formally what the adversary optimizes, under which constraints, with what knowledge: this is the central discipline of the module.
• Beware of “miracle defenses”: train yourself to identify why a defense not evaluated against an adaptive attack proves nothing.
• Redo the numbers of the cost of robustness (accuracy, training time) on a small model: the trade-off is better seen than read.
• Reread Graph ML ch. 5 (the GCN layer) and LLM 9.4 at the start of the module, plus Advanced GNN ch. 7 and the Byzantine part of Emerging ML if you took them: the graphs, LLM and poisoning chapters build on them.

0. Recap: the foundation and the vocabulary

One cannot discuss attacks on a model without agreeing on what the model is, what “robust” means and who the adversary is. This chapter recalls the prerequisites from the M1 modules and separates three properties that are often confused.

Notation. This page follows the site-wide notation conventions. The attack budget, the radius of the threat ball, is written \varepsilon_{\mathrm{atk}} (ch. 2–4, 9–10); \sigma is the standard deviation of the smoothing noise \xi (ch. 5); a bare \varepsilon is always a privacy budget of differential privacy (Definition 5.4, ch. 7–8). \delta is the input perturbation in ch. 2–5 and the failure probability of DP in Definition 5.4.

1. Threat model and taxonomy of attacks

A fraud detector resists a one-step attack (FGSM) in the validation report, then falls to an iterated attack (PGD) after deployment. This chapter shows that such surprises come from an implicit threat model, and sets out the explicit one that every later chapter uses.

2. Adversarial examples: the phenomenon and its distances

A network with 99% test accuracy can be made to misclassify almost any image by a change invisible to the eye. This chapter defines these adversarial examples, the distances in which “invisible” is measured, and the simple linear reason why they exist.

3. Optimization-based attacks: FGSM, PGD, CW, AutoAttack

Chapter 2 showed that adversarial examples exist; an attacker, or an evaluator, must actually find them. This chapter writes the search as an optimization problem and derives the standard attacks, from the one-step FGSM to the AutoAttack ensemble.

# FGSM and PGD in about ten lines (PyTorch)
def fgsm(model, x, y, eps):
    x.requires_grad_(True)
    loss = F.cross_entropy(model(x), y)
    loss.backward()
    return (x + eps * x.grad.sign()).clamp(0, 1).detach()

def pgd(model, x, y, eps, alpha, iters, restarts=1):
    worst, worst_loss = None, float("-inf")
    for _ in range(restarts):
        d = torch.zeros_like(x).uniform_(-eps, eps)      # random start
        for _ in range(iters):
            d.requires_grad_(True)
            loss = F.cross_entropy(model((x + d).clamp(0, 1)), y)
            g = torch.autograd.grad(loss, d)[0]          # gradient of the inner max
            with torch.no_grad():
                d = (d + alpha * g.sign()).clamp(-eps, eps)
        x_adv = (x + d).clamp(0, 1).detach()
        l = F.cross_entropy(model(x_adv), y).item()      # candidate loss
        if l > worst_loss:                               # keep the worst case
            worst, worst_loss = x_adv, l
    return worst

4. Defenses: adversarial training

If attacks are optimization problems, the natural defense is to train the model on the worst perturbations it may face. This chapter formulates that defense as a min-max problem, proves what it does on a linear model, and lists the ways it fails.

# full lab: PGD-AT on MNIST (the min-max, line by line)
import torch, torch.nn as nn, torch.nn.functional as F

model = nn.Sequential(                                 # small CNN
    nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(), nn.Linear(64 * 7 * 7, 128), nn.ReLU(), nn.Linear(128, 10))

def pgd(model, x, y, eps, steps, alpha):               # the inner MAX (def. 3.3)
    d = torch.zeros_like(x).uniform_(-eps, eps)        # random start
    for _ in range(steps):
        d.requires_grad_(True)
        loss = F.cross_entropy(model((x + d).clamp(0, 1)), y)
        g = torch.autograd.grad(loss, d)[0].sign()
        with torch.no_grad():
            d = (d + alpha * g).clamp(-eps, eps)
    return (x + d).clamp(0, 1).detach()

opt = torch.optim.Adam(model.parameters(), lr=1e-3)
eps, steps, alpha = 0.3, 7, 0.3 * 2.5 / 7             # cheap 7 steps, rule 2.5·eps/K (Madry et al.:
                                                       # 40 steps of 0.01 on MNIST, 7 of 2/255 on CIFAR-10)
for epoch in range(10):
    for x, y in train_loader:
        x_adv = pgd(model, x, y, eps, steps, alpha)    # approximate the max...
        loss = F.cross_entropy(model(x_adv), y)        # ...then minimize (outer MIN)
        opt.zero_grad(); loss.backward(); opt.step()

def accuracy(model, x, y, attack=None):                # honest evaluation
    xb = x if attack is None else attack(model, x, y)
    return (model(xb).argmax(1) == y).float().mean().item()

# mandatory final report: BOTH numbers (def. 10.1)
print("standard:", accuracy(model, x_test, y_test))
print("robust  :", accuracy(model, x_test, y_test,
      lambda m, x, y: pgd(m, x, y, eps, 50, 0.01)))    # strong PGD in evaluation

5. Certification: randomized smoothing and formal verification

Chapters 3-4 bound robustness from below (attacks) — no guarantee about what has not been found. Certification bounds it from above: a radius within which no perturbation of a given budget can change the prediction.

6. Trojans and backdoors

An attacker who can touch the training data does not need to perturb inputs at test time: they can plant a hidden rule in the model itself. This chapter describes such backdoors and the two main ways of detecting them.

7. Data poisoning

A backdoor is one goal of an attacker who controls part of the data; more generally they may simply want the model to be worse, or biased against a group. This chapter formalizes poisoning, shows which modifications hurt most, and gives the tool that traces a prediction back to the training points.

8. Model stealing and membership inference

A prediction API answers questions, and each answer leaks a little about the model and its training data. This chapter shows how an attacker turns these answers into a copy of the model, or into the knowledge that a given person was in the training set.

9. Beyond images: LLMs and graphs

The attacks of chapters 2–8 were stated for images and tables; the master’s main objects are texts and graphs. This chapter first recalls the LLM attacks of the mandatory LLM course, then shows what carries over to texts and graphs and what changes when the input is discrete; Exercise 10 is its graph lab.

# Lab skeleton (Exercise 10): Nettack-style direct structure attack on a linearized GCN
import numpy as np, scipy.sparse as sp
from sklearn.linear_model import LogisticRegression

def gcn_propagator(A):                        # Â = D̃^-1/2 (A + I) D̃^-1/2 (Graph ML, Derivation 5.2)
    At = A + sp.eye(A.shape[0])
    d = np.asarray(At.sum(1)).ravel() ** -0.5
    return sp.diags(d) @ At @ sp.diags(d)

def fit_surrogate(A, X, y, train):            # logits ² X W + b: W fitted by logistic regression
    P = gcn_propagator(A)
    clf = LogisticRegression(max_iter=2000).fit((P @ (P @ X))[train], y[train])
    return X @ clf.coef_.T, clf.intercept_    # XW (one column per class), b

def margin(A, XW, b, u, y_u):                 # > 0 while u is correctly classified
    P = gcn_propagator(A)
    z = np.asarray(P[u] @ (P @ XW)).ravel() + b   # row u of the logits only
    return z[y_u] - np.delete(z, y_u).max()

def flip(A, u, v):                            # add the edge (u, v) if absent, remove it if present
    A = A.tolil(copy=True)
    A[u, v] = A[v, u] = 1 - A[u, v]
    return A.tocsr()

def direct_attack(A, XW, b, u, y_u, budget, candidates):
    for _ in range(budget):                   # greedy: best marginal gain, then repeat
        scores = {v: margin(flip(A, u, v), XW, b, u, y_u) for v in candidates if v != u}
        v = min(scores, key=scores.get)
        A = flip(A, u, v)
        if scores[v] < 0:                     # the target is misclassified: stop
            break
    return A

10. Evaluating and deploying

A robustness number is only meaningful if the attack, the budget and the protocol behind it are known, and a robust model is only one layer of a secure system. This chapter fixes the evaluation protocol and the deployment checklist.

Exercises / Labs

1. ★ FGSM/PGD on MNIST (ch. 2-3 — opening lab). Train a small CNN; implement FGSM and PGD (thm. 3.2, def. 3.3); plot accuracy vs \varepsilon_{\mathrm{atk}} for K \in \{1, 7, 20, 50\} steps and 1/5/10 restarts; visualize the examples (MNIST grid at \varepsilon_{\mathrm{atk}} \in \{0.05, 0.1, 0.3\}); measure white→black transfer to a second model (ch. 2).

   Proof. Guidance (expected results):

   • clean CNN ≈ 99%;
   • at \varepsilon_{\mathrm{atk}} = 0.3, FGSM drops accuracy below 10% (each pixel moves by a third of the dynamic range — thm. 3.2 in action);
   • K = 1 (one-step PGD, null start) ≈ FGSM;
   • the gain saturates quickly in K (7 steps suffice), restarts add only a few tenths of a point at this budget — the real role of restarts appears on defended models (labs 2 and 4);
   • transfer degrades but does not disappear (30-60% of the attack carries over to the target model): the “black box by secrecy of the weights” defense is futile. ∎

2. ★ Adversarial training (ch. 4 — central lab). PGD-AT (def. 4.1, 7 steps) on MNIST then CIFAR-10 at \varepsilon_{\mathrm{atk}} = 8/255; plot standard and robust accuracy per epoch on the same figure — highlight the robust overfitting (def. 4.3) and the trade-off of thm. 4.2; compare to TRADES (error decomposition); report the RobustBench grid (standard vs robust).

   Proof. Guidance (expected results):

   • MNIST — standard accuracy ~98% (the trade-off of thm. 4.2 is small), robust at \varepsilon_{\mathrm{atk}} = 0.3 raised from ~0% to ≈ 90%;
   • CIFAR-10 — standard ~95% → ~80% with AT, robust at 8/255 from ~0% to 45-50% (more with the augmentations of Rebuffi et al., 2021);
   • robust overfitting is visible on CIFAR: test robustness peaks around the middle of training then decreases by several points while train robustness rises — early stopping and massive augmentation mitigate it (def. 4.3);
   • TRADES gains a few robust points at equal standard by reading the boundary error. ∎

3. ★ Extraction (ch. 8).

   1. Exact extraction of a logistic regression exposed by an API (prop. 8.2): verify the exact reconstruction, then with rounded probabilities (linear programming);
   2. distillation attack (Jagielski): measure accuracy and fidelity as a function of the query budget;
   3. discuss the defenses (rounding, top-1, quotas) and circumvent them.

4. ★★ Standardized evaluation (ch. 3, 10). Take the models from lab 2 and evaluate them with the AutoAttack library (APGD-CE, APGD-T, Square); compare to the “home-grown” PGD numbers; discuss the gap — then break a naive defense (e.g. re-training on FGSM adversaries only) with an adaptive attack (Athalye/Tramèr, ch. 4) and write the evaluation report in RobustBench format.

   Proof. Guidance (expected results):

   • on an adversarially trained model, AutoAttack and the home-grown PGD are close (the model has no masked gradient — a gap on the order of one point);
   • on the naive defense (trained against FGSM only), the “home-grown” PGD overestimates robustness by tens of points and AutoAttack reveals the collapse: FGSM training masks the gradient (piecewise degraded, thm. 3.2 trapped) — the adaptive attack (reparameterization, smoothing) restores it;
   • the honest report: standard accuracy, robust, budget \varepsilon_{\mathrm{atk}}, attack, restarts, cost — without these elements, no conclusion (def. 10.1). ∎

5. ★★ Certification (ch. 5). Implement randomized smoothing: estimation of g(x) and of the bounds \underline{p_A}, \overline{p_B} (Clopper-Pearson, example 5.3) on MNIST at \sigma \in \{0.25, 0.5, 1.0\}; plot certified accuracy vs radius (thm. 5.2) and overlay it with the empirical accuracy (L_2 attacks); discuss the certification/empirical gap and the \sigma accuracy/radius trade-off.

   Proof. Guidance (expected results):

   • at \sigma = 0.25, the smoothed classifier keeps a high clean accuracy but the certified radius quickly vanishes (R_max ≈ 0.3-0.4 on MNIST);
   • at \sigma = 1.0, the certified radius reaches ~1 but the clean accuracy of the smoothed classifier drops — the certified accuracy vs radius curve is below the empirical curve everywhere, the gap measuring our ignorance (imperfect attacks from below, conservative certification from above);
   • with n = 10^4 draws and \alpha = 0.01, Clopper-Pearson costs ~2 points of radius — the draw budget is a hyperparameter of the guarantee (example 5.3). ∎

6. ★★ Backdoor and detection (ch. 6). Insert a BadNets trigger (3×3 patch) on 5% of MNIST; check clean accuracy vs trigger success rate; implement the Neural Cleanse reverse-engineering (def. 6.3: minimize \|\delta_c\|_1 per class, detect the MAD anomaly, reconstruct the trigger); repair by pruning then unlearning and measure the clean/trigger/repair triangle.

   Proof. Guidance (expected results):

   • with 5% of contaminated data, clean accuracy ≈ unchanged (98-99% — the attack is invisible to standard metrics, def. 6.1) and trigger success rate ≈ 99%;
   • Neural Cleanse (def. 6.3) finds \|\delta_t\|_1 abnormally small for the target class only (MAD > 3, the shortcut already exists);
   • pruning the hyperactive neurons under the reconstructed trigger drops the trigger success but eats into clean accuracy from ~10% of removed neurons — hence the targeted unlearning (fine-tuning on reconstructed inputs) which restores the clean behavior without resurrecting the trigger;
   • conclusion: repair is a trade-off, not a restoration. ∎

7. ★★ Poisoning and influence (ch. 7). Label flipping on a tabular MLP: compare random flips vs “large margin” flips (prop. 7.2) at equal budget; compute the influence functions (Hessian-vector products) and show that the highest-influence flips explain the degradation; test the defenses: filtering by influence, truncated loss, regularization — conclude on the federated setting (referral to Emerging ML module ch. 4).

8. ★★ Membership inference (ch. 8). On a tabular MLP: shadow models + confidence attack (Shokri) then simplified LiRA (Carlini); plot ROC and TPR at FPR ∈ {0.1%, 1%, 10%}; retrain with DP-SGD (Opacus) at \varepsilon \in \{1, 8\} and plot utility vs attack advantage against the bound of Definition 5.4 — the trade-off curve of Privacy and Uncertainty ch. 4, revisited from the attacker’s side.

9. ★★★ Final synthesis (ch. 9-10). Write the complete threat analysis (def. 1.1) of a proposed real system (bank scoring, content filter, internal RAG agent): surface, capabilities, goals, applicable attacks chapter by chapter, defenses in depth (ch. 10) with their costs — a 10-minute defense with demonstrations drawn from the previous labs. The bank-scoring version is the threat model of the trust work package of the end-of-year project Y2.5 (fraud alerts on a transaction graph); Exercise 10 rehearses its graph attack on Cora.

10. ★★★ Graph attack lab (ch. 9). On Cora (Planetoid split, loaded with PyTorch Geometric), fit the linearized surrogate of the ch. 9 skeleton and train a two-layer GCN (Graph ML ch. 5). Follow the Nettack protocol: 40 correctly classified test targets (the 10 with the largest margin, the 10 with the smallest, 20 at random), budget \Delta_u = \deg(u) + 2.

    1. Run the direct attack and an influencer attack, which may only flip edges of 5 random neighbors of u (all of them if u has fewer), never touching u — the paper’s influencer setting.
    2. Measure the fraction of targets still correct: on the surrogate, on the clean-trained GCN fed the perturbed graph (evasion), and on a GCN retrained on it (poisoning).
    3. Open part: a defense that drops edges between nodes with dissimilar features (Jaccard similarity of X below a threshold, Wu et al., 2019), then an adaptive attack that only adds edges between similar nodes. Report both numbers, as in def. 10.1.

    Proof. Guidance (expected results):

    • the direct attack flips almost every target within its budget, usually with fewer flips than \Delta_u, and the targets with the smallest clean margin need the fewest flips;
    • most flips add edges from u to nodes of the class that wins — the attack exploits the low-pass averaging of the GCN (Graph ML ch. 5) rather than deleting edges;
    • influencer attacks with the same budget succeed noticeably less often: each flip reaches u only through one more aggregation;
    • the attack transfers: the trained GCN misclassifies most of the targets that fool the surrogate, in evasion and in poisoning;
    • the Jaccard filter removes many of the added edges and restores part of the accuracy, until the adaptive attack restricted to similar nodes is run — it then needs more flips but typically succeeds again: a defense is worth the adaptive attack it survives (ch. 4). ∎

Projects and assessment

The fiche displays no assessment: the course description is not published, and the S3/S4 tracks of the SynapseS 2026–2027 catalog list no course yet. The subjects, exam and rubric below are therefore proposals of these notes, not official rules. Every subject starts with a threat model in the format of Definition 1.1 — the format of the trust work package of every Year 2 end-of-year project — and reports its numbers in the format of Definition 10.1.

The projects exercise mainly O8 (trustworthy AI: attacks, defenses, guarantees), with O4 (attacks on graph models), O7 (hands-on evaluation and deployment decisions) and O3 (attacks on and through language models). The written exam is a mock for self-assessment.

Solo projects

1. [Solo · ≈ 20 h] Paper study: Obfuscated Gradients Give a False Sense of Security (Athalye, Carlini and Wagner, ICML 2018). Does an input-transformation defense survive an attack that knows it?
   • Work: a ResNet-18 trained on CIFAR-10, wrapped in the JPEG-compression, bit-depth-reduction and random crop-and-rescale defenses of Guo et al. (2018); attacked with home-grown PGD (Definition 3.3), then AutoAttack, then BPDA — the defense replaced by the identity on the backward pass — with EOT for the random crop.
   • Data: CIFAR-10 (Krizhevsky, 2009), 1,000 test images, \varepsilon_{\mathrm{atk}} = 8/255 in L_\infty.
   • Success criterion: under BPDA the robust accuracy falls to within a few points of the undefended model’s, as in the paper; the gaps between the three attacks explained by gradient masking (ch. 4).
   • Deliverables: code, a 4–6 page report in RobustBench format, a 10-minute talk; chapters 1, 3, 4 and 10, Exercise 4.
2. [Solo · ≈ 20 h] Paper study: Certified Adversarial Robustness via Randomized Smoothing (Cohen, Rosenfeld and Kolter, ICML 2019). How far below the attacks does the certificate sit, and what does the sampling budget cost?
   • Work: the certification procedure of Example 5.3 with Clopper–Pearson bounds, on the noise-trained models released with the paper at \sigma \in \{0.25, 0.5, 1.0\}; certified accuracy against radius for n \in \{10^3, 10^4, 10^5\}; an L_2 PGD attack on the smoothed classifier with Monte Carlo gradients (Salman et al., 2019) for the empirical curve.
   • Data: CIFAR-10, 200 test images, on a free GPU tier.
   • Success criterion: certified curves within 5 points of the paper’s at the same \sigma; the radius cap of about 3.8\,\sigma at n = 10^5 and \alpha = 0.001 observed; the certified curve below the empirical one at every radius.
   • Deliverables: code, report, talk; chapters 2, 5 and 10, Exercise 5.
3. [Solo · ≈ 25 h] Adversarial suffixes on a small open LLM. Does the PGD logic carry over to tokens, and does a perplexity filter stop it?
   • Work: GCG (Zou et al., 2023; the ch. 9 recall) on Qwen2.5-0.5B-Instruct with 20 suffix tokens and benign targets only — making the model begin its answer with a fixed canary phrase it never produces otherwise; batches of 128 candidates and at most 200 steps per run, stopping at the first success, keep the 150 runs within a few hours of a free GPU; then the perplexity filter of Jain et al. (2023, “Baseline Defenses for Adversarial Attacks Against Aligned Language Models”) and an adaptive attack that adds a fluency penalty.
   • Data: 50 instructions from databricks-dolly-15k (Conover et al., 2023).
   • Success criterion: attack success rate against iterations over 3 seeds; the filter’s false-alarm rate on clean prompts; the adaptive attack’s success with the filter on.
   • Deliverables: code, report, talk; chapters 1, 3, 9 and 10. It applies LLM 9.4 and prepares Explainability, Security, Privacy of LLMs ch. 5.

Group projects

1. [Group of 3 · ≈ 30 h each] Robustness audit of a GNN fraud detector. Which detector survives a budgeted structure attack on a transaction graph, and what can be certified?
   • Work: GCN and GAT (Graph ML ch. 5) against a gradient-boosting baseline on the node features, with a strictly temporal split; direct and influencer attacks (the ch. 9 skeleton, or DeepRobust) on illicit targets from a few attacker-controlled transactions (nodes); edge pruning by feature similarity (cosine similarity, because the Elliptic features are continuous; Exercise 10 uses Jaccard on Cora’s binary features) and randomized smoothing on the features; a per-tree leaf-enumeration bound for the boosting fallback (ch. 9 note).
   • Data: the Elliptic Data Set (Weber et al., KDD 2019 Workshop on Anomaly Detection in Finance; on Kaggle): 203,769 Bitcoin transactions over 49 time steps, about 2 % labelled illicit, attacked one time step at a time.
   • Split: models and temporal protocol; attacks; defenses, certification and the deployment plan (ch. 10).
   • Success criterion: recall at a fixed alert budget against the attack budget, over 5 seeds with intervals; a threat model and a fallback decision that a reviewer can check.
   • Deliverables: the threat model, code, an 8–10 page joint report, a 20-minute talk; chapters 1, 5, 9 and 10, Exercises 9–10. It is a first version of the trust work package of Y2.5 and extends Advanced GNN ch. 7.
2. [Group of 3 · ≈ 30 h each] Backdoors in a fine-tuning supply chain, images and text. How cheap is a backdoor in a fine-tuning pipeline, and which detector catches which?
   • Work: the BadNets patch (Gu et al., 2017) and the clean-label feature collision of Poison Frogs (Shafahi et al., 2018) when fine-tuning an ImageNet-pretrained ResNet-18; instruction poisoning of a small LLM fine-tuned with LoRA (Wan et al., ICML 2023, “Poisoning Language Models During Instruction Tuning”); detection by Neural Cleanse (Definition 6.3), spectral signatures and influence scores (Definition 7.3); repair by pruning or unlearning.
   • Data: CIFAR-10; databricks-dolly-15k with 1 % of the examples carrying a trigger phrase, on Qwen2.5-0.5B-Instruct.
   • Split: image attacks; text attack; detection and repair across both.
   • Success criterion: the clean / trigger / repair triangle per modality over 3 seeds; each detector’s detection rate at a fixed false-alarm rate.
   • Deliverables: code, joint report, talk; chapters 6 and 7, Exercises 6–7. It connects to Explainability, Security, Privacy of LLMs ch. 5 (LLM backdoors) and MLOps ch. 9 (supply chain).
3. [Group of 4 · ≈ 25 h each] Privacy red team of a prediction API. What does a score API leak, and what does each defense cost?
   • Work: an MLP served behind a small API; extraction — exact recovery of a logistic model (Proposition 8.2), then a distillation attack scored by accuracy and fidelity against the query budget (Jagielski et al., 2020); membership inference — shadow models, then LiRA with 32 shadow models (Carlini et al., 2022); defenses — rounding, top-1 labels, query quotas, DP-SGD with Opacus at \varepsilon \in \{1, 8\}.
   • Data: Purchase-100 (Shokri et al., 2017, built from Kaggle’s Acquire Valued Shoppers data): 600 binary features, 100 classes.
   • Split: API and defenses; extraction; membership inference; utility and cost accounting.
   • Success criterion: TPR at 0.1 % FPR for each defense, with intervals, set against the bound of Definition 5.4; fidelity against queries; the utility each defense costs.
   • Deliverables: a threat analysis with costs in the form of Exercise 9, code, joint report, talk; chapters 1, 5 and 8, Exercises 3 and 8. It connects to Privacy and Uncertainty ch. 4 and Explainability, Security, Privacy of LLMs ch. 8.

Written exam (proposal)

Two hours, one handwritten A4 sheet and a simple calculator allowed; four questions, 20 points. Given: \Phi^{-1}(0.95) = 1.645, \Phi^{-1}(0.98) = 2.054, \Phi^{-1}(0.9931) = 2.46, \ln(0.001) = -6.91.

1. Question (5 points). A team reports that its defended CIFAR-10 classifier keeps 70 % accuracy under FGSM at \varepsilon_{\mathrm{atk}} = 8/255. (a) Write the threat model that this number assumes. (b) Give two reasons why it does not show robustness, and the evaluation you would require. (c) List what the final report must contain.

   • Expected: (a) Definition 1.1 — white-box attacker, inference time (evasion), untargeted error, L_\infty ball of radius 8/255, and implicitly a one-step attacker.
   • (b) FGSM maximizes a linearization (Theorem 3.2): it only gives a lower bound on the inner max. A defense can also mask the gradient (Athalye et al., 2018), so the first-order attack fails while the model is not robust. Require PGD with restarts, AutoAttack and an adaptive attack that knows the defense (BPDA, EOT), plus sanity checks: accuracy falls to 0 as the budget grows, and black-box transfer is not stronger than white-box.
   • (c) Definition 10.1: standard and robust accuracy, norm and budget, attacks, restarts, training cost.

2. Question (5 points). A linear classifier predicts the sign of f(x) = w^\top x + b, with w = (3, -1, 2) and b = -1. Take the point x = (0.5, 0.2, 0.1), of label y = +1.

   (a) Compute f(x). (b) Give the FGSM perturbation of the logistic loss at \varepsilon_{\mathrm{atk}} = 0.1 in L_\infty, and the new score. (c) Give the smallest L_\infty and L_2 budgets that change the prediction. (d) Give the clean and the robust hinge losses at the same budget.

   • Expected: (a) f(x) = 1.5 - 0.2 + 0.2 - 1 = 0.5, so the prediction is correct.
   • (b) The gradient of \log(1 + e^{-y f(x)}) in x is -y\,\sigma(-y f(x))\, w, of sign -\mathrm{sign}(w), hence \delta^\star = (-0.1, 0.1, -0.1) and a new score 0.5 - 0.1 \times 6 = -0.1: the prediction flips.
   • (c) Proposition 2.3: the score moves by at most \varepsilon_{\mathrm{atk}}\|w\|_1, so the L_\infty threshold is 0.5 / 6 \approx 0.083; in L_2 the closest boundary point is at distance 0.5 / \|w\|_2 = 0.5/\sqrt{14} \approx 0.134.
   • (d) Clean hinge 1 - 0.5 = 0.5; the point is inside the original margin, so by Theorem 4.2 the robust hinge is 0.5 + 0.6 = 1.1.

3. Question (5 points). Randomized smoothing with \sigma = 0.25.

   (a) Monte Carlo gives \underline{p_A} = 0.95, and one takes \overline{p_B} = 1 - \underline{p_A}: give the certified radius. (b) Same with a separate bound \overline{p_B} = 0.02.

   (c) With n = 1000 noisy copies, all classified c_A, and \alpha = 0.001, what is the largest radius one can certify? (d) Why is a larger \sigma not free?

   • Expected: (a) by Example 5.3, R = \sigma\,\Phi^{-1}(0.95), that is 0.25 \times 1.645 \approx 0.41.
   • (b) R = \tfrac{0.25}{2}(1.645 + 2.054) \approx 0.46: a tighter bound on the runner-up buys radius.
   • (c) The Clopper–Pearson lower bound is \alpha^{1/n} = e^{-6.91/1000} \approx 0.9931, so R \leq 0.25 \times 2.46 \approx 0.62, that is 2.46\,\sigma; certifying 3.8\,\sigma needs about 10^5 copies.
   • (d) R grows linearly in \sigma, but the base classifier sees noisier inputs: p_A falls, the smoothed classifier abstains or errs more often, and the certified accuracy at small radii drops.

4. Question (5 points). A model is trained by an (\varepsilon, \delta)-DP algorithm M.

   (a) Prove that every membership test on a fixed record z satisfies \mathrm{TPR} \leq e^{\varepsilon}\,\mathrm{FPR} + \delta . (b) Evaluate the bound at an FPR of 0.1 % for \varepsilon = 1 and for \varepsilon = 8, with \delta = 10^{-5}, and comment.

   • Expected: (a) Let D' be the training set without z and D = D' \cup \{z\}: they are neighbors. The test is a function T of the trained model; let S be the set of models on which T answers “member”. Then \mathrm{TPR} = \mathbb{P}(M(D) \in S), \qquad \mathrm{FPR} = \mathbb{P}(M(D') \in S), and Definition 5.4 applied to S gives the inequality — the post-processing argument.
   • (b) At \varepsilon = 1 the bound is 2.718 \times 0.001 + 10^{-5}, about 0.27\,\%: a strong guarantee. At \varepsilon = 8: e^8 \approx 2981, so the bound is about 2.98 > 1 and says nothing at this FPR — the low-FPR regime of LiRA (Definition 8.3) is exactly where large budgets give no protection.

Grading rubric (proposal)

References

• Foundations and the phenomenon. C. Szegedy, W. Zaremba, I. Sutskever, et al., Intriguing Properties of Neural Networks, ICLR 2014; I. Goodfellow, J. Shlens, C. Szegedy, Explaining and Harnessing Adversarial Examples (FGSM), ICLR 2015; A. Kurakin, I. Goodfellow, S. Bengio, Adversarial Examples in the Physical World, ICLR 2017 (workshop); A. Ilyas, S. Santurkar, D. Tsipras, et al., Adversarial Examples Are Not Bugs, They Are Features, NeurIPS 2019; B. Biggio & F. Roli, Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning, Pattern Recognition 2018.
• Attacks and evaluation. N. Carlini & D. Wagner, Towards Evaluating the Robustness of Neural Networks (CW), IEEE S&P 2017; F. Croce & M. Hein, Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-Free Attacks (AutoAttack), ICML 2020; F. Croce, M. Andriushchenko, V. Sehwag, et al., RobustBench: A Standardized Adversarial Robustness Benchmark, NeurIPS 2021 (Datasets and Benchmarks); F. Tramèr, N. Carlini, W. Brendel, A. Kurakin, On Adaptive Attacks to Adversarial Example Defenses, NeurIPS 2020.
• Defenses. A. Madry, A. Makelov, L. Schmidt, et al., Towards Deep Learning Models Resistant to Adversarial Attacks (PGD-AT), ICLR 2018; H. Zhang, Y. Yu, J. Yi, et al., Theoretically Principled Trade-off between Robustness and Accuracy (TRADES), ICML 2019; L. Rice, E. Wong, J. Zico Kolter, Overfitting in Adversarially Robust Deep Learning, ICML 2020; L. Schmidt, S. Santurkar, D. Tsipras, et al., Adversarially Robust Generalization Requires More Data, NeurIPS 2019; A. Athalye, N. Carlini, D. Wagner, Obfuscated Gradients Give a False Sense of Security, ICML 2018; C. Guo, M. Rana, M. Cissé, L. van der Maaten, Countering Adversarial Images Using Input Transformations, ICLR 2018; S.-A. Rebuffi, S. Gowal, D. A. Calian, et al., Fixing Data Augmentation to Improve Adversarial Robustness, NeurIPS 2021.
• Certification. J. Cohen, E. Rosenfeld, J. Zico Kolter, Certified Adversarial Robustness via Randomized Smoothing, ICML 2019; M. Lecuyer, V. Atlidakis, R. Geambasu, et al., Certified Robustness to Adversarial Examples with Differential Privacy (PixelDP), IEEE S&P 2019; G. Katz, C. Barrett, D. Dill, et al., Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks, CAV 2017; A. Kantchelian, J. D. Tygar, A. D. Joseph, Evasion and Hardening of Tree Ensemble Classifiers, ICML 2016; H. Salman, G. Yang, J. Li, et al., Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers, NeurIPS 2019.
• Backdoors and poisoning. T. Gu, B. Dolan-Gavitt, S. Garg, BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, 2017; B. Wang, Y. Yao, S. Shan, et al., Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks, IEEE S&P 2019; B. Tran, J. Li, A. Madry, Spectral Signatures in Backdoor Attacks, NeurIPS 2018; A. Shafahi, W. Ronny Huang, M. Najibi, et al., Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks, NeurIPS 2018; B. Biggio, B. Nelson, P. Laskov, Poisoning Attacks against Support Vector Machines, ICML 2012; P. Koh & P. Liang, Understanding Black-box Predictions via Influence Functions, ICML 2017; A. Wan, E. Wallace, S. Shen, D. Klein, Poisoning Language Models During Instruction Tuning, ICML 2023.
• Extraction and privacy. F. Tramèr, F. Zhang, A. Juels, et al., Stealing Machine Learning Models via Prediction APIs, USENIX Security 2016; M. Jagielski, N. Carlini, D. Berthelot, et al., High Accuracy and High Fidelity Extraction of Neural Networks, USENIX Security 2020; R. Shokri, M. Stronati, C. Song, V. Shmatikov, Membership Inference Attacks Against Machine Learning Models, IEEE S&P 2017; N. Carlini, S. Chien, M. Nasr, et al., Membership Inference Attacks From First Principles (LiRA), IEEE S&P 2022; C. Dwork & A. Roth, The Algorithmic Foundations of Differential Privacy, FnT-TCS 2014 (recap from the Emerging ML module ch. 5).
• LLMs and graphs. A. Zou, Z. Wang, N. Carlini, et al., Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG), 2023; N. Jain, A. Schwarzschild, Y. Wen, et al., Baseline Defenses for Adversarial Attacks Against Aligned Language Models, 2023; D. Zügner, A. Akbarnejad, S. Günnemann, Adversarial Attacks on Neural Networks for Graph Data (Nettack), KDD 2018; H. Wu, C. Wang, Y. Tyshetskiy, et al., Adversarial Examples for Graph Data: Deep Insights into Attack and Defense (Jaccard defense, Exercise 10), IJCAI 2019; M. Weber, G. Domeniconi, J. Chen, et al., Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Forensics (Elliptic Data Set), KDD 2019 Workshop on Anomaly Detection in Finance; for the extension: M2 module Explainability, Security and Privacy of LLMs (jailbreaks, injections, watermarking) and M2 module Advanced GNN ch. 7 (robustness and privacy on graphs).
• Framework and tools. Regulation (EU) 2024/1689 — AI Act; NIST AI Risk Management Framework 1.0, 2023; AutoAttack (code); RobustBench (leaderboard); randomized smoothing (code); IBM Adversarial Robustness Toolbox; Opacus (DP-SGD).