Advanced Learning for Text and Graph Data (ALTEGRAD)

Contents

• Introduction and overview
• 0. Review: probability, linear algebra, optimization and graphs
• 1. Text as data: bags of words, n-grams and subwords
• 2. Word embeddings: Word2Vec, GloVe and the PMI matrix
• 3. Graph-of-words and graph degeneracy for text mining
• 4. Recurrent sequence models and attention
• 5. Deep discriminative architectures for classification
• 6. The Transformer engine
• 7. Self-supervised pretraining and transfer
• 8. Autoregressive pretraining at scale
• 9. Evaluation, compression and multilingual benchmarks
• 10. Downstream adaptation and parameter efficiency
• 11. Alignment and post-training
• 12. Retrieval-augmented and graph-augmented generation
• 13. Quantization and efficient inference
• 14. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with an M1 in mathematics, computer science or physics who has already trained a neural network, and who wants to derive, implement and judge the models behind modern text processing, from word vectors to aligned, retrieval-augmented and quantized large language models.

Empirical first: on a synthetic review corpus in which a negation flips the polarity, a bag-of-words logistic regression stays at chance (51 % test accuracy) and a bigram model reaches 100 %. As soon as an intensifier separates the negation from the adjective (“not very good”), the bigram model falls back to chance (48 %), while a one-layer convolution over word vectors scores 100 % (code of ch. 1 and ch. 5).

Meaning is not additive in words: “not” changes what “good” means. Linear models on counts cannot represent such interactions, n-gram features capture them only between adjacent words, and learned representations of context — the subject of most of this course — capture them in general. The chapters follow the path of natural language processing over the last fifteen years: sparse counts and their graph counterpart, the graph-of-words; learned dense vectors; recurrent and convolutional encoders with attention; the Transformer and self-supervised pretraining; and finally the engineering of large language models, from scaling laws to alignment, retrieval, graph augmentation and quantization.

The Transformer and pretraining chapters share tools with Foundations of Large Language Models and Deep Learning. The graph chapters lead to Graphs in Machine Learning and Kernel Methods for Machine Learning, and the last chapter prepares Training and Deploying Large-Scale Models.

What you will be able to do

• Build bag-of-words, TF-IDF and n-gram representations, estimate smoothed language models and compare them by perplexity.
• Train subword tokenizers (BPE, WordPiece, unigram) and word embeddings, and prove that skip-gram with negative sampling factorizes a shifted PMI matrix.
• Represent a document as a graph-of-words, extract keywords by k-core decomposition and compare documents with a shortest-path graph kernel.
• Derive back-propagation through time, explain vanishing gradients, and implement LSTM and GRU encoders with additive and multiplicative attention.
• Design convolutional, bidirectional and hierarchical attention classifiers for sentences and documents.
• Implement scaled dot-product and multi-head attention with sinusoidal, rotary or ALiBi positions, and prove their symmetry properties.
• Choose a pretraining objective (masked, denoising or autoregressive) and size a model and its data with compute-optimal scaling laws.
• Evaluate generated text with n-gram and learned metrics, distil an expensive metric, and design a multilingual benchmark with honest error bars.
• Adapt a pretrained model by instruction tuning, LoRA or prefix tuning, and align it by RLHF or DPO.
• Build a retrieval-augmented and graph-augmented generation pipeline, and quantize a model for inference.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com presents ALTEGRAD as a 28-hour overview of state-of-the-art machine learning and AI methods for text and graph data, with a significant focus on applications. Each session combines two hours of lecture with two hours of programming. Grading is based on a final data challenge plus a lab-based evaluation.

The same page states that registration through a form is necessary and points to a course page on the Moodle of IP Paris. The 2026–2027 syllabus has four parts — introduction to NLP and the attention mechanism, deep learning for NLP, LLMs, and advanced LLM topics — and fifteen items. The Université Paris-Saclay course detail adds the volume (14 h of lectures, 14 h of labs) and the nature of the assessment. The table maps every official item to a section of this page.

Beyond the syllabus, this page adds a review of the M1 tools (ch. 0), complete proofs of the main structural results (shifted PMI, k-core peeling, rotary positions, compute-optimal allocation, DPO), and a chapter of research directions (ch. 14, self-study).

Where this course fits in the programme

Official prerequisites: not displayed on the official pages. The page assumes the M1 level of the MVA admission: probability, linear algebra, a first course in machine learning, and fluent Python.

• Refreshers — Probability and Statistics (maximum likelihood, entropy, conditional distributions), Linear Algebra, Analysis and Optimization (singular value decomposition, gradients, Lagrange multipliers) and Scientific Python, NumPy and PyTorch (every code block of this page).
• Related courses of the same semester — Foundations of Large Language Models, Deep Learning, Theoretical Foundations of Deep Learning, Introduction to Statistical Learning, Introduction to Probabilistic Graphical Models and Deep Generative Models, Foundations of Distributed and Large-Scale Computing Optimization and Turing Seminar: An Introduction to AGI Safety (alignment).
• Courses that build on it in semester 2 — Training and Deploying Large-Scale Models, Large Language Models for Code and Proof, Algorithms for Speech and Natural Language Processing, Graphs in Machine Learning, Kernel Methods for Machine Learning (graph kernels), Reinforcement Learning (policy gradients behind PPO), Multimodal Explainable AI, Responsible Machine Learning and Regulating AI.

Course outline

• Review: probability, linear algebra, optimization and graphs — cross-entropy and softmax, the Eckart–Young theorem, Adam, breadth-first search.
• Text as data — bags of words, TF-IDF, the limits of linear models on counts, n-gram language models and perplexity, subword tokenization.
• Word embeddings — skip-gram and CBOW, negative sampling as a shifted PMI factorization, GloVe, analogies, subword vectors.
• Graph-of-words and graph degeneracy — sliding-window graphs, k-cores, keyword extraction, TW-IDF, the shortest-path and core-based kernels for document similarity.
• Recurrent sequence models and attention — RNNs, vanishing gradients, LSTM, GRU, encoder–decoder models, Bahdanau and Luong attention.
• Deep discriminative architectures — 1D convolutions with max-over-time pooling, bidirectional encoders, hierarchical attention networks.
• The Transformer engine — scaled dot-product and multi-head attention, the block, causal masking, sinusoidal, rotary and ALiBi positions.
• Self-supervised pretraining and transfer — masked language modelling, denoising sequence-to-sequence pretraining, causal language modelling, BARThez and French models.
• Autoregressive pretraining at scale — training dynamics, compute accounting, Kaplan and Chinchilla scaling laws, tokenization at scale.
• Evaluation, compression and multilingual benchmarks — BLEU, ROUGE, BERTScore, knowledge distillation, FrugalScore, native-sourced benchmarks.
• Downstream adaptation and parameter efficiency — instruction tuning, adapters, LoRA, prefix tuning.
• Alignment and post-training — Bradley–Terry reward models, the KL-regularized objective, PPO, DPO, AI feedback.
• Retrieval-augmented and graph-augmented generation — BM25, dense passage retrieval, RAG, hallucination, knowledge graphs, GraphRAG, text-to-SQL.
• Quantization and efficient inference — uniform quantization, LLM.int8() and NF4, GPTQ, AWQ, the KV cache, pruning.
• Research directions and open problems — long context, evaluation, alignment, multilingual models, graph and language models (self-study).

Session plan

28 h in class = 7 sessions of 4 h. The volume is that of the Université Paris-Saclay programme table (lectures 14 h, labs 14 h), and master-mva.com organises each session as two hours of lecture followed by two hours of programming; the plan keeps that rhythm. The assignment of topics to sessions is a proposal.

Level 1 = foundations and first examples, 2 = core models and algorithms, 3 = advanced material close to current research; the level never decreases.

Self-study, outside the 28 h (about 3 h in total, plus the data challenge): ch. 14 Research directions and open problems (≈ 3 h, with Ex. 16).

How to study this module

• Read ch. 0 before session 1: the softmax gradient (Proposition 0.2) and the Eckart–Young theorem (Theorem 0.3) return in ch. 2, 7, 9, 10 and 12.
• Each session has the official rhythm: two hours of lecture, then two hours of programming on the codes listed in the Practice column. Do the listed exercises before the next session.
• Keep one notebook per lab; the official grade includes a lab-based evaluation, so record observations and numbers as you go.
• Start the data challenge early (see Projects and assessment): the group subjects reuse the labs of sessions 2, 5 and 7.
• Python environment: NumPy, SciPy, scikit-learn, NetworkX and PyTorch (CPU). Every block runs in a few seconds on a laptop, with synthetic data and fixed seeds; no pretrained model is downloaded.

0. Review: probability, linear algebra, optimization and graphs

Four M1 tools carry the course: the cross-entropy of a softmax model, the best low-rank approximation of a matrix, stochastic first-order optimization, and breadth-first search on a graph. This chapter states them in the form used later.

1. Text as data: bags of words, n-grams and subwords

A learning algorithm needs vectors, and a text is a sequence of symbols. This chapter builds the first representations — counts of words and of short word sequences — measures what they lose, and explains how modern systems cut text into subword units.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
fillers = ["the", "movie", "plot", "was", "really", "and", "acting", "honestly"]

def review(intensifier):
    pol, neg = rng.integers(2), rng.integers(2)     # polarity, negation
    core = (["not"] if neg else []) + (["very"] if intensifier else [])
    core += ["good" if pol else "bad"]
    words = list(rng.choice(fillers, 4)) + core + list(rng.choice(fillers, 3))
    return " ".join(words), int(pol != neg)          # negation flips the label

def data(n, intensifier):
    pairs = [review(intensifier) for _ in range(n)]
    return [p[0] for p in pairs], np.array([p[1] for p in pairs])

for ngram in [(1, 1), (1, 2)]:
    for inten in [False, True]:
        Xtr, ytr = data(2000, inten); Xte, yte = data(1000, inten)
        vec = CountVectorizer(ngram_range=ngram, token_pattern=r"\S+")
        clf = LogisticRegression(max_iter=2000).fit(vec.fit_transform(Xtr), ytr)
        print(ngram, "not very good" if inten else "not good",
              "test accuracy %.3f" % clf.score(vec.transform(Xte), yte))

import numpy as np
from collections import Counter

train = ["<s> the cat sat </s>", "<s> the dog sat </s>", "<s> the cat ran </s>"]
pairs, hist = Counter(), Counter()
for s in train:
    w = s.split()
    hist.update(w[:-1])
    pairs.update(zip(w[:-1], w[1:]))
V = len({x for s in train for x in s.split()} - {"<s>"})   # 6 possible next tokens

def p_add_k(prev, nxt, k=1.0):
    return (pairs[prev, nxt] + k) / (hist[prev] + k * V)

test = "<s> the dog ran </s>".split()
logp = sum(np.log(p_add_k(a, b)) for a, b in zip(test[:-1], test[1:]))
print("add-one perplexity: %.3f" % np.exp(-logp / (len(test) - 1)))   # 3.969

Word-level vocabularies have two defects: they are huge, and they still meet unknown words. Subword tokenizers fix both by learning a vocabulary of frequent character sequences.

from collections import Counter

def learn_bpe(word_freqs, n_merges):
    """Byte-pair encoding (Sennrich et al., 2016) on a {word: count} dictionary."""
    corpus = {tuple(w): c for w, c in word_freqs.items()}   # words as symbol tuples
    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for symbols, c in corpus.items():
            for a, b in zip(symbols, symbols[1:]):
                pairs[a, b] += c
        (a, b), count = pairs.most_common(1)[0]
        merges.append((a + b, count))
        new = {}
        for symbols, c in corpus.items():
            out, i = [], 0
            while i < len(symbols):
                if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == (a, b):
                    out.append(a + b); i += 2
                else:
                    out.append(symbols[i]); i += 1
            new[tuple(out)] = c
        corpus = new
    return merges, corpus

merges, corpus = learn_bpe({"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}, 3)
print(merges)          # [('ug', 20), ('un', 16), ('hug', 15)]
print(list(corpus))    # hug | p ug | p un | b un | hug s

2. Word embeddings: Word2Vec, GloVe and the PMI matrix

A bag of words treats “cat” and “dog” as orthogonal coordinates. Word embeddings place words in a space of a few hundred dimensions in which words used in similar contexts are close — the distributional hypothesis of Harris (1954) made geometric.

import numpy as np

rng = np.random.default_rng(0)
topics = {"animals": ["cat", "dog", "horse", "cow", "fur", "farm", "pet", "eats"],
          "computers": ["cpu", "gpu", "ram", "disk", "chip", "fast", "memory", "data"]}
shared = ["the", "a", "is", "very"]
vocab = sorted({w for ws in topics.values() for w in ws} | set(shared))
idx = {w: i for i, w in enumerate(vocab)}

sents = []
for _ in range(3000):
    t = topics["animals" if rng.random() < 0.5 else "computers"]
    words = list(rng.choice(t, 4)) + list(rng.choice(shared, 2))
    rng.shuffle(words)
    sents.append(words)

C = np.zeros((len(vocab), len(vocab)))           # window of 2 on each side
for s in sents:
    for i, w in enumerate(s):
        for j in range(max(0, i - 2), min(len(s), i + 3)):
            if j != i:
                C[idx[w], idx[s[j]]] += 1

total = C.sum()
pmi = np.log((C * total) / (C.sum(1, keepdims=True) * C.sum(0, keepdims=True)) + 1e-12)
U, S, _ = np.linalg.svd(np.maximum(pmi, 0))
W = U[:, :2] * np.sqrt(S[:2])                     # 2-dimensional word vectors

def cos(a, b):
    u, v = W[idx[a]], W[idx[b]]
    return u @ v / (np.linalg.norm(u) * np.linalg.norm(v))

print("cos(cat, dog) = %.2f" % cos("cat", "dog"))   # 1.00
print("cos(cat, gpu) = %.2f" % cos("cat", "gpu"))   # -0.01

3. Graph-of-words and graph degeneracy for text mining

A bag of words assumes that terms are independent. The graph-of-words keeps which terms occur near each other, and graph degeneracy turns that structure into keyword scores, retrieval weights and document similarities. This line of work was developed largely in the DaSciM group by Rousseau, Tixier, Nikolentzos, Vazirgiannis and coauthors.

import re
import networkx as nx

text = ("Graph degeneracy offers a simple view of a document. The graph of words links "
        "two terms when they co-occur in a sliding window. The k-core decomposition of "
        "the graph of words keeps the dense and cohesive part of the graph, and the "
        "terms of the main core are good keyword candidates for the document.")
stop = {"a", "of", "the", "two", "when", "they", "in", "and", "are", "for", "offers", "keeps"}
terms = [t for t in re.findall(r"[a-z]+(?:-[a-z]+)?", text.lower()) if t not in stop]

def graph_of_words(terms, window=3):
    G = nx.Graph()
    for i, t in enumerate(terms):
        for u in terms[i + 1:i + window]:             # the next window-1 terms
            if u != t:
                w = G.get_edge_data(t, u, {"weight": 0})["weight"]
                G.add_edge(t, u, weight=w + 1)
    return G

G = graph_of_words(terms)
core = nx.core_number(G)
k_max = max(core.values())
print("terms:", G.number_of_nodes(), "edges:", G.number_of_edges(), "degeneracy:", k_max)
print("main core:", sorted(t for t in core if core[t] == k_max))
corerank = {t: sum(core[u] for u in G[t]) for t in G}
print("top-5 CoreRank:", sorted(corerank, key=corerank.get, reverse=True)[:5])

Graphs-of-words also feed neural models: MPAD (Nikolentzos, Tixier and Vazirgiannis, 2020) runs message passing, defined in ch. 12, on them for document classification.

4. Recurrent sequence models and attention

Bags of words and graphs-of-words look at a whole text at once. Recurrent networks read it in order and carry a state; attention then lets a decoder look back at every position of the input instead of a single summary vector.

import torch
torch.manual_seed(0); torch.set_num_threads(1)

T, d = 60, 32
x = torch.randn(1, T, d, requires_grad=True)
for name, net in [("RNN (tanh)", torch.nn.RNN(d, d, batch_first=True)),
                  ("LSTM", torch.nn.LSTM(d, d, batch_first=True))]:
    if name == "LSTM":                               # forget-gate bias set to 1
        with torch.no_grad():
            net.bias_ih_l0[d:2 * d].fill_(1.0)
    out, _ = net(x)
    out[0, -1].sum().backward()                      # loss read at the last step
    g = x.grad[0].norm(dim=1)                        # sensitivity to the input x_t
    print(f"{name:10s} |dL/dx_t| at t = T-1, T-10, T-30: "
          f"{g[-1]:.2e} {g[-10]:.2e} {g[-30]:.2e}")
    x.grad = None

5. Deep discriminative architectures for classification

Classifying a sentence or a document needs a fixed-size vector built from a variable-length sequence. This chapter covers three designs: convolutions that detect n-grams anywhere, bidirectional encoders that read both ways, and hierarchical attention for long documents.

A single width-3 filter therefore detects “not very good” anywhere, a pattern that defeats bigram counts. Max pooling is itself a non-linearity: on presence features it computes an or, so even width-1 filters can represent the exclusive or of Proposition 1.3 (Exercise 8). Wider filters are needed when the label depends on order, for example “good bad” against “bad good”.

The weights show which words and sentences drive a decision, a useful diagnostic. They are not a faithful explanation in general: Jain and Wallace (2019) and Wiegreffe and Pinter (2019) discuss when they are.

import numpy as np, torch, torch.nn as nn

rng = np.random.default_rng(0); torch.manual_seed(0); torch.set_num_threads(1)
fillers = ["the", "movie", "plot", "was", "really", "and", "acting", "honestly"]
vocab = {w: i for i, w in enumerate(fillers + ["not", "very", "good", "bad"])}

def data(n):
    X, y = [], []
    for _ in range(n):
        pol, neg = rng.integers(2), rng.integers(2)
        core = (["not"] if neg else []) + ["very", "good" if pol else "bad"]
        words = list(rng.choice(fillers, 4)) + core + list(rng.choice(fillers, 3))
        words += ["the"] * (10 - len(words))                       # pad to length 10
        X.append([vocab[w] for w in words]); y.append(int(pol != neg))
    return torch.tensor(X), torch.tensor(y)

class TextCNN(nn.Module):
    def __init__(self, V, d=16, n_filters=16, width=3):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.conv = nn.Conv1d(d, n_filters, width)
        self.out = nn.Linear(n_filters, 2)
    def forward(self, x):
        h = torch.relu(self.conv(self.emb(x).transpose(1, 2)))   # (batch, filters, positions)
        return self.out(h.max(dim=2).values)                     # max-over-time pooling

Xtr, ytr = data(2000); Xte, yte = data(1000)
model = TextCNN(len(vocab))
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for epoch in range(10):
    for b in range(0, 2000, 100):
        loss = nn.functional.cross_entropy(model(Xtr[b:b + 100]), ytr[b:b + 100])
        opt.zero_grad(); loss.backward(); opt.step()
acc = (model(Xte).argmax(1) == yte).float().mean().item()
print("CNN test accuracy on 'not very good': %.3f" % acc)       # 1.000

6. The Transformer engine

The Transformer (Vaswani and coauthors, 2017) drops recurrence. Every position attends to every other in one step, so a layer runs in parallel over the sequence and any two tokens are one step apart. This chapter builds the layer and the three ways of telling it where tokens are.

import torch
torch.manual_seed(0); torch.set_num_threads(1)

def sdpa(Q, K, V, causal=False):
    S = Q @ K.transpose(-2, -1) / K.shape[-1] ** 0.5
    if causal:                                    # position i only sees j <= i
        n = S.shape[-1]
        S = S.masked_fill(torch.triu(torch.ones(n, n, dtype=torch.bool), 1), float("-inf"))
    return torch.softmax(S, dim=-1) @ V

def rope(x, base=10000.0):
    """Rotate each pair (x_2i, x_2i+1) of row m by the angle m * theta_i."""
    n, d = x.shape
    theta = base ** (-torch.arange(0, d, 2) / d)
    ang = torch.arange(n)[:, None] * theta[None, :]
    cos, sin = torch.cos(ang), torch.sin(ang)
    x1, x2 = x[:, 0::2], x[:, 1::2]
    out = torch.empty_like(x)
    out[:, 0::2], out[:, 1::2] = x1 * cos - x2 * sin, x1 * sin + x2 * cos
    return out

n, d = 6, 8
X = torch.randn(n, d)
P = torch.eye(n)[torch.randperm(n)]
print("permutation-equivariant:",
      torch.allclose(sdpa(P @ X, P @ X, P @ X), P @ sdpa(X, X, X), atol=1e-6))
q, k = torch.randn(d), torch.randn(d)
Qr, Kr = rope(q.repeat(n, 1)), rope(k.repeat(n, 1))   # same vectors at every position
print("RoPE score depends on m - n only:", torch.allclose(Qr[3] @ Kr[1], Qr[5] @ Kr[3], atol=1e-5))
out = sdpa(X, X, X, causal=True)
print("first causal output = first value:", torch.allclose(out[0], X[0]))

7. Self-supervised pretraining and transfer

Labelled data are scarce; raw text is not. Self-supervised pretraining learns from raw text by predicting parts of it from the rest, and the pretrained network is then fine-tuned on the task. Around 2018 this replaced task-specific feature engineering in most of NLP.

import numpy as np
rng = np.random.default_rng(1)

def mlm_corrupt(tokens, vocab, p=0.15):
    """BERT: select 15 % of positions; 80 % -> [MASK], 10 % -> random, 10 % kept."""
    x, targets = list(tokens), {}
    for i in np.flatnonzero(rng.random(len(tokens)) < p):
        targets[int(i)] = tokens[i]                 # the loss is computed here only
        u = rng.random()
        x[i] = "[MASK]" if u < 0.8 else (rng.choice(vocab) if u < 0.9 else tokens[i])
    return x, targets

def text_infilling(tokens, mask_ratio=0.3, lam=3.0):
    """BART: spans with Poisson(3) lengths, each replaced by ONE [MASK]."""
    x, budget = list(tokens), int(round(mask_ratio * len(tokens)))
    while budget > 0:
        L = min(rng.poisson(lam), budget)
        start = rng.integers(0, len(x))
        if "[MASK]" in x[start:start + L]:
            continue                                # avoid nesting spans
        x[start:start + L] = ["[MASK]"]              # L = 0 inserts a mask
        budget -= max(L, 1)
    return x

sent = "the graph of words links two terms when they co-occur in a window".split()
print(mlm_corrupt(sent, vocab=sent))
print(" ".join(text_infilling(sent)))

8. Autoregressive pretraining at scale

Large language models are decoder-only Transformers trained with the causal loss of Definition 7.5 on trillions of tokens. At this scale the questions become quantitative: how much compute a run needs, how to split it between parameters and data, and how the tokenizer changes the bill.

E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28   # fit of Hoffmann et al. (2022)

def optimum(C):
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
    N = G * (C / 6) ** (beta / (alpha + beta))
    D = (C / 6) / N
    return N, D, E + A / N**alpha + B / D**beta

for C in [1e21, 1e23, 1e25]:
    N, D, L = optimum(C)
    print(f"C = {C:.0e}: N* = {N:.1e}, D* = {D:.1e}, D*/N* = {D / N:5.1f}, loss {L:.3f}")

for name, N, D in [("Gopher", 280e9, 300e9), ("Chinchilla", 70e9, 1.4e12)]:
    print(f"{name:10s} C = {6 * N * D:.1e}, predicted loss {E + A / N**alpha + B / D**beta:.3f}")

9. Evaluation, compression and multilingual benchmarks

A generation model is only as good as the way it is measured. This chapter covers surface and learned metrics for generated text, knowledge distillation as the common tool to make models — and metrics — cheaper, and the design of benchmarks for languages other than English.

import numpy as np, torch, torch.nn as nn
from scipy.stats import kendalltau
rng = np.random.default_rng(0); torch.manual_seed(0); torch.set_num_threads(1)

V, n_syn, d, L = 200, 40, 32, 10
syn = rng.integers(n_syn, size=V)                          # synonym classes
E = rng.normal(size=(n_syn, d))[syn] + 0.3 * rng.normal(size=(V, d))
E /= np.linalg.norm(E, axis=1, keepdims=True)              # the teacher's word vectors

def teacher(c, r):          # expensive metric: greedy matching F1
    S = E[c] @ E[r].T
    P, R = S.max(1).mean(), S.max(0).mean()
    return 2 * P * R / (P + R)

def unigram_f1(c, r):       # cheap surface metric, a ROUGE-1 proxy
    m = len(set(c) & set(r))
    return 2 * m / (len(set(c)) + len(set(r)))

def make_pair():            # a reference and a corrupted candidate
    r = rng.integers(V, size=L); c = r.copy()
    for i in range(L):
        u = rng.random()
        if u < 0.4:   c[i] = rng.choice(np.flatnonzero(syn == syn[r[i]]))   # synonym
        elif u < 0.6: c[i] = rng.integers(V)                                # error
    return c, r

pairs = [make_pair() for _ in range(6000)]
Y = torch.tensor([teacher(c, r) for c, r in pairs], dtype=torch.float32)
C = torch.tensor(np.array([p[0] for p in pairs])); R = torch.tensor(np.array([p[1] for p in pairs]))

class Student(nn.Module):
    def __init__(self, k=8):
        super().__init__()
        U, S, _ = np.linalg.svd(E - E.mean(0), full_matrices=False)
        self.emb = nn.Embedding.from_pretrained(
            torch.tensor(U[:, :k] * S[:k], dtype=torch.float32), freeze=False)
        self.head = nn.Linear(3, 1)
    def forward(self, c, r):
        a = nn.functional.normalize(self.emb(c), dim=-1)
        b = nn.functional.normalize(self.emb(r), dim=-1)
        S = a @ b.transpose(1, 2)
        P, R = S.max(2).values.mean(1), S.max(1).values.mean(1)
        return self.head(torch.stack([P, R, P * R], 1)).squeeze(1)

student = Student()
opt = torch.optim.Adam(student.parameters(), lr=1e-2)
for epoch in range(5):                                   # distillation = regression on Y
    for b in range(0, 5000, 250):
        loss = ((student(C[b:b + 250], R[b:b + 250]) - Y[b:b + 250]) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()

with torch.no_grad():
    pred = student(C[5000:], R[5000:]).numpy()
surface = [unigram_f1(c, r) for c, r in pairs[5000:]]
print("Kendall tau with the teacher: unigram F1 %.2f, student %.2f"
      % (kendalltau(surface, Y[5000:])[0], kendalltau(pred, Y[5000:])[0]))

10. Downstream adaptation and parameter efficiency

A pretrained language model continues text; an assistant must follow instructions, and a domain system needs specialised behaviour. Fine-tuning all the weights of a large model is costly to train and to store, one copy per task. This chapter covers instruction tuning and the methods that train a tiny fraction of the weights.

import torch, torch.nn as nn
torch.manual_seed(0); torch.set_num_threads(1)

class LoRALinear(nn.Module):
    """h = W0 x + (alpha / r) B A x, with W0 frozen (Hu et al., 2022)."""
    def __init__(self, base: nn.Linear, r=4, alpha=8):
        super().__init__()
        self.base, self.scale = base, alpha / r
        for p in self.base.parameters():
            p.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, base.in_features) / base.in_features ** 0.5)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))     # BA = 0 at start
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

d = 128
base = nn.Linear(d, d)                                   # the "pretrained" layer
delta = torch.randn(d, 4) @ torch.randn(4, d) / d        # the task needs a rank-4 change
X = torch.randn(1024, d)
Y = base(X).detach() + X @ delta.T

for r in [1, 4, 16]:
    layer = LoRALinear(base, r=r)
    opt = torch.optim.Adam([layer.A, layer.B], lr=1e-2)
    for step in range(200):
        loss = ((layer(X) - Y) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    n_train = layer.A.numel() + layer.B.numel()
    print(f"r = {r:2d}: {n_train:5d} trainable parameters "
          f"({100 * n_train / (d * d):.1f} % of W0), final MSE {loss.item():.1e}")

11. Alignment and post-training

Instruction tuning teaches a model the format of helpful answers; it does not say which of two plausible answers people prefer. Post-training on preferences does, with reinforcement learning (RLHF, PPO) or without it (DPO). This chapter derives both from one objective.

import torch
torch.manual_seed(0); torch.set_num_threads(1)

K, beta = 5, 0.5
r_true = torch.tensor([1.0, 0.5, 0.0, -0.5, -1.0])               # hidden rewards
log_ref = torch.log(torch.tensor([0.1, 0.2, 0.4, 0.2, 0.1]))     # reference policy

i, j = torch.randint(K, (20000,)), torch.randint(K, (20000,))
keep = i != j
i, j = i[keep], j[keep]
i_wins = torch.rand(len(i)) < torch.sigmoid(r_true[i] - r_true[j])
yw, yl = torch.where(i_wins, i, j), torch.where(i_wins, j, i)

logits = log_ref.clone().requires_grad_(True)                     # start at the reference
opt = torch.optim.Adam([logits], lr=0.05)
for step in range(500):
    logp = torch.log_softmax(logits, 0)
    margin = beta * ((logp[yw] - log_ref[yw]) - (logp[yl] - log_ref[yl]))
    loss = -torch.nn.functional.logsigmoid(margin).mean()        # DPO loss
    opt.zero_grad(); loss.backward(); opt.step()

print("DPO policy:     ", [round(p, 3) for p in torch.softmax(logits, 0).tolist()])
print("closed form pi*:", [round(p, 3) for p in torch.softmax(log_ref + r_true / beta, 0).tolist()])

12. Retrieval-augmented and graph-augmented generation

The knowledge of a language model is frozen in its weights at the end of pretraining, cannot be cited, and is sometimes reconstructed wrongly when it is missing. Retrieval-augmented generation fetches evidence at query time and conditions the answer on it; graph-augmented generation fetches structured evidence that supports multi-step reasoning.

import math, re
from collections import Counter
import networkx as nx

docs = ["BARThez is a French sequence-to-sequence model pretrained like BART.",
        "BART was introduced by Lewis and coauthors at Facebook AI.",
        "FrugalScore distils expensive evaluation metrics into small models.",
        "The k-core of a graph of words keeps its densest terms.",
        "Facebook AI Research has a laboratory in Paris."]
tok = lambda s: re.findall(r"[a-z0-9\-]+", s.lower())
D = [tok(d) for d in docs]
avgdl, N = sum(map(len, D)) / len(D), len(D)
df = Counter(t for d in D for t in set(d))

def bm25(query, k1=1.5, b=0.75):
    scores = []
    for d in D:
        tf, s = Counter(d), 0.0
        for t in tok(query):
            if t in tf:
                idf = math.log(1 + (N - df[t] + 0.5) / (df[t] + 0.5))
                s += idf * tf[t] * (k1 + 1) / (tf[t] + k1 * (1 - b + b * len(d) / avgdl))
        scores.append(s)
    return sorted(range(N), key=lambda i: -scores[i])

q = "In which city does the lab behind the model that BARThez imitates work?"
print("BM25 top-2:", bm25(q)[:2])            # [0, 3]: first hop only

KG = nx.DiGraph()                            # triples extracted from the documents
for h, r, t in [("BARThez", "pretrained_like", "BART"), ("BART", "introduced_by", "Facebook AI"),
                ("Facebook AI", "has_lab_in", "Paris"), ("FrugalScore", "distils", "metrics")]:
    KG.add_edge(h, t, rel=r)
linked = [e for e in KG if e.lower() in tok(q)]             # naive linking: exact token match
sub = nx.compose_all([nx.ego_graph(KG, e, radius=3) for e in linked])
context = [f"{u} {KG[u][v]['rel']} {v}." for u, v in nx.bfs_edges(sub, linked[0])]
print("graph context for the LLM:", " ".join(context))

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE model(name TEXT, base TEXT, language TEXT);
INSERT INTO model VALUES ('BART', NULL, 'English'), ('BARThez', 'BART', 'French'),
    ('mBART', 'BART', 'multilingual'), ('CamemBERT', 'RoBERTa', 'French');
""")
schema = "model(name, base, language)"                        # shown to the LLM
question = "Which French models are derived from BART?"
generated = "SELECT name FROM model WHERE base = 'BART' AND language = 'French'"
reference = "SELECT m.name FROM model AS m WHERE m.language = 'French' AND m.base = 'BART'"
rows = db.execute(generated).fetchall()
print(rows, "| exact string match:", generated == reference,
      "| execution match:", set(rows) == set(db.execute(reference).fetchall()))

13. Quantization and efficient inference

A 7-billion-parameter model stored in 16-bit floats needs 14 GB for its weights alone, and decoding reads all of them for every generated token. Quantization stores weights, and sometimes activations, on 8 or 4 bits; the key–value cache and pruning address the other costs of inference.

import numpy as np
rng = np.random.default_rng(0)

def quant(w, scale, bits=4):
    qmax = 2 ** (bits - 1) - 1
    return scale * np.clip(np.round(w / scale), -qmax, qmax)

d_in, d_out, n = 128, 64, 2048
W = rng.normal(size=(d_out, d_in)) / np.sqrt(d_in)
W[:, :2] *= 8                                              # two outlier input channels
mix = rng.normal(size=(d_in, d_in)) * np.logspace(0, -2, d_in)
X = mix @ rng.normal(size=(d_in, n))                       # correlated calibration inputs
err = lambda Wq: np.linalg.norm((W - Wq) @ X) ** 2 / np.linalg.norm(W @ X) ** 2

def scales(W, group):                                      # one scale per row and group
    s = np.abs(W.reshape(len(W), -1, group)).max(axis=2) / 7
    return np.repeat(s, group, axis=1)

def gptq(W, X, S, damp=0.01):
    W = W.copy()
    H = 2 * X @ X.T
    H += damp * np.mean(np.diag(H)) * np.eye(len(H))       # dampening
    Hinv = np.linalg.inv(H)
    for q in range(W.shape[1]):
        wq = quant(W[:, q], S[:, q])
        e = (W[:, q] - wq) / Hinv[q, q]
        W[:, q:] -= np.outer(e, Hinv[q, q:])               # Theorem 13.5
        W[:, q] = wq
        Hinv -= np.outer(Hinv[:, q], Hinv[q, :]) / Hinv[q, q]   # eliminate column q
    return W

for group in [d_in, 32]:
    S = scales(W, group)
    print(f"group size {group:3d}: RTN error {err(quant(W, S)):.4f}, "
          f"GPTQ error {err(gptq(W, X, S)):.4f}")

14. Research directions and open problems

This chapter is for self-study. It points to active questions that extend each part of the course, with entry papers; it is a map, not a survey.

Exercises

Levels: ★ apply the course, ★★ combine or prove a short result, ★★★ open-ended or research-flavoured. The Practice column of the session plan says when to do each one.

1. ★ TF-IDF, cosine and perplexity by hand. Three documents: d_1 = “graph graph kernel”, d_2 = “graph words”, d_3 = “words vectors”.

   1. Compute the TF-IDF vectors with natural logarithms and N = 3. (b) Compute the cosine of d_1 with d_2 and with d_3. (c) A model gives probabilities 0.5, 0.25, 0.125 and 0.125 to the four tokens of a test sentence; what is its perplexity?

   Solution: (a) “graph” has \mathrm{df} = 2 and “kernel” \mathrm{df} = 1, so d_1 has weights 2 \log 1.5 = 0.811 and \log 3 = 1.099.

   The document d_2 has \log 1.5 = 0.405 for “graph” and for “words”; d_3 has 1.099 for “vectors” and 0.405 for “words”.

   2. The norms are 1.365 and 0.573, and only “graph” is shared, so \cos(d_1, d_2) = \frac{0.811 \times 0.405}{1.365 \times 0.573} \approx 0.42, \qquad \cos(d_1, d_3) = 0.

   3. The product of the probabilities is 1/512, so the perplexity is 512^{1/4} = 2^{9/4} \approx 4.76.

2. ★ BPE by hand. Run four BPE merges on the word counts “low” 5, “lowest” 2, “newer” 6, “wider” 3, “new” 2 (break ties by first occurrence). Segment “lower” and “newest” with the learned merges, and check with the code of Example 1.10.

   Solution: the initial pair counts are 9 for “e r”, 8 for “w e”, “n e” and “e w”, and 7 for “l o” and “o w”. The merges are “er” (9), “ne” (8), “new” (8) and “lo” (7); after “er”, the pair “w e” only remains in “lowest” (count 2).

   “lower” becomes “lo w er” and “newest” becomes “new e s t”. The code prints the merges [('er', 9), ('ne', 8), ('new', 8), ('lo', 7)].

3. ★ Cores by hand. A graph-of-words has the edges A–B, A–C, B–C, A–D, B–D and D–E. Run the peeling algorithm of Theorem 3.3, give every core number, the degeneracy, the main core and the CoreRank scores.

   Solution: remove E (degree 1, so \kappa = 1); then C and D have degree 2. Remove C (\kappa = 2), then D (degree 2), then A and B (degrees 1 and 0, \kappa stays 2).

   Core numbers: E 1, and A, B, C, D 2. The degeneracy is 2 and the main core is \{A, B, C, D\}. CoreRank: A 6, B 6, D 5, C 4, E 2, which orders the terms of the main core.

4. ★ Additive, dot and scaled attention. Take the data of Example 4.8.

   Compute the weights and the context for additive attention with W_a = U_a = I and v = (1, 1), and for scaled dot-product attention with d_k = 2. Compare with Example 4.8.

   Solution: additive scores v^\top \tanh(s + h_j) are 1.726, 1.523 and 0.995, so the weights are (0.435, 0.355, 0.210) and the context is (0.854, 0.790).

   Scaled scores are (0.707, 0, 1.414), with weights (0.284, 0.140, 0.576) and context (1.436, 0.424).

   The additive score ranks h_1 first: the tanh saturates the large coordinate of s + h_3, so it rewards keys active in both coordinates. The \sqrt{d_k} scaling keeps the ranking of Example 4.8 and only softens it (0.576 instead of 0.665 on h_3).

5. ★ LoRA and KV-cache budgets. A decoder has 32 layers, model width 4096 and heads of dimension 128. (a) How many parameters does LoRA of rank 16 on W_Q and W_V of every layer train, and how many megabytes do they take in 16 bits? (b) What is the KV-cache memory for a batch of 8 sequences of 2048 tokens in 16 bits, with 32 key–value heads, and with grouped-query attention using 8?

   Solution: (a) Each matrix gets 16 \times (4096 + 4096) = 131{,}072 parameters, so 2 \times 32 \times 131{,}072 \approx 8.4 million in total, about 16.8 MB (Proposition 10.3).

   2. By Proposition 13.7, 2 \times 32 \times 32 \times 128 \times 2048 \times 8 \times 2 bytes = 8 GiB with 32 heads, and 2 GiB with 8 key–value heads.

6. ★ A compute budget. You may spend 10^{22} operations. (a) With the rule of 20 tokens per parameter and C = 6ND, what model size and how many tokens? (b) How long does training take on 256 accelerators sustaining 4 \times 10^{14} operations per second each? (c) Using the code of Example 8.6, what does the fitted law of Theorem 8.4 recommend instead?

   Solution: (a) C = 6 N \cdot 20 N = 120 N^2, so N \approx 9.1 \times 10^9 parameters and D \approx 1.8 \times 10^{11} tokens. (b) The 256 accelerators deliver about 10^{17} operations per second, so training takes about 9.8 \times 10^4 seconds, a little more than a day.

   3. The fitted law gives N^* \approx 5.2 \times 10^9 parameters and D^* \approx 3.2 \times 10^{11} tokens, a smaller model trained on more data (about 63 tokens per parameter): the two estimation methods disagree, which is why Example 8.6 discusses the replication.

7. ★★ SGNS with a smoothed noise law. Redo the proof of Theorem 2.4 with negatives drawn from P_n(c) = \#(c)^{3/4} / Z. Show that the optimum is a PMI computed with a smoothed context distribution, shifted by \log k. Explain why rare contexts get smaller PMI values than with the unigram law, and what a finite dimension changes.

   Solution: the term of the pair (w, c) becomes f(x) = \#(w, c) \log \sigma(x) + k\, \#(w)\, P_n(c) \log \sigma(-x), concave in x, with maximum at x^* = \log \frac{\#(w, c)}{\#(w)\, P_n(c)} - \log k. Raising counts to the power 3/4 increases the probability of rare contexts, which lowers their PMI and damps the known bias of PMI towards rare pairs.

   With a finite dimension, the inner products cannot reach all the targets: SGNS then computes a weighted low-rank approximation, with weights given by the counts, unlike the unweighted SVD of Proposition 2.5.

8. ★★ Max pooling, exclusive or, and order (computational). (a) Show that a width-1 convolution with max pooling and a linear output can represent the label of Example 1.4: positive when exactly one of “not” and “good” occurs. (b) Run Example 5.5 with width=1. (c) Change the generator so that the core is “good bad” (label 1) or “bad good” (label 0), and compare widths 1 and 2.

   Solution: (a) With one-hot vectors, a filter that responds 1 to “not” and to “good” gives, after max pooling, the or of the two presences A and B; two more filters give A and B. The output 2\,\mathrm{OR} - A - B equals the exclusive or.

   2. The width-1 network also reaches 1.000: the task depends only on which words occur, and the non-linearity of max pooling suffices. (c) Both classes now have the same bag of words: width 1 stays at chance (0.458 in our run), width 2 reaches 1.000.

9. ★★ Positions: sinusoids versus rotations. (a) Show that, for each frequency, the pair (\sin, \cos) of Definition 6.6 at position p + k is the pair at p rotated by the angle k \omega_i.

   2. For d_k = 2, write the RoPE score of Theorem 6.8 with the angles \varphi_q, \varphi_k of q and k. (c) Show that adding sinusoidal codes to the inputs does not make attention scores depend on the offset only.

   Solution: (a) The addition formulas give \sin((p + k)\omega) = \sin(p\omega)\cos(k\omega) + \cos(p\omega)\sin(k\omega), and similarly for the cosine: a rotation matrix that does not depend on p.

   2. The score is \|q\|\, \|k\| \cos\big(\varphi_k - \varphi_q + (n - m)\theta\big). For q = k = (1, 0), \theta = \pi/6, m = 1 and n = 3, it is \cos(\pi/3) = 0.5.

   3. With x_m + p_m as input, the score (x_m + p_m)^\top W_Q^\top W_K (x_n + p_n) contains the cross terms x_m^\top W_Q^\top W_K\, p_n, which depend on the absolute position n.

10. ★★ Pretraining objectives compared. (a) How many supervised predictions does a 512-token sequence give under masked language modelling with a 15 % rate, and under causal language modelling? (b) Why can a decoder generate with a cache of past keys and values, while a bidirectional encoder must recompute everything when a token is appended? (c) Why does BART text infilling sometimes insert a mask that hides zero tokens?

    Solution: (a) About 0.15 \times 512 \approx 77 predictions against 511: causal pretraining extracts more signal per token, one reason why it dominates at scale.

    2. Under the causal mask, the states of past positions do not depend on later tokens, so they can be cached and reused. In an encoder, every state depends on every token, so appending one changes all of them.

    3. Since a mask may hide zero, one or several tokens, the model must predict the length of each span, not only its content; this prepares it for generation of variable-length text.

11. ★★ Metrics and distillation (computational). (a) Compute \mathrm{BLEU}_2 of “the cat sits on the mat” against “the cat is on the mat”. (b) Prove the gradient formula of Proposition 9.5. (c) Run Example 9.7 with student dimensions k = 2, 4, 8, 16 and plot Kendall’s tau against k.

    Solution: (a) As in Example 9.2, p_1 = 5/6 and p_2 = 3/5, so \mathrm{BLEU}_2 \approx 0.707: the near-synonym is treated as an error.

    2. The KL term equals a cross-entropy with soft targets p minus a constant; by Proposition 0.2 applied to the logits z^S / \tau, its gradient in z^S is (q - p)/\tau.

    3. With the seeds of the example, tau is 0.28, 0.51, 0.72 and 0.86 for k = 2, 4, 8, 16. The unigram F1 (0.40) is beaten from k = 4 on: a small learned model captures the synonym structure that exact matching misses.

12. ★★ DPO at initialisation and in the deterministic limit. (a) Show that adding any function f(x) to the reward changes neither \pi^* nor the preference probabilities. (b) At \pi_\theta = \pi_{\mathrm{ref}}, give the value of the DPO loss and its gradient. (c) In Example 11.8, suppose the better answer always wins. What happens to the DPO solution?

    Solution: (a) \exp(f(x)/\beta) cancels in the normalisation of Theorem 11.3, and f(x) cancels in the reward difference of Definition 11.1.

    2. The margin is 0, the loss is \log 2 \approx 0.693, and every pair weighs the same in the gradient: \nabla_\theta \mathcal{L}_{\mathrm{DPO}} = -\frac{\beta}{2}\, \mathbb{E}\big[\nabla \log \pi_\theta(y_w) - \nabla \log \pi_\theta(y_l)\big].

    3. The Bradley–Terry likelihood is then maximised only by infinite reward gaps, so the logits diverge and the policy collapses on the best answer, whatever \beta. The KL regularisation is lost; this overfitting motivates the IPO variant of Azar and coauthors (2024).

13. ★★ Quantization error and outliers (computational). Draw 4096 weights from a standard normal law, set the first one to 20, and quantize on 4 bits with absmax scales per tensor, per group of 128 and per group of 32. Compare the mean squared error with the prediction s^2/12 of Proposition 13.2, and give the storage cost in bits per weight.

    Solution: with numpy.random.default_rng(0), the errors are 0.612, 0.034 and 0.013, against mean predictions 0.680, 0.034 and 0.015; the costs are 4.004, 4.125 and 4.5 bits per weight.

    Per tensor, the step is 20/7 \approx 2.9, so almost every weight rounds to zero and the error is close to the variance of the weights. The prediction fails there because errors are not uniform when the step exceeds the spread of the data.

14. ★★★ The shortest-path kernel for documents. Implement Definition 3.7 with windows of 3 and d_{\max} = 2, compute the normalised Gram matrix of “the cat sat on the mat”, “on the mat the cat sat” and “the dog chased the cat”, check its eigenvalues, and discuss the cost and the roles of the window and of d_{\max}.

    Hints and expected result: one implementation uses breadth-first search from every node, as below.

    import numpy as np, networkx as nx
    from collections import Counter
    
    def gow(text, window=3):
        t = text.lower().split()
        G = nx.Graph(); G.add_nodes_from(t)
        G.add_edges_from((t[i], u) for i in range(len(t)) for u in t[i + 1:i + window] if u != t[i])
        return G
    
    def sp_features(G, d_max=2):
        phi = Counter()
        for a, dist in nx.all_pairs_shortest_path_length(G, cutoff=d_max):
            for b, l in dist.items():
                if a < b:
                    phi[a, b, l] += 1
        return phi
    
    def sp_kernel(G1, G2):
        f1, f2 = sp_features(G1), sp_features(G2)
        k = lambda f, g: sum(v * g[key] for key, v in f.items())
        return k(f1, f2) / np.sqrt(k(f1, f1) * k(f2, f2))
    
    docs = ["the cat sat on the mat", "on the mat the cat sat", "the dog chased the cat"]
    G = [gow(d) for d in docs]
    K = np.array([[sp_kernel(a, b) for b in G] for a in G])
    print(np.round(K, 3), np.round(np.linalg.eigvalsh(K), 3))

    The off-diagonal values are 0.700 for the two permuted sentences and 0.129 for the third. The eigenvalues 0.300, 0.955 and 1.745 are non-negative, as Proposition 3.8 guarantees.

    Breadth-first search from every node costs O(n(n + m)) per document. A larger window adds edges and shortens distances; a larger d_{\max} adds features and makes the kernel closer to a bag of term pairs.

15. ★★★ A multi-hop retrieval benchmark. Generate a synthetic knowledge graph of about 200 entities and 5 relations, turn every triple into a one-sentence passage, and generate two-hop questions whose answer needs two triples. Compare BM25 over passages (Definition 12.1) with entity linking plus k-hop expansion (Example 12.7), by recall at k of the two supporting passages.

    Hints and expected result: BM25 retrieves the first-hop passage, which shares the entity name with the question, but rarely the second, which shares no word with it. Graph expansion retrieves both when entity linking succeeds, at the price of many distractors when nodes have high degree.

    A good report shows recall at k against k for both methods, and a hybrid — BM25 for the first hop, the graph for the next — that combines their strengths. This is the motivation of the pipelines of Definition 12.6.

16. ★★★ Tokenizer fertility across languages. Build two synthetic languages with different syllable inventories and word-length distributions, train BPE (Example 1.10) on a mixture that contains 90 % of language A, and measure the fertility (Definition 8.7) on each language. Vary the mixture and the vocabulary size, and relate the results to Petrov and coauthors (2023).

    Hints and expected result: fertility on the minority language is clearly higher, and the gap shrinks as its share or the vocabulary size grows. Express the cost of a fixed text in tokens and in context-window usage for both languages, and discuss what a sovereign model gains by training its own tokenizer (Remark 9.10).

Projects and assessment

The official assessment (master-mva.com) is a final data challenge plus a lab-based evaluation; the Université Paris-Saclay course detail describes it as non-integral continuous assessment (“Evaluation Continue non Intégrale”). No weights and no written exam are displayed. The group subjects below are shaped like a data challenge, the solo subjects are paper studies and lab extensions, and the written exam is only a self-assessment proposal.

Solo projects

1. [Solo · ≈ 20 h] Paper study: keyword extraction by graph degeneracy (Rousseau and Vazirgiannis, 2015; Tixier, Malliaros and Vazirgiannis, 2016). Question: do main-core retention and CoreRank beat TF-IDF and TextRank (Mihalcea and Tarau, 2004) on single-document keyword extraction?
   • Implement the weighted graph-of-words, main-core retention, CoreRank with a rule for the number of keywords, TextRank (PageRank on the same graph) and a TF-IDF baseline (ch. 1, 3).
   • Evaluate precision, recall and F1 with stemmed exact matching on the public abstracts of Hulth (2003) or the SemEval-2010 keyphrase data (Kim and coauthors, 2010), with an ablation on the window size.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Success: the paper’s ranking of the methods is reproduced or its failure explained.
2. [Solo · ≈ 20 h] Paper study: DPO against reward modelling and policy gradient (Rafailov and coauthors, 2023). Question: on a controlled problem with a known reward, when do the two routes reach the same policy?
   • Extend Example 11.8 to many prompts with a tabular policy. Implement reward-model fitting followed by KL-regularised policy gradient, DPO, and the IPO variant (Azar and coauthors, 2024).
   • Plot reward against KL divergence from the reference, sample efficiency, and the effect of nearly deterministic preferences (Exercise 12). Chapters 0, 11.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Success: both routes reach the same frontier on Bradley–Terry data, and the degeneration of DPO on deterministic preferences is shown and corrected.
3. [Solo · ≈ 20 h] How low can rank and bits go? Question: what LoRA rank and how many bits does a small open model need on a classification task?
   • Fine-tune GPT-2 small (124 million parameters, public weights) with LoRA of rank 1 to 64 on SST-2 (Socher and coauthors, 2013), on a free GPU tier (ch. 10).
   • Merge the adapters, quantize the model with round-to-nearest and the GPTQ procedure of Example 13.8 at 8, 4 and 3 bits, and report SST-2 accuracy. As a check of general ability, also report the WikiText-2 perplexity (Merity and coauthors, 2017) of the pretrained model quantized the same way (ch. 13).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Success: an accuracy-versus-rank curve with a clear knee, and GPTQ at least as good as round-to-nearest at 4 and 3 bits.

Group projects

1. [Group of 2–3 · ≈ 30 h each] Data challenge: classifying papers from their text and their citations. Use the ogbn-arxiv citation graph of the Open Graph Benchmark (Hu and coauthors, 2020) with the public titles and abstracts of its papers; the task is the subject area, on the official split.
   • Member 1: text models — TF-IDF with logistic regression, then a small pretrained encoder, frozen or fine-tuned with LoRA (ch. 1, 7, 10).
   • Member 2: graph models — label propagation and message passing over citations (Definition 12.6), with graph-of-words features (ch. 3).
   • Member 3: fusion, hidden test set, leaderboard and error analysis.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Success: the fused model beats the group’s best text-only and best graph-only models on the hidden split, with ablations.
2. [Group of 3 · ≈ 30 h each] A grounded, graph-augmented question-answering system. Use the distractor setting of HotpotQA (Yang and coauthors, 2018), whose questions need two supporting paragraphs.
   • Retrieval: BM25 and a small public dense encoder, evaluated by recall at k of the supporting paragraphs (ch. 12).
   • Graph: an entity graph built from the paragraphs, with k-hop expansion from the entities of the question (Example 12.7).
   • Generation: a small open instruction-tuned model, or an extractive reader, prompted to cite paragraphs; exact match, F1 and citation correctness (ch. 9, 12).
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Success: graph expansion raises supporting-paragraph recall on bridge questions over BM25 alone, and the answer metrics follow.
3. [Group of 2–3 · ≈ 30 h each] Paper study: distilling an evaluation metric (Kamal Eddine and coauthors, 2022). Reproduce FrugalScore at small scale.
   • Member 1: generate candidate–reference pairs from a public summarisation corpus such as CNN/DailyMail (Hermann and coauthors, 2015) and score them with BERTScore (Zhang and coauthors, 2020) as teacher.
   • Member 2: train compact pretrained students (Turc and coauthors, 2019) to regress the teacher scores (Proposition 9.5, Example 9.7).
   • Member 3: meta-evaluation against the human judgements of SummEval (Fabbri and coauthors, 2021), and speed benchmarks.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Success: the student keeps most of the teacher’s Kendall correlation with human judgements while running at least ten times faster.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet allowed, no electronic device, four questions.

1. Question (4 points). Define the k-core of a graph, the core number of a node and the degeneracy. Describe main-core keyword extraction on a graph-of-words, explain why it needs no corpus statistics, unlike TF-IDF, and give one limitation.

   Expected: Definitions 3.1–3.2 and the peeling algorithm of Theorem 3.3; the score uses only the document’s own graph (Definition 3.4). Limitations: on short texts the main core is coarse, which CoreRank refines; the result depends on the window size and on preprocessing.

2. Question (5 points). (a) Under the rule of 20 tokens per parameter, how many tokens and operations does a model of 10^9 parameters need? (b) How many parameters does LoRA of rank 8 add to a 2048 \times 2048 matrix? (c) What is the KV-cache memory of one sequence of 4096 tokens for 24 layers, 16 key–value heads of dimension 128, in 16 bits?

   Expected: (a) D = 2 \times 10^{10} tokens and C = 6ND = 1.2 \times 10^{20} operations. (b) 8 \times (2048 + 2048) = 32{,}768 parameters, 0.78 % of the matrix. (c) By Proposition 13.7, 2 \times 24 \times 16 \times 128 \times 4096 \times 2 bytes, that is 768 MiB.

3. Question (5 points). One query attends to three keys with d_k = 2: q = (1, 0); \quad k_1 = (1, 0),\ k_2 = (0, 1),\ k_3 = (-1, 0); \quad v_1 = (1, 0),\ v_2 = (0, 1),\ v_3 = (1, 1). Compute the attention weights and the output. Then apply a causal mask, the query being at position 2, and recompute. Finally, explain why self-attention without positions cannot distinguish “good bad” from “bad good”.

   Expected: scaled scores (0.707, 0, -0.707), weights (0.576, 0.284, 0.140), output (0.716, 0.424). With the mask only k_1, k_2 remain: weights (0.670, 0.330) and output (0.670, 0.330). Without positions, self-attention is permutation-equivariant (Proposition 6.4), so any pooled output is the same for both orders.

4. Question (6 points). For a reward r, a reference policy and \beta > 0, prove that the maximiser of the KL-regularized objective is the reference tilted by \exp(r/\beta). Deduce the DPO loss under the Bradley–Terry model, and explain what happens when the policy equals the reference.

   Expected: the Gibbs argument of Theorem 11.3, writing the objective as -\beta\, \mathrm{KL}(\pi \,\|\, \pi^*) + \beta \log Z; inversion of the optimal policy and cancellation of \log Z(x) in the reward difference (Theorem 11.6). At the reference, the margin is 0 and the loss is \log 2 (Exercise 12).

Grading rubric (proposal)

References

Works behind the methods named in the official syllabus. The official pages list no bibliography; these are the papers that introduced the methods the 2026–2027 syllabus names explicitly (GreekMMLU is named there without a reference).

• M. Kamal Eddine, A. J.-P. Tixier, M. Vazirgiannis, “BARThez: a Skilled Pretrained French Sequence-to-Sequence Model”, EMNLP, 2021.
• M. Kamal Eddine, G. Shang, A. J.-P. Tixier, M. Vazirgiannis, “FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation”, ACL, 2022.
• T. Mikolov, K. Chen, G. Corrado, J. Dean, “Efficient Estimation of Word Representations in Vector Space”, ICLR Workshop, 2013; T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, “Distributed Representations of Words and Phrases and their Compositionality”, NeurIPS, 2013.
• J. Pennington, R. Socher, C. Manning, “GloVe: Global Vectors for Word Representation”, EMNLP, 2014.
• D. Bahdanau, K. Cho, Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, ICLR, 2015; M.-T. Luong, H. Pham, C. Manning, “Effective Approaches to Attention-based Neural Machine Translation”, EMNLP, 2015.
• Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, E. Hovy, “Hierarchical Attention Networks for Document Classification”, NAACL, 2016.
• J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding”, arXiv, 2021 (Neurocomputing, 2024); O. Press, N. Smith, M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation”, ICLR, 2022.
• J. Hoffmann and coauthors, “Training Compute-Optimal Large Language Models”, NeurIPS, 2022.
• E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR, 2022; X. L. Li, P. Liang, “Prefix-Tuning: Optimizing Continuous Prompts for Generation”, ACL, 2021.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, “Proximal Policy Optimization Algorithms”, arXiv, 2017; R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. Manning, C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, NeurIPS, 2023.
• V. Karpukhin and coauthors, “Dense Passage Retrieval for Open-Domain Question Answering”, EMNLP, 2020.
• E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”, ICLR, 2023; J. Lin and coauthors, “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration”, MLSys, 2024; T. Dettmers, M. Lewis, Y. Belkada, L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, NeurIPS, 2022.

Textbooks.

• D. Jurafsky, J. H. Martin, Speech and Language Processing, 3rd edition draft, Stanford University, online.
• Y. Goldberg, Neural Network Methods for Natural Language Processing, Morgan & Claypool, 2017.
• C. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval, Cambridge University Press, 2008.
• W. L. Hamilton, Graph Representation Learning, Morgan & Claypool, 2020.
• I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016.

Chapters 0–3: text representations and graphs of words.

• D. Kingma, J. Ba, “Adam: A Method for Stochastic Optimization”, ICLR, 2015; Z. Harris, “Distributional Structure”, Word, 1954.
• G. Salton, C. Buckley, “Term-Weighting Approaches in Automatic Text Retrieval”, Information Processing & Management, 1988; S. Chen, J. Goodman, “An Empirical Study of Smoothing Techniques for Language Modeling”, ACL, 1996.
• R. Sennrich, B. Haddow, A. Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL, 2016; T. Kudo, “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”, ACL, 2018; M. Schuster, K. Nakajima, “Japanese and Korean Voice Search”, ICASSP, 2012.
• O. Levy, Y. Goldberg, “Neural Word Embedding as Implicit Matrix Factorization”, NeurIPS, 2014; O. Levy, Y. Goldberg, I. Dagan, “Improving Distributional Similarity with Lessons Learned from Word Embeddings”, TACL, 2015.
• P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, “Enriching Word Vectors with Subword Information”, TACL, 2017.
• F. Rousseau, M. Vazirgiannis, “Graph-of-word and TW-IDF: New Approach to Ad Hoc IR”, CIKM, 2013; F. Rousseau, M. Vazirgiannis, “Main Core Retention on Graph-of-Words for Single-Document Keyword Extraction”, ECIR, 2015.
• A. J.-P. Tixier, F. Malliaros, M. Vazirgiannis, “A Graph Degeneracy-based Approach to Keyword Extraction”, EMNLP, 2016.
• S. Seidman, “Network Structure and Minimum Degree”, Social Networks, 1983; V. Batagelj, M. Zaversnik, “An O(m) Algorithm for Cores Decomposition of Networks”, arXiv, 2003.
• K. Borgwardt, H.-P. Kriegel, “Shortest-Path Kernels on Graphs”, ICDM, 2005; G. Nikolentzos, P. Meladianos, F. Rousseau, Y. Stavrakas, M. Vazirgiannis, “Shortest-Path Graph Kernels for Document Similarity”, EMNLP, 2017.
• G. Nikolentzos, P. Meladianos, S. Limnios, M. Vazirgiannis, “A Degeneracy Framework for Graph Similarity”, IJCAI, 2018; G. Nikolentzos, A. J.-P. Tixier, M. Vazirgiannis, “Message Passing Attention Networks for Document Understanding”, AAAI, 2020.
• R. Mihalcea, P. Tarau, “TextRank: Bringing Order into Texts”, EMNLP, 2004.

Chapters 4–7: sequence models, Transformers and pretraining.

• Y. Bengio, P. Simard, P. Frasconi, “Learning Long-Term Dependencies with Gradient Descent is Difficult”, IEEE Transactions on Neural Networks, 1994; R. Pascanu, T. Mikolov, Y. Bengio, “On the Difficulty of Training Recurrent Neural Networks”, ICML, 2013.
• S. Hochreiter, J. Schmidhuber, “Long Short-Term Memory”, Neural Computation, 1997; F. Gers, J. Schmidhuber, F. Cummins, “Learning to Forget: Continual Prediction with LSTM”, Neural Computation, 2000; R. Jozefowicz, W. Zaremba, I. Sutskever, “An Empirical Exploration of Recurrent Network Architectures”, ICML, 2015.
• K. Cho and coauthors, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation”, EMNLP, 2014; I. Sutskever, O. Vinyals, Q. Le, “Sequence to Sequence Learning with Neural Networks”, NeurIPS, 2014.
• Y. Kim, “Convolutional Neural Networks for Sentence Classification”, EMNLP, 2014; M. Schuster, K. Paliwal, “Bidirectional Recurrent Neural Networks”, IEEE Transactions on Signal Processing, 1997.
• S. Jain, B. Wallace, “Attention is not Explanation”, NAACL, 2019; S. Wiegreffe, Y. Pinter, “Attention is not not Explanation”, EMNLP, 2019.
• A. Vaswani and coauthors, “Attention Is All You Need”, NeurIPS, 2017.
• J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, NAACL, 2019.
• M. Lewis and coauthors, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”, ACL, 2020; C. Raffel and coauthors, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, JMLR, 2020.
• A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, “Language Models are Unsupervised Multitask Learners”, OpenAI technical report, 2019; T. Brown and coauthors, “Language Models are Few-Shot Learners”, NeurIPS, 2020.
• Y. Liu and coauthors, “Multilingual Denoising Pre-training for Neural Machine Translation”, TACL, 2020; L. Martin and coauthors, “CamemBERT: a Tasty French Language Model”, ACL, 2020; H. Le and coauthors, “FlauBERT: Unsupervised Language Model Pre-training for French”, LREC, 2020.

Chapters 8–10: scale, evaluation and adaptation.

• I. Loshchilov, F. Hutter, “Decoupled Weight Decay Regularization”, ICLR, 2019.
• J. Kaplan and coauthors, “Scaling Laws for Neural Language Models”, arXiv, 2020; T. Besiroglu, E. Erdil, M. Barnett, J. You, “Chinchilla Scaling: A Replication Attempt”, arXiv, 2024.
• A. Petrov, E. La Malfa, P. Torr, A. Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS, 2023; P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, I. Gurevych, “How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models”, ACL, 2021.
• K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation”, ACL, 2002; C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries”, ACL Workshop Text Summarization Branches Out, 2004.
• T. Zhang, V. Kishore, F. Wu, K. Weinberger, Y. Artzi, “BERTScore: Evaluating Text Generation with BERT”, ICLR, 2020.
• G. Hinton, O. Vinyals, J. Dean, “Distilling the Knowledge in a Neural Network”, NeurIPS Deep Learning Workshop, 2015; V. Sanh, L. Debut, J. Chaumond, T. Wolf, “DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter”, NeurIPS EMC² Workshop, 2019.
• D. Hendrycks and coauthors, “Measuring Massive Multitask Language Understanding”, ICLR, 2021; S. Gururangan and coauthors, “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks”, ACL, 2020.
• J. Wei and coauthors, “Finetuned Language Models Are Zero-Shot Learners”, ICLR, 2022; N. Houlsby and coauthors, “Parameter-Efficient Transfer Learning for NLP”, ICML, 2019; B. Lester, R. Al-Rfou, N. Constant, “The Power of Scale for Parameter-Efficient Prompt Tuning”, EMNLP, 2021.
• A. Aghajanyan, L. Zettlemoyer, S. Gupta, “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning”, ACL, 2021; J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, G. Neubig, “Towards a Unified View of Parameter-Efficient Transfer Learning”, ICLR, 2022.

Chapters 11–13: alignment, retrieval and efficient inference.

• R. A. Bradley, M. E. Terry, “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons”, Biometrika, 1952.
• P. Christiano and coauthors, “Deep Reinforcement Learning from Human Preferences”, NeurIPS, 2017; L. Ouyang and coauthors, “Training Language Models to Follow Instructions with Human Feedback”, NeurIPS, 2022.
• Y. Bai and coauthors, “Constitutional AI: Harmlessness from AI Feedback”, arXiv, 2022; L. Gao, J. Schulman, J. Hilton, “Scaling Laws for Reward Model Overoptimization”, ICML, 2023.
• S. Robertson, H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, Foundations and Trends in Information Retrieval, 2009; J. Johnson, M. Douze, H. Jégou, “Billion-Scale Similarity Search with GPUs”, IEEE Transactions on Big Data, 2019.
• P. Lewis and coauthors, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS, 2020; Z. Ji and coauthors, “Survey of Hallucination in Natural Language Generation”, ACM Computing Surveys, 2023; N. Liu and coauthors, “Lost in the Middle: How Language Models Use Long Contexts”, TACL, 2024.
• J. Gilmer, S. Schoenholz, P. Riley, O. Vinyals, G. Dahl, “Neural Message Passing for Quantum Chemistry”, ICML, 2017; X. He and coauthors, “G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering”, NeurIPS, 2024.
• T. Yu and coauthors, “Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task”, EMNLP, 2018.
• D. Edge and coauthors, “From Local to Global: A Graph RAG Approach to Query-Focused Summarization”, arXiv, 2024; V. Traag, L. Waltman, N. J. van Eck, “From Louvain to Leiden: Guaranteeing Well-Connected Communities”, Scientific Reports, 2019.
• B. Hassibi, D. Stork, “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon”, NeurIPS, 1993; T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs”, NeurIPS, 2023.
• N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need”, arXiv, 2019; J. Ainslie and coauthors, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, EMNLP, 2023; W. Kwon and coauthors, “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP, 2023.
• E. Frantar, D. Alistarh, “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot”, ICML, 2023; M. Sun, Z. Liu, A. Bair, J. Z. Kolter, “A Simple and Effective Pruning Approach for Large Language Models”, ICLR, 2024.

Chapter 14 and projects.

• G. Shang and coauthors, “Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization”, ACL, 2018.
• T. Dao, D. Fu, S. Ermon, A. Rudra, C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”, NeurIPS, 2022; A. Gu, T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, arXiv, 2023.
• M. G. Azar and coauthors, “A General Theoretical Paradigm to Understand Learning from Human Preferences”, AISTATS, 2024; Z. Shao and coauthors, “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”, arXiv, 2024.
• L. Zheng and coauthors, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, NeurIPS Datasets and Benchmarks, 2023; S. Pan and coauthors, “Unifying Large Language Models and Knowledge Graphs: A Roadmap”, IEEE Transactions on Knowledge and Data Engineering, 2024.
• J. Chee, Y. Cai, V. Kuleshov, C. De Sa, “QuIP: 2-Bit Quantization of Large Language Models With Guarantees”, NeurIPS, 2023.
• A. Hulth, “Improved Automatic Keyword Extraction Given More Linguistic Knowledge”, EMNLP, 2003; S. N. Kim, O. Medelyan, M.-Y. Kan, T. Baldwin, “SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientific Articles”, SemEval, 2010.
• R. Socher and coauthors, “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”, EMNLP, 2013; S. Merity, C. Xiong, J. Bradbury, R. Socher, “Pointer Sentinel Mixture Models”, ICLR, 2017.
• W. Hu and coauthors, “Open Graph Benchmark: Datasets for Machine Learning on Graphs”, NeurIPS, 2020; Z. Yang and coauthors, “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”, EMNLP, 2018.
• K. M. Hermann and coauthors, “Teaching Machines to Read and Comprehend”, NeurIPS, 2015; I. Turc, M.-W. Chang, K. Lee, K. Toutanova, “Well-Read Students Learn Better: On the Importance of Pre-training Compact Models”, arXiv, 2019; A. Fabbri and coauthors, “SummEval: Re-evaluating Summarization Evaluation”, TACL, 2021.