LLGA · Course notes

From Boosting to Foundation Models: learning with Tabular Data

M2 · P1 (announced) · Year 2 · Period 1

Contents

Introduction and overview

Module From Boosting to Foundation Models: learning with Tabular Data
Code Not published — the module is announced in period 1 by the master’s brochure; the 2026-2027 SynapseS course description has not been published (the S3/S4 tracks of the catalog are currently empty).
Team and workload Instructor, teaching hours and ECTS: not published · why GBDTs still dominate most industrial data, what exactly XGBoost, LightGBM, and CatBoost do, how to encode, augment, and generate columns, and how foundation models (TabPFN, LLMs) move the needle — with an honest benchmark in the lab.
Period M2 · period 1 (announced)
Status in the program M2Outside recommended track M2 program being finalized by the course instructors; the official description will not be communicated until the course description is published.

Audience: a tabular / CRM / fraud engineer who still sees XGBoost beat deep nets. Empirical first: leakage via sloppy target encoding; LightGBM blazing fast but biased on rare categories — then we name GBDT, GOSS, CatBoost, TabPFN.

Why tables resist deep learning: boosting math → tabular foundations.

What you will be able to do

Official objectives and syllabus

Official syllabus: course description not published — this module appears neither in the S3 track nor in the S4 track of the 2026-2027 SynapseS catalog (both currently empty), and the master’s wiki indicates that the detailed M2 curriculum is still under development: description, teaching hours and instructor are therefore “not published”. The themes below are those carried by the module in the master’s brochure, as restated by these course materials, mapped to the sections that cover them.

Announced theme (brochure, as restated by the materials) Course section covering it
The dominance of GBDTs on tabular data ch. 1 “Why GBDTs (still) dominate tabular data”
Boosting as functional gradient descent ch. 2 “Boosting: functional gradient descent”
The engineering of XGBoost, LightGBM, CatBoost ch. 3 “XGBoost, LightGBM, CatBoost: three engineering approaches”
Encoding categorical and numerical columns ch. 4 “Encoding columns: categorical and numerical”
Deep tabular: regularized MLPs, FT-Transformer ch. 5 “Deep tabular: regularized MLPs and FT-Transformer”
Tabular foundation models: in-context TabPFN ch. 6 “TabPFN: learning tables in-context”
LLMs and tables: serialization, text-to-SQL ch. 7 “LLMs and tables: serialization, text-to-SQL”
Synthetic tabular data ch. 8 “Synthetic tabular data”
Honest benchmarking and decision guide ch. 9 “Honest benchmarking” and ch. 10 “Choosing your method: guide and open questions”

The exact split between brochure content and the materials’ own extensions can only be settled once the official course description is published; in any case, chapter 0 (reminders: trees, forests, boosting) is part of the common structure of the repository’s course materials.

Where this course fits in the program

M2 module offered in period 1, outside the recommended track, designed as an applied deepening accessible right after M1. Prerequisites: chapter 6 (trees, forests, boosting) of Machine Learning is the direct foundation — ch. 0 consolidates it before pushing the theory; the scikit-learn practice acquired in M1 suffices for the labs. The few SQL notions needed in ch. 7 are recalled there, since the Databases course is not on the LLGA list.

Downstream modules: data versioning, rigorous benchmarking, and computation costs of the ML Ops and LLM Engineering Principles module naturally extend this course when tables become pipelines; and for your internships as for your first jobs, it is tabular data you will handle — this is the module you will use the soonest outside the School.

Course outline

Session plan

24 h in class (assumed) = 8 sessions of 3 h. The SynapseS 2026–2027 catalogue publishes no sheet and no teaching hours for this module; 24 h is the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS) and is assumed here, and the session split below is our proposal for covering these notes in that time.

Level 1 = foundations (reminders, first results), 2 = core methods (boosting engines, encodings, deep tabular models), 3 = recent research and open questions (tabular foundation models, LLMs on tables, benchmarking controversies); the level never goes down from one session to the next.

Session Hours Level Content Practice
1 3 h 1 ch. 0 Reminders: trees, forests, boosting; ch. 1 Why GBDTs still dominate tabular data Ex. 3
2 3 h 2 ch. 2 Boosting as functional gradient descent, AdaBoost as its exponential-loss case; ch. 3.1 XGBoost: leaf weights and split gain Ex. 1
3 3 h 2 ch. 3.2–3.3 LightGBM (histograms, GOSS, EFB) and CatBoost (oblivious trees, ordered statistics) Ex. 4
4 3 h 2 ch. 4 Encoding categorical and numerical columns; target leakage Ex. 2 (central lab)
5 3 h 2 ch. 5 Deep tabular: regularized MLPs, FT-Transformer, TabNet, NODE Ex. 7 (start: GBDT and FT-Transformer baselines)
6 3 h 3 ch. 6 TabPFN: prior-fitted networks, the prior-data likelihood, in-context learning on tables Ex. 8
7 3 h 3 ch. 7 LLMs and tables: serialization, TabLLM, text-to-SQL and execution accuracy; ch. 8 Synthetic tabular data Ex. 5, 6
8 3 h 3 ch. 9 Honest benchmarking; ch. 10 Choosing your method, open questions Ex. 7 (conclusions and decision guide)

Self-study, outside the 24 h: none — every chapter is covered in class; the labs (Ex. 2 and 7 in particular) are the personal work expected alongside.

How to study this module

The LLGA perspective A master dedicated to LLMs and graphs might give the impression that tabular data is outdated; it is the opposite: it is the ground where foundation models have the most to prove, and where the industrial value of AI remains the most massive. By crossing the “statistics/trees” culture of data scientists with the “generative foundations” culture of LLGA, this module makes you able to choose the dominant tool in industry and to judge, without a manual, the next wave of tabular models.

Independent note Place in the two-year path: like the M1 modules, this course starts from concrete failure modes, then names the methods and proves their formulas.

0. Reminders: trees, forests, boosting

This module studies the models that work best on tables, starting from the tree ensembles of M1. This chapter lists what is assumed from that course and fixes the notation used afterwards.

Reminders from the Machine Learning module (M1 P1, ch. 4 and 6) — what is assumed acquired

  • CART trees: recursive partitions of the space by univariate thresholds x_j \leq t, splits chosen by maximizing the impurity decrease (Gini/variance), depth = control of the bias-variance trade-off;
  • Bias-variance decomposition: \mathrm{Err} = \mathrm{bias}^2 + \mathrm{variance} + \mathrm{noise}, and the role of ensembles: bagging and random forests reduce variance by averaging decorrelated trees;
  • Gradient boosting as functional gradient descent (ML section 6.3, Theorem 6.4): at each iteration a small tree is fit by regression to the pseudo-residuals (minus the derivative of the loss in the prediction), then added with a step \eta > 0. The quadratic loss gives the plain residuals, the logistic loss gives y_i - p_i, and early stopping on the number of trees is the main regularization;
  • Not assumed: AdaBoost, which M1 does not present — ch. 2 defines it and derives it as the exponential-loss case of the same descent;
  • Evaluation (ch. 4): cross-validation, data leakage, benchmark protocols — we return to this in ch. 9 of the present module.

Definition 0.1 — Notation of the module A tabular dataset is a matrix X \in \mathbb{R}^{n \times p} whose columns (features) are heterogeneous: numerical (price, age, log-ratio), categorical (country, product code), ordinal, binary, dates, sometimes textual; the target is y \in \mathcal{Y} (classification) or y \in \mathbb{R} (regression).

We write F for a fitted model (for boosting, the ensemble F_M), f_m for a single tree, \ell(\hat{y}, y) for the loss — prediction first, as in the Machine Learning module — and \mathcal{D} = \{(x_i, y_i)\}_{i=1}^n for the sample. A calligraphic \mathcal{L} always denotes an objective that is minimized.

The number of features is p rather than the site-wide d, because on this page d already denotes a tree depth (ch. 2–3) and an embedding or token dimension (ch. 4–5); from ch. 3 on, g_i and h_i are the first and second derivatives of the loss at example i.

A single row can contain scales ranging from 10^{-3} to 10^{7} and massively skewed columns — it is this heterogeneity that structures the whole module.

The module’s storyline M1 already derived boosting as functional gradient descent, with logistic pseudo-residuals and early stopping (Machine Learning section 6.3, Theorem 6.4). This module starts from that result and makes boosting a central object of study:

  1. we recall the first-order derivation briefly, add AdaBoost as a special case, and prove the second-order formulas of the industrial engines (optimal leaf weights, split gain, GOSS, ordered statistics);
  2. we explain why they win on tables, based on the large empirical studies of 2021-2025;
  3. we explore the two deep counter-attacks — attention over features (ch. 5) and tabular foundations (TabPFN, ch. 6) or linguistic ones (ch. 7);
  4. we finish with synthetic generation (ch. 8) and an honest comparison protocol (ch. 9), extending ch. 4 of the Machine Learning module.

1. Why GBDTs (still) dominate tabular data

Concrete scene: a leaderboard is won… thanks to temporal leakage. This chapter explains, with three large empirical studies and one simple invariance property, why gradient-boosted trees still dominate on tables.

What breaks In-fold target encoding; ignoring trees’ monotone invariance; comparing deep vs GBDT without a feature budget.

Images, text, speech, and graphs have been conquered by deep networks. Tables — which remain the majority format of enterprise data (CRM, sensors, finance, health, logs) — resist. This finding was established rigorously by three major empirical studies, which serve as the module’s guiding thread.

Definition 1.1 — GBDT (extended reminder) A gradient boosted decision tree is an additive model F_M(x) = \sum_{m=1}^{M} \eta\, f_m(x), where each f_m is a shallow regression tree fit on the negative of the gradient of the loss with respect to the current prediction (ch. 2). The structural hyperparameters: number of trees M, learning rate \eta, maximum depth (≈ order of interaction), subsampling rate (rows/columns), leaf regularization.

Study 1.2 — Grinsztajn, Oyallon & Varoquaux (NeurIPS 2022) “Why do tree-based models still outperform deep learning on typical tabular data?”: over dozens of medium-sized OpenML datasets (up to \sim\!10,000 rows, regression and classification), tree-based models (XGBoost, scikit-learn GBDT) systematically beat deep tabular networks, even after reasonable tuning. The paper isolates two mechanistic causes:

  • networks are sensitive to uninformative features: removing as many as possible helps networks far more than trees, which naturally “isolate” useful columns (a tree can simply never split on a useless column);
  • networks struggle with irregular target functions: neither smooth nor monotone, with “steps” and local interactions — the natural terrain of threshold splits.

A random rotation experiment on the columns confirms the diagnosis: after rotation, trees degrade sharply (their axis-aligned splits lose their meaning) whereas networks — rotation-invariant by construction — fare better: the axis-aligned structure of real tables is information that trees exploit and MLPs ignore.

Proposition 1.3 — Trees are invariant under monotone transformation of columns Let f be a tree and, for each column j, g_j : \mathbb{R} \to \mathbb{R} a strictly increasing function. If \tilde{x}_j = g_j(x_j), then the tree trained on the transformed data induces the same partition of the space (up to the thresholds) and makes the same predictions. In particular, log-transforming, standardizing or rank-transforming a skewed column — each a strictly increasing map applied column by column — changes nothing for a GBDT, up to threshold placement and binning (ch. 3).

Two common transforms are not covered. Clipping is only non-decreasing: it merges the clipped tail into one value, so the splits inside that tail disappear. Whitening (x \mapsto \Sigma^{-1/2} x) mixes columns, so it is not column-wise at all: it breaks the invariance, as the rotation experiment of study 1.2 shows.

Proof. A split x_j \leq t partitions the examples into \{i : x_{ij} \leq t\} and its complement. Since g_j is strictly increasing, x_{ij} \leq t \iff g_j(x_{ij}) \leq g_j(t) : every split on x_j is a split on \tilde{x}_j, with the image threshold g_j(t).

By recursion over the depth, the partitions coincide, so the leaves receive the same examples and the predictions (leaf means/majorities) are identical. ∎

Why this proposition matters It explains a robust empirical result: GBDTs win especially when features have skewed, heavy, multimodal distributions — precisely the cases where a network, which adds and multiplies activations, suffers from scales and distribution tails, whereas a tree only looks at the order of values.

McElfresh et al. (TabZilla, NeurIPS 2023: When do neural nets outperform boosted trees on tabular data?) quantifies this over 176 datasets and 19 algorithms: GBDTs handle skewed or heavy-tailed features much better than networks. It also finds that, on many datasets, the gap between the two families is negligible or smaller than the effect of lightly tuning the GBDT — tuning matters more than the choice between a GBDT and a network.

Example 1.4 — The 2021-2023 debate in three studies

  1. Shwartz-Ziv & Armon (Tabular data: Deep learning is not all you need, Information Fusion 2022): XGBoost beats deep models on most of their datasets, and tuning does not close the gap; GBDT+network ensembles = the best of both.
  2. Grinsztajn et al. (2022): the gap is real at medium sizes and stems from inductive biases (study 1.2).
  3. McElfresh et al. (TabZilla, 2023): on the largest benchmark then available, no family dominates everywhere; light tuning of a GBDT matters more than the choice between a GBDT and a network; and the best average performance goes to TabPFN (v1, ch. 6), even with its training sets subsampled to 3,000 rows.

Moral: the question is not “which is best” but “in which regime” — the subject of ch. 10.

2. Boosting: functional gradient descent

Boosting is most cleanly written as gradient descent in the space of functions — a view due to Friedman (2001). The Machine Learning module derived it in section 6.3 (Theorem 6.4); this chapter recalls it in the notation of the page, then shows that AdaBoost (Freund & Schapire, 1997), which M1 does not cover, is the same descent for the exponential loss.

Definition 2.1 — Functional gradient descent (recall of ML Theorem 6.4) We look for F^\star = \arg\min_F \sum_i \ell(F(x_i), y_i) in a space of functions. The “gradient at point x_i” is the variable

Key result — the pseudo-residual r_i = -\left[\dfrac{\partial \ell(F(x_i), y_i)}{\partial F(x_i)}\right]_{F = F_{m-1}} .

The boosting iteration:

  1. compute the r_i;
  2. fit a weak learner f_m (shallow tree) by least squares to the r_i;
  3. update F_m = F_{m-1} + \eta\, f_m .

Proposition 2.2 — Residuals for the quadratic loss For \ell(\hat{y}, y) = \tfrac{1}{2}(\hat{y} - y)^2, one has exactly r_i = y_i - F_{m-1}(x_i): “fitting on the residuals” is a special case of definition 2.1.

Proof. \partial \ell / \partial \hat{y} = \hat{y} - y, so -\partial \ell(F(x_i), y_i)/\partial F = y_i - F(x_i). ∎

The role of each ingredient

  • \eta (shrinkage): a small step ⇒ each tree corrects what the previous ones left behind — equivalent to a regularization of the descent step, to be combined with more trees.
  • Subsampling (Friedman, 2002): fitting each tree on a fraction of the rows injects bagging into the descent and decorrelates the trees (direct link with the M1 forests).
  • Limited depth: a tree of depth d models only order-d interactions — boosting learns a sum of “simple piecewise” functions, which matches the irregularity of tabular targets (study 1.2).
  • Early stopping: M chosen on validation — the analogue of truncating an expansion.

Example 2.3 — One iteration by hand Regression, quadratic loss, current model F_0 \equiv \bar{y}.

  1. Residuals r_i = y_i - \bar{y}.
  2. A depth-1 tree fit to the residuals chooses the column j^\star and the threshold t^\star minimizing the within-leaf variance of the r_i — that is, best explaining what the constant model could not say.
  3. New prediction F_1(x) = \bar{y} + \eta\,(c_1\, \mathbf{1}[x_{j^\star} \leq t^\star] + c_2\,\mathbf{1}[x_{j^\star} > t^\star]), where c_1, c_2 are the per-leaf residual means.

The following iterations attack the residuals of F_1, and so on.

AdaBoost came first, as a reweighting heuristic for classifiers; it is worth seeing why it is a functional gradient method in disguise.

Definition 2.4 — AdaBoost (Freund & Schapire, 1997) Binary labels y_i \in \{-1, +1\}, weak classifiers with values in \{-1, +1\}, initial weights w_i = 1/n. For m = 1, \dots, M:

  1. fit f_m to minimize the weighted error \mathrm{err}_m = \frac{\sum_i w_i\, \mathbf{1}[f_m(x_i) \neq y_i]}{\sum_i w_i} ;
  2. set the vote weight \alpha_m = \tfrac{1}{2}\log\frac{1 - \mathrm{err}_m}{\mathrm{err}_m} ;
  3. reweight w_i \leftarrow w_i\, e^{-\alpha_m y_i f_m(x_i)}, so misclassified examples gain weight.

The prediction is the weighted vote \mathrm{sign}(F_M(x)), with F_M = \sum_m \alpha_m f_m. Freund and Schapire’s original vote weight is 2\alpha_m: it gives the same vote and, after normalization, the same weights.

Proposition 2.5 — AdaBoost is stagewise descent on the exponential loss (Friedman, Hastie & Tibshirani, 2000) Take the exponential loss \ell(\hat{y}, y) = e^{-y \hat{y}} and the weights w_i = e^{-y_i F_{m-1}(x_i)}. Over classifiers f with values in \{-1,+1\} and steps \alpha > 0, the minimizer of \sum_i \ell\big(F_{m-1}(x_i) + \alpha f(x_i),\, y_i\big) is the pair (f_m, \alpha_m) of definition 2.4, as long as the best weighted error is below 1/2. The weights of the next round are e^{-y_i F_m(x_i)}.

Proof.

  1. Factor the current model out: the objective equals \sum_i w_i\, e^{-\alpha y_i f(x_i)}.
  2. Since y_i f(x_i) = \pm 1, with W = \sum_i w_i it equals W e^{-\alpha} + (e^{\alpha} - e^{-\alpha}) \sum_i w_i \mathbf{1}[f(x_i) \neq y_i] . For \alpha > 0 the bracket is positive, so the best f minimizes the weighted error.
  3. Dividing by W leaves e^{-\alpha}(1 - \mathrm{err}) + e^{\alpha}\,\mathrm{err}, whose derivative vanishes when e^{2\alpha} = (1 - \mathrm{err})/\mathrm{err}, which gives \alpha_m.
  4. Finally e^{-y_i F_m(x_i)} = w_i\, e^{-\alpha_m y_i f_m(x_i)}: the reweighting rule. ∎

The link with definition 2.1 is direct: the pseudo-residual of the exponential loss is r_i = y_i\, e^{-y_i F_{m-1}(x_i)} = y_i w_i. The AdaBoost weights are the sizes of the functional gradient, and the next classifier follows its sign.

What ch. 3 adds Definition 2.1 says what to fit; XGBoost, LightGBM, and CatBoost optimize how: closed-form leaf weights via a second-order Taylor expansion, histogram-based splits, large-gradient sampling, symmetric trees, ordered statistics for categoricals. All choices that take the GBDT from “nice idea” to industrial default on tables.

3. XGBoost, LightGBM, CatBoost: three engineering approaches to gradient boosting

Chapter 2 says what each new tree must fit; the three industrial libraries differ in how they fit it fast and without overfitting. This chapter derives the formulas each one relies on — second-order leaf weights, histogram splits and gradient-based sampling, ordered statistics for categories — and compares the resulting biases.

3.1. XGBoost: the second derivative and explicit regularization

Definition 3.1 — XGBoost’s regularized objective At iteration m, write \hat{y}_i = F_{m-1}(x_i) for the current prediction and g_i = \partial_{\hat{y}}\, \ell(\hat{y}_i, y_i) \quad\text{and}\quad h_i = \partial^2_{\hat{y}}\, \ell(\hat{y}_i, y_i) for the first and second derivatives of the loss in the prediction (h_i is a curvature, not a tree). One minimizes \sum_{i=1}^{n}\Big[ g_i\, f_m(x_i) + \tfrac{1}{2} h_i\, f_m(x_i)^2 \Big] + \Omega(f_m), \qquad \Omega(f) = \gamma\, T + \tfrac{\lambda}{2}\sum_{j=1}^{T} w_j^2, where f_m is a tree with T leaves of weights w_1, \dots, w_T.

This is the second-order Taylor expansion of the loss, exact for the quadratic loss, very good for the log-loss.

Theorem 3.2 — Optimal weights and split gain (XGBoost) For a fixed tree, write G_j = \sum_{i \in \text{leaf } j} g_i and H_j = \sum_{i \in \text{leaf } j} h_i. Then w_j^\star = -\frac{G_j}{H_j + \lambda}, \qquad \mathcal{L}^\star = -\frac{1}{2}\sum_{j=1}^{T}\frac{G_j^2}{H_j + \lambda} + \gamma T, and splitting a parent node P into a left and a right child improves the objective by

Key result — XGBoost split gain \mathrm{Gain} = \frac{1}{2}\left[\frac{G_{\mathrm{left}}^2}{H_{\mathrm{left}} + \lambda} + \frac{G_{\mathrm{right}}^2}{H_{\mathrm{right}} + \lambda} - \frac{(G_{\mathrm{left}} + G_{\mathrm{right}})^2}{H_{\mathrm{left}} + H_{\mathrm{right}} + \lambda}\right] - \gamma.

Proof. The objective factorizes per leaf: \sum_j [\,w_j G_j + \tfrac{1}{2} w_j^2 (H_j + \lambda)\,] + \gamma T (the \lambda completes the square). Each term is a trinomial in w_j with minimum w_j^\star = -G_j/(H_j + \lambda), \quad \text{of value} \quad -\tfrac{1}{2}G_j^2/(H_j + \lambda). Hence \mathcal{L}^\star.

A split replaces leaf P (sums G_P = G_{\mathrm{left}} + G_{\mathrm{right}}, and likewise H_P) with two leaves: the gain is the drop of \mathcal{L}^\star from the parent to the split, i.e. the stated expression; one splits only if Gain > 0, i.e. beyond the “price” \gamma. ∎

The other pieces of XGBoost

  • Weighted quantile sketch: proposing split thresholds via quantiles weighted by the h_i (points with high curvature count more), with an approximation guarantee.
  • Sparsity-aware: each node learns a “default direction” for missing values.
  • Column/row sampling: random-forest-style bagging applied to boosting.

The whole made XGBoost (2016) the reference engine of tabular competitions from 2015 onward.

3.2. LightGBM: histograms, GOSS, EFB

Definition 3.3 — Histogram-based splits Each numerical feature is pre-discretized into B bins (typically 255); the (g_i, h_i) are accumulated per bin, and the candidate thresholds are the bin boundaries. Cost per feature: O(n) to build the histogram, instead of O(n \log n) for sorting; in memory, the child histogram is derived from the parent by subtraction. The approximation error is bounded by the bin width — imperceptible in practice.

Definition 3.4 — GOSS (Gradient-based One-Side Sampling) Examples with small gradients are well predicted; one keeps the a \cdot n examples with the largest |g_i| (set A), randomly samples b \cdot n examples from the rest (B'), and amplifies their gradients by a factor (1-a)/b. The split gain is computed on this weighted selection.

Proposition 3.5 — GOSS (approximately) preserves the gain With the weights \tilde{g}_i = \frac{1-a}{b}\, g_i on B', the estimator \tilde{G}_j = \sum_{i \in A \cap j} g_i + \sum_{i \in B' \cap j} \tilde{g}_i is an unbiased estimator of G_j = \sum_{i \in j} g_i, and likewise for the H_j — hence the gain of theorem 3.2 is estimated consistently.

Proof. Write G_j = G_j^{A} + G_j^{\bar{A}}. Uniform sampling of B' in \bar{A} gives \mathbb{E}[\sum_{i \in B' \cap j} \tilde{g}_i] = \frac{1-a}{b}\, \mathbb{E}[\sum_{i \in B' \cap j} g_i] = \frac{1-a}{b} \cdot \frac{b}{1-a} G_j^{\bar{A}} = G_j^{\bar{A}}, hence the expectation G_j^{A} + G_j^{\bar{A}} = G_j.

The rest follows by replacing the sums with their estimators in the gain formula (second-order bias, negligible compared to the gain itself). ∎

Definition 3.6 — EFB (Exclusive Feature Bundling) In sparse data (massive one-hot), two columns are rarely nonzero simultaneously: they can share a “merged” bin. One builds a conflict graph (edge if frequent co-nonzero), greedily colors it to form groups of exclusive features; each bundle becomes a single histogram column. Combined with GOSS, this is the pair that gives LightGBM its speed gains.

3.3. CatBoost: oblivious trees and ordered statistics

Definition 3.7 — Oblivious trees An oblivious tree applies the same split to all nodes of a given level: depth d gives exactly d tests (j_1, t_1), \dots, (j_d, t_d) and the result is indexed like an array of size 2^d. Advantages: evaluation in O(d), trivially vectorized execution, and strong regularization (higher bias, lower variance — the same logic as the limited depth of ch. 2).

Definition 3.8 — Ordered target statistics For a categorical c and a random permutation \sigma of the rows, the encoded value of row i uses only earlier rows: \hat{x}_i = \frac{\sum_{\sigma(k) < \sigma(i)} \mathbf{1}[c_k = c_i]\, y_k + \kappa\, \bar{y}}{\sum_{\sigma(k) < \sigma(i)} \mathbf{1}[c_k = c_i] + \kappa}, with \bar{y} a global prior (the target mean) and \kappa a smoothing weight — the pseudo-count of def. 4.1. This is target encoding (ch. 4) without leakage: row i is never encoded with its own target. Ordered boosting generalizes the idea to residuals: each example is scored only by models that have not seen it — correcting the “prediction shift” of classical GBDTs (Prokhorenkova et al., 2018).

XGBoost LightGBM CatBoost
Split histogram (hist, the default since v2.0; exact and approx remain), 2nd order histogram, leaf by leaf (leaf-wise) oblivious, level by level
Acceleration quantile sketch, sparsity-aware GOSS + EFB (defs 3.4-3.6) symmetric trees, native GPU
Categoricals native partition-based splits (enable_categorical, since v1.6) or one-hot partition of categories by gradient statistics native ordered TS (def. 3.8)
Inductive bias regularization \gamma, \lambda leaf-wise growth (beware overfitting on small n) anti-leakage + symmetry 
# The same problem seen by the three engines (nearly identical APIs)
import xgboost as xgb, lightgbm as lgb, catboost as cb
from sklearn.model_selection import train_test_split
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=0.2, random_state=0)

m1 = xgb.XGBClassifier(n_estimators=2000, learning_rate=0.02, max_depth=6,
                       early_stopping_rounds=100).fit(Xtr, ytr, eval_set=[(Xva, yva)])
m2 = lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.02, num_leaves=63,
                        subsample=0.8, subsample_freq=1,  # bagging is off unless freq > 0
                        colsample_bytree=0.8).fit(
                        Xtr, ytr, eval_set=[(Xva, yva)],
                        callbacks=[lgb.early_stopping(100)])
m3 = cb.CatBoostClassifier(iterations=2000, learning_rate=0.02, depth=8,
                           cat_features=cat_idx, early_stopping_rounds=100).fit(
                           Xtr, ytr, eval_set=(Xva, yva), verbose=False)

4. Encoding columns: categorical and numerical

On tables, a massive share of the performance is decided before the model: GBDTs barely digest massive one-hots, networks require clean numerical inputs. This chapter surveys the encodings, with the central pitfall: target leakage.

Definition 4.1 — Target encoding (mean encoding) For a high-cardinality categorical c (postal codes, SKUs, users), one replaces the category with a statistic of the target; smoothed version of Micci-Barreca (2001): \mathrm{enc}(v) = \frac{n_v\, \bar{y}_v + \kappa\, \bar{y}}{n_v + \kappa}, where \bar{y}_v is the target mean within category v, n_v its count, \bar{y} the global mean and \kappa a pseudo-count: rare categories are pulled toward the global prior.

Example 4.2 — Leakage, demonstrated Suppose a column “identifier” unique per row (n_v = 1 everywhere) and a target independent of everything (pure noise). Naive target encoding (computed on the whole train set, row included) gives \mathrm{enc}(v_i) = y_i: the feature contains exactly the target. Any model reaches a training AUC of 1.0, while the test AUC falls back to 0.5. The model has learned nothing: it copies. With intermediate cardinality, the leakage is partial (\mathrm{enc}(v) mixes y_i and \bar{y}_v) — even more dangerous, because invisible on naive metrics.

Proposition 4.3 — Anti-leakage: out-of-fold and random order Two standard repairs: (i) K-fold out-of-fold: split the train set into K blocks, encode each block with the statistics of the other K-1; (ii) random order (CatBoost, def. 3.8): each row encoded using only earlier rows. In both cases, \mathrm{enc}(v_i) becomes independent of y_i conditionally on the category — the leakage disappears at the cost of a slight estimation noise, absorbed by the smoothing \kappa.

Definition 4.4 — One-hot vs embeddings One-hot: K binary columns: no similarity assumption between categories, but dimension K (disastrous for large K, and cf. EFB).

Entity embeddings (Guo & Berkhahn, 2016): each category v receives a learned vector z_v \in \mathbb{R}^{d} (d \ll K), via a table embedded in the network: categories that behave similarly with respect to the target move closer — the embeddings of “Bavaria” and “Baden-Württemberg” (Rossmann competition) end up neighbors.

This is the natural bridge between categoricals and networks; GBDTs, for their part, prefer native target encoding or splits by category partitions.

Definition 4.5 — Numerical feature embeddings Three families to “textualize” a scalar in the eyes of a network (Gorishniy et al., NeurIPS 2022):

  • Linear: e_j = b_j + x_j w_j (the FT-Transformer tokenizer, ch. 5);
  • Periodic: e_j = [\sin(2\pi c_j x_j), \cos(2\pi c_j x_j)]_{c_j \sim \mathcal{N}(0, \sigma^2)}, parameters \sigma (or the c_j) learned — a kernel-type building block;
  • Piecewise (piecewise-linear binning): x_j \to interpolation coordinates over B bins defined by quantiles — a controlled return to the “histogram logic” of GBDTs.

Massive empirical result: a simple MLP with good numerical embeddings rivals sophisticated architectures.

Engineering, not magic Networks inherit the M1 reflexes: standardize per column (never globally), transform heavy distributions (rank/quantile), extract dates (day of week, holidays, \sin/\cos cyclicity of the hour), cross business columns (income/spending), and handle missing values by imputation + indicator (networks have no XGBoost-style default direction — GBDTs do; cf. ch. 3). Proposition 1.3 reminds us why all this only benefits networks: trees do not care.

Encoding Good cardinality Leakage? GBDT Networks
One-hot low no mediocre (sparse) well (dense layer)
Ordinal — no native CatBoost/LightGBM compatible false ordering — to avoid
Naive target encoding high yes (ex. 4.2) trap trap
OOF / ordered target high no excellent well
Entity embeddings high no (learned) via pre-training excellent (def. 4.4)

5. Deep tabular: regularized MLPs and FT-Transformer

Two deep answers to the dominance of trees: better regularize MLPs, and treat the columns as tokens of a transformer — attention playing the role of a differentiable interaction engine.

Definition 5.1 — Feature tokenizer (FT-Transformer) Each column becomes a token of dimension d: for a numerical one, e_j = b_j + x_j\, w_j (b_j, w_j \in \mathbb{R}^d learned); for a categorical one, a row of an embedding table. To this one adds a [\mathrm{CLS}] token.

The following block is a standard transformer (multi-head attention, cf. Deep Learning P1 ch. 6 or Deep Learning P2 ch. 0): \mathrm{Att}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V, computed between features of the same row. The prediction is read from [\mathrm{CLS}] after n_{\mathrm{blocks}} transformer blocks.

Why feature-wise attention changes things An MLP must sample interactions via its weights; a tree builds them greedily; the transformer learns, for each pair of columns, a conditional interaction weight (the heads saturate on useful pairs and “switch off” useless columns — a partial answer to the sensitivity to uninformative features of study 1.2). Gorishniy et al. (NeurIPS 2021) show that the FT-Transformer is the most consistent tabular network, at an equal tuning budget — and that the tokenizer/def. 4.5 accounts for much of it.

Example 5.2 — Two other families

  • TabNet (Arik & Pfister, AAAI 2021): per-instance selection of a small group of features (sparse masks learned via a sparsemax) at each step — a differentiable echo of the trees’ “column selection”, with the masks offered as interpretability.
  • NODE (Popov et al., 2020): differentiable oblivious trees (entmax splits over thresholds) trained end-to-end — boosting rewritten as a network.
  • Regularized MLPs (Kadra et al., 2021): a well-tuned “cocktail” of regularizations suffices to hold the comparison — confirming that on tables, the essence of the game is played on inductive biases and tuning.

Peculiarities of tables (summary)

  • Heterogeneous columns: mixed types, scales, semantics — no “natural” convolution kernel;
  • No locality, no natural column order: permuting the pixels of an image destroys its local structure, whereas a fixed permutation of the columns changes nothing for an MLP or a tree. What breaks trees is mixing columns: the rotation experiment of ch. 1 shows that the axis-aligned meaning of each column is a richness that trees exploit and MLPs do not;
  • Irregular distributions: skewness, heavy tails, multimodality, “round” regulatory thresholds;
  • Many uninformative features and informative missing values;
  • Small/medium n by vision standards: the risk of overfitting the architecture is real (cf. TabZilla, ch. 9).

6. TabPFN: learning tables in-context

The 2023-2025 paradigm shift: no longer train a model on the table, but train a transformer once and for all on millions of synthetic tables, then predict in-context — the tabular analogue of LLM few-shot learning.

Definition 6.1 — Prior-fitted network (PFN) Let a prior \mathfrak{p} over datasets: a law over functions f (and noises, features, sizes) from which synthetic datasets \mathcal{D}_{sim} are drawn.

A PFN is a transformer q_\theta trained to predict, for an example x presented next to a set \mathcal{D}_{train} inside its input window, the law p(y \mid x, \mathcal{D}_{train}). Its training objective is the prior-data negative log-likelihood: draw a synthetic dataset, hold out one of its points (x, y), and score the prediction on it,

Key result — the PFN training objective \mathcal{L}_{\mathrm{PFN}}(\theta) = \mathbb{E}_{\mathcal{D} \cup \{(x, y)\} \sim \mathfrak{p}} \big[ -\log q_\theta(y \mid x, \mathcal{D}) \big] .

For classification this is a cross-entropy; for regression, q_\theta outputs a histogram over bins of y (the “bar distribution” of Müller et al.). The language-side counterpart — a model that picks up a task from demonstrations placed in its prompt — is defined in Large Language Models section 4.0.

Proposition 6.2 — The PFN objective targets the posterior predictive (Müller et al., ICLR 2022) Write p(\cdot \mid x, \mathcal{D}) for the Bayesian posterior predictive of the prior \mathfrak{p}, p(y \mid x, \mathcal{D}) = \int p(y \mid x, \phi)\, p(\phi \mid \mathcal{D})\, d\phi , where \phi indexes the data-generating functions of the prior. Then \mathcal{L}_{\mathrm{PFN}}(\theta) = \mathbb{E}_{x, \mathcal{D}}\Big[\mathrm{KL}\big(p(\cdot \mid x, \mathcal{D}) \,\|\, q_\theta(\cdot \mid x, \mathcal{D})\big)\Big] + \mathbb{E}_{x, \mathcal{D}}\Big[H\big(p(\cdot \mid x, \mathcal{D})\big)\Big] . The entropy term does not depend on \theta, so a rich enough q_\theta that minimizes the objective equals the posterior predictive: in-context learning is amortized Bayesian inference, done in a single forward pass.

Proof.

  1. Condition on (x, \mathcal{D}): under the prior, the held-out y follows p(\cdot \mid x, \mathcal{D}).
  2. Add and subtract \log p(y \mid x, \mathcal{D}) inside the expectation over y: \mathbb{E}_{y}[-\log q_\theta] = \mathbb{E}_{y}[-\log p] + \mathbb{E}_{y}\Big[\log \frac{p}{q_\theta}\Big] = H(p) + \mathrm{KL}(p \,\|\, q_\theta) .
  3. Take the expectation over (x, \mathcal{D}). The KL is non-negative and vanishes only when q_\theta = p. ∎

Definition 6.3 — TabPFN’s causal prior TabPFN (Hollmann et al., ICLR 2023; v2: Nature 2025) draws its synthetic tables from random structural causal models: random DAG over p variables, nonlinear mechanisms (networks, piecewise transformations), heterogeneous noises, then monotone marginal transformations — mimicking the “irregular distributions” of ch. 5 — and adding uninformative features and missing values.

By training on millions of such tables, the model learns to learn: in use, one deposits train + the row to predict in the context; no gradient descent is run on the user’s table.

Proposition 6.4 — What in-context buys

  1. Zero pipeline hyperparameters (no trees, rates, or depths): the “tuning” is amortized in pre-training;
  2. seconds of inference for tables up to \sim\!10,000 rows (v2), classification and regression, missing values and outliers tolerated;
  3. free uncertainty: q_\theta(y \mid x, \mathcal{D}) is a predictive law — usable for abstention.

Trade-offs: a context window (beyond \sim\!10,000 rows, fall back to “classical” training minibatches), limited types (no rich textual columns; cf. ch. 7), and a dependence on the proximity between the real table and the prior of def. 6.3.

Example 6.5 — A shifting benchmark TabPFN v1 already had the best average performance in TabZilla (2023), but for classification only and on training sets subsampled to 3,000 rows. Version 2 (Nature 2025) extends the model to regression and to tables of up to 10,000 rows and 500 features.

On the reference suites of medium-sized datasets, v2 beats tuned GBDTs on average — the strongest “deep” result in this regime so far, three years after Grinsztajn et al.

Ch. 9 nonetheless requires reading these results with a symmetric protocol (GBDT tuning budget, same folds, same seeds): the “trees vs foundations” debate remains open for large n, textual columns, and industrial constraints.

7. LLMs and tables: serialization, text-to-SQL

An LLM does not read a table: it reads text. Everything goes through a serialization — and this engineering choice is as decisive as the prompt itself.

Definition 7.1 — Usual serializations of a row For a row (x_1, \dots, x_p) with named columns c_1, \dots, c_p: CSV aligned on a header; JSON key-value; Markdown; textual template (“The patient is x_{age} years old, receives x_{dose} mg…”); the “schema” (column names, types, description) can precede the series.

Two critical properties: the column order influences the prediction (the model is sensitive to token positions), and the column names activate pre-training knowledge (“country”, “dosage”) — a semantic prior that no GBDT has.

Study 7.2 — TabLLM (Hegselmann et al., 2023) In few-shot (typically \leq 64 examples), an LLM (T0/GPT-3.5, textual serializations) predicts tables better than dedicated deep methods and rivals GBDTs; the advantage reverses as soon as data abounds — the per-row token cost becomes prohibitive and the semantic prior no longer compensates for the statistical capacity. The LLM wins where every example is expensive (medical annotation, industry); the GBDT where data is massive.

When the table lives in a relational database, the LLM’s job changes from predicting to querying. The few SQL notions this needs fit in six lines.

SQL in six lines (the Databases course is not on the LLGA list)

SELECT c.name, SUM(o.amount) AS revenue            -- columns to return, with aggregates
FROM clients AS c                                  -- a table (a relation), with an alias
JOIN orders AS o ON o.client_id = c.id             -- join: pair the rows whose keys match
WHERE o.year = 2025 AND o.status <> 'refund'       -- filter rows before grouping
GROUP BY c.id, c.name HAVING SUM(o.amount) > 1000  -- one row per client; filter the groups
ORDER BY revenue DESC LIMIT 5;                     -- sort, keep the first 5

Each table has a primary key (clients.id); a foreign key (orders.client_id) points to another table’s key, and a join follows it. A join along the wrong key, a join that duplicates rows before a SUM, or a GROUP BY on a non-unique column such as a name (two clients called Martin merge) returns a plausible but wrong number without any error.

Definition 7.3 — Text-to-SQL and schema linking When the table is a database, the LLM becomes an interface: producing SQL from a question in natural language.

Schema linking denotes the step of aligning question ↔︎ schema: entities of the question to cells of the tables, terms to candidate columns/tables, then plausible joins.

Benchmarks: Spider (Yu et al., 2018; clean schemas, academic questions) then BIRD (Li et al., 2023; 95 large “dirty” databases, dirty values, external knowledge, query efficiency requirement) — on which the performance drop of models between the two worlds is precisely measured.

Text-to-SQL is scored by running the queries, not by reading them.

Definition 7.4 — Execution accuracy For N questions with gold queries s_k^\star and generated queries \hat{s}_k, run both on the database \mathrm{DB}: \mathrm{EX} = \frac{1}{N}\sum_{k=1}^{N} \mathbf{1}\big[\mathrm{exec}(\hat{s}_k, \mathrm{DB}) = \mathrm{exec}(s_k^\star, \mathrm{DB})\big], where results are compared as multisets of rows, or as ordered lists when the question asks for an order.

EX accepts every correct rewrite, unlike exact-match accuracy, which compares SQL strings. It can also accept a wrong query that happens to return the right rows on this one database; test-suite accuracy (Zhong, Yu & Klein, 2020) reduces these false positives by running both queries on several database instances. BIRD adds a valid efficiency score that also rewards fast queries.

A text-to-SQL error taxonomy When auditing generated queries, classify each failure:

  1. schema linking: wrong table or column (the product’s name instead of the client’s);
  2. joins: a missing join, a wrong join path, or a join that duplicates rows and inflates a SUM;
  3. values and conditions: a literal that does not match the stored value (‘France’ vs ‘FR’), a wrong comparison or date format — BIRD’s “dirty values”;
  4. aggregation and grouping: a missing GROUP BY column, COUNT(*) instead of COUNT(DISTINCT ...), WHERE instead of HAVING;
  5. nesting and set operations: a missing subquery, NOT IN over a column with NULLs, a wrong EXCEPT;
  6. external knowledge: a business definition that the schema does not contain (“margin = price − cost”);
  7. invalid SQL: a syntax error or a column that does not exist.

An execution-feedback loop (run the query, return the error message to the LLM, regenerate) repairs class 7 and part of class 1. Classes 2–6 fail silently, so only EX on a labelled set reveals them.

# Three postures of the LLM facing a table
# 1) Predict: serialized row -> label (TabLLM, few-shot)
prompt = f"Column 'dosage_mg': {x['dose']}; 'age': {x['age']}; ...\nAnswer (0/1)?"
# 2) Engineer: propose features / clean / document the columns
#    (the LLM writes the feature engineering code, the GBDT predicts)
# 3) Query: natural language question -> SQL (schema linking + BIRD)
question = "top 5 clients by margin in 2025, excluding refunds"
# -> SELECT c.id, SUM(o.margin) FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT 5;

Structural limits Tokenization of numbers (scalars are chopped into pieces — fine arithmetic remains fragile); context window (millions of rows will never fit); confidentiality (sending customer rows to an API; cf. Explainability, Security and Privacy of LLMs); cost and non-determinism in production (cf. ML Ops ch. 4).

Hence the dominant posture: hybrid — the LLM prepares, summarizes, links, programs; the tabular model (GBDT/TabPFN) predicts.

8. Synthetic tabular data

Real tables are often too small, too unbalanced or too sensitive to be shared. This chapter defines what a good synthetic table must achieve, presents the three families of generators, and explains why synthetic data are not anonymous by default.

Definition 8.1 — The fidelity-utility-privacy triangle A tabular generator \mathcal{D}_{syn} \sim G(\mathcal{D}_{real}) is judged on three axes in tension:

  • fidelity (marginals and dependencies reproduced; KS/TV distances per column, correlations, detectability by a real-vs-synthetic classifier),
  • utility (does a model trained on the synthetic perform on the real? the train-synthetic-test-real protocol, TSTR), and
  • privacy (no synthetic row should copy an individual; nearest-neighbor distance metrics and membership inference attacks — cf. Privacy and Uncertainty ch. 5, or the shadow-model attack of Emerging ML ch. 5).

Example 8.2 — Three families of generators

  • GAN/VAE: CTGAN/TVAE (Xu et al., 2019) — “mode-specific” normalization of multimodal columns (a Gaussian mixture per column) and, for imbalanced categoricals, a conditional generator with training-by-sampling: each row of a batch is conditioned, through a conditional vector, on a category drawn in proportion to the log of its frequency, so rare categories are seen far more often than in the data.
  • Diffusion: TabDDPM (Kotelnikov et al., ICML 2023) — Gaussian diffusion for numericals, multinomial diffusion for categoricals; the paper also checks privacy through the distance to the closest real record. The diffusion foundation is in Multimodal Generative AI ch. 3, Deep Learning P2 section 7.3 or Deep Learning P1 section 9.4.
  • Language models: GReaT (Borisov et al., ICLR 2023) — one fine-tunes an LM on rows serialized as text with random column order; generating = sampling conditioned completions — the most faithful on dependencies, the most exposed to memorization (cf. Large Language Models section 9.7 on extracting memorized text).

Proposition 8.3 — Synthetic ≠ anonymous A generator trained by maximum likelihood memorizes: rare rows (atypical individuals) can be reproduced almost verbatim.

The formal safeguard is differential privacy (cf. Emerging ML ch. 5 or Privacy and Uncertainty ch. 1–4): training the generator in DP (Gaussian mechanism on gradients) guarantees an (\varepsilon, \delta) — at the price of a measurable fidelity loss.

In practice one combines: DP when the guarantee is required, membership inference audits and nearest-record distances otherwise, and documentation of both (regulation; cf. ML Ops).

Secondary uses Data augmentation to rebalance rare classes, pipeline testing (“twin datasets” for ML regression tests), cross-entity sharing (banks/suppliers without exchanging real rows), and — back to ch. 6 — synthetic generation is also the training technology of PFNs: TabPFN’s prior is nothing other than a controlled causal table generator. The loop is closed between “learning from synthetic” and “generating synthetic”.

9. Honest benchmarking

The “trees vs deep vs foundations” debate has produced more contradictory conclusions than any other area of ML — almost always because of asymmetric protocols. This chapter sets the method required in all the module’s labs (an extension of ch. 4 of the Machine Learning module).

Protocol rules

  1. Same folds, same preprocessing for everyone: seeded splits, encodings computed only inside the CV loop (the OOF target encoding of prop. 4.3 is inside the CV, never before);
  2. Symmetric tuning budget: same number of trials, same search, same selection metric — TabZilla finds that lightly tuning a GBDT often matters more than the family choice (ch. 1);
  3. Multiple seeds (≥ 5) and report mean ± standard deviation: on medium tables, the inter-seed std of a GBDT can approach the gap between families;
  4. Counted cost: training + inference + tuning time (TabPFN’s “zero-tuning” is compared to a tuned XGBoost — but also to default XGBoost);
  5. Dataset deduplication: OpenML suites share variants of the same dataset — a “win by average” benchmark may count the same dataset three times;
  6. Test reuse: forbidden — a test set that served to select a family no longer measures anything (benchmark overfitting).

Example 9.1 — Calibrating an “X% better” On 20 OpenML datasets: mean GBDT AUC 0.812 ± 0.004 (5 seeds), FT-Transformer 0.807 ± 0.006, TabPFN 0.818 ± 0.000 (deterministic). Before writing “TabPFN wins”:

  1. paired per-dataset test (permutation or corrected t; cf. Machine Learning section 4.9);
  2. check that the GBDT had its budget;
  3. look at the distribution — gaining 0.006 everywhere is not gaining 0.06 on half the datasets and losing on the other half;
  4. report the total time to prediction.

The honest conclusion is often “statistically tied, choose by cost and constraint” (ch. 10).

Where this protocol is used next The Year 2 transverse project Y2.5 “Fraud alerts you can defend” starts from a tuned GBDT on the Elliptic transaction data and asks which graph signals add value over it. Rules 1–6 apply as written, with one change: the folds become a strictly temporal split, since shuffling time steps leaks the future. Y2.4 applies the same protocol to gradient boosting for travel-time prediction.

10. Choosing your method: guide and open questions

The previous chapters compared model families regime by regime; this chapter condenses them into a default choice for each situation, then lists the questions that research has not yet settled.

Regime Recommended default Why
n \lesssim 10,000, no tuning resources TabPFN zero hyperparameters, seconds, uncertainty included (prop. 6.4)
Heterogeneous, skewed tables, medium/large n GBDT (XGBoost/LightGBM/CatBoost) monotone invariance (prop. 1.3), irregularity (study 1.2), cost
Very large n, multimodal columns (linked text/image), differentiable stack Tabular network (FT-Transformer) + embeddings shared representations, joint gradients (ch. 5)
Few-shot with semantic columns, free text LLM (prediction or feature engineering) pre-training prior (study 7.2)
Anonymization / sharing / rebalancing Synthetic generator + audit triangle of def. 8.1

Open questions 2026

  1. Scaling of tabular foundations: does pre-training on synthetic priors follow exploitable scaling laws, and how far can one go in n/p?
  2. Transfer across schemas: can a model trained “cold” on millions of tables specialize in-context to a new business schema?
  3. Theory of tabular in-context: what does the PFN exactly compute — a true Bayesian average, a kernel regression, something else?
  4. Benchmarks: deduplicated suites, counted cost, industrial constraints (drift, structured missing values); cf. ch. 9.

These questions feed the master’s internship/lab topics.

Exercises / Labs

  1. ★ Theorem 3.2 by hand (ch. 2-3). Implement in pure NumPy (without a library) a depth-1 tree boosting: residuals, leaf weights w_j^\star = -G_j/(H_j+\lambda), split gain — verify on a toy dataset that the objective of theorem 3.2 decreases by exactly the announced Gain at each split.
  2. ★ Target encoding leakage (ch. 4 — central lab). On a dataset where the target is pure noise: (i) naive target encoding with an identifier column — measure train/test AUC; (ii) intermediate cardinality (100 categories) — show that partial leakage skews standard CV; (iii) repair with K-fold OOF and Micci-Barreca smoothing (sweep \kappa); (iv) rerun CatBoost with native ordered TS and compare.
  3. ★ Rotation and useless features (ch. 1). Reproduce the two diagnostics of Grinsztajn et al. on 3 datasets: (i) random column rotation — GBDT vs MLP; (ii) adding noise features then importance-based pruning — who suffers most? Reread proposition 1.3 in light of the results.
  4. ★★ GOSS and EFB (ch. 3). In a toy GBDT: implement GOSS (a = 0.2, b = 0.1) and numerically verify proposition 3.5 (compare estimated gains with exact gains over 100 splits); build a sparse one-hot and measure the achievable EFB fusion rate (greedy coloring of the conflict graph).
  5. ★★ LLM on a table (ch. 7). Serialize a tabular dataset in 4 formats (CSV/JSON/Markdown/template): measure the effect of format and column order in few-shot (16/64 examples); compare with a GBDT at the same sample sizes — reproduce the crossover of study 7.2; finish with 10 text-to-SQL questions on a mini-database: report the execution accuracy (def. 7.4) and classify every failure with the error taxonomy of ch. 7.
  6. ★★ Synthetic and privacy (ch. 8). Generate data with CTGAN, TabDDPM, and a serialized LM (GReaT-like); evaluate fidelity (KS, correlations, detectability) and utility (TSTR vs TRTR); launch a membership inference attack and a nearest-record-distance analysis; redo the generator with light DP and plot the fidelity-privacy curve.
  7. ★★★ Symmetric benchmark (ch. 5-6-9). On 8-10 OpenML datasets (Grinsztajn suite): XGBoost/LightGBM/CatBoost with an equal tuning budget (Optuna, 100 trials), FT-Transformer (tokenizer + periodic embeddings), TabPFN. Same folds, 5 seeds, mean ± standard deviation, paired test, total time reported — write the “TabZilla-style” conclusion (ch. 9) then the decision guide (ch. 10) on these data.
  8. ★★★ Prior and in-context (ch. 6). Generate tables from random SCMs (def. 6.3 — DAG skeleton, piecewise mechanisms, marginal transformations); train a small PFN transformer to predict p(y \mid x, \mathcal{D}); show in-context learning (performance curve vs context size) and test sensitivity to the prior (Gaussian vs causal prior — when does the PFN fail?).

Projects and assessment

Official assessment: not displayed. This module has no published SynapseS 2026–2027 sheet: code, hours, credits, instructor and assessment are all “not published” (see the fiche above). The proposals below are ours: a project, solo or in a group, plus a written exam, weighted as the instructor decides. They exercise O1 (boosting, encodings and deep tabular models), O7 (honest benchmarks on real tables) and O3 (LLMs on tables, text-to-SQL); the fraud project also touches O4 and the mini-PFN project O8.

The fraud project rehearses the Year 2 transverse project Y2.5, and the ch. 9 protocol that every project follows is also the one Y2.4 uses for travel-time prediction.

The projects draw on Graph ML and Large Language Models; exam questions 2 and 3 take Machine Learning Theorem 6.4 to second order and to AdaBoost.

Solo projects

  1. [Solo · ≈ 20 h] Leakage forensics: an audit kit for target, temporal and group leakage. How much does each kind of leakage inflate a cross-validated score, and does the right split or encoding remove all of it?
    • Implement: a small Python library that injects and detects three leaks — naive versus out-of-fold versus ordered target encoding (Ex. 2), temporal leakage from shuffled folds on time-ordered rows (the opening scene of ch. 1), and group leakage (the same patient in train and test). The detector compares random-split scores with grouped or temporal-split scores.
    • Data: Amazon_employee_access (OpenML, high-cardinality categoricals); the Bike Sharing dataset (Fanaee-T & Gama, 2013, UCI, hourly rentals); Diabetes 130-US Hospitals (Strack et al., 2014, UCI, repeated encounters per patient).
    • Deliverables: the library with unit tests, a 4-page report, a 10-minute talk.
    • Chapters: ch. 1, 4, 9.
    • Success: for each leak and dataset, the inflation (leaky minus honest score) with a 5-seed interval; the inflation vanishes, within the seed spread, under the correct split or encoding; the detector flags every injected leak and none on clean controls.
  2. [Solo · ≈ 25 h] Paper study: a mini-PFN against the exact posterior (Müller et al., 2022). Does a small transformer trained on the prior-data likelihood really approach the Bayesian posterior predictive, as “Transformers Can Do Bayesian Inference” (ICLR 2022) reports?
    • Implement: a 4-layer PFN in PyTorch trained on the objective of def. 6.1 for regression tasks drawn from a Gaussian-process prior (RBF kernel, fixed hyperparameters), with a Gaussian or bar-distribution head; compare its test NLL with the closed-form GP posterior at context sizes 5 to 200. Then switch to an SCM prior (def. 6.3) for classification.
    • Data: synthetic priors; five OpenML-CC18 tables of at most 1,000 rows (credit-g, diabetes, blood-transfusion, ilpd, wdbc).
    • Deliverables: code, a 4-page report stating which claims were reproduced, a 10-minute talk.
    • Chapters: ch. 6 (prop. 6.2), ch. 9; Privacy and Uncertainty ch. 7 and ch. 9 for calibration and conformal prediction.
    • Success: the NLL gap to the exact GP posterior per context size, with 3 seeds, shrinking with training; on the real tables, the calibration error (ECE) of the PFN against that of a tuned GBDT, and the coverage and mean set size of split conformal prediction at 90 % built on each.

Group projects

  1. [Group of 4 · ≈ 30 h each] A symmetric tabular benchmark: GBDTs, FT-Transformer, TabPFN v2 and an LLM. In which regimes (rows, cardinality, skew, textual columns) does each family win once the protocol is symmetric?
    • Implement: Ex. 7 at full scale. XGBoost, LightGBM and CatBoost under an equal Optuna budget; FT-Transformer with periodic embeddings; TabPFN v2 on at most 10,000 training rows; a TabLLM-style few-shot LLM (Qwen2.5-7B-Instruct, 4-bit) at 16, 64 and 256 shots, scored on 500 test rows per dataset wherever the prompt fits its 32k-token context, prompted as in Large Language Models section 4.0.
    • Data: 15–20 datasets from the benchmark suites of Grinsztajn et al. (2022) on OpenML, deduplicated; one industrial table, Home Credit Default Risk (Kaggle, 2018; about 300,000 loans, over 100 columns).
    • Split: GBDTs and the tuning harness; deep tabular models; TabPFN and the LLM; protocol, statistics and deduplication.
    • Deliverables: a repository that reruns everything with one command, a 6-page TabZilla-style report, a 15-minute talk.
    • Chapters: ch. 3, 5, 6, 7, 9, 10; Machine Learning section 4.9 for the paired tests.
    • Success: every claimed win backed by a paired test across datasets (Wilcoxon signed-rank with Holm correction), 5 seeds, the total time to prediction; each row of the ch. 10 guide confirmed or amended with evidence.
  2. [Group of 3 · ≈ 30 h each] Graph-augmented fraud detection: from a GBDT on transaction features to GNN embeddings. Do graph signals add detection value over a tuned GBDT, and does the gain survive a regime shift? A dress rehearsal for Y2.5.
    • Implement: four set-ups under a strictly temporal protocol — a tuned GBDT on the transaction features; the same GBDT plus graph features (degrees, PageRank, node2vec embeddings from Graph ML ch. 3); an end-to-end GNN (Graph ML ch. 5); TabPFN on per-time-step slices.
    • Data: the Elliptic dataset (Weber et al., 2019): 203,769 Bitcoin transactions, 166 features, 49 time steps, about 2 % labelled illicit; train on steps 1–34, test on 35–49, as in the paper.
    • Split: GBDT and protocol; graph features and embeddings; GNN.
    • Deliverables: a repository, a 6-page report, a 15-minute talk.
    • Chapters: ch. 1, 3, 4, 6, 9; Advanced GNN ch. 6 for temporal graphs.
    • Success: illicit-class F1 and recall at a fixed alert budget per time step, 5 seeds; a paired test of each graph addition against the GBDT; the drop after the dark-market shutdown at step 43 measured and discussed.
  3. [Group of 3 · ≈ 25 h each] A text-to-SQL analyst over a dirty database. How far does an open LLM get on realistic databases, which errors remain, and does execution feedback help?
    • Implement: schema linking, SQL generation and an execution-feedback self-correction loop with Qwen2.5-Coder-7B-Instruct, in the ReAct style of Large Language Models section 8.1; a home-made SQLite CRM with dirty values (mixed country codes, NULLs, duplicates); a GBDT churn model exposed through the same natural-language interface (“the LLM prepares, the tabular model predicts”).
    • Data: 300 questions of the BIRD development set (Li et al., 2023); the IBM Telco Customer Churn sample (7,043 customers) for the CRM.
    • Split: schema linking and retrieval; generation and self-correction; the CRM, the churn model and the evaluation.
    • Deliverables: a repository with a one-command evaluation, a 6-page report, a live demo.
    • Chapters: ch. 3, 4, 7 (def. 7.4 and the error taxonomy), 9; Large Language Models section 7.4 for text-to-SQL as a structured-query variant of retrieval.
    • Success: execution accuracy with a bootstrap interval; the effect of self-correction measured by a paired McNemar test; an error table by class before and after correction; identical churn scores through the interface and the direct API.

Written exam (proposal)

Format: 2 hours; one handwritten A4 sheet allowed; no electronic device; 4 questions — a course question, a computation and two short proofs.

  1. Question (5 points). Explain why GBDTs beat MLPs on typical medium-sized tables. State the invariance of proposition 1.3 and its limits, give the two mechanistic causes identified by Grinsztajn et al. (2022), and say what a random rotation of the columns does to each family.

    Expected:

    • A strictly increasing map of a column sends every split x_j \le t to the split at threshold g_j(t), so partitions and predictions are unchanged: logs, standardization and ranks do not matter to a tree (2 pts).
    • Clipping merges the tail into one value; whitening mixes columns and breaks the invariance (1 pt).
    • Networks are hurt by uninformative features and by irregular, non-smooth targets (1 pt).
    • After a random rotation, trees lose their axis-aligned splits and degrade sharply, while MLPs, rotation-invariant by construction, change little (1 pt).

  2. Question (6 points). Six examples sorted by one feature have labels (1, 1, 1, 0, 0, 0). The current raw score is 0 for all of them; the loss is the logistic loss; \lambda = 1, \gamma = 0.5. (a) Give g_i and h_i (1 pt). (b) Compute the gain of the split between the third and fourth examples (2 pts). (c) Compute the gain of the split between the second and third, and say which split XGBoost makes (1.5 pts). (d) Give the two leaf weights, and the new probability on the left with \eta = 0.3 (1.5 pts).

    Expected: (a) g_i = \sigma(0) - y_i = \mp 0.5, h_i = 0.25. (b) G = \mp 1.5, H = 0.75 on each side, parent G = 0: \mathrm{Gain} = \tfrac{1}{2}\big[\tfrac{2.25}{1.75} \cdot 2\big] - 0.5 \approx 0.786 .

    1. Left G = -1, H = 0.5; right G = 1, H = 1: \tfrac{1}{2}\big[\tfrac{1}{1.5} + \tfrac{1}{2}\big] - 0.5 \approx 0.083 ; the 3–3 split wins, since by symmetry the 4–2 split also gives 0.083 and the 1–5 and 5–1 splits have negative gain (about −0.344). (d) w = \pm 1.5/1.75 \approx \pm 0.857; new score 0.257, so p = \sigma(0.257) \approx 0.564.

  3. Question (5 points). Labels y_i \in \{-1, +1\}, exponential loss \ell(\hat{y}, y) = e^{-y\hat{y}}, current model F_{m-1}, weights w_i = e^{-y_i F_{m-1}(x_i)}.

    • (a) For a fixed classifier f with values in \{-1, +1\} and weighted error \mathrm{err} < 1/2, show that the best step is \alpha = \tfrac{1}{2}\log\frac{1 - \mathrm{err}}{\mathrm{err}} \quad \text{(3 pts)}.
    • (b) Show that for any fixed \alpha > 0 the best f minimizes the weighted error, and give the weights of the next round (2 pts).

    Expected: (a) The objective is \sum_i w_i e^{-\alpha y_i f(x_i)}; dividing by W = \sum_i w_i gives e^{-\alpha}(1 - \mathrm{err}) + e^{\alpha}\mathrm{err}, a convex function of \alpha whose derivative vanishes at e^{2\alpha} = (1 - \mathrm{err})/\mathrm{err}. (b) The objective equals W e^{-\alpha} + (e^{\alpha} - e^{-\alpha}) \textstyle\sum_i w_i \mathbf{1}[f(x_i) \neq y_i], increasing in the weighted error when \alpha > 0; the new weights are w_i e^{-\alpha y_i f(x_i)} (proposition 2.5).

  4. Question (4 points). A column customer_id takes a distinct value on every training row, and the binary target is independent of all features. (a) Give its smoothed target encoding (def. 4.1) computed on the full training set, with pseudo-count \kappa > 0 and global mean \bar{y} (1 pt). (b) Show that one split on this column classifies the training set perfectly for every \kappa, and give the test AUC (2 pts). (c) What does out-of-fold encoding give, and why does the leakage disappear (1 pt)?

    Expected: (a) n_v = 1 and \bar{y}_v = y_i, so the encoding is (y_i + \kappa\bar{y})/(1 + \kappa). (b) It takes two values that differ by 1/(1 + \kappa) > 0, the larger one exactly on y_i = 1: a threshold between them separates the classes; smoothing shrinks the gap but never closes it. Test identifiers are unseen, so the encoding is the constant \bar{y} and the AUC is 0.5. (c) Each identifier is absent from the other folds, so every training row is encoded by the out-of-fold mean, which does not depend on its own label: train and test scores agree.

Grading rubric (proposal)

Criterion Weight What earns the marks
Question and baselines 15 % A precise question; a tuned GBDT baseline under the same budget as every other model
Correct implementation 20 % Code that runs with one command; formulas checked on a toy case (the split gain of theorem 3.2, the encodings of ch. 4)
Protocol and statistics 25 % The rules of ch. 9: same folds, 5 or more seeds, paired tests, no test reuse, counted cost
Data hygiene and leakage 10 % Encodings fitted inside the folds; temporal or grouped splits where the data require them; leakage checks reported
Decision and limits 10 % A conclusion stated by regime (ch. 10), with its limits and the cases where the evidence is a tie
Report and talk 20 % A clear 4–6-page report, figures with error bars, a 10–15-minute talk that answers the question

References