Reinforcement Learning and Autonomous Agents

Contents

• Introduction and overview
• 0. Refresher and probabilistic reasoning: Markov chains, Bayesian networks, dynamic programming
• 1. Markov decision processes
• 2. Dynamic programming (known model)
• 3. Learning: TD, SARSA, Q-learning (unknown model)
• 4. Bandits, exploration and Monte-Carlo tree search
• 5. Policy gradients, actor-critic and gradient-free policy search
• 6. Deep RL and advanced methods: DQN, continuous control, offline, distributional, model-based, imitation
• 7. Autonomous agents in the era of LLMs
• 8. Course research notes (Jesse Read)
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: an engineer who has trained a classifier / network and wants to move from predicting to deciding (recommendation, pricing, agents). Empirical first: Q-learning on a gridworld that stalls because ε-greedy is too low; DQN that diverges when the replay buffer leaks the distribution; then we name MDP, TD, policy gradient, RLHF.

From MDPs to dynamic programming, deep RL and LLM agents. Mandatory P2 module: the glue between supervision and LLM alignment (LLMs in M2).

What you will be able to do

• Formalize a sequential problem as a Markov decision process: states, actions, transitions, rewards, policy.
• Solve a known MDP by dynamic programming (value iteration, policy iteration).
• Implement model-free learning methods: TD, SARSA, Q-learning.
• Write policy gradient algorithms and actor-critic variants.
• Carry out deep RL: DQN, Rainbow, DDPG/SAC, and offline learning (offline RL).
• Master the exploration/exploitation trade-off, in particular in the bandit setting, and plan with Monte-Carlo tree search.
• Represent uncertain knowledge with a Bayesian network and take a single-step decision by maximum expected utility.
• Optimize policies without gradients (CEM, evolution strategies) and place a problem among the stochastic, multi-objective and constrained settings.
• Explain and compare distributional, model-based, imitation/inverse and sequence-modeling (decision transformer) approaches.
• Analyze autonomous agents in the era of LLMs: tool use, LLM agents, multi-agent systems, read as POMDPs and Markov games.

Official objectives and syllabus (2026-2027 catalog)

The official 2026–2027 sheet (X3A session “Reinforcement Learning and Autonomous Agents”, period X-3A-P2, from 04/01/2027 to 19/03/2027, taught in English) presents a course on the development of autonomous agents: systems that perceive, learn and act independently to achieve goals in uncertain and dynamic environments, drawing on probabilistic decision-making, inference and planning.

It develops Markov decision processes and their key concepts, deep Q-learning and its variants, the policy gradient and actor-critic architectures, then model-based learning, gradient-free optimization, imitation learning and other routes to sequential decision-making such as model-based control and decision transformers, and the specialized topics of offline RL, distributional RL, imitation and inverse reinforcement learning — with an emphasis on modern deep architectures that scale, on the practical consequences of architectural and algorithmic choices, and on formulating decision problems well. Students implement, study and compare algorithms in several settings.

Official prerequisites: a course such as Machine Learning and/or Deep Learning and the associated concepts, plus Python and its scientific libraries.

Assessment: the LLGA section of the sheet displays none; the description announces a course project (50%) and practical work (50%), and the other programs list homework (50%) and a project (50%, oral).

The sheet publishes a nine-block programme, matched below.

Every block of the official programme now has a section in these notes. Our extensions, beyond the official syllabus: the refresher sections 0.1–0.3 (Markov chains, martingales, dynamic programming), section 5.5 (neural combinatorial optimisation, for logistics), chapter 7 (autonomous agents in the era of LLMs: tool use, LLM agents, multi-agent systems and Markov games) and chapter 8 (the research work of Jesse Read, self-study).

Where this course fits in the program

Mandatory module of period 2 of M1 (36 h, 4.5 ECTS), of which Jesse Read is the official head. Official prerequisites: a course such as Machine Learning and/or Deep Learning and the concepts associated with these courses, as well as knowledge of Python and its scientific libraries. It constitutes the third pillar of machine learning alongside supervised and unsupervised learning, and directly feeds the second year.

• Prerequisites — Machine Learning for models, losses and generalization, and one Deep Learning course for networks: Deep Learning P1, or Deep Learning P2 taken in parallel (read its ch. 0 first).
• Markov chains — the finite-Markov-chain section of Refresher Statistics (section 2.7) and the refresher chapter 0 of this page, which also covers martingales and dynamic programming. Probability Theory for ML is an optional deepening (notes kept for reference: it is not in the 2026–27 LLGA lists).
• Downstream modules — RL directly prepares the RLHF chapter of the Large Language Models module (M2): PPO, DPO and GRPO are used there without detour; it also sheds light on the autonomous agents of Real-time AI in Video Games: decisive & collaborative actions (same period, real-time decision; the Markov games of section 7.1 frame its multi-agent chapter).

Course outline

• Refresher and probabilistic reasoning — Markov chains, martingales, dynamic programming, then Bayesian networks and single-step decisions by inference (official block 1).
• Markov decision processes — the formal framework: agent, environment, policy, value function.
• Dynamic programming — solving a known MDP: value and policy iterations.
• Learning: TD, SARSA, Q-learning — learning without a model of the environment, by interaction.
• Bandits, exploration and Monte-Carlo tree search — the exploration/exploitation trade-off in the simplest setting, then MCTS, where every tree node is a bandit.
• Policy gradients, actor-critic and gradient-free search — differentiating the policy to optimize it directly, or optimizing it as a black box (CEM, evolution strategies, multi-objective settings, architecture search).
• Deep RL and advanced methods — DQN, Rainbow and prioritized replay, DDPG/SAC, offline and safe RL, distributional RL, model-based RL and MPC, imitation and inverse RL, decision transformers.
• Autonomous agents in the era of LLMs — tool use, LLM agents, Markov games and multi-agent learning: where RL meets language models.
• Course research notes (Jesse Read) — a tour of the instructor’s research work (self-study).
• Exercises, Projects and assessment and References — systematic training, project subjects with an exam proposal, and sources to go further.

Session plan

36 h in class = 12 sessions of 3 h. The volume is the one of the SynapseS 2026–2027 sheet (36 h of in-person classes, no split between lectures and tutorials displayed).

The sheet publishes a nine-block programme; the plan below follows its order (probabilistic reasoning → MDPs and dynamic programming → TD and deep Q-learning → actor-critic → optimization → advanced architectures) with two changes, both made so that no session relies on a notion taught later.

Block 3 (bandits and MCTS) comes right after TD learning (sessions 5–6) rather than before the MDP chapters, because MCTS backs up values in an MDP; and block 2 (sequence models, planning, architecture search) is distributed over sessions 8 and 11, because architecture search is taught as an instance of policy search and decision transformers and model-predictive control are offline and model-based RL methods.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core algorithms and their convergence proofs, 3 = advanced methods and current practice; the level never goes down from one session to the next.

Self-study, outside the 36 h (about 3 h in total): ch. 8 Course research notes — classifier chains, extreme classification, stream learning and concept drift (≈ 3 h).

Read ahead for the team project (it starts before the last sessions): chapter 7 and its section 7.1, and Algorithm 4.5 with the router note that follows it, by project week 4.

How to study this module

• Assimilate the refresher section before anything else: without Markov chains or dynamic programming, MDPs and TD-learning remain opaque recipes.
• Implement each algorithm on a small environment (gridworld, bandits): in RL, one only truly understands by coding the agent-environment loop.
• Always plot learning curves over several random seeds: the variance of runs is in itself a lesson of the module.
• Classify the algorithms by their assumptions — on-policy or off-policy, online or offline, value or policy: this mental table is worth more than any mnemonic device.
• Read the section on LLM agents early, even though it comes at the end of the course: it gives a concrete and modern motivation for everything that precedes it.
• Redo the exercises of each section before moving on to the next; the classic pitfalls (insufficient exploration, divergence of the critic) are encountered there first.

0. Refresher and probabilistic reasoning: Markov chains, Bayesian networks, dynamic programming

Reinforcement learning algorithms are random processes that try to reach the fixed point of an equation. Proving that they work takes three tools, recalled in this chapter: Markov chains (the trajectory of an agent with a fixed policy is one), stochastic approximation (learning from noisy updates), and dynamic programming (the recursive structure of optimal decisions). Section 0.4 then adds the first block of the official programme: representing uncertain knowledge with a Bayesian network and taking a single decision by inference.

0.1. Classification of states, ergodicity

0.2. Martingales and stochastic approximation

0.3. Deterministic dynamic programming

0.4. Probabilistic reasoning: Bayesian networks and decisions as inference

This section is the first block of the official programme: before any sequence of decisions, an agent must represent what it knows about an uncertain world and choose one action well. A Bayesian network is the standard way to write that knowledge down, and the single-step decision rule it supports — maximize expected utility under the posterior — is exactly what the Bellman equations of chapter 1 will repeat over time.

1. Markov decision processes

Concrete scene: a pricing agent updates its estimates Q(s,a) online. After 10,000 episodes its learning curve rises; with another random seed the same code collapses, and the variance of the return is three times larger from one run to the next. Before any deep-learning trick, one must state precisely what is being optimized: this chapter sets up the Markov decision process.

2. Dynamic programming (known model)

When the transition kernel P and the reward r are known, the Bellman equations of ch. 1 can be solved by computation alone, without any interaction. This chapter gives the two classical algorithms, value iteration and policy iteration, with their convergence proofs: everything that follows is a sampled or approximated version of them.

import numpy as np

# 4x4 gridworld: states 0..15, actions 0..3 (up, down, left, right)
S, A, gamma = 16, 4, 0.95
goal = 15
P = np.zeros((S, A, S)); r = -np.ones((S, A))
def step(s, a):
    i, j = divmod(s, 4)
    di, dj = [(-1,0),(1,0),(0,-1),(0,1)][a]
    return 4*min(max(i+di,0),3) + min(max(j+dj,0),3)
for s in range(S):
    for a in range(A):
        s2 = step(s, a)
        P[s, a, s2] = 1.0
        if s2 == goal or s == goal: r[s, a] = 0.0
r[goal] = 0.0

def T_op(V):                      # optimal Bellman operator
    return (r + gamma * (P @ V)).max(axis=1)

# --- Value iteration (th. 2.1) ---
V = np.zeros(S)
for _ in range(200):
    V_new = T_op(V)
    gap = np.abs(V_new - V).max()
    V = V_new                                   # keep V_{k+1}: the iterate the bound covers
    if gap < 1e-10: break                       # stopping criterion th. 2.1
Q = r + gamma * (P @ V)
print("VI :", V.reshape(4,4).round(2))

# --- Policy iteration (th. 2.2) ---
pi = np.zeros(S, dtype=int); V = np.zeros(S)
while True:
    # (i) exact evaluation: (I - gamma P^pi) V = r^pi
    R = r[np.arange(S), pi]
    Ppi = P[np.arange(S), pi]
    V = np.linalg.solve(np.eye(S) - gamma * Ppi, R)
    # (ii) greedy improvement
    Q = r + gamma * (P @ V)
    pi_new = Q.argmax(axis=1)
    if (pi_new == pi).all(): break              # stable policy: optimal
    pi = pi_new
print("PI :", V.reshape(4,4).round(2))

3. Learning: TD, SARSA, Q-learning (unknown model)

In practice we know neither P nor r: we interact. Temporal-difference learning learns by correcting each prediction toward the observation:

4. Bandits, exploration and Monte-Carlo tree search

Before learning in a world with many states, consider the simplest decision problem in which exploration matters: a single situation, several actions whose mean rewards are unknown, and a budget of T trials. Every trial spent on a poor action is lost, yet without trying the actions one never learns which is best; this chapter measures that trade-off by the regret and gives the algorithms that handle it optimally.

4.1. Exploring beyond bandits

• Three families of exploration: \varepsilon-greedy (blind, learns nothing from the structure); optimism (UCB, count-based: bonus \beta/\sqrt{N(s,a)} — explore what is uncertain); posterior (Thompson: explore what is plausible). In MDPs, optimism yields the UCT/Monte-Carlo Tree Search algorithms (the heart of AlphaGo).

• Intrinsic exploration in high dimension: when the count N(s,a) is empty (states almost all distinct — images), replace novelty by a prediction error.

  RND (Burda et al., 2019, sketch): a random frozen target network f and a predictor g trained to imitate f on visited states; the intrinsic reward r_{\mathrm{int}}(s) = \|f(s) - g(s)\|_2^2 is large on new states (never seen by g) and fades with familiarity — a similarity-based generalization of count-based exploration;

• Exploration in LLMs: RLHF and LLM evaluation (best-of-n, jury voting) are bandit problems (arms = answers, context = prompt).

  RLVR (RL from verifiable rewards) uses exploration for reasoning-chain search (verifiable reward = binomial oracle); policy entropy and sampling temperature play the role of the temperature \alpha of SAC (ch. 6) — a topic taken up again in the M2 Large Language Models module.

4.2. Monte-Carlo tree search

The official programme moves from bandits to Monte-Carlo tree search (MCTS), and the link is direct: MCTS treats every node of a search tree as a bandit. It is a planning method — it needs a simulator of the environment (a game engine, a learned model as in section 6.4) — that spends a budget of simulations from the current state and concentrates them on the promising branches.

5. Policy gradients, actor-critic and gradient-free policy search

Value-based methods learn Q and act greedily, which requires a maximization over the actions — awkward for continuous actions, and unable to represent a deliberately random policy. Policy gradient methods parameterize the policy \pi_\theta directly and climb the gradient of the expected return J(\theta).

5.1. REINFORCE: the proof step by step

5.2. Actor-critic

5.3. From trust region to PPO

# REINFORCE with baseline skeleton (PyTorch, CartPole problem)
logits = policy_net(states)                 # (B, n_actions)
dist = torch.distributions.Categorical(logits=logits)
actions = dist.sample()
logp = dist.log_prob(actions)
returns = compute_returns(rewards, gamma)   # G_t
advantage = returns - value_net(states).squeeze(-1)   # learned baseline
loss = -(logp * advantage.detach()).mean() + 0.5 * advantage.pow(2).mean()
optimizer.zero_grad(); loss.backward(); optimizer.step()
# PPO: on a stored batch (states, actions, logp_old, returns, adv_gae), use instead
#   logp = dist.log_prob(actions)                 # stored actions under the current policy
#   ratio = torch.exp(logp - logp_old)            # rho_t(theta); logp_old stored, no gradient
#   adv = adv_gae.detach()                        # precomputed GAE advantages (Prop. 3.4)
#   l1 = ratio * adv; l2 = torch.clamp(ratio, 1-eps, 1+eps) * adv
#   v_loss = (returns - value_net(states).squeeze(-1)).pow(2).mean()   # critic regresses on returns
#   loss = -torch.min(l1, l2).mean() + 0.5 * v_loss - 0.01 * dist.entropy().mean()

5.4. Gradient-free and evolutionary policy search

The official programme devotes a week to autonomous agents seen as optimization problems, with a focus on methods for direct policy search that do not differentiate the policy at all. They treat \theta \mapsto J(\theta) as a black box that can only be evaluated, noisily, by running episodes — which is enough to optimize it.

5.5. Neural combinatorial optimisation: routing as policy search

Logistics problems — vehicle routing, dispatch, scheduling — are combinatorial: a solution is a sequence of discrete choices. Building it is therefore a one-episode MDP, and a policy network trained by REINFORCE becomes a fast heuristic solver that generalizes across instances.

POMO (Kwon et al., 2020) exploits a symmetry of the TSP: a tour can start from any of its n cities. It samples N = n tours per instance, each forced to start at a different city, and uses their mean length as a shared baseline.

This baseline needs no extra rollout. Since the N tours are sampled independently, it only rescales the expected gradient by (N-1)/N.

The same estimator, with capacities and time windows added to the mask, trains vehicle-routing policies: this is the entry point for the fleet-dispatch subject (Y1.2) and the language-driven dispatcher of Year 2. Exercise 17 trains a TSP-20 policy.

6. Deep RL and advanced methods: DQN, continuous control, offline, distributional, model-based, imitation

The tabular methods of ch. 3 store one value per state-action pair, which is impossible for Atari screens or robot joints. Deep RL replaces the table by a neural network — and thereby meets the deadly triad of ch. 3.

This chapter presents the stabilizers that make it work, the extensions to continuous actions, and learning from a fixed dataset; sections 6.3–6.6 then cover the advanced families named by the official programme: distributional RL, model-based RL and planning, imitation and inverse RL, and decision transformers.

Several of these methods, and the RLHF of the LLM course, regularize a policy toward a reference one. One lemma gives their optimum over unrestricted policies; we state it once here.

# Minimal DQN (PyTorch) -- the three stabilizers in evidence
replay = deque(maxlen=100_000)            # 1. experience replay
def train_step(batch=128, gamma=0.99):
    s, a, r, s2, done = sample(replay, batch)
    with torch.no_grad():                 # 2. target network (frozen theta^-)
        y = r + gamma * (1 - done) * target_net(s2).max(1).values
    q = policy_net(s).gather(1, a).squeeze(1)
    loss = F.smooth_l1_loss(q, y)         # 3. TD loss (Huber, more robust)
    optimizer.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(policy_net.parameters(), 10)
    optimizer.step()
# Double DQN (algorithm 3.6): replace the 'y' line with
#   a2 = policy_net(s2).argmax(1, keepdim=True)          # selection: current network
#   y = r + gamma * (1 - done) * target_net(s2).gather(1, a2).squeeze(1)  # evaluation: target

6.1. Continuous actions: DDPG and SAC

6.2. Offline RL and sim-to-real

6.3. Distributional RL

Q-learning predicts the mean return. Two actions with the same mean can carry very different risks, and learning the whole distribution also gives the network a richer training signal — the C51 line of Rainbow’s table.

In practice the distribution is parameterized. C51 puts probability masses on 51 fixed atoms z_j evenly spaced in [V_{\min}, V_{\max}], projects the shifted target r + \gamma z_j back onto the atoms, and minimizes a cross-entropy; KL is not a contraction for \mathcal{T}^\pi, but the projection makes the algorithm work well.

QR-DQN (Dabney et al., 2018) instead learns N quantiles with the quantile-regression loss, which directly minimizes a Wasserstein-1 error.

import numpy as np
rng = np.random.default_rng(0)
# Risky toy MDP: from s=0, "road" (a=0) takes 3 steps of -1 then +10 at the goal;
# "bridge" (a=1) takes 1 step of -1, then +10 w.p. 0.96 or a fall worth -10 w.p. 0.04.
def step(s, a):                                   # returns s', r, done
    if s == 0: return (1, -1.0, False) if a == 0 else (3, -1.0, False)
    if s == 1: return 2, -1.0, False
    if s == 2: return 4, 10.0, True
    return (4, 10.0, True) if rng.random() > 0.04 else (4, -10.0, True)
S, A, N, vmin, vmax = 5, 2, 51, -15.0, 15.0
z = np.linspace(vmin, vmax, N); dz = z[1] - z[0]
P = np.zeros((S, A, N)); P[:, :, N // 2] = 1.0     # C51: one categorical law per (s, a), Dirac at 0
Q = np.zeros((S, A)); n = np.zeros((S, A))        # Q-learning: the DQN target without a network
def project(r, done, probs):                      # distributional Bellman target on the atoms
    b = (np.clip(r + (0.0 if done else 1.0) * z, vmin, vmax) - vmin) / dz
    lo, hi = np.floor(b).astype(int), np.ceil(b).astype(int)
    m = np.zeros(N)
    np.add.at(m, lo, probs * (hi - b + (lo == hi))); np.add.at(m, hi, probs * (b - lo))
    return m
def cvar(probs, alpha=0.1):                       # mean of the worst alpha fraction of returns
    c = np.cumsum(probs); w = np.minimum(c, alpha) - np.minimum(c - probs, alpha)
    return (w * z).sum() / alpha
for episode in range(20000):
    s, done = 0, False
    while not done:
        a = rng.integers(A) if rng.random() < 0.3 else int(Q[s].argmax())   # epsilon-greedy
        s2, r, done = step(s, a); n[s, a] += 1; lr = 1.0 / n[s, a]          # sample-average step
        Q[s, a] += lr * (r + (0.0 if done else Q[s2].max()) - Q[s, a])
        a2 = int((P[s2] @ z).argmax())                                       # greedy on the mean
        P[s, a] = (1 - lr) * P[s, a] + lr * project(r, done, P[s2, a2])
        s = s2
print("Q-learning Q(0,.)  :", Q[0].round(2))
print("C51 means at s=0   :", (P[0] @ z).round(2))
print("C51 CVaR_0.1 at s=0:", [round(float(cvar(P[0, a])), 2) for a in range(A)])

6.4. Model-based RL and planning

The methods of chapters 3, 5 and 6 so far are model-free: they never estimate P or r. Model-based RL learns a model \hat M = (\hat P, \hat r) from transitions — an ordinary supervised regression on (s,a) \mapsto (s', r) — and then plans in it, reusing the dynamic programming of chapter 2 or the tree search of section 4.2.

Interactions with the real environment are expensive; computations in the model are cheap.

6.5. Imitation learning and inverse RL

When a reward is hard to write but an expert can demonstrate the task (driving, surgery, a dialogue), one learns from demonstrations \mathcal{D}_E = \{(s_i, a_i)\} produced by an expert policy \pi_E.

import numpy as np
rng = np.random.default_rng(0)
# Lane keeping in a side wind: x = lateral offset, actions u in {-1, 0, +1}, off-road when |x| > 1.
def step(x, u): return x + 0.1 * u + 0.03 + 0.03 * rng.standard_normal()
def expert(x): return -1 if x > 0.05 else (1 if x < -0.25 else 0)
NC = 300; edges = np.linspace(-3, 3, NC + 1)            # tabular learner: cells of width 0.02
def cell(x): return int(np.clip(np.digitize(x, edges) - 1, 0, NC - 1))
def fit(data):                                         # majority expert label in each cell
    counts = np.zeros((NC, 3))
    for x, u in data: counts[cell(x), u + 1] += 1
    seen = counts.sum(1) > 0
    return lambda x: int(counts[cell(x)].argmax()) - 1 if seen[cell(x)] else 0  # unseen: do nothing
def rollout(policy, T):                                # returns (steps off-road, expert-labelled states)
    x, cost, labelled = 0.0, 0, []
    for _ in range(T):
        labelled.append((x, expert(x)))                # the expert labels the states *visited*
        x = step(x, policy(x)); cost += abs(x) > 1
    return cost, labelled
T, budget = 200, 2000                                  # horizon, number of expert labels
bc_data = sum((rollout(expert, T)[1] for _ in range(budget // T)), [])
bc = fit(bc_data)                                      # behavioural cloning
data = rollout(expert, T)[1]; dagger = fit(data)
while len(data) < budget:                              # DAgger: run the learner, label, aggregate
    data += rollout(dagger, T)[1]; dagger = fit(data)
for name, pi in [("expert", expert), ("BC", bc), ("DAgger", dagger)]:
    off = np.mean([rollout(pi, T)[0] for _ in range(200)]) / T
    print(f"{name:7s} fraction of steps off-road: {off:.3f}")

6.6. Decision transformers: RL as sequence modeling

The official programme names decision transformers among the sequence-modeling approaches to sequential decisions. The idea is to drop Bellman backups altogether and treat offline RL (section 6.2) as conditional sequence prediction, with the same architecture as a language model.

7. Autonomous agents in the era of LLMs

An LLM agent — a language model that reads observations, calls tools and acts in a loop — is a reinforcement learning agent whose state is only partially observed. This chapter gives the matching formal framework, the POMDP, and uses it to read the design of modern agent systems; section 7.1 then lets several agents learn at once, with Markov games.

• Defining an agent: perceive → decide → act → observe, in a loop (the agent = policy \pi over a POMDP); perception = context (text, memory), action = tool calls, APIs, code. The memory/state estimation of the POMDP materializes as: context window (short-term memory — bounded), vector memories (similarity retrieval = approximate belief estimation), recurrent networks/hidden state (literally a learned filter);
• LLM + tools: the LLM is the decision engine; the “prompt” = observation; function calling = action space. The ReAct loop (Yao et al., 2023) alternates explicit reasoning and actions:

# ReAct trace (excerpt, translated) -- question: "Did the stock price of X
# go up after supplier Y's latest flagship product?"
Thought: I must identify Y's latest flagship product, then the release
  date, then compare the price of X around that date.
Action: search[Y's recent products]
Observation: Y launched product P on March 14, 2026.
Thought: I now look up the price of X around March 14, 2026.
Action: lookup[stock X price March 2026]
Observation: X traded at 102.4 on March 13; 109.9 on March 15.
Thought: I have both data points, I can conclude (+7.3% in two sessions).
Action: finish[yes, up ~7%]

The function calling / tool use format declares the action space (JSON structuring, validated schemas):

{
  "name": "get_stock_price",
  "description": "Closing price of a stock",
  "parameters": {
    "type": "object",
    "properties": {
      "ticker": {"type": "string"},
      "date":   {"type": "string", "format": "date"}
    },
    "required": ["ticker"]
  }
}
// agent's reply: {"ticker": "X", "date": "2026-03-15"}
// -> the orchestrator executes, reinjects the result as an observation

• Memory and RAG: the state exceeds the context window ⇒ external memory (vector stores). GraphRAG: index knowledge as a graph (entity-relation) and query by neighborhood/multi-hop — retrieval becomes a query over a knowledge graph; exact intersection of the master’s program: NLP × Graph ML × LLM. Prerequisite: the minimal RAG pipeline (chunk, embed, retrieve, prompt) of Text Mining and NLP, section 7.4; GraphRAG itself is in Graph ML, ch. 10;

• Multi-agent: agent systems = graphs — nodes: specialized agents (each with its own context and prompt), edges: information flows, DAG or cyclic topology (orchestrator-workers, debates, verifiers).

  Self-play (the agent learns against itself — AlphaGo/AlphaZero, Silver et al.; section 7.1) is the training version; LLM agent teams (generator/critic/verifier) are its inference version.

  Current “deep research” workflows orchestrate >100 agents in parallel phases; this “graph engineering” is one of the career paths of the master’s program (cf. LangGraph, AutoGen); orchestration is itself a policy — a meta-level sequential decision (cross-reference chapter 5);

• Distribution, robustness, confidentiality of multi-agent systems: when workers are distributed (or malicious), aggregating their outputs raises the same questions as aggregating gradients — cf. Emerging Subjects in ML module, chapters 2 (distributed) and 4 (Byzantine);

• Evaluating agents: environments (tool-use benchmarks), verifiable rewards (test programs, formal constraints — RLVR link, chapter 4), costs (tokens, calls, latency — an agent = an MDP whose every step is costly: optimize the return per token);

• Safety: per-tool permissions (whitelist of allowed actions = action space restriction), execution sandboxing (code, network), human supervision at critical points (human-in-the-loop = policy with interruption), full trajectory logging for audit.

7.1. Markov games: several learning agents

When several agents learn in the same environment — the vehicles of a fleet, the players of a game, LLM agents that negotiate — the environment of each one contains the others, and they change as they learn. Markov games are the formal frame for this situation.

• Independent learners and non-stationarity. The simplest method runs one Q-learner per agent over its own actions, the others being part of the environment. From agent i’s viewpoint the transition kernel is P^i(s' \mid s, a^i) = \sum_{a^{-i}} \boldsymbol\pi^{-i}(a^{-i} \mid s)\, P(s' \mid s, a^i, a^{-i}), and it changes as the others learn. The environment is non-stationary, the contraction argument of Theorem 3.2 no longer applies, and independent learners can cycle or settle on a poor equilibrium (Claus & Boutilier, 1998; Exercise 16).
• Self-play. In a two-player zero-sum game, one policy is trained against copies of itself or of its past versions. TD-Gammon reached master level this way, and AlphaZero (with the MCTS of section 4.2) superhuman play; a population of past opponents (league training, as in AlphaStar) prevents cycling between strategies that beat each other.
• A taxonomy of multi-agent RL. By rewards: cooperative, competitive (zero-sum) or mixed (general-sum). By training: independent learners, centralized training with decentralized execution (CTDE, for instance QMIX or MADDPG; see Real-time AI in Video Games, ch. 8), or fully centralized. By observation: a team that shares one reward but sees partial observations is a Dec-POMDP, the multi-agent version of Definition 7.1.

8. Course research notes (Jesse Read)

Jesse Read (professor at LIX, École Polytechnique) is one of the founding authors of multi-label learning and a central player in stream learning (data streams). His themes — multi-label and extreme classification, adaptive stream learning — are directly connected to the program of this course: they are problems of sequential decisions under non-stationarity.

8.1. Multi-label classification: classifier chains

8.2. Stream learning: concept drift and adaptation

Exercises

1. ★ Bellman on paper. Analytically solve a 2-state MDP (continue at gain c per step, stop at reward 0): write the Bellman equations and compute V^\star as a function of \gamma and c; interpret the indifference threshold and its link with \gamma (variant: stopping yields b once).
2. ★ Value vs policy iteration. On the gridworld of chapter 2, compare the number of iterations and the cost per iteration; verify that the policies coincide.
3. ★ Tabular Q-learning. Implement Q-learning with \varepsilon-greedy on Cliff Walking (Sutton & Barto, ex. 6.6): compare the trajectories learned by Q-learning (optimal, cliff edge) and SARSA (cautious); plot the return per episode.
4. ★ Overestimation bias on an example. Let two actions with true values Q(s, a_1) = 0 and Q(s, a_2) = 0, estimated with independent Gaussian noises \mathcal{N}(0, \sigma^2). Compute \mathbb{E}[\max(\hat Q_1, \hat Q_2)] (literally \sigma/\sqrt{\pi}) and comment in light of Theorem 3.5.
5. ★ REINFORCE, variance. On CartPole, compare REINFORCE with and without baseline; plot the variance of the empirical gradients and the learning curve.
6. ★ PPO-clip on paper. Plot \min(\rho\hat A, \mathrm{clip}(\rho, 1{-}\epsilon, 1{+}\epsilon)\hat A) as a function of the ratio \rho for \hat A = \pm 1; locate the zones where the gradient is zero and relate to the proof of Theorem 5.5.
7. ★ Bandits. Simulate a 3-armed Bernoulli bandit (0.2, 0.5, 0.7); compare \varepsilon-greedy, UCB, Thompson; plot the cumulative regret over 5000 steps and verify the \log T behavior of the last two (compare with the Lai-Robbins bound of Theorem 4.3).
8. ★ Synthetic LinUCB. Generate contexts x \sim \mathcal{N}(0, I_d), d = 5, reward x^\top\theta^\star + noise; implement LinUCB and \varepsilon-greedy; plot the regret and the evolution of the bonus \sqrt{x^\top A^{-1} x}.
9. ★ ReAct agent. Write by hand a complete ReAct trace (at least 3 Thought/Action/Observation cycles) for a multi-hop question of your choice; identify for each cycle: the observation, the action, the state estimation (POMDP belief, Def. 7.1).
10. ★★ Contraction and stopping criterion. Prove the “max-Lipschitz” lemma of Theorem 1.6, then the stopping criterion of Theorem 2.1: \|V_k - V^\star\|_\infty \leq \frac{\gamma}{1-\gamma}\|V_k - V_{k-1}\|_\infty .
11. ★★ TD(\lambda) piecewise. Verify on an example that \lambda = 0 gives back TD(0) and \lambda = 1 the Monte-Carlo return; prove the telescoping G_t^{(n)} - V(S_t) = \sum_{l<n} \gamma^l \delta_{t+l} (step 1 of Proposition 3.4).
12. ★★ Soft policy. Derive \pi^\star(a\mid s) \propto \exp(Q(s,a)/\alpha) from the decomposition F(\pi) = -\alpha\,\mathrm{KL}(\pi\|\pi^\star) + \alpha \log Z (Algorithm 6.4), and verify that \alpha \to 0 gives back the greedy policy.
13. ★★ Gaussian Thompson. Take up the \mathcal{N}(m_n, v_n) update of Algorithm 4.4. (a) Show that the posterior mean is a shrinkage estimator, \mathbb{E}[m_n \mid \mu_a] = \frac{n\tau^2}{n\tau^2 + \sigma^2}\, \mu_a , so it is biased toward the prior mean 0; compute the bias and show that it vanishes when \tau^2 \to \infty or n \to \infty. (b) Describe the posterior when \tau^2 \to \infty, then when n \to \infty.
14. ★★ UCT by hand. Continue Example 4.8 for three more simulations, assuming they return 1, 0, 1 to whichever action is selected; give the counts and means after each backup and the action finally recommended (most visited). Then redo the first step with c = 0.5 and explain how c shifts the balance between exploration and exploitation.
15. ★★ Evolution strategies vs CEM. (a) Repeat Example 5.8 for J(\theta) = -\theta^4 and show that the ES gradient equals -4\theta^3 - 12\sigma^2\theta, i.e. the gradient of the smoothed objective, not of J. (b) Implement CEM (Algorithm 5.6) on CartPole with a linear policy and compare the number of episodes needed with REINFORCE (exercise 5).
16. ★★ Independent Q-learners on a penalty game. Two agents share the payoff of the matrix game below (rows: agent 1, columns: agent 2; Claus & Boutilier, 1998): R = \begin{pmatrix} 10 & 0 & k \\ 0 & 2 & 0 \\ k & 0 & 10 \end{pmatrix}, \qquad k \leq 0 .
    1. For k = -10, list the pure Nash equilibria (Definition 7.3) and say which are optimal. (b) Run two independent stateless Q-learners (\alpha = 0.1, \varepsilon-greedy with \varepsilon = 0.1, ties broken at random) for 5000 rounds and 100 seeds, for k \in \{0, -10, -30\}; report how often each joint action is reached. (c) Explain the outcome with the non-stationarity of section 7.1: why does a larger penalty push the learners toward the safe equilibrium? (d) Let each agent observe the other’s action and learn Q-values over joint actions; compare.
17. ★★★ Neural TSP-20. (a) Show that the greedy-rollout baseline of Algorithm 5.10 keeps the gradient unbiased, and that POMO’s shared baseline rescales it by (N-1)/N. (b) Implement Algorithm 5.10 with a small attention encoder (2 layers, width 64) on uniform random instances with n = 20. Compare greedy decoding with the nearest-neighbour heuristic and with OR-Tools, as the mean optimality gap on 1000 test instances. (c) Swap in POMO’s shared baseline at an equal number of sampled tours and compare the learning curves over 3 seeds.
18. ★★★ Agent project. Extend the guiding lab: add a verification loop (critic agent) and measure the impact on report quality (human grading or LLM-judge) and on total token cost.

Projects and assessment

The official assessment, from the fiche: the LLGA section of the 2026–2027 sheet displays no modality; the course description announces “a course project (50%) and practical work (50%)”, and the other programs list homework (50%) and a project (50%, oral assessment), with a resit.

The proposals below fit that split: one project, solo or in a group, for the 50% project with an oral defence; the lab sheets of sessions 10–11 and the exercises as practical work; and a written exam for the resit. The projects exercise objectives O1 (foundations, here those of RL), O3 (RLHF and LLM agents) and O7 (hands-on practice), and touch O4–O5 through graph retrieval and O8 through constraints and reward hacking.

Solo projects

1. [Solo · ≈ 25 h] Bandits for recommendation, evaluated offline by replay. Do UCB1, Thompson sampling and LinUCB keep their simulated ranking on real logged clicks?
   • Implement: UCB1 and Algorithms 4.4–4.5; the replay estimator of Li, Chu, Langford & Wang (WSDM 2011), unbiased when the logging policy is uniform; a sliding-window variant for drift (section 8.2).
   • Data: the Open Bandit Dataset (Saito et al., NeurIPS 2021 Datasets and Benchmarks), its uniformly random logs; synthetic Bernoulli arms to compare the empirical regret with the Lai–Robbins constant (Theorem 4.3).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 4, ch. 8.
   • Success: replay click rates with bootstrap intervals over 5 seeds; LinUCB beats the best context-free policy, or the report shows why not; on the synthetic arms, R_T/\ln T is plotted up to T = 10^5 against the Lai–Robbins constant: Thompson sampling’s curve rises toward it, UCB1’s ends well above it, and the report explains both.
2. [Solo · ≈ 25 h] Paper study: Proximal Policy Optimization (Schulman, Wolski, Dhariwal, Radford & Klimov, 2017). Which ingredients of PPO matter: clipping, the GAE parameter \lambda, advantage normalization, the entropy bonus?
   • Implement: REINFORCE with a baseline, A2C (Algorithm 5.3), then PPO-clip with GAE (Proposition 3.4, Theorem 5.5) in PyTorch, using “The 37 Implementation Details of PPO” (Huang et al., ICLR Blog Track 2022) as a checklist.
   • Data: Gymnasium CartPole-v1 and LunarLander-v3, on a laptop CPU.
   • Deliverables: code, learning curves over 5 seeds, an ablation table, a 4–6 page report, a 10-minute talk. Chapters: ch. 3, ch. 5.
   • Success: CartPole reaches a return of 500 within 2 \cdot 10^5 steps on at least 4 seeds out of 5; LunarLander reaches a mean return of 200; every ablation is reported with confidence bands.

Group projects

1. [Group of 3 · ≈ 30 h each] From PPO to GRPO: RL fine-tuning of a small language model with verifiable rewards. At an equal sampling budget, does a critic (PPO) beat group-relative baselines (GRPO), and when does the policy hack the reward?
   • Implement: a token-level policy-gradient loop on a small instruction model (Qwen2.5-0.5B-Instruct with LoRA, free GPU tier), reward = exact match of the final answer, a KL penalty to \pi_{\mathrm{ref}} (lemma of ch. 6); PPO with a value head versus GRPO (Shao et al., DeepSeekMath, 2024).
   • Data: GSM8K (Cobbe et al., 2021), train and test splits.
   • Split: one member on the PPO core (validated first on LunarLander), one on rollouts and rewards, one on evaluation (accuracy, KL drift, entropy, reward-hacking audit).
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 3, ch. 5, ch. 6. Continues in Large Language Models, ch. 5.
   • Success: test accuracy above the starting model with a 95 % interval; KL and entropy curves; at least one documented format or reward exploit and its fix.
2. [Group of 3 · ≈ 30 h each] AlphaZero-lite: from UCT to PUCT with self-play on Connect Four. How many simulations does a learned policy and value prior save over random rollouts?
   • Implement: UCT (Algorithm 4.7) with random rollouts; then PUCT with a small residual network trained on root visit counts from self-play, after Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play” (Science, 2018).
   • Data: the Connect Four environment of PettingZoo, or a 40-line implementation.
   • Split: search engine; network and self-play loop; evaluation (Elo, ablations on c and the budget).
   • Deliverables: code, a 6–8 page report, a 15-minute talk with a live game. Chapters: ch. 4, ch. 5, ch. 7 (section 7.1).
   • Success: PUCT with 50 simulations wins at least 60 % of 200 games, alternating first player, against UCT with 500 random-rollout simulations.
3. [Group of 4 · ≈ 30 h each] Offline RL shoot-out: behavioural cloning, DAgger, CQL and the decision transformer. On datasets of mixed quality, which method stitches sub-optimal trajectories, and how fast do cloning errors compound?
   • Implement: BC and DAgger (Definition 6.11, Theorem 6.12, lab sheet of session 11) with a scripted expert; CQL (section 6.2; Kumar et al., 2020); a decision transformer (Definition 6.15; Chen et al., 2021).
   • Data: Minari datasets of the Farama Foundation — the D4RL PointMaze mazes for stitching, MuJoCo Hopper medium and expert data.
   • Split: one member per method family (BC and DAgger, CQL, decision transformer); the fourth builds the shared datasets and evaluation harness.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 6.
   • Success: normalized scores over 3 seeds per method and dataset; a stitching test that separates the decision transformer from CQL, or an explanation of why it does not; the compounding-error curve of BC against the horizon.
4. [Group of 3 · ≈ 30 h each] A tool-using research agent as a constrained POMDP, with a learned retrieval router (the RL work package of Y1.1). Can a router learn, per question, which retrieval mode to call so as to maximize accuracy under a token budget?
   • Implement: the guiding lab of ch. 7 and Exercise 18 (orchestrator, workers, verifier, tool whitelist); a LinUCB router (Algorithm 4.5) over question features, rewarded by correctness minus \lambda \times tokens, with \lambda adapted by the primal–dual rule of Definition 6.5.
   • Data: 2WikiMultiHopQA (Ho et al., COLING 2020); BM25 and dense retrieval from Text Mining and NLP (ch. 2, section 7.4), GraphRAG from Graph ML, ch. 10; a small open LLM.
   • Split: agent and tools; retrieval back-ends; router and evaluation.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 4, ch. 6, ch. 7.
   • Success: on at least 500 questions, the router matches or beats the best fixed mode at no greater mean token cost (bootstrap intervals); the budget is met within 5 %; unsafe tool calls are counted.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet of notes allowed, no electronic device; four exercises, 20 points.

1. Question (4 points). Explain why Q-learning is off-policy and SARSA on-policy, state the deadly triad, and name one stabilizer of DQN for each danger it creates. Expected: Q-learning bootstraps on \max_{a'} Q, the greedy policy, whatever the behaviour policy, so it learns Q^\star; SARSA bootstraps on the action actually played and evaluates the \varepsilon-greedy policy (Cliff Walking). Triad: function approximation, bootstrapping and off-policy data can diverge together (Baird). Stabilizers: replay (correlations), a target network (moving target), double Q-learning (the max bias of Theorem 3.5).
2. Question (6 points). Two states s_1, s_2, discount 0.9. In the first, stay gives reward 1 and stays; go gives 0 and moves to the second. In the second, the only action gives 2 and stays. (a) Compute V^\star and \pi^\star. (b) Run two steps of value iteration from V_0 = 0. (c) What does the stopping criterion of Theorem 2.1 guarantee after V_2? Compare with the true error. Expected:
   • 1. V^\star(s_2) = 2/(1-\gamma) = 20; in s_1, staying forever is worth 10 and going is worth 0.9 \cdot 20 = 18, so V^\star(s_1) = 18 and the optimal action in s_1 is go.
   • 2. V_1 = (1, 2) and V_2 = (1.9, 3.8); greedy with respect to V_2, go (3.42) already beats stay (2.71).
   • 3. the residual is \varepsilon = \|V_2 - V_1\|_\infty = 1.8, so \|V_2 - V^\star\|_\infty \leq 0.9 \cdot 1.8 / 0.1 = 16.2. The true errors are 16.1 and 16.2: the bound is attained in s_2.
3. Question (5 points). Prove the KL-regularised policy improvement lemma of chapter 6, then deduce the optimal policy of SAC for a fixed Q. Expected:
   • Write r = \beta\log(\pi^\star/\pi_0) + \beta\log Z, so the objective equals \beta\log Z - \beta\,\mathrm{KL}(\pi\,\|\,\pi^\star); Gibbs’ inequality gives the unique maximizer \pi^\star and the value \beta\log Z.
   • For SAC, \pi_0 is uniform and \beta = \alpha, because the entropy is \log|\mathcal{A}| minus the KL to the uniform law; hence \pi^\star \propto e^{Q/\alpha}.
4. Question (5 points). Bernoulli arms with means 0.5 and 0.4. (a) At t = 10, arm 1 has been pulled 6 times with empirical mean 0.5 and arm 2 four times with empirical mean 0.25: which arm does UCB1 play? (b) Compute the Lai–Robbins constant and compare it with the logarithmic coefficient of the UCB1 bound. Expected:
   • 1. \sqrt{2\ln 10/6} \approx 0.876 and \sqrt{2\ln 10/4} \approx 1.073, so the indices are 1.376 and 1.323: UCB1 plays arm 1, by a narrow margin.
   • 2. the divergence between the two Bernoulli laws is 0.4\ln 0.8 + 0.6\ln 1.2 \approx 0.0201, so the constant is 0.1/0.0201 \approx 4.97. The UCB1 bound gives 8/\Delta = 80 — a factor 16 above, which KL-UCB and Thompson sampling remove.

Grading rubric (proposal)

References

• R. Sutton, A. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018 — free online; the textbook of the course (chapters 3-6 for ch. 1-3, chapter 13 for policy gradients).
• M. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 2005 — the reference for chapters 1-2 (MDP, LP, modified policy iteration).
• D. Bertsekas, J. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, 1996 — contraction, asynchronous DP, deadly triad.
• C. Szepesvári, Algorithms for Reinforcement Learning, Morgan & Claypool, 2010 — compact theoretical synthesis.
• C. Watkins, P. Dayan, “Q-learning”, Machine Learning 8, 1992; J. Tsitsiklis, “Asynchronous Stochastic Approximation and Q-Learning”, Mathematics of OR, 1994 — the convergences of chapter 3.
• R. Sutton, D. McAllester, S. Singh, Y. Mansour, “Policy Gradient Methods for RL with Function Approximation”, NeurIPS 1999; R. Williams, “Simple Statistical Gradient-Following Algorithms for Connectionist RL” (REINFORCE), Machine Learning, 1992 — chapter 5.
• S. Kakade, J. Langford, “Approximately Optimal Approximate Reinforcement Learning” (CPI), ICML 2002 — the monotone improvement behind TRPO/PPO.
• J. Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation (GAE), ICLR 2016; Trust Region Policy Optimization, ICML 2015; Proximal Policy Optimization Algorithms, 2017 — the heart of RLHF.
• V. Mnih et al., Human-level control through deep reinforcement learning (DQN), Nature, 2015; Asynchronous Methods for Deep Reinforcement Learning (A3C), ICML 2016.
• H. van Hasselt, A. Guez, D. Silver, Deep Reinforcement Learning with Double Q-learning, AAAI 2016 — the max bias (Theorem 3.5) and its correction.
• M. Hessel et al., Rainbow: Combining Improvements in Deep Reinforcement Learning, AAAI 2018 — and the original papers cited (dueling, C51, PER, noisy nets).
• T. Lillicrap et al., Continuous Control with Deep Reinforcement Learning (DDPG), ICLR 2016; T. Haarnoja et al., Soft Actor-Critic, ICML 2018 — chapter 6.
• S. Levine et al., Offline Reinforcement Learning: Tutorial, Review, and Perspectives, 2020; A. Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2020.
• P. Auer, N. Cesa-Bianchi, P. Fischer, “Finite-time Analysis of the Multiarmed Bandit Problem”, Machine Learning 47, 2002 (UCB1); T. Lai, H. Robbins, “Asymptotically Efficient Adaptive Allocation Rules”, Advances in Applied Mathematics, 1985; O. Chapelle, L. Li, An Empirical Evaluation of Thompson Sampling, NeurIPS 2011; W. Li et al., A Contextual-Bandit Approach to Personalized News Article Recommendation (LinUCB), WWW 2010 — chapter 4.
• S. Bubeck, N. Cesa-Bianchi, Regret Analysis of Stochastic and Nonstochastic Bandit Problems, FnT-ML, 2012 — for bandits.
• Y. Burda et al., Exploration by Random Network Distillation, ICLR 2019 — intrinsic exploration.
• D. Silver et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search”, Nature, 2016 — self-play and UCT.
• S. Yao et al., ReAct, ICLR 2023; S. Schick et al., Toolformer, NeurIPS 2023 — and the Anthropic/LangChain multi-agent reports on graph engineering — for chapter 7.
• Research group: J. Read et al., Classifier Chains for Multi-label Classification, Machine Learning, 2011; J. Read, F. Perez-Cruz, Deep Learning for Multi-label Classification, 2014; H. Gomes, J. Read et al., Machine learning for streaming data, SIGKDD Explorations, 2019; J. Read, I. Žliobaitė, Learning from Data Streams: An Overview and Update, 2022; MEKA, JMLR 2016 — chapter 8.
• D. Koller, N. Friedman, Probabilistic Graphical Models, MIT Press, 2009 — Bayesian networks and decisions (section 0.4); S. Levine, Reinforcement Learning and Control as Probabilistic Inference, 2018.
• L. Kocsis, C. Szepesvári, “Bandit Based Monte-Carlo Planning”, ECML 2006 (UCT); D. Silver et al., “Mastering the game of Go without human knowledge” (AlphaZero line), Nature, 2017 — section 4.2.
• T. Salimans et al., Evolution Strategies as a Scalable Alternative to Reinforcement Learning, 2017; B. Zoph, Q. Le, Neural Architecture Search with Reinforcement Learning, ICLR 2017; E. Real et al., Regularized Evolution for Image Classifier Architecture Search, AAAI 2019 — section 5.4.
• T. Schaul et al., Prioritized Experience Replay, ICLR 2016; E. Altman, Constrained Markov Decision Processes, Chapman & Hall, 1999 — chapter 6.
• M. Bellemare, W. Dabney, R. Munos, A Distributional Perspective on Reinforcement Learning, ICML 2017; W. Dabney et al., Distributional RL with Quantile Regression, AAAI 2018 — section 6.3.
• R. Sutton, “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming” (Dyna), ICML 1990; D. Hafner et al., Dream to Control, ICLR 2020; J. Schrittwieser et al., MuZero, Nature, 2020 — section 6.4.
• S. Ross, J. Bagnell, “Efficient Reductions for Imitation Learning”, AISTATS 2010; S. Ross, G. Gordon, J. Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger), AISTATS 2011; B. Ziebart et al., “Maximum Entropy Inverse Reinforcement Learning”, AAAI 2008; J. Ho, S. Ermon, Generative Adversarial Imitation Learning, NeurIPS 2016 — section 6.5.
• L. Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021; M. Janner, Q. Li, S. Levine, Offline Reinforcement Learning as One Big Sequence Modeling Problem, NeurIPS 2021 — section 6.6.
• W. Kool, H. van Hoof, M. Welling, Attention, Learn to Solve Routing Problems!, ICLR 2019; Y.-D. Kwon et al., POMO: Policy Optimization with Multiple Optima for Reinforcement Learning, NeurIPS 2020 — section 5.5.
• L. Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022 — RLHF (introduction, section 6.5).
• L. S. Shapley, “Stochastic Games”, PNAS 39, 1953; A. M. Fink, “Equilibrium in a stochastic n-person game”, Journal of Science of the Hiroshima University 28, 1964; M. Littman, “Markov games as a framework for multi-agent reinforcement learning”, ICML 1994; C. Claus, C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems”, AAAI 1998 — section 7.1.
• Gymnasium — the environments of the labs.