Speech Technology for Conversational AI

Contents

• Introduction and overview
• 0. Recap: the speech signal and its representations
• 1. Classical ASR: the GMM-HMM pipeline
• 2. CTC: the loss, derived
• 3. Encoder-decoder: LAS, Conformer, RNN-T
• 4. Whisper: robustness and multi-task
• 5. Streaming: chunking, VAD, latency vs quality
• 6. Vocoders: from WaveNet to HiFi-GAN
• 7. Neural TTS: Tacotron 2, VITS, diffusion and flow
• 8. Expressivity: prosody, style tokens, SSML
• 9. Voice conversion and cloning: ethics and watermarking
• 10. The conversational loop: VAD, turn-taking, full-duplex
• 11. Speech LLMs and multilingualism
• Exercises / Labs
• Projects and assessment
• References

Introduction and overview

Audience: an NLP or signal-processing engineer who wants to master the full voice loop. Empirical first: the module starts from concrete failures — a word error rate of 4% in the studio that becomes 28% on a noisy phone line, a TTS system that clones a voice without any watermark — and only then names the tools (Mel features, CTC, Whisper, vocoders).

The course goes from recognition (ASR) to synthesis (TTS) to speech LLMs, and bridges Signal Processing (M1) and Large Language Models (M2).

What you will be able to do

• Choose and compute the representations of the speech signal (Mel spectrogram, MFCC) according to the use;
• Explain the classical GMM-HMM pipeline and what caused its replacement;
• Derive the CTC loss (sum over alignments, forward-backward) and implement it;
• Compare LAS, Conformer and RNN-T according to the quality/latency trade-off, and explain Whisper’s multi-task robustness;
• Size a streaming system: chunking, voice activity detection, latency vs quality;
• Explain the modern TTS chain: vocoders (WaveNet to HiFi-GAN), Tacotron 2, VITS, diffusion and flow approaches;
• Control the expressivity of a synthesis (prosody, style tokens, SSML) and master the ethical issues of voice cloning;
• Describe the complete conversational loop (VAD, turn-taking, full-duplex) and how multilingual speech LLMs work.

Official objectives and syllabus

Official course description not published (S3/S4 catalog empty): as of the start of the 2026 academic year, the “LLGA 2 – Semestre 3” and “LLGA 2 – Semestre 4” tracks of the 2026-2027 SynapseS catalog still list no courses, and the master’s wiki indicates that the detailed M2 program is “under development” (period 2: “to be completed soon”).

The program below therefore reflects the master’s brochure (period 2), with an indicative volume of ~24 h; these course materials are calibrated to that volume, i.e. about two hours of work per chapter, labs included.

The following table maps each brochure theme to the course sections; the materials’ own additions are marked “going beyond the syllabus”.

Where this course fits in the program

An M2 module announced for period 2 and preselected by the AI-systems route of the home page: it is the master’s “multimodal” module devoted to the audio modality, designed as the joint production deployment of two prerequisites. Prerequisites: Signal Processing (windowing, Fourier transform, spectrograms — revisited in ch. 0) and Large Language Models, whose multimodal chapter (ch. 8) announces the speech LLMs covered here at the end of the course.

Downstream: the module gives its voice component to the conversational assistant — the LLGA’s application through-line, which crosses the agents of Advanced Topics in Large Language Models (ch. 9) and the production constraints of ML Ops and LLM Engineering Principles (P99 latency, costs). It extends to audio the text watermark of LLMs ch. 9.5; readers of the Graph × LLM route find the same ideas in Control in Generative AI, which this page never requires.

Course outline

• Recap: the speech signal and its representations — windowing, Mel spectrograms and MFCC: the raw material of the whole course.
• Classical ASR: the GMM-HMM pipeline — the historical architecture, still useful to understand what neural methods replaced.
• CTC: the loss, derived — learning without frame/phoneme alignment: the complete derivation of the loss that opened up deep ASR.
• Encoder-decoder: LAS, Conformer, RNN-T — three architectural answers to the quality/latency trade-off.
• Whisper: robustness and multi-task — what very large-scale supervised learning changes (and costs).
• Streaming: chunking, VAD, latency vs quality — going from file to stream: split, detect speech, own the latency.
• Vocoders: from WaveNet to HiFi-GAN — generating the waveform itself, and the path from slow autoregressive models to fast GANs.
• Neural TTS: Tacotron 2, VITS, diffusion and flow — modern synthesis from text to spectrogram then to signal.
• Expressivity: prosody, style tokens, SSML — not only saying, but how to say it.
• Voice conversion and cloning: ethics and watermarking — the promises of cloning, its misuse, and how to mark generated audio.
• The conversational loop: VAD, turn-taking, full-duplex — listening, waiting one’s turn, interrupting oneself: real-time orchestration.
• Speech LLMs and multilingualism — the speech signal fed directly into the language model: architectures and issues for low-resource languages.
• Exercises / Labs — align a CTC, measure a WER, synthesize and watermark a voice clip.
• Projects and assessment — solo and group subjects, a written-exam proposal and a grading rubric, all labelled as proposals.
• References — the founding papers of neural ASR and TTS.

Session plan

24 h in class (announced) = 8 sessions of 3 h. The SynapseS 2026–2027 sheet publishes no volume for this module; 24 h is the indicative volume quoted with the brochure programme in the fiche above, and also the M2 volume published for the Large Language Models course (CSC_53432_EP: 24 h, 2 ECTS); the session split below is our proposal for covering these notes in that time.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods, 3 = advanced material (recent research: full-duplex dialogue, speech LLMs); the level never goes down from one session to the next.

Self-study, outside the 24 h: no chapter — every chapter is covered in class; the four labs (Ex. 1, 2, 7 and 8) each need about 2 h of personal work.

How to study this module

• Listen to every example audio of the course (originals, degraded, synthesized): the ear is here a measuring instrument to train as much as the WER.
• Write the CTC forward-backward yourself on a mini-example by hand: it is the obligatory step to never again confuse alignment and decoding.
• For each architecture, place it on the “latency/quality” axis and note what is computed online versus offline: it is the module’s reading grid.
• Wire up a mini “microphone → ASR → response → TTS → speaker” loop early, even a basic one, then improve it chapter after chapter. This loop is the speech front end of the Year 2 transverse project Y2.3, a voice-first assistant for 3D products.
• Read ch. 9 (cloning, ethics) with the eye of a future professional: questions of consent and audio watermarking will arise in your projects from the internship onward.

0. Recap: the speech signal and its representations

Every model of this module reads or writes a spectrogram, so the course starts from the signal. This chapter recalls what the Signal Processing module established, then states the two problems of the module — recognition and synthesis — and how recognition is scored.

1. Classical ASR: the GMM-HMM pipeline

A recognizer tuned on call-center audio collapses as soon as the accent or the microphone changes. To understand why, and what neural systems changed, this chapter presents the architecture that dominated speech recognition for thirty years: an acoustic model of sounds, a lexicon and a language model, combined by Bayesian decoding.

The historical pipeline (1980-2015) survives only as technical memories — but its formalism (emission model, chain of states, Bayesian decoding) reappears everywhere, up to the CTC forward-backward (ch. 2). We cover it in one structured chapter.

2. CTC: the loss, derived

The central problem of neural ASR: the output (sequence of tokens y) is shorter than the input (T frames) and unaligned — there is no label per frame. The solution of Graves et al. (ICML 2006): marginalize over all possible alignments. We derive everything: this is technical chapter no. 1 of the module.

3. Encoder-decoder: LAS, Conformer, RNN-T

CTC assumes that the frames are conditionally independent and has no language model of its own. This chapter presents the three architectures that lift these limits in different ways — by learning the alignment with attention (LAS), by improving the encoder (Conformer), or by adding a prediction network while keeping streaming possible (RNN-T).

4. Whisper: robustness and multi-task

The architectures of chapter 3 were trained on a few thousand carefully transcribed hours and broke on real-world audio. This chapter shows how Whisper obtained robustness from a plain encoder-decoder trained on a very large amount of imperfectly labelled web audio.

5. Streaming: chunking, VAD, latency vs quality

Conversing means answering fast: the ASR must produce words while one speaks. This chapter deals with the “offline \to streaming” transformation, whose cost is measured in WER added per second gained.

6. Vocoders: from WaveNet to HiFi-GAN

Synthesis ends with a waveform: tens of thousands of samples per second, whose phase is not contained in the spectrogram. This chapter presents the vocoders that generate it, from the slow but faithful WaveNet to the fast HiFi-GAN used in production.

7. Neural TTS: Tacotron 2, VITS, diffusion and flow

Given a vocoder, it remains to turn text into a spectrogram — a one-to-many problem, since a sentence can be said in many ways. This chapter follows the three generations of answers: an attention model that writes the Mel frame by frame, an end-to-end stochastic model, and iterative diffusion and flow models.

8. Expressivity: prosody, style tokens, SSML

A synthesis can be intelligible and still sound flat or say a sentence with the wrong intention. This chapter describes what expressivity is made of and the two ways of controlling it: soft control through learned style latents, hard control through markup.

9. Voice conversion and cloning: ethics and watermarking

The same models that give an assistant a pleasant voice can give anyone’s voice to a fraudster from a few seconds of recording. This chapter explains how conversion and cloning work, what the law and the engineer must do about them, and how generated audio is watermarked.

10. The conversational loop: VAD, turn-taking, full-duplex

Recognition and synthesis are not yet a conversation: the system must also know when the user has finished, answer within a human delay, and cope with interruptions. This chapter measures the human speaking turn and shows how systems went from a pipeline with thresholds to full-duplex models.

11. Speech LLMs and multilingualism

The last step removes the boundary between speech and text models: if audio can be cut into discrete tokens, a language model can read and write it directly. This chapter shows how audio is tokenized, the three ways of grafting audio onto an LLM, and what this changes for the thousands of languages with little data.

Exercises / Labs

1. ★ Lab: Whisper transcription and WER (the module’s ASR lab). With transformers:

   1. load openai/whisper-tiny as an ASR pipeline, transcribe two French excerpts from Common Voice (one read, one spontaneous);

   2. implement the WER (word-level Levenshtein alignment: S + D + I against the reference);

   3. compare tiny/small;

   4. pre-filter with Silero VAD (cut the silences) and measure the effect on the hallucinations of (a);

   5. redo on English to situate French.

   Proof. Guidance (expected results): tiny is 2-3× faster but clearly more fragile; the spontaneous French WER is 1.5-3× read English; the VAD massively reduces the hallucinated loops on silences (Whisper “invents” text without energy — prop. 4.2 experienced); beware of normalizations (capitalization, punctuation: normalize before aligning, otherwise the WER is artifactual). ∎

2. ★ Lab: TTS with transformers (the synthesis lab).

   1. synthesize a French text with MMS-TTS (facebook/mms-tts-fra, a single-speaker VITS), and an English one with SpeechT5 (microsoft/speecht5_tts, English only; vocoder microsoft/speecht5_hifigan; a speaker x-vector from CMU ARCTIC);

   2. measure the RTF (computation time / audio duration) and the first-sample latency of both;

   3. try three trap sentences on MMS-TTS: “1 234 567 €”, “le SARS-CoV-2”, “à Saint-Étienne, il est 19 h58” — record the normalization failures;

   4. inject a different speaker x-vector into SpeechT5 and compare (MMS-TTS has no speaker input);

   5. break the Mel contract of ch. 6: copy-synthesize a real 16 kHz recording through the SpeechT5 vocoder, its Mel computed by the SpeechT5 feature extractor (80 bins, 16 ms hop), then again from a 22.05 kHz file left unresampled, or with the hop changed to 10 ms.

   Proof. Guidance: RTF < 1 on GPU, on the order of 1-3 on CPU for SpeechT5, which generates its Mel autoregressively, and much lower for the parallel VITS of MMS-TTS; numbers and acronyms fail reproducibly — discuss neural vs rule-based front-end (def. 8.3); the x-vector shifts the timbre but not the accent: perfect zero-shot cloning does not exist with a single vector (ch. 9: one needs the sequential prompt, VALL-E style); in (e) the vocoder still outputs a voice, but slowed, shifted or buzzy — the Mel parameters are an interface, not a detail. ∎

3. ★ CTC by hand. Take up example 2.2 (T = 4, y = \texttt{ab}) with, for t = 1..4, p_t(\epsilon) = 0.5, \qquad p_t(a) = 0.3, \qquad p_t(b) = 0.2 . Compute the 15 products, sum them, check that the sum equals the \alpha_4(5) + \alpha_4(4) obtained by the recursions of Theorem 2.3 (fill in the \alpha table row by row).

   Proof. Guidance: the \alpha table has 4 rows \times 5 columns (with structural zeros — one cannot reach \ell_4 before t=2). Its last row and the result are (0.0625, 0.0816, 0.171, 0.0864, 0.06), \qquad p(y \mid x) = 0.0864 + 0.06 = 0.1464, the sum of the 15 products.

   Six paths tie as the most probable (two blanks, one a, one b: 0.5^2 \times 0.3 \times 0.2 = 0.015 each), whereas the frame-by-frame argmax is \epsilon\epsilon\epsilon\epsilon, which collapses to the empty string — the best path is not the best labelling.

   Check that the bound T \geq |y| (here 2) is necessary and explain the jump condition \mathbf 1\{\ell_s \neq \ell_{s-2}\} on a counter-example “aa”. ∎

4. ★★ CTC in code, gradient verified. Implement \alpha, \beta (thm. 2.3) in NumPy for a mini-vocabulary; compute \partial \mathcal L/\partial p_t(k) by the closed formula; check against automatic differentiation (autograd/PyTorch of the sum of paths) to within \varepsilon = 10^{-6}.

   Proof. Guidance: the convention (\beta with or without the p_t factor) is the classic error — the autograd check detects it immediately; measured complexity \mathcal O(T S) versus enumeration (15 paths here; enumeration explodes beyond T \approx 15); conclude on the differentiation of the full model’s objective (softmax + recursions) — this is exactly what PyTorch does, vectorized. ∎

5. ★★ Latency budget of a voice assistant. A pipeline: VAD (30 ms window) \to chunking ASR (640 ms chunk, 160 ms lookahead) \to LLM (time-to-first-token 250 ms) \to TTS (first Mel chunk 120 ms + vocoder 20 ms).

   1. compute the first-sound latency of the response (component-by-component detail);

   2. identify the two dominant terms and propose three reductions;

   3. compare with Moshi’s full-duplex (160 ms theoretical, about 200 ms in practice): what does it remove?

   Proof. Guidance:

   1. sum the delays after the user’s last word:

      • VAD 30 ms;
      • endpointing — absent from the list but unavoidable — a silence threshold of 300-500 ms;
      • ASR chunking: the last word is transcribed only once its 640 ms chunk and the 160 ms lookahead have been heard, i.e. on average half a chunk plus the lookahead (320 + 160 = 480 ms) and up to 800 ms;
      • LLM 250 ms;
      • TTS 120 + 20 = 140 ms.

      Total, of which endpointing and chunking make about two thirds: \approx 30 + 400 + 480 + 250 + 140 \approx 1.3 \text{ s on average (up to } \approx 1.7 \text{ s)};

      this serial sum is an upper estimate: when the endpointer and the streaming ASR read the same audio in parallel, their two waits overlap and the LLM starts at the later of the two (about 0.54 s on average), for a total near 0.9 s;

   2. reduce the threshold by end-of-turn prediction (prop. 10.2), ASR with a smaller chunk, speculative LLM (referral to LLMs M2 ch. 6);

   3. Moshi removes VAD/endpointing (continuous stream prediction), ASR/TTS (native audio tokens) and the LLM queue: what remains is network time + one model step. ∎

6. ★★ Audio watermarking: ROC under attack (bonus). Synthesize 20 sentences (lab 2); mark half of them with a public audio watermark (e.g. AudioSeal); plot the detector’s ROC (true positive / false positive) on: clean audio, mp3 re-encoding at 64 kbps, light noise addition; conclude on the end-to-end provenance channel (ch. 9).

   Proof. Guidance: the clean curve is nearly perfect (AUC > 0.99); mp3 degrades moderately (the watermark survives usual compressions — it is designed for that); strong additive noise attacks harder; relate to the z-score of Theorem 9.3: marking strength \uparrow = imperceptibility \downarrow — the same triple trade-off as in text (Control in Generative AI ch. 10, optional). ∎

7. ★★ Lab: the WER(\lambda) curve of a streaming ASR. Simulate a chunked encoder (Theorem 5.2, item 1) with the CTC model facebook/wav2vec2-base-960h, whose output frames are 20 ms apart, on the first 200 utterances of LibriSpeech test-clean.

   1. cut each utterance into chunks of L frames; run the encoder on 2 s of left context, the chunk and R frames of lookahead, and keep the chunk’s logits only;

   2. concatenate the kept logits, decode greedily (prop. 2.4), and compute the WER for L \in \{8, 16, 32, 64\} and R \in \{0, 4, 8, 16\};

   3. plot WER against \lambda = (L + R)\,\Delta with \Delta = 20 ms, next to the offline WER, and pick the elbow (Proposition 5.3);

   4. measure the RTF of each setting.

   Proof. Guidance: collapse after concatenation, not chunk by chunk, or a letter whose frames straddle a border is emitted twice. Expect the WER to be far above the offline value at R = 0 and small L, where the frames at chunk borders see no future, then to fall quickly as the lookahead grows to a few hundred ms, then to plateau near the offline WER. A model trained with the same chunk masks (Theorem 5.2) loses less at small \lambda than this offline one. The RTF grows as L shrinks, since the context is recomputed for every chunk. ∎

8. ★★ Lab: per-language WER, Whisper against MMS. On the FLEURS test sets (Conneau et al., IEEE SLT 2022: the same read sentences in 102 languages) of French, English, Swahili, Yoruba and one language of your choice:

   1. transcribe 100 utterances per language with openai/whisper-small and with facebook/mms-1b-all, after loading its adapter for the language;

   2. normalize both references and hypotheses (case, punctuation, digits), then compute WER and character error rate (CER) per language;

   3. report the mean over languages next to the per-language values, and find the language where the two systems differ most (Proposition 11.3).

   Proof. Guidance: expect Whisper to lead on English and French and MMS on the low-resource languages, where Whisper can exceed 50% WER or answer in the wrong language (force the language token). The mean hides these collapses: report per-language values and confidence intervals over utterances. CER is steadier than WER where word segmentation is uncertain. Normalization choices move WER by several points: state them (Example 4.3). ∎

Projects and assessment

The official assessment of this course is not displayed: its course description is not published, and the LLGA 2 – Semestre 3 and Semestre 4 tracks of the 2026–2027 SynapseS catalog list no courses yet, so no modality, weight or ECTS value is known. The subjects below are proposals of these notes, not official rules: a project can stand for continuous assessment, and the exam proposal for a final written exam if one is announced.

The projects exercise mainly O6 (the audio modality end to end: recognition, synthesis, dialogue), O2 (CTC as a marginal likelihood, flows, diffusion and flow-matching TTS), O7 (latency budgets and deployment on real data) and O8 (consent, watermarking, disaggregated evaluation), with O3 wherever an LLM joins the loop. Every subject that records voices follows the consent rule of ch. 9: written consent, recordings deleted at the end of the project, no clone of anyone who has not agreed.

Solo projects

1. [Solo · ≈ 20 h] Paper study: CTC, from the forward-backward to forced alignment (Graves, Fernández, Gomez and Schmidhuber, 2006). Does your own forward-backward give the library’s loss and gradient, and how precise are the word boundaries it yields?
   • Work: \alpha and \beta in log space (Theorem 2.3) in NumPy, checked against torch.nn.functional.ctc_loss and autograd (Ex. 4); Viterbi forced alignment over the extended label sequence, on the emissions of facebook/wav2vec2-base-960h; prefix beam search with shallow fusion of the official LibriSpeech 4-gram language model (OpenSLR resource 11).
   • Data: LibriSpeech test-clean (Panayotov et al., 2015); reference word timings from the Montreal Forced Aligner (McAuliffe et al., 2017).
   • Success criterion: loss and gradient within 10^{-6} (relative) of the library; median word-start deviation under 50 ms from the reference timings, with its distribution; a lower WER for beam search with the LM than for greedy decoding.
   • Deliverables: code, a 4–6 page report, a 10-minute talk; chapters 1 and 2, Ex. 3–4.
2. [Solo · ≈ 20 h] The latency/quality curve of streaming ASR, with asynchronous correction. Where is the elbow of WER(\lambda), and how much of the gap to the offline WER does a second pass recover?
   • Work: Ex. 7 extended to 5–8 values of L and of R, on test-clean and test-other, with bootstrap intervals over utterances; then each endpointed segment re-transcribed by Whisper (Theorem 5.2, item 3), measuring the WER recovered and the delay of the correction.
   • Data: LibriSpeech test-clean and test-other; a noisy copy mixed with MUSAN noise (Snyder, Chen and Povey, 2015) at 10 dB SNR.
   • Success criterion: WER(\lambda) curves with confidence bands and a justified elbow; the share of the streaming-to-offline gap recovered by the second pass, with its median delay.
   • Deliverables: code, report, talk; chapters 3–5, Ex. 7. It is the streaming block of the Y2.3 project.
3. [Solo · ≈ 20 h] Paper study: diffusion against flow matching for the Mel decoder (Popov et al., 2021; Mehta, Tu, Beskow, Székely and Henter, 2024). How many sampling steps does each family need for the same quality?
   • Work: first, both objectives of Theorem 7.3 on a 2-D toy distribution; then the released LJSpeech checkpoints of Grad-TTS and of Matcha-TTS (a conditional-flow-matching decoder), with a HiFi-GAN vocoder trained on LJSpeech, swept over 1, 2, 4, 10 and 50 steps.
   • Data: LJSpeech (Ito and Johnson, 2017), 100 held-out sentences.
   • Success criterion: per step count, the RTF, the WER of Whisper on the synthesized speech (intelligibility) and the UTMOS predicted MOS (Saeki et al., 2022), with intervals; the step count where each curve flattens.
   • Deliverables: code, report, talk with audio samples; chapters 6 and 7.

Group projects

1. [Group of 3 · ≈ 30 h each] A low-latency voice assistant with barge-in. Can a cascade of open models answer a spoken question within the turn-taking budget of ch. 10, and stop talking when it is interrupted?
   • Work: microphone → Silero VAD → streaming faster-whisper → a small instruction-tuned LLM with retrieval over a document set (served by vLLM or llama.cpp) → Piper (VITS) TTS, every stage traced; a sweep of the endpointing threshold \tau_s (Proposition 10.2); barge-in, where the user’s voice stops the TTS; a qualitative comparison with the public Moshi demo.
   • Data: 60 spoken questions recorded by consenting members in quiet and noisy rooms, plus TTS renders mixed with MUSAN noise; the course pages as the retrieval corpus.
   • Split: speech front end and endpointing; LLM, retrieval and serving; TTS, barge-in and the evaluation harness.
   • Success criterion: P50 and P95 time to first audio against the budget of Ex. 5 (target: P95 at most 1.5 s); the false cut-in rate as a function of \tau_s; the TTS stopped within 300 ms of a barge-in.
   • Deliverables: code and a live demo, an 8–10 page joint report, a 20-minute talk; chapters 4, 5, 7 and 10. It connects to LLMs ch. 8 and to the serving SLOs of MLOps ch. 3, and rehearses the Y2.3 project.
2. [Group of 3 · ≈ 30 h each] A provenance audit of voice cloning, with consented voices only. Does an audio watermark survive the attacks a fraudster would use, and does a spoof detector catch what it misses?
   • Work: zero-shot cloning of the members’ own voices with YourTTS; marking with AudioSeal and WavMark; attacks by mp3 at 64 kbps, white noise at 20 dB SNR, speed change of ±5 %, and re-recording through a loudspeaker and a phone; ROC and AUC per attack (Theorem 9.3, Ex. 6).
   • Detection: speaker similarity with an ECAPA-TDNN verifier (Desplanques, Thienpondt and Demuynck, 2020); a spoof countermeasure trained on ASVspoof 2019 LA (Wang et al., 2020).
   • Split: consent protocol and cloning; watermarking and attacks; verification, spoof detection and a one-page note on AI Act art. 50.
   • Success criterion: true-positive rate at 1 % false positives per attack, with bootstrap intervals, set against the figures reported for AudioSeal (San Roman et al., 2024); the spoof detector’s equal error rate on the clones.
   • Deliverables: code, joint report, talk; chapters 6 and 9. The correlation detector rests on Signal Processing ch. 7; the attack budget connects to Security and Robustness ch. 10 and Explainability, Security and Privacy of LLMs ch. 9.
3. [Group of 2 · ≈ 30 h each] French accents and a low-resource language: disaggregated WER, then LoRA. Which speakers does a multilingual recognizer fail, and does a light adaptation help them without hurting the others?
   • Work: Whisper (two sizes) and MMS on Common Voice French grouped by its self-reported accent field, and on one low-resource language of FLEURS; a stated text normalization (numbers, liaison spellings, Example 4.3); LoRA on the Whisper encoder (LLMs 4.3), trained on the train split of the weakest groups.
   • Data: Common Voice French (Ardila et al., 2020) and FLEURS (Conneau et al., 2022), official splits.
   • Split: benchmarking and normalization; adaptation and regression tests.
   • Success criterion: per-group WER with intervals before and after adaptation; no group losing more than one WER point.
   • Deliverables: code, joint report, talk; chapters 4 and 11, Ex. 1 and 8.

Written exam (proposal)

Two hours, one handwritten A4 sheet allowed, a simple calculator allowed; four exercises of 5 points. It would stand for a final written exam, if the course announces one.

1. Question (5 points). Compare CTC, LAS, RNN-T and Whisper on three axes: how the alignment is handled, whether the model has an internal language model, and whether it can stream. Then say where a causal RNN-T gets its right context from.
   • Expected: CTC marginalizes monotone alignments of conditionally independent frames, has no internal LM and streams with a causal encoder. LAS learns the alignment by attention, has an autoregressive decoder (an internal LM), and cannot stream strictly since it reads all of x first.
   • RNN-T sums over a T \times U lattice, its prediction network is a partial internal LM, and it streams with a causal encoder. Whisper generates text and timestamps as tokens over 30 s windows: no native streaming.
   • A causal RNN-T has no acoustic right context: its prediction network only adds left context over the emitted tokens. Accuracy is bought back with a few lookahead frames, paid in latency \lambda = (L + R)\,\Delta.
2. Question (5 points). CTC on T = 3 frames for the one-letter label y = \texttt{a}, with at every frame p_t(\epsilon) = 0.6, \qquad p_t(\texttt{a}) = 0.4 . List the paths that collapse to \texttt{a} and sum their probabilities; fill the \alpha table of Theorem 2.3 and check; give the loss and its derivative with respect to p_2(\texttt{a}); compare with greedy decoding.
   • Expected: \ell = (\epsilon, \texttt{a}, \epsilon) and six paths, writing - for \epsilon — aaa (0.064), aa- and -aa (0.096 each), a--, -a- and --a (0.144 each) — so p(y \mid x) = 0.688.
   • Rows of \alpha: (0.6, 0.4, 0), (0.36, 0.4, 0.24), (0.216, 0.304, 0.384), and \alpha_3(3) + \alpha_3(2) = 0.688.
   • The loss is -\ln 0.688 \approx 0.374. With \beta_2(2) = 1, the derivative of p(y \mid x) is \alpha_2(2)\beta_2(2)/p_2(\texttt{a}) = 1, so the loss derivative is -1/0.688 \approx -1.45.
   • Greedy decoding picks \epsilon at every frame and outputs the empty string, whose probability is 0.216 < 0.688: the best path is not the best labelling.
3. Question (5 points). Let f be the affine coupling layer of Definition 7.2b. Prove that f is invertible with an explicit inverse and that its log-determinant is \sum_j s_j(z_a). Deduce \log p_Z(z) when u = f(z) \sim \mathcal N(0, I_d), and say why VITS takes s = 0.
   • Expected: z_a = y_a, then z_b = (y_b - m(y_a)) \odot \exp(-s(y_a)); no network is inverted.
   • The Jacobian is block lower-triangular, with diagonal blocks I and \mathrm{diag}(e^{s(z_a)}), so its determinant is \prod_j e^{s_j(z_a)}.
   • By the change of variables: \log p_Z(z) = -\tfrac12 \|f(z)\|^2 - \tfrac d2 \log 2\pi + \sum_j s_j(z_a).
   • With s = 0 the layer preserves volume: the log-determinant vanishes, training is simpler and stabler, expressiveness comes from the shifts and the stacking, and the density stays exact.
4. Question (5 points). A chunked streaming encoder with frame step \Delta = 10 ms gives, on a fixed test set, these WERs for the chunk and lookahead sizes (L, R) in frames: (16, 0) 14.0 %; (16, 4) 10.5 %; (16, 8) 9.6 %; (32, 8) 9.1 %; (64, 16) 8.8 %; offline 8.5 %. Give \lambda for each setting, choose the best one under a 300 ms budget, locate the elbow, and explain what an asynchronous second pass changes.
   • Expected: \lambda = 160, 200, 240, 400 and 800 ms (Definition 5.1).
   • Under 300 ms, (16, 8) at 240 ms with 9.6 %. The elbow is near 200–240 ms: the WER drops 4.4 points from 160 to 240 ms, then only 0.8 point for 560 ms more.
   • A second pass shows the streaming text at once and replaces it, a few hundred ms after each endpoint, by an offline transcription (Theorem 5.2, item 3): the final WER approaches 8.5 % while the reactivity stays that of the first pass, at the price of extra compute and text that corrects itself.

Grading rubric (proposal)

References

• Signal and representations: L. Rabiner, B.-H. Juang, Fundamentals of Speech Recognition, Prentice Hall, 1993; L. Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proc. IEEE 77(2), 1989; S. Davis, P. Mermelstein, Comparison of Parametric Representations… (MFCC), IEEE TASSP, 1980 — foundations in the Signal Processing (M1) module ch. 4-8.
• End-to-end ASR: A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, ICML 2006; A. Graves, Sequence Transduction with Recurrent Neural Networks (RNN-T), 2012; W. Chan et al., Listen, Attend and Spell, ICASSP 2016; A. Gulati et al., Conformer: Convolution-augmented Transformer for Speech Recognition, Interspeech 2020; N. Moritz, T. Hori, J. Le Roux, Streaming Automatic Speech Recognition with the Transformer Model, ICASSP 2020.
• Self-supervised representations and data at scale: A. Baevski et al., wav2vec 2.0, NeurIPS 2020; W.-N. Hsu et al., HuBERT, IEEE/ACM TASLP 2021; A. Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), ICML 2023; V. Pratap et al., Scaling Speech Technology to 1000+ Languages (MMS), 2023; Mozilla Common Voice; V. Panayotov et al., LibriSpeech: an ASR Corpus Based on Public Domain Audio Books, ICASSP 2015; A. Conneau et al., FLEURS, IEEE SLT 2022.
• TTS and vocoders: A. van den Oord et al., WaveNet: A Generative Model for Raw Audio, 2016; J. Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2), ICASSP 2018; J. Kong et al., HiFi-GAN, NeurIPS 2020; J. Kim et al., Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (VITS), ICML 2021; V. Popov et al., Grad-TTS, ICML 2021; S. Mehta, R. Tu, J. Beskow, É. Székely, G. E. Henter, Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching, ICASSP 2024; M. Le et al., Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale, 2023; Y. Wang et al., Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis, ICML 2018; L. Dinh, J. Sohl-Dickstein, S. Bengio, Density Estimation Using Real NVP (affine coupling), ICLR 2017; Y. Lipman et al., Flow Matching for Generative Modeling, ICLR 2023; W3C, Speech Synthesis Markup Language (SSML) 1.1.
• Cloning, ethics, watermarking: E. Casanova et al., YourTTS, ICML 2022; C. Wang et al., Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E), 2023; L. Wan et al., Generalized End-to-End Loss for Speaker Verification, ICASSP 2018; G. Chen et al., WavMark: Watermarking for Audio Generation, 2023; R. San Roman et al., Proactive Detection of Voice Cloning with Localized Watermarking (AudioSeal), ICML 2024; Regulation (EU) 2024/1689 — AI Act (art. 50) — the text watermark and its z-test in LLMs ch. 9.5 (J. Kirchenbauer et al., ICML 2023); extended by the Explainability module ch. 9 and, on the Graph × LLM route, the Control in Generative AI module ch. 6 and 10.
• Conversation and speech LLMs: G. Skantze, Turn-taking in Conversational Systems and Human-Robot Interaction: A Review, Computer Speech & Language 67, 2021; T. Stivers et al., Universals and Cultural Variation in Turn-Taking, PNAS 2009; T. Nguyen et al., Generative Spoken Dialogue Language Modeling (dGSLM), TACL 2023; Z. Borsos et al., AudioLM: A Language Modeling Approach to Audio Generation, IEEE/ACM TASLP 2023; Y.-A. Chung et al., w2v-BERT, ASRU 2021; N. Zeghidour et al., SoundStream: An End-to-End Neural Audio Codec, IEEE/ACM TASLP 2022; A. Défossez et al., High Fidelity Neural Audio Compression (EnCodec), 2022; A. Défossez et al., Moshi: a Speech-Text Foundation Model for Real-Time Dialogue, 2024 (code); J. Ao et al., SpeechT5: Unified-Modal Encoder-Decoder for Speech Language Processing, ACL 2022; D. Zhang et al., SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities, Findings of EMNLP 2023; C. Tang et al., SALMONN: Towards Generic Hearing Abilities for Large Language Models, ICLR 2024 — the multimodal layer is in the LLMs (M2) module ch. 8.
• Lab tools: HuggingFace transformers (ASR/TTS pipelines); Silero VAD; jiwer (WER); AudioSeal.