Object Recognition and Computer Vision

Contents

• Introduction and overview
• 0. Review: images, filters, least squares and gradient descent
• 1. Camera geometry
• 2. Local invariant features
• 3. Correspondence and robust estimation
• 4. Efficient visual search
• 5. Category-level recognition: bags of features, sparse coding and dictionary learning
• 6. Neural networks and optimisation methods
• 7. Convolutional neural networks for visual recognition
• 8. Motion and human actions
• 9. 3D object recognition
• 10. Weakly-supervised learning for visual recognition
• 11. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with an M1 in mathematics, computer science or physics who wants to understand, and to build, the systems that recognise objects, scenes and actions in photographs and videos.

Empirical first: photograph the same building from two street corners. Compared pixel by pixel, the two images look unrelated; yet a few hundred local feature matches, filtered by RANSAC, recover the geometric relation between the views in milliseconds. In 2012, a convolutional network cut the top-5 error of the ImageNet challenge from 26.2 % (the second-best entry) to 15.3 % (Krizhevsky, Sutskever and Hinton, 2012). This course explains both successes, and where each still fails.

The course goes from the geometry of a single camera to networks that learn from weak labels. It follows the official themes in order: instance-level recognition, category-level recognition, neural networks, convolutional networks, motion and actions, 3D objects and weak supervision.

It shares tools with Deep Learning, 3D Computer Vision and The Machine Intelligence of Images, taught in the same semester.

What you will be able to do

• Model image formation with the pinhole camera, and estimate cameras, homographies and fundamental matrices from point correspondences.
• Detect corners and blobs (Harris, difference of Gaussians), describe them with SIFT, and prove which transformations each step is covariant or invariant to.
• Match features robustly with the ratio test and RANSAC, and size RANSAC for a given outlier rate.
• Build a large-scale visual search engine with visual words, inverted files and product quantisation.
• Encode images with bags of features, spatial pyramids and sparse codes, and learn a dictionary.
• Derive backpropagation, and train networks with stochastic gradient methods and a sound initialisation.
• Design and analyse convolutional networks for classification and detection: receptive fields, equivariance, residual connections, IoU and non-maximum suppression.
• Estimate optical flow, and explain the main families of action-recognition models.
• Align 3D shapes (Kabsch, ICP), estimate an object pose from 2D–3D matches, and classify point clouds with a permutation-invariant network.
• Learn localisation from image-level labels with multiple-instance learning and class activation maps, and explain contrastive and language supervision.
• Read, reimplement and extend a recent computer vision paper, as the final project requires.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com presents automated object recognition — and, more generally, scene analysis — from photographs and videos as the great challenge of computer vision. Its objective is to give a coherent introductory overview of the image, object and scene models, and of the methods and algorithms, used today to address this challenge.

The same page announces 10 lectures of 3 hours each, with all materials and lectures in English, and lists seven themes. The Université Paris-Saclay course detail gives 30 h of lectures and an assessment type, and displays no list of contents. Each theme, and each part of a composite theme, has a row below.

Beyond the official themes, these notes add the evaluation of detectors (ch. 7), a self-contained review of the mathematical tools (ch. 0) and a self-study chapter on research directions (ch. 11).

Where this course fits in the programme

The official prerequisites are basic linear algebra, analysis and probability. The course is taught in the first period of semester 1 by Gül Varol (ENPC), Ivan Laptev (INRIA), Jean Ponce (ENS Ulm), Cordelia Schmid (INRIA), Josef Sivic (INRIA) and Mathieu Aubry (ENPC), as listed on master-mva.com.

• Preparation — the refreshers Linear Algebra, Analysis and Optimization (SVD, least squares, gradient descent), Probability and Statistics (Bayes’ rule, binomial and chi-square laws) and Scientific Python, NumPy and PyTorch (the language of the assignments).
• Related courses in semester 1 — Deep Learning, 3D Computer Vision, The Machine Intelligence of Images, Convex Optimization, Introduction to Statistical Learning, Theoretical Foundations of Deep Learning, Sub-Pixel Image Processing and Geometric Data Analysis.
• Courses that build on it or share its tools in semester 2 —Point Clouds and 3D Modelling, Generative Models for Images, Deep Learning in Practice, AI and Computer Vision for Cultural Heritage, Representation Learning for Computer Vision and Medical Imaging, Deep Learning for Medical Imaging, Remote Sensing Data, Multimodal Explainable AI and Kernel Methods for Machine Learning.
• Research — the final project is a first research experience, and a natural step towards a vision topic for the research internship.

Course outline

• Review: images, filters, least squares and gradient descent — the four tools used everywhere else, with their short proofs.
• Camera geometry — the pinhole camera, calibration, homographies, epipolar geometry and triangulation.
• Local invariant features — Harris corners, scale space, scale selection, SIFT and affine regions.
• Correspondence and robust estimation — matching, the ratio test, RANSAC and Hough voting.
• Efficient visual search — visual words, inverted files, vocabulary trees and product quantisation.
• Category-level recognition — bags of features, kernels, spatial pyramids, sparse coding and dictionary learning.
• Neural networks and optimisation methods — backpropagation, stochastic gradient methods, initialisation and regularisation.
• Convolutional neural networks — equivariance, receptive fields, residual networks, transfer, detection and segmentation.
• Motion and human actions — optical flow and the families of action-recognition models.
• 3D object recognition — pose estimation, rigid alignment, point-cloud networks and recognition with 3D models.
• Weakly-supervised learning — multiple-instance learning, weakly supervised detection, class activation maps, contrastive and language supervision.
• Research directions and open problems — foundation models, geometry and learning, video and language (self-study).

Session plan

30 h in class = 10 sessions of 3 h.

The volume is that of the Université Paris-Saclay programme table (30 h of lectures), and it matches the organisation announced on master-mva.com, 10 lectures of 3 hours each. The split of the themes into sessions below is a proposal; it follows the official order of the themes, which is also the order of difficulty.

Level 1 = foundations (reviews, definitions, first worked examples), 2 = core methods and their proofs, 3 = advanced material close to current research; the level never decreases from one session to the next.

Self-study, outside the 30 h (about 4 h in total): ch. 11 Research directions and open problems (≈ 3 h) · Exercise 18 on repeatability under viewpoint change (≈ 1 h).

How to study this module

• Read the chapters in order: each one uses only the previous ones, and ch. 0 can be skipped by a reader fluent in SVD and convolution.
• After each session, do the exercises listed in its row of the session plan, then run the Python block of the chapter and change one parameter.
• The Python environment is the one of the scientific Python refresher: NumPy, SciPy, scikit-learn and PyTorch on CPU. Every code block runs in less than 20 seconds on a laptop CPU, with synthetic data or the digits data set bundled with scikit-learn.
• Start the final project early (see Projects and assessment): choose a paper after session 7, once the instance-level, category-level and deep learning tools are in place.
• Keep a page of formulas: projection P = K[R \mid t], epipolar constraint, RANSAC iteration count, receptive field recursion, Lucas–Kanade normal equations.

0. Review: images, filters, least squares and gradient descent

Every later chapter uses the same few tools: images as arrays, linear filtering, the singular value decomposition and gradient descent. This review states them once, with the short proofs that the course relies on.

The refreshers on linear algebra and optimisation and on scientific Python cover the same ground at a slower pace.

This is why convolution is the natural first layer of a vision model: it is the only linear way to process every position of the image identically (ch. 7).

Every estimation problem of ch. 1 — camera, homography, fundamental matrix, triangulation — reduces to this proposition. Stack one or two linear equations per correspondence into a matrix A, and take its last right singular vector.

Sparse coding (ch. 5) and the analysis of stochastic gradient descent (ch. 6) both start from this inequality.

The three facts above can be checked numerically in a few lines.

import numpy as np
from scipy import ndimage

rng = np.random.default_rng(0)
I = rng.random((64, 64))

# 1. Convolution commutes with (circular) translations
k = rng.random((5, 5))
conv = lambda img: ndimage.convolve(img, k, mode="wrap")
shift = lambda img: np.roll(img, (3, -2), axis=(0, 1))
print("equivariance error:", np.abs(conv(shift(I)) - shift(conv(I))).max())

# 2. Derivative-of-Gaussian gradient of a linear ramp 0.5 x + 0.25 y
y, x = np.mgrid[0:64, 0:64].astype(float)
ramp = 0.5 * x + 0.25 * y
gx = ndimage.gaussian_filter(ramp, 2.0, order=(0, 1))   # d/dx (along columns)
gy = ndimage.gaussian_filter(ramp, 2.0, order=(1, 0))   # d/dy (along rows)
print("gradient at the centre:", gx[32, 32].round(3), gy[32, 32].round(3))

# 3. Homogeneous least squares: minimise ||A x|| subject to ||x|| = 1
A = rng.normal(size=(20, 4))
x_true = np.array([1.0, -2.0, 0.5, 1.0]) / np.sqrt(6.25)
A -= np.outer(A @ x_true, x_true)           # x_true is now an exact null vector
A += 1e-3 * rng.normal(size=A.shape)        # small noise
_, s, Vt = np.linalg.svd(A)
x_hat = Vt[-1] * np.sign(Vt[-1] @ x_true)   # fix the sign ambiguity
print("smallest singular value:", s[-1].round(5))
print("error on x:", np.linalg.norm(x_hat - x_true).round(5))

The equivariance error is exactly zero, the filtered gradient of the ramp is (0.5, 0.25), and the last right singular vector recovers the null vector to 2 \times 10^{-4}.

1. Camera geometry

Instance-level recognition asks whether this object — this book cover, this façade — appears in an image. A rigid object obeys the laws of perspective, so geometry gives the strongest test: a correct set of matches must be explained by a camera. This chapter builds the camera model and the constraints between two views.

Vanishing points calibrate a camera from a single image of a building. With zero skew and square pixels, the principal point is the orthocentre of the triangle formed by the vanishing points of three orthogonal directions, and the focal length follows (Hartley and Zisserman, 2004).

In practice one films a planar checkerboard in several positions (Zhang, 2000). Each view gives a homography between the board and the image, and each homography gives two linear constraints on K.

Part 1 is why a book cover, a painting or a façade can be verified with a homography. Part 2 is the basis of panorama stitching.

Without step 1, pixel coordinates of order 10^3 make the columns of the system differ by six orders of magnitude, and the solution becomes very sensitive to noise (Hartley, 1997). The code fits a homography to 50 correspondences corrupted by 0.5 px of noise.

import numpy as np
rng = np.random.default_rng(1)
np.set_printoptions(suppress=True, precision=4)

def normalise(p):
    """Similarity T: centroid to 0, mean distance to sqrt(2) (Hartley)."""
    c = p.mean(0)
    s = np.sqrt(2) / np.linalg.norm(p - c, axis=1).mean()
    return np.array([[s, 0, -s * c[0]], [0, s, -s * c[1]], [0, 0, 1]])

def dlt_homography(x, y):
    """H with y ~ H x from n >= 4 correspondences (n x 2 arrays)."""
    Tx, Ty = normalise(x), normalise(y)
    xh = np.c_[x, np.ones(len(x))] @ Tx.T
    yh = np.c_[y, np.ones(len(y))] @ Ty.T
    rows = []
    for (a, b, c), (u, v, w) in zip(xh, yh):      # two rows of y x (H x) = 0
        rows.append([0, 0, 0, -w*a, -w*b, -w*c, v*a, v*b, v*c])
        rows.append([w*a, w*b, w*c, 0, 0, 0, -u*a, -u*b, -u*c])
    h = np.linalg.svd(np.array(rows))[2][-1]       # Proposition 0.4
    H = np.linalg.inv(Ty) @ h.reshape(3, 3) @ Tx   # undo the normalisation
    return H / H[2, 2]

def apply(H, x):
    p = np.c_[x, np.ones(len(x))] @ H.T
    return p[:, :2] / p[:, 2:]

H_true = np.array([[1.1, 0.05, 30], [-0.08, 0.95, 12], [2e-4, -1e-4, 1]])
x = rng.uniform(0, 640, size=(50, 2))
y = apply(H_true, x) + rng.normal(scale=0.5, size=(50, 2))   # 0.5 px noise
H = dlt_homography(x, y)
err = np.linalg.norm(apply(H, x) - y, axis=1)
print("estimated H:\n", H)
print("mean reprojection error (px):", err.mean().round(3))

The mean residual is 0.59 px. For two-dimensional Gaussian noise of standard deviation 0.5 px per coordinate, the mean distance is 0.5\sqrt{\pi/2} \approx 0.63 px: the fit is as good as the data allow.

With calibrated cameras, E has 5 degrees of freedom and two equal non-zero singular values (Exercise 9). Five correspondences then suffice (Nistér, 2004), which matters inside RANSAC (ch. 3).

With noise, the two back-projected rays do not meet. The statistically optimal estimate minimises the reprojection error in both images. Refining all cameras and all points jointly in this way is bundle adjustment, the core of structure from motion (Snavely, Seitz and Szeliski, 2006).

2. Local invariant features

Two photographs of the same object differ by viewpoint, scale, rotation, lighting and occlusion. Local features sidestep most of these nuisances: detect many small distinctive regions in a covariant way, describe each one by an invariant vector, and match the vectors.

The code detects corners on a synthetic image of two rectangles.

import numpy as np
from scipy import ndimage

# Synthetic image: two bright rectangles on a dark background
I = np.zeros((100, 100))
I[20:50, 20:50] = 1.0
I[60:85, 55:90] = 1.0

def harris(I, sigma_d=1.0, sigma_i=2.0, kappa=0.05):
    Ix = ndimage.gaussian_filter(I, sigma_d, order=(0, 1))
    Iy = ndimage.gaussian_filter(I, sigma_d, order=(1, 0))
    Sxx = ndimage.gaussian_filter(Ix * Ix, sigma_i)    # entries of the structure tensor
    Syy = ndimage.gaussian_filter(Iy * Iy, sigma_i)
    Sxy = ndimage.gaussian_filter(Ix * Iy, sigma_i)
    return Sxx * Syy - Sxy**2 - kappa * (Sxx + Syy)**2

R = harris(I)
peaks = (R == ndimage.maximum_filter(R, size=7)) & (R > 0.1 * R.max())
print("corners (row, col):", np.argwhere(peaks).tolist())
print("response on an edge / in a flat zone:", R[35, 20].round(5), R[5, 5].round(5))

# Scale selection: scale-normalised Laplacian at the centre of a Gaussian blob of std 4
yy, xx = np.mgrid[-50:51, -50:51]
blob = np.exp(-(xx**2 + yy**2) / (2 * 4.0**2))
sigmas = np.linspace(1, 10, 91)
resp = [-(s**2) * ndimage.gaussian_laplace(blob, s)[50, 50] for s in sigmas]
print("selected scale:", sigmas[int(np.argmax(resp))].round(2))

The detector returns the eight corners, each one pixel inside the rectangle. The response is negative on the edges and zero in flat zones, as Proposition 2.3 predicts.

The Harris detector is not covariant to scaling: a sharp corner seen from far away becomes a rounded curve when the image is enlarged. Scale covariance needs a representation of the image at all scales.

Clipping limits the influence of a few large gradients, caused for instance by saturation, which the affine model of intensity does not cover.

3. Correspondence and robust estimation

Matching descriptors gives putative correspondences, a large fraction of which are wrong: repeated textures, occlusions and features without a counterpart all produce false matches. This chapter turns putative matches into a verified geometric relation.

The breakdown point of least squares is therefore 0: an arbitrarily small fraction of outliers can make the estimate arbitrarily bad. The code at the end of this chapter shows the same failure for a homography.

The code runs RANSAC on 200 matches, half of which are outliers, and compares it with a least-squares fit on all matches.

import numpy as np
rng = np.random.default_rng(2)

def dlt(x, y):                       # minimal DLT (coordinates already in [0, 1])
    A = []
    for (a, b), (u, v) in zip(x, y):
        A.append([0, 0, 0, -a, -b, -1, v*a, v*b, v])
        A.append([a, b, 1, 0, 0, 0, -u*a, -u*b, -u])
    H = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
    return H / H[2, 2]

def apply(H, x):
    p = np.c_[x, np.ones(len(x))] @ H.T
    return p[:, :2] / p[:, 2:]

def ransac(x, y, thresh, p=0.99, max_iter=10_000):
    best, N, it = np.zeros(len(x), bool), max_iter, 0
    while it < N:
        idx = rng.choice(len(x), 4, replace=False)          # minimal sample
        inliers = np.linalg.norm(apply(dlt(x[idx], y[idx]), x) - y, axis=1) < thresh
        if inliers.sum() > best.sum():
            best, w = inliers, inliers.mean()
            N = min(max_iter, np.log(1 - p) / np.log(1 - w**4 + 1e-12))  # Prop. 3.4
        it += 1
    for _ in range(3):                                       # refit on all inliers, re-classify
        H = dlt(x[best], y[best])
        best = np.linalg.norm(apply(H, x) - y, axis=1) < thresh
    return H, best, it

# 200 putative matches, 50 % outliers, 1 px noise at a 640 px image scale
H_true = np.array([[1.05, 0.1, 0.05], [-0.05, 0.9, 0.1], [0.2, -0.1, 1.0]])
x = rng.uniform(0, 1, (200, 2))
y = apply(H_true, x) + rng.normal(scale=1 / 640, size=(200, 2))
outlier = rng.random(200) < 0.5
y[outlier] = rng.uniform(0, 1, (outlier.sum(), 2))
H, inl, iters = ransac(x, y, thresh=np.sqrt(5.99) / 640)
print("iterations:", iters, "| inliers kept:", inl.sum(), "| true inliers:", (~outlier).sum())
print("false inliers:", (inl & outlier).sum(), "| max |H - H_true|:", np.abs(H - H_true).max().round(4))
H_ls = dlt(x, y)                                             # plain least squares on all matches
print("least squares on all matches, max |H - H_true|:", np.abs(H_ls - H_true).max().round(2))

RANSAC keeps 91 of the 97 true inliers, about the 95 % expected from Proposition 3.6, and accepts no outlier. It runs 437 iterations rather than the 72 of Example 3.5: a model fitted to four noisy points finds fewer inliers than the true model. The best minimal model has 64 inliers, so the running estimate is w = 0.32 instead of 0.49. The least-squares fit on all matches is useless, with entries off by 1.76.

4. Efficient visual search

Matching a query image against one database image costs about 10^6 descriptor comparisons, with 1 000 features on each side. Against a million images, that is 10^{12} comparisons per query. Large-scale search borrows the tools of text retrieval (Sivic and Zisserman, 2003): quantise descriptors into visual words, index the words, and verify geometry only for the best candidates.

For SIFT descriptors (d = 128) with m = 8 and k^* = 256, each vector takes 64 bits instead of 4 096 bits in single precision, and the implicit codebook has 2^{64} centroids. The code implements k-means and product quantisation on synthetic descriptors that lie near an 8-dimensional subspace.

import numpy as np
rng = np.random.default_rng(3)

def kmeans(X, k, iters=20):
    """Lloyd's algorithm (Proposition 4.2): assign, then recentre."""
    C = X[rng.choice(len(X), k, replace=False)]
    for _ in range(iters):
        lab = ((X[:, None, :] - C[None]) ** 2).sum(-1).argmin(1)
        C = np.array([X[lab == j].mean(0) if np.any(lab == j) else C[j] for j in range(k)])
    return C

d, m, ks = 32, 8, 64                   # dimension, sub-vectors, centroids per sub-quantiser
ds = d // m
basis = rng.normal(size=(8, d))        # data near an 8-dimensional subspace, like real descriptors
X = rng.normal(size=(10000, 8)) @ basis + 0.3 * rng.normal(size=(10000, d))
Q = rng.normal(size=(100, 8)) @ basis + 0.3 * rng.normal(size=(100, d))

# One k-means per sub-space; each database vector becomes m one-byte codes
books, codes = [], np.empty((len(X), m), dtype=np.uint8)
for j in range(m):
    sub = X[:, j * ds:(j + 1) * ds]
    books.append(kmeans(sub[:3000], ks))
    codes[:, j] = ((sub[:, None, :] - books[j][None]) ** 2).sum(-1).argmin(1)

def adc(q):
    """Asymmetric distances: m tables of ks entries, then m look-ups per database vector."""
    tables = [((books[j] - q[j * ds:(j + 1) * ds]) ** 2).sum(1) for j in range(m)]
    return sum(tables[j][codes[:, j]] for j in range(m))

hits = [np.argmin(((X - q) ** 2).sum(1)) in np.argsort(adc(q))[:10] for q in Q]
print(f"memory per vector: {m} bytes instead of {4 * d} bytes (float32)")
print("recall@10 of the exact nearest neighbour:", np.mean(hits))

With 8 bytes per vector instead of 128, the exact nearest neighbour is ranked among the first ten for 89 % of the queries. Combined with a coarse quantiser that selects a few cells to scan, this is the IVFADC scheme of Jégou, Douze and Schmid (2011), which later systems scaled to billions of vectors (Johnson, Douze and Jégou, 2019).

5. Category-level recognition: bags of features, sparse coding and dictionary learning

Category-level recognition asks whether an image contains a car, any car. Geometric verification no longer applies, because two different cars are not related by a homography. From 2004 to 2012, the dominant answer was statistical: describe the image by the distribution of its local features, and learn a classifier on that description.

K-SVD updates the atoms one at a time with a rank-one SVD (Aharon, Elad and Bruckstein, 2006). Online dictionary learning processes one mini-batch at a time and scales to millions of patches (Mairal, Bach, Ponce and Sapiro, 2009). Dictionaries learned on natural image patches resemble oriented edge detectors, like the receptive fields of the primary visual cortex (Olshausen and Field, 1996).

Sparse codes, max-pooled over a spatial pyramid and fed to a linear SVM, matched non-linear kernel SVMs on bags of features at a fraction of the cost (Yang et al., 2009). The code compares these representations on the 8 × 8 digits bundled with scikit-learn, using all 4 × 4 patches as local descriptors.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(5)

imgs, y = load_digits(return_X_y=True)
imgs = imgs.reshape(-1, 8, 8) / 16.0
# all 4 x 4 patches (5 x 5 positions), centred: shape (n, 25, 16)
P = np.stack([imgs[:, i:i + 4, j:j + 4].reshape(len(imgs), 16)
              for i in range(5) for j in range(5)], 1)
P = P - P.mean(-1, keepdims=True)
cell = np.array([(i > 2) * 2 + (j > 2) for i in range(5) for j in range(5)])  # 2 x 2 grid

def pyramid(F, pool):
    """Level 0 (whole image) + level 1 (2 x 2 cells) pooling of patch codes F: (n, 25, K)."""
    return np.concatenate([pool(F, 1)] + [pool(F[:, cell == c], 1) for c in range(4)], 1)

def ista(X, D, lam, iters=50):
    """Lasso codes for every row of X (Algorithm 5.6), step 1/L with L = ||D||_2^2."""
    L = np.linalg.norm(D, 2) ** 2
    A = np.zeros((len(X), D.shape[1]))
    for _ in range(iters):
        Z = A - (A @ D.T - X) @ D / L
        A = np.sign(Z) * np.maximum(np.abs(Z) - lam / L, 0)          # soft-thresholding
    return A

K, flat = 32, P.reshape(-1, 16)
train = flat[rng.choice(len(flat), 2000, replace=False)]
# (a) bag of features: k-means vocabulary, hard assignment, sum pooling
C = train[rng.choice(len(train), K, replace=False)]
for _ in range(20):
    lab = ((train[:, None] - C[None]) ** 2).sum(-1).argmin(1)
    C = np.array([train[lab == k].mean(0) if np.any(lab == k) else C[k] for k in range(K)])
H = np.eye(K)[((P[..., None, :] - C) ** 2).sum(-1).argmin(-1)]       # one-hot codes
# (b) sparse coding: alternate ISTA codes and least-squares dictionary updates
D = train[:K].T / np.linalg.norm(train[:K], axis=1)
for _ in range(5):
    A = ista(train, D, lam=0.05)
    D = np.linalg.lstsq(A, train, rcond=None)[0].T
    D /= np.maximum(np.linalg.norm(D, axis=0), 1e-8)
S = np.abs(ista(flat, D, lam=0.05)).reshape(len(imgs), 25, K)

feats = {"raw pixels": imgs.reshape(len(imgs), -1),
         "BoF, sum pooling, level 0": H.sum(1),
         "BoF, spatial pyramid": pyramid(H, np.sum),
         "sparse codes, max pooling, pyramid": pyramid(S, np.max)}
for name, F in feats.items():
    F = F / np.linalg.norm(F, axis=1, keepdims=True)                 # L2-normalise
    Ftr, Fte, ytr, yte = train_test_split(F, y, test_size=0.3, random_state=0, stratify=y)
    acc = LinearSVC(C=1.0, max_iter=20000).fit(Ftr, ytr).score(Fte, yte)
    print(f"{name:36s} test accuracy {acc:.3f}")

The orderless bag of features loses information (88.0 % against 96.3 % for raw pixels). The spatial pyramid recovers it (96.1 %), and sparse codes with max pooling do best (98.9 %).

6. Neural networks and optimisation methods

The pipelines of ch. 5 learn only their last stage, the classifier; descriptors, coding and pooling are designed by hand. Neural networks learn every stage jointly, by gradient descent on a loss computed from the labels. This chapter covers the model, the computation of its gradient and the optimisers.

One hidden layer already suffices to approximate any continuous function on a compact set: Cybenko (1989) and Hornik (1991) prove it for sigmoidal and bounded activations, and Leshno et al. (1993) for every non-polynomial one, the ReLU included. Depth makes such approximations far more economical, and it is what makes features reusable across tasks.

Reverse-mode automatic differentiation, as in PyTorch, applies the same recursion to any computation graph.

The first term is the rate of deterministic gradient descent; the second is a noise floor proportional to the step. This explains learning-rate decay schedules, and the effect of the batch size: averaging b independent per-example gradients divides their variance by b (Bottou, Curtis and Nocedal, 2018).

The code implements Theorem 6.3 in NumPy, checks it against a finite difference, and trains with momentum on the two-moons data set.

import numpy as np
from sklearn.datasets import make_moons
rng = np.random.default_rng(0)
X, y = make_moons(400, noise=0.2, random_state=0)
Y = np.eye(2)[y]                                      # one-hot targets

def init(sizes):                                      # He initialisation (Proposition 6.6)
    return [(rng.normal(0, np.sqrt(2 / a), (a, b)), np.zeros(b)) for a, b in zip(sizes, sizes[1:])]

def forward(params, X):
    acts = [X]
    for i, (W, b) in enumerate(params):
        z = acts[-1] @ W + b
        acts.append(np.maximum(z, 0) if i < len(params) - 1 else z)
    return acts

def loss_and_grads(params, X, Y):
    acts = forward(params, X)
    z = acts[-1] - acts[-1].max(1, keepdims=True)
    p = np.exp(z) / np.exp(z).sum(1, keepdims=True)  # softmax
    loss = -np.mean(np.sum(Y * np.log(p), 1))
    delta = (p - Y) / len(X)                          # Proposition 6.2
    grads = []
    for i in reversed(range(len(params))):            # backward pass (Theorem 6.3)
        grads.append((acts[i].T @ delta, delta.sum(0)))
        if i > 0:
            delta = (delta @ params[i][0].T) * (acts[i] > 0)
    return loss, grads[::-1]

params = init([2, 32, 32, 2])
loss, grads = loss_and_grads(params, X, Y)
eps = 1e-6
params[0][0][0, 0] += eps                             # finite-difference check of one weight
loss_eps, _ = loss_and_grads(params, X, Y)
params[0][0][0, 0] -= eps
print(f"backprop {grads[0][0][0, 0]:.6f}   finite difference {(loss_eps - loss) / eps:.6f}")

vel = [(np.zeros_like(W), np.zeros_like(b)) for W, b in params]
for step in range(2000):                              # mini-batch SGD with momentum 0.9
    idx = rng.choice(len(X), 32, replace=False)
    _, grads = loss_and_grads(params, X[idx], Y[idx])
    for (W, b), (gW, gb), (vW, vb) in zip(params, grads, vel):
        vW *= 0.9; vW -= 0.05 * gW; W += vW
        vb *= 0.9; vb -= 0.05 * gb; b += vb
print("training accuracy:", (forward(params, X)[-1].argmax(1) == y).mean())

The backpropagated derivative and the finite difference agree to six decimals, and the network separates the two moons with 97.5 % training accuracy.

7. Convolutional neural networks for visual recognition

A fully connected layer from a 224 × 224 colour image to the 112 × 112 × 64 feature map of a standard first layer would need about 10^{11} weights (Example 7.2). Convolutional networks build in two properties of images, locality and the equivalence of positions. Since 2012 they have dominated recognition (Krizhevsky, Sutskever and Hinton, 2012).

Strides and pooling break exact equivariance: a one-pixel shift of the input can change the output of a strided layer, a form of aliasing (Zhang, 2019). Recognition networks are therefore only approximately shift-invariant, and data augmentation fills the gap.

Three 3 \times 3 convolutions with stride 1 have the receptive field 1 + 2 + 2 + 2 = 7 of a single 7 \times 7 layer. They use 27C^2 weights instead of 49C^2 for C channels, and three non-linearities instead of one: this is the design principle of VGG networks (Simonyan and Zisserman, 2015).

The code trains a small CNN with global average pooling on the 8 × 8 digits, and checks Proposition 7.3 on a circular convolution.

import torch, torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
torch.manual_seed(0)

X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32).reshape(-1, 1, 8, 8)
y = torch.tensor(y)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

net = nn.Sequential(
    nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(),     # 16 x 8 x 8
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),    # 32 x 8 x 8
    nn.MaxPool2d(2),                               # 32 x 4 x 4
    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),    # 64 x 4 x 4
    nn.AdaptiveAvgPool2d(1), nn.Flatten(),         # global average pooling -> 64
    nn.Linear(64, 10))
print("parameters:", sum(p.numel() for p in net.parameters()))
opt = torch.optim.AdamW(net.parameters(), lr=3e-3, weight_decay=1e-4)
for epoch in range(30):
    perm = torch.randperm(len(Xtr))
    for i in range(0, len(Xtr), 64):
        idx = perm[i:i + 64]
        loss = nn.functional.cross_entropy(net(Xtr[idx]), ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    print(f"test accuracy: {(net(Xte).argmax(1) == yte).float().mean():.3f}")
    # Proposition 7.3: a circular convolution commutes with a circular shift
    conv = nn.Conv2d(1, 4, 3, padding=1, padding_mode="circular")
    s = lambda t: torch.roll(t, shifts=(1, 2), dims=(2, 3))
    print("equivariance error:", (conv(s(Xte[:5])) - s(conv(Xte[:5]))).abs().max().item())

With 23 946 parameters, the network reaches about 98 % test accuracy, against 96 % for a linear classifier on raw pixels (ch. 5). The equivariance error of the circular convolution is exactly zero.

import numpy as np

def iou(a, B):
    """IoU between box a and each row of B; boxes are (x1, y1, x2, y2)."""
    w = np.clip(np.minimum(a[2], B[:, 2]) - np.maximum(a[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(a[3], B[:, 3]) - np.maximum(a[1], B[:, 1]), 0, None)
    area = lambda r: (r[..., 2] - r[..., 0]) * (r[..., 3] - r[..., 1])
    return w * h / (area(a) + area(B) - w * h)

def nms(boxes, scores, thr=0.5):
    """Greedy non-maximum suppression (Algorithm 7.8)."""
    order, keep = np.argsort(-scores), []
    while len(order):
        i, order = order[0], order[1:]
        keep.append(int(i))
        order = order[iou(boxes[i], boxes[order]) < thr]
    return keep

boxes = np.array([[0, 0, 10, 10], [1, 1, 11, 11], [5, 5, 15, 15], [20, 20, 30, 30]], float)
scores = np.array([0.9, 0.8, 0.7, 0.6])
print("IoU of box 0 with boxes 1-3:", iou(boxes[0], boxes[1:]).round(3))
print("boxes kept by NMS:", nms(boxes, scores))

Boxes 0 and 2 overlap on a 5 \times 5 square, so their IoU is 25/175 \approx 0.143; box 1, shifted by one pixel, has IoU 0.681 with box 0 and is suppressed. NMS keeps boxes 0, 2 and 3.

8. Motion and human actions

A video adds time. Motion separates objects from the background, reveals 3D structure, and is the main cue for recognising what people do. This chapter first estimates the motion between two frames, the optical flow, then studies models that recognise human actions.

Seen through a small aperture, a straight edge looks the same whatever its motion along the edge: this is the barber-pole illusion.

The points that can be tracked reliably are therefore those where both eigenvalues of M are large — the corners of ch. 2. This is the criterion of the KLT tracker, “good features to track” (Shi and Tomasi, 1994). Large motions violate the linearisation; they are handled coarse to fine on a Gaussian pyramid, and by iterating the estimate after warping, as in the code below.

import numpy as np
from scipy import ndimage
rng = np.random.default_rng(4)

I0 = ndimage.gaussian_filter(rng.random((128, 128)), 3.0)        # smooth random texture
d_true = np.array([0.6, -0.4])                                   # (row, column) motion in pixels
I1 = ndimage.shift(I0, d_true, order=3, mode="wrap")             # I1(x) = I0(x - d)

def structure(I, sigma=1.0, window=6.0):
    """Gradients and window-averaged structure tensor M (Definition 2.2)."""
    Iy = ndimage.gaussian_filter(I, sigma, order=(1, 0))
    Ix = ndimage.gaussian_filter(I, sigma, order=(0, 1))
    w = lambda f: ndimage.gaussian_filter(f, window)
    M = np.array([[w(Iy * Iy), w(Iy * Ix)], [w(Ix * Iy), w(Ix * Ix)]]).transpose(2, 3, 0, 1)
    return Iy, Ix, w, M

def lucas_kanade(I0, I1, n_iter=3):
    Iy, Ix, w, M = structure(I0)
    d = np.zeros(I0.shape + (2,))
    grid = np.indices(I0.shape).astype(float)
    for _ in range(n_iter):                                       # Gauss-Newton refinement
        warped = ndimage.map_coordinates(I1, grid + d.transpose(2, 0, 1), order=3, mode="wrap")
        It = ndimage.gaussian_filter(warped - I0, 1.0)
        b = -np.stack([w(Iy * It), w(Ix * It)], -1)
        d += np.linalg.solve(M, b[..., None])[..., 0]             # normal equations (Theorem 8.3)
    return d

flow = lucas_kanade(I0, I1)
print("median flow (row, col):", np.median(flow.reshape(-1, 2), 0).round(3), " true:", d_true)

# Aperture problem: on vertical stripes the structure tensor is singular
stripes = np.tile(np.sin(np.arange(128) / 3.0), (128, 1))
print("eigenvalues of M on stripes:", np.linalg.eigvalsh(structure(stripes)[3][64, 64]).round(6))

The estimated motion, (0.599, -0.400), matches the true one to a thousandth of a pixel. On vertical stripes one eigenvalue of M vanishes: the vertical component of the motion is invisible, which is the aperture problem.

Human pose is another strong cue for actions. Rendered images of synthetic bodies provide training data for body-part segmentation and depth estimation without manual annotation (Varol et al., 2017).

9. 3D object recognition

Objects are three-dimensional. Recognising one fully means recovering its category, its 3D pose and sometimes its shape, from images or from 3D sensors. This chapter combines the geometry of ch. 1 with the learned features of ch. 7.

import numpy as np
from scipy.spatial import cKDTree
from scipy.spatial.transform import Rotation
rng = np.random.default_rng(5)

def kabsch(P, Q):
    """R, t minimising sum ||R p_i + t - q_i||^2 over rotations (Theorem 9.3)."""
    p0, q0 = P.mean(0), Q.mean(0)
    U, _, Vt = np.linalg.svd((P - p0).T @ (Q - q0))
    D = np.diag([1.0, 1.0, np.sign(np.linalg.det(Vt.T @ U.T))])   # no reflection
    R = Vt.T @ D @ U.T
    return R, q0 - R @ p0

def icp(P, Q, iters=40):
    """Iterative closest point (Algorithm 9.4)."""
    tree, R, t = cKDTree(Q), np.eye(3), np.zeros(3)
    for _ in range(iters):
        _, nn = tree.query(P @ R.T + t)          # closest-point correspondences
        R, t = kabsch(P, Q[nn])
    return R, t

P = rng.normal(size=(500, 3)) * [3.0, 2.0, 1.0]                     # anisotropic cloud
R_true = Rotation.from_euler("xyz", [10, -15, 20], degrees=True).as_matrix()
Q = P @ R_true.T + [0.5, -0.2, 0.3] + 0.01 * rng.normal(size=P.shape)
err = lambda R: np.degrees(Rotation.from_matrix(R.T @ R_true).magnitude()).round(4)
R, t = kabsch(P, Q)                                                 # known correspondences
print("Kabsch rotation error (deg):", err(R))
Q = Q[rng.permutation(len(Q))]                                      # correspondences now unknown
R, t = icp(P, Q)
print("ICP rotation error (deg):", err(R), " translation:", t.round(3))

Kabsch recovers the rotation to about 0.01°. ICP, started from the identity, reaches the same solution without correspondences, because the initial rotation is moderate and the cloud is anisotropic.

The code trains a small PointNet to tell spheres, cubes and cylinders apart, each shape seen under a random orthogonal transformation.

import torch, torch.nn as nn
torch.manual_seed(0)

def sample(kind, n=128):
    """n points on a sphere (0), a cube (1) or a cylinder (2)."""
    if kind == 0:
        x = torch.randn(n, 3)
        return x / x.norm(dim=1, keepdim=True)
    if kind == 1:
        x = torch.rand(n, 3) * 2 - 1
        x[torch.arange(n), torch.randint(3, (n,))] = torch.sign(torch.randn(n))
        return x
    theta, z = torch.rand(n) * 2 * torch.pi, torch.rand(n) * 2 - 1
    return torch.stack([theta.cos(), theta.sin(), z], 1)

def batch(m):
    y = torch.randint(3, (m,))
    X = torch.stack([sample(int(k)) for k in y])
    R = torch.linalg.qr(torch.randn(m, 3, 3))[0]        # random orthogonal transform per shape
    return X @ R, y

class PointNet(nn.Module):                              # f(X) = g(max_i h(x_i)), Prop. 9.5
    def __init__(self):
        super().__init__()
        self.h = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 128), nn.ReLU())
        self.g = nn.Sequential(nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, 3))
    def forward(self, X):                               # X: (batch, points, 3)
        return self.g(self.h(X).max(dim=1).values)

net = PointNet()
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
for step in range(300):
    X, y = batch(32)
    loss = nn.functional.cross_entropy(net(X), y)
    opt.zero_grad(); loss.backward(); opt.step()
X, y = batch(300)
with torch.no_grad():
    acc = (net(X).argmax(1) == y).float().mean().item()
    gap = (net(X) - net(X[:, torch.randperm(X.shape[1])])).abs().max().item()
print(f"test accuracy {acc:.3f}; largest output change under a point permutation {gap:.1e}")

The three shapes are separated almost perfectly, and permuting the points changes the output by exactly 0, as Proposition 9.5 guarantees.

10. Weakly-supervised learning for visual recognition

Boxes and masks are expensive to annotate, object by object and image by image, and video multiplies the cost by the number of frames. Images and videos, however, come with weak signals: image-level tags, captions, subtitles, narration. Weakly supervised learning extracts localised recognition from such signals.

The same property is the weakness of max pooling. An early mistake — the most discriminative part, such as a face instead of the whole person, or a context region, such as water for boats — is reinforced rather than corrected.

The code trains an instance scorer from bag labels only, with the three poolings. A bag holds ten points of the plane and is positive when one of them lies near (2, 2).

import math, torch, torch.nn as nn
torch.manual_seed(0)

def make_bags(m, n=10):
    """Bags of n points in the plane; a bag is positive iff one point lies near (2, 2)."""
    X = 1.5 * torch.randn(m, n, 2)
    near = (X - torch.tensor([2.0, 2.0])).norm(dim=-1) < 1.0
    X[near] = X[near] - 3.0                                   # keep negatives away from (2, 2)
    y, k = torch.randint(2, (m,)), torch.randint(n, (m,))
    pos = y == 1
    X[pos, k[pos]] = torch.tensor([2.0, 2.0]) + 0.3 * torch.randn(int(pos.sum()), 2)
    return X, y, k                                           # k = index of the key instance

def pool(s, kind, r=5.0):
    if kind == "max":
        return s.max(1).values
    if kind == "mean":
        return s.mean(1)
    return (torch.logsumexp(r * s, 1) - math.log(s.shape[1])) / r   # LSE, Proposition 10.4

for kind in ["mean", "lse", "max"]:
    torch.manual_seed(0)
    f = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 1))  # instance scorer
    opt = torch.optim.Adam(f.parameters(), lr=1e-2)
    for step in range(300):
        X, y, _ = make_bags(128)
        loss = nn.functional.binary_cross_entropy_with_logits(pool(f(X).squeeze(-1), kind), y.float())
        opt.zero_grad(); loss.backward(); opt.step()
    X, y, k = make_bags(2000)
    with torch.no_grad():
        s = f(X).squeeze(-1)
        bag_acc = ((pool(s, kind) > 0).long() == y).float().mean().item()
        found = (s[y == 1].argmax(1) == k[y == 1]).float().mean().item()
    print(f"{kind:>4} pooling: bag accuracy {bag_acc:.3f}, key instance ranked first {found:.3f}")

Mean pooling dilutes the single positive instance among ten, and classifies only 80 % of the bags correctly. Max and LSE pooling classify about 97 % of the bags and rank the key instance first in about 98 % of the positive bags, although no instance label was ever used.

Oquab et al. (2015) obtained localisation from image labels with global max pooling over the score maps of a fully convolutional network: the multiple-instance view of Proposition 10.3, applied to positions.

SimCLR learns ImageNet features in this way that a linear classifier uses almost as well as supervised ones (Chen et al., 2020). CLIP applies the same loss to 400 million image–caption pairs collected on the web, with an image encoder and a text encoder (Radford et al., 2021). Classification then needs no training: embed a prompt such as “a photo of a dog” for each class and pick the closest one, which is zero-shot recognition.

11. Research directions and open problems

This chapter is for self-study, after the ten sessions. It points to active research lines that extend each theme of the course, and to questions that remain open. It cites only published work, with authors and year, so that each line can be followed from its source.

Visual foundation models

Self-supervised vision transformers learn features that transfer without labels. Work by Caron and coauthors (2021) shows that their attention maps segment objects without any mask supervision. DINOv2 (Oquab and coauthors, 2024) scales this recipe into general-purpose frozen features, and masked autoencoders reconstruct hidden patches as a pretext task (He and coauthors, 2022).

Segment Anything (Kirillov and coauthors, 2023) segments any object from a point or a box prompt, trained on a data engine that alternates model predictions and human corrections. The open questions concern what such features fail to encode: fine geometry, counting, spatial relations, and robustness to distribution shifts such as the corruptions of ImageNet-C (Hendrycks and Dietterich, 2019).

Vision and language

CLIP (ch. 10) made recognition open-vocabulary. Visual language models such as Flamingo connect a frozen image encoder to a language model and answer questions about images from a few examples (Alayrac and coauthors, 2022). Narrated videos train video question answering without manual annotation (Yang and coauthors, 2021).

Grounding remains open: a model can describe an image fluently without localising what it describes, and it can describe objects that are not there. Long videos, of minutes or hours, remain hard for every current architecture.

Geometry and learning

Neural radiance fields represent a scene by a network that maps a 3D point and a viewing direction to a colour and a density, trained from posed images by differentiable volume rendering (Mildenhall and coauthors, 2020). 3D Gaussian splatting replaces the network by explicit Gaussians and renders in real time (Kerbl and coauthors, 2023).

DUSt3R regresses dense 3D point maps directly from pairs of uncalibrated images, bypassing the explicit camera estimation of ch. 1 (Wang and coauthors, 2024). Learned matchers such as SuperGlue (Sarlin and coauthors, 2020) and LoFTR (Sun and coauthors, 2021) replace the ratio test and part of RANSAC (ch. 3). A recurring open question is when learned components beat classical geometry such as COLMAP (Schönberger and Frahm, 2016), and how to give them calibrated uncertainty.

Search, synthesis and robustness

Billion-scale similarity search on GPUs builds on product quantisation (Johnson, Douze and Jégou, 2019), and graph-based indexes offer another trade-off between speed and recall (Malkov and Yashunin, 2020). Synthetic data, such as rendered human bodies, provide unlimited labels (Varol and coauthors, 2017), but the gap between synthetic and real images is still closed by fine-tuning rather than by theory.

Open problems

• Learning to localise rare categories from weak or noisy labels, with guarantees on what multiple-instance learning can recover.
• Recognising actions over long time scales, and anticipating them, from narrated or unlabelled video.
• Reconciling learned 3D reconstruction with the exact geometry of ch. 1: consistency, scale and uncertainty.
• Evaluating recognition beyond benchmark accuracy: robustness, spatial reasoning and calibration.
• Reducing the data and energy needed to train visual foundation models.

Exercises

1. ★ Projecting by hand. A camera has K with f_x = f_y = 800, principal point (320, 240), zero skew, and R = I, t = 0.

   1. Project the point (1, 0.5, 5). (b) Compute the vanishing point of the direction d = (1, 0, 1).

   Solution: (a) u = 800 \times 1/5 + 320 = 480 and v = 800 \times 0.5/5 + 240 = 320, so the image point is (480, 320).

   2. By Proposition 1.4, \tilde{v} \sim Kd = (800 + 320,\ 240,\ 1), so the vanishing point is (1120, 240), outside the 640-pixel-wide image, on the horizon row of the camera.

2. ★ Homography of a panning camera. Same K; the camera rotates by \theta = 10° about its vertical axis, with R = \begin{pmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{pmatrix}. Where does the homography of Theorem 1.6 send the principal point?

   Solution: K^{-1}(320, 240, 1)^\top = (0, 0, 1)^\top, and R(0, 0, 1)^\top = (\sin\theta, 0, \cos\theta)^\top. Applying K and dividing by \cos\theta gives (800 \tan\theta + 320,\ 240) = (461.1,\ 240). A pan of 10° moves the image content by about 141 pixels horizontally.

3. ★ Harris by hand. A window of four pixels with uniform weights has gradients (1, 0), (1, 0), (0, 1), (0, 1). Compute M and the Harris response for \kappa = 0.05. Repeat with four gradients equal to (1, 0), and with the first configuration rotated by 45°.

   Solution: in the first case M = \operatorname{diag}(2, 2) and R = 4 - 0.05 \times 16 = 3.2 > 0: a corner.

   In the second case M = \operatorname{diag}(4, 0) and R = -0.8 < 0: an edge. After a rotation by 45°, the gradients become (1, 1)/\sqrt2 twice and (-1, 1)/\sqrt2 twice, and M is again \operatorname{diag}(2, 2), as Proposition 2.3 predicts.

4. ★ Sizing RANSAC. With p = 0.99 and 40 % of outliers, how many iterations do the 4-point homography, the 7-point and the 8-point fundamental matrix solvers need?

   Solution: with w = 0.6, Proposition 3.4 gives N = \lceil \log 0.01 / \log(1 - 0.6^s) \rceil: N = 34 \ (s = 4), \qquad 163 \ (s = 7), \qquad 272 \ (s = 8).

5. ★ tf-idf by hand. A database has four images over the words a, b, c: image 1 = \{a, a, b\}, image 2 = \{a, c\}, image 3 = \{a, b, b\}, image 4 = \{a, a, a\}. Rank them for the query \{a, b, c\} by the cosine similarity of tf-idf vectors.

   Solution: the idf weights are \log(4/4) = 0 for a, \log 2 for b and \log 4 = 2\log 2 for c.

   The query vector is proportional to (0, 1, 2).

   Images 1 and 3 are proportional to (0, 1, 0), image 2 to (0, 0, 1), and image 4 has a zero vector. The scores are 2/\sqrt5 \approx 0.894 for image 2, 1/\sqrt5 \approx 0.447 for images 1 and 3, and 0 for image 4. The word a, present everywhere, plays no role.

6. ★ CNN arithmetic. An input of size 32 \times 32 \times 3 goes through a 5 \times 5 convolution with 6 filters (no padding), a 2 \times 2 max pooling with stride 2, a 5 \times 5 convolution with 16 filters, and a second pooling. Give the output shapes, the parameter counts and the final receptive field.

   Solution: shapes 28 \times 28 \times 6, 14 \times 14 \times 6, 10 \times 10 \times 16 and 5 \times 5 \times 16. Parameters: 5 \cdot 5 \cdot 3 \cdot 6 + 6 = 456 and 5 \cdot 5 \cdot 6 \cdot 16 + 16 = 2416.

   By Proposition 7.4, the receptive field grows as 1 \to 5 \to 6 \to 14 \to 16, with jumps 1, 1, 2, 2, 4: each final unit sees a 16 \times 16 patch.

7. ★ Lucas–Kanade in a window. In a window with uniform weights, two pixels have \nabla I = (1, 0) and I_t = -0.5, and two have \nabla I = (0, 1) and I_t = 0.3. Compute the flow. What happens if all four pixels have \nabla I = (1, 0) and I_t = -0.5?

   Solution: M = \operatorname{diag}(2, 2) and b = \sum I_t \nabla I = (-1, 0.6), so u^\star = -M^{-1}b = (0.5, -0.3).

   In the second case M = \operatorname{diag}(4, 0) is singular. Only the normal flow u_n = 0.5 along the first axis is determined (Proposition 8.2); the second component is arbitrary.

8. ★ Gradients of LSE pooling. Show that the gradient of \operatorname{LSE}_r is \operatorname{softmax}(rs), and compute it for s = (2, 1, 0) with r = 1 and r = 5.

   Solution: differentiating \frac1r \log \sum_i e^{rs_i} gives e^{rs_i} / \sum_j e^{rs_j}; the constant -\frac1r\log n does not matter.

   For r = 1, the gradient is (0.665, 0.245, 0.090); for r = 5, it is (0.9933, 0.0067, 0.00005). Large r behaves like max pooling (Proposition 10.3), small r spreads the update.

9. ★★ Singular values of the essential matrix. Prove that E = [t]_\times R has singular values (\|t\|, \|t\|, 0). How should a noisy estimate of E be projected onto the essential matrices?

   Solution: since RR^\top = I and [t]_\times^\top = -[t]_\times, EE^\top = [t]_\times RR^\top [t]_\times^\top = -[t]_\times^2 . For any v, t \times (t \times v) = (t^\top v)\, t - \|t\|^2 v, so -[t]_\times^2 = \|t\|^2 I - tt^\top.

   Its eigenvalues are \|t\|^2 (twice, on t^\perp) and 0 (on t), which gives the singular values.

   The closest essential matrix in Frobenius norm replaces the singular values (\sigma_1, \sigma_2, \sigma_3) of the estimate by (\bar\sigma, \bar\sigma, 0), with \bar\sigma = (\sigma_1 + \sigma_2)/2.

10. ★★ Scale selection in one dimension. For a 1D Gaussian blob of standard deviation s, find the scale that maximises \sigma^\gamma |L''(0, \sigma)|. For which \gamma is \sigma = s selected? Compare with Theorem 2.5.

    Hints and expected result: L(\cdot, \sigma) = G_\tau with \tau^2 = s^2 + \sigma^2, and G_\tau''(0) = -1/(\sqrt{2\pi}\, \tau^3). With u = \sigma^2, maximise u^{\gamma/2}(s^2 + u)^{-3/2}; the derivative of the logarithm vanishes at u = \frac{\gamma\, s^2}{3 - \gamma} . With \gamma = 2, the selected scale is \sigma = \sqrt2\, s; \sigma = s needs \gamma = 3/2.

    In 2D the same computation gives u = \gamma s^2 / (4 - \gamma), so \gamma = 2 selects \sigma = s. In every case the selected scale is proportional to s, so covariance holds.

11. ★★ The spatial pyramid kernel is positive definite. Prove that the kernel of Definition 5.3 is positive definite, and give an explicit feature map for histograms with integer entries.

    Solution: each I_\ell is a sum of intersection kernels, positive definite by Proposition 5.2, and k_L is a combination of them with positive weights. For an integer a \leq A, let u(a) \in \{0, 1\}^A have a leading ones; then \min(a, b) = u(a)^\top u(b).

    Concatenating, over all cells and words, the vectors u(h_j) multiplied by the square root of the level weight gives an explicit map \Phi with k_L(h, h') = \Phi(h)^\top \Phi(h').

12. ★★ Backpropagation through a convolution. For a 1D convolution layer y = x * k (finite signals, zero padding) and a loss \mathcal{L}, express \partial\mathcal{L}/\partial x and \partial\mathcal{L}/\partial k in terms of g = \partial\mathcal{L}/\partial y.

    Solution: y(n) = \sum_m x(n - m)\, k(m), so y(n) depends on x(p) with coefficient k(n - p), and on k(m) with coefficient x(n - m). Hence \frac{\partial\mathcal{L}}{\partial x(p)} = \sum_n g(n)\, k(n - p), \qquad \frac{\partial\mathcal{L}}{\partial k(m)} = \sum_n g(n)\, x(n - m). The first is the cross-correlation of g with k, a convolution with the flipped kernel (the “transposed convolution”); the second is the cross-correlation of g with x. The backward pass therefore costs two convolutions, as Theorem 6.3 predicts.

13. ★★ Kabsch and reflections. Let q_i = Sp_i with S = \operatorname{diag}(1, 1, -1), where the centred points p_i have covariance \operatorname{diag}(a, b, c) with a > b > c > 0. Compute H, its SVD, the unconstrained orthogonal minimiser and the rotation of Theorem 9.3.

    Solution: H = n \operatorname{diag}(a, b, -c), so one can take U = I, \Sigma = n\operatorname{diag}(a, b, c) and V = S. The unconstrained minimiser VU^\top = S is a reflection with zero residual, not a rotation.

    Since \det(VU^\top) = -1, Theorem 9.3 gives R^\star = S\, S = I, with residual \sum_i \|p_i - Sp_i\|^2 = 4\sum_i z_i^2 = 4nc. The correction flips the axis of smallest variance, which costs least.

14. ★★ Normalised eight-point algorithm (Python). Draw 100 points uniformly in the box [-2, 2] \times [-1.5, 1.5] \times [4, 10] of the first camera frame, and view them with two cameras (K as in Exercise 1, second camera rotated by 15° about the vertical axis and translated by (1, 0.1, 0.2)). Add 0.5 px of noise to both images, and implement Algorithm 1.10 with and without normalisation. Compare the median distance of the points to their epipolar lines.

    Hints and expected result: build each row as the Kronecker product of \tilde{x}' and \tilde{x}, take the last right singular vector, project to rank 2 with an SVD, and undo the normalisation with F = T'^\top \hat{F} T. With normalisation, the median epipolar distance is about 0.4–0.5 px, at the noise level. Without it, it is a few pixels (1.5–2.6 px over five seeds in our runs).

Measure also the distance of the noise-free points to the epipolar lines of each noisy estimate. It is about 0.1 px with normalisation, but stays at a few pixels without it: the error lies in the estimated F, not in the data. On exact data, both versions recover F to machine precision.

15. ★★ Dictionary size and pooling (Python). In the code of ch. 5, vary the number of atoms K \in \{16, 32, 64\}, and compare sum and max pooling of the sparse codes over the spatial pyramid.

    Hints and expected result: put the dictionary learning and the evaluation inside a loop over K and over the pooling function. With max pooling, accuracy rises from about 96 % at K = 16 to about 99 % at K = 32 and 64. With 16 atoms the two poolings are equivalent; from 32 atoms, max pooling is better (about 98.9 % against 97.0 % at K = 32), as Yang et al. (2009) observed on real images.

16. ★★★ Class activation maps on synthetic images (Python). Generate 32 \times 32 images of noise, each with a bright horizontal bar as a distractor; in half of them, add a bright 6 \times 6 square, and label the image by the presence of the square. Train a small CNN ending with global average pooling, compute the class activation map of Proposition 10.6, and measure how often its maximum falls inside the square.

    Hints and expected result: three convolutional layers with one pooling, 400 Adam steps on batches of 64, are enough. The image-level accuracy is close to 100 %, and the maximum of the map falls within 4 pixels of the centre of the square in more than 90 % of the positive images, although no position was given. Then make the square always appear next to the bar, and check whether the map still points at the square or moves to the bar — the failure mode discussed after Proposition 10.3.

17. ★★★ Designing a billion-scale search engine. You must index 10^9 SIFT descriptors on one server with 64 GB of RAM and answer a query in a few milliseconds. Propose a design based on ch. 4, with memory and time estimates.

    Hints and expected result: raw descriptors take 512 GB in single precision, so compression is mandatory. Product quantisation with m = 8 bytes per vector uses 8 GB. Exhaustive asymmetric distances would cost 8 \times 10^9 look-ups per query, about seconds.

    A coarse quantiser with 2^{16} cells and an inverted file reduces the scan: visiting 64 cells touches about 10^6 vectors, that is, 8 \times 10^6 look-ups, a few milliseconds. Encoding the residual to the coarse centroid improves accuracy (the IVFADC scheme of Jégou, Douze and Schmid, 2011); re-ranking the best few hundred candidates with finer codes recovers precision.

18. ★★★ Repeatability under viewpoint change. Take a textured image (a synthetic random texture or any photograph), warp it by affine maps that simulate a growing tilt of the viewing direction, and measure the repeatability of Harris corners and of difference-of-Gaussian blobs: the fraction of points detected in the reference image whose transformed position is detected again within 2 pixels.

    Hints and expected result: use the ground-truth affine maps to transfer points, and plot repeatability against the tilt. Both detectors degrade as the tilt grows, because they are covariant to similarities only. Adding affine adaptation by the structure tensor, as in the note of ch. 2, flattens the curve; compare your curves with those of Mikolajczyk and Schmid (2004).

Projects and assessment

Officially, the course is assessed by three programming assignments, worth 50 % of the grade, and by a research-oriented final project, including a report and a presentation, worth the other 50 % (master-mva.com); the Université Paris-Saclay record lists partial continuous assessment. The solo subjects below are sized like an assignment or a light final project, and the group subjects like a full final project. Each subject is graded with the rubric at the end of this section, against its stated success criterion. The written exam is a self-assessment proposal, since the official assessment includes no written exam.

Solo projects

1. [Solo · ≈ 20 h] Panorama stitching from scratch. Question: how accurately can a rotating camera be registered from local features alone?
   • Implement Harris or DoG detection, a simple gradient-histogram descriptor, ratio-test matching, normalised DLT and RANSAC; warp and blend at least three of your own photographs taken from one point.
   • Success criterion: inlier reprojection error below 1 px, and a mosaic without visible seams.
   • Deliverables: code, a 4–6 page report with the error statistics, a 10-minute talk. Chapters: ch. 1, 2, 3.
   • Evaluation: the rubric below, with weight on the RANSAC analysis.
2. [Solo · ≈ 20 h] Paper study: Video Google (Sivic and Zisserman, 2003). Question: how much do vocabulary size, tf-idf weighting and spatial re-ranking contribute to retrieval?
   • Data: the public Oxford Buildings data set (Philbin et al., 2007), with SIFT features from OpenCV or VLFeat.
   • Implement k-means vocabularies of several sizes, an inverted file, tf-idf scoring and RANSAC re-ranking of the top 100 results.
   • Success criterion: a curve of mean average precision against vocabulary size, and a measured gain from re-ranking.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 2, 3, 4.
3. [Solo · ≈ 20 h] Paper study: class activation maps (Zhou et al., 2016). Question: how well does a classifier trained with image labels localise objects?
   • Data: CUB-200-2011 birds, which provide boxes for evaluation only (Wah et al., 2011); a pre-trained ResNet-18 fine-tuned on a free GPU tier.
   • Compute class activation maps, threshold them into boxes, and measure the fraction of images with IoU above 0.5; compare max, average and LSE pooling.
   • Success criterion: reproduce the qualitative finding that global average pooling localises the whole object better than global max pooling.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 7, 10.
4. [Solo · ≈ 20 h] Optical flow, classical against learned. Question: where does Lucas–Kanade fail, and does a coarse-to-fine pyramid fix it?
   • Data: the public MPI-Sintel training set, with ground-truth flow (Butler et al., 2012).
   • Implement pyramidal Lucas–Kanade and Horn–Schunck, measure the average end-point error by region (textured, untextured, occlusion boundaries), and compare with published RAFT numbers.
   • Success criterion: a measured reduction of the end-point error by the pyramid on pixels moving more than 5 px, and error maps whose large errors coincide with a small eigenvalue \lambda_2 of the structure tensor (report the correlation), as Proposition 8.2 predicts.
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters: ch. 2, 8.

Group projects

1. [Group of 2–3 · ≈ 30 h each] Two-stream action recognition on UCF101. Question: how much does motion add to appearance for action classification?
   • Data: UCF101, split 1 (Soomro, Zamir and Shah, 2012); a free GPU tier. Computing the flow of all 13 320 clips is the bottleneck: sample 5 frame pairs per clip, or restrict the study to the first 25 classes.
   • Member 1 computes optical flow and trains the flow stream; member 2 fine-tunes an ImageNet-pretrained RGB stream; member 3 inflates the RGB network into a small 3D CNN (Proposition 8.7) and runs the fusion and the evaluation.
   • Success criterion: report the accuracy of each stream and of their fusion, and a measured gain from fusion, as in Simonyan and Zisserman (2014).
   • Deliverables: code, a joint 6–8 page report, a 15-minute talk with a short individual part each. Chapters: ch. 7, 8.
2. [Group of 2–3 · ≈ 30 h each] Paper study: weakly supervised detection (Bilen and Vedaldi, 2016). Question: can a detector be trained with image labels only?
   • Data: PASCAL VOC 2007 (Everingham et al., 2010), with region proposals from selective search or edge boxes, and an ImageNet-pretrained backbone.
   • Member 1 builds the proposal pipeline and the evaluation (CorLoc and AP); member 2 implements the two-stream WSDDN head of Definition 10.5; member 3 studies the failure modes (parts, context) and a fix such as a spatial regulariser or LSE pooling.
   • Success criterion: reach, with the same backbone, a CorLoc within a few points of the value reported by Bilen and Vedaldi (2016), and explain the main failure modes with examples.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 7, 10.
3. [Group of 2–3 · ≈ 30 h each] Point clouds against multiple views on ModelNet10. Question: which 3D representation is most accurate and most robust?
   • Data: ModelNet10 (Wu et al., 2015).
   • Member 1 trains a PointNet on sampled point clouds; member 2 trains a multi-view CNN on rendered views; member 3 builds the robustness benchmark (missing points, noise, rotations) and an ICP-based nearest-shape baseline.
   • Success criterion: test accuracies for the three methods and robustness curves, with the effect of rotation augmentation measured.
   • Deliverables: code, a 6–8 page report, a 15-minute talk. Chapters: ch. 3, 7, 9.

Written exam (proposal)

Format: 2 hours, one handwritten A4 sheet of notes allowed, no electronic devices, four exercises.

1. Question (5 points). Describe the full pipeline of instance-level recognition, from two images to a verified detection: detection, description, matching, robust estimation. For each step, state the invariance or the robustness it provides. Expected: covariant detection with scale selection; an invariant descriptor (rotation by the dominant orientation, illumination by normalisation); the ratio test; RANSAC with a homography or a fundamental matrix; acceptance on the number of inliers; the number of RANSAC iterations as a function of the inlier ratio.

2. Computation (5 points). A camera has f = 1000 px, principal point (500, 400) and zero skew. (a) Project the point (0.2, -0.1, 2) given in the camera frame. (b) The camera pans by 5° about its vertical axis: where does the principal point go? (c) How many RANSAC iterations does a homography need at 60 % of outliers and confidence 0.99?

   Expected: (a) (600, 350); (b) (1000\tan 5° + 500,\ 400) \approx (587.5, 400); (c) w^4 = 0.0256, so N = \lceil \log 0.01 / \log 0.9744 \rceil = 178.

3. Short proof (5 points). Prove that the Lucas–Kanade problem has a unique solution if and only if the structure tensor of the window is invertible, and give the solution. Explain why corners are good features to track, and relate the result to the aperture problem. Expected: the argument of Theorem 8.3 (quadratic form, normal equations, kernel of M forces \nabla I^\top v = 0); both eigenvalues large at corners; on an edge M has rank 1 and only the normal flow is determined.

4. Convolutional networks (5 points). (a) Prove that a convolutional layer with circular padding and stride 1 commutes with translations, and that global average pooling makes a network invariant. (b) Compute the receptive field and the parameter count of three stacked 3 \times 3 convolutions with 64 channels, and compare with one 7 \times 7 convolution. Expected: (a) Proposition 7.3; (b) receptive field 7 in both cases; 3 \times (9 \cdot 64 \cdot 64 + 64) = 110\,784 against 49 \cdot 64 \cdot 64 + 64 = 200\,768 parameters, and three non-linearities against one.

Grading rubric (proposal)

References

Official references (master-mva.com):

• D. A. Forsyth and J. Ponce, Computer Vision: A Modern Approach, Prentice-Hall, 2nd edition, 2011.
• J. Ponce, M. Hebert, C. Schmid and A. Zisserman (eds.), Toward Category-Level Object Recognition, Lecture Notes in Computer Science 4170, Springer-Verlag, 2007.
• R. Szeliski, Computer Vision: Algorithms and Applications, Springer, 2010. Available online at http://szeliski.org/Book/.

Textbooks:

• R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2nd edition, 2004.
• I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016.
• G. H. Golub and C. F. Van Loan, Matrix Computations, Johns Hopkins University Press, 4th edition, 2013.

Camera geometry, local features and matching:

• Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11), 1330–1334, 2000.
• R. Hartley, “In defense of the eight-point algorithm”, IEEE Transactions on Pattern Analysis and Machine Intelligence 19(6), 580–593, 1997.
• D. Nistér, “An efficient solution to the five-point relative pose problem”, IEEE Transactions on Pattern Analysis and Machine Intelligence 26(6), 756–770, 2004.
• N. Snavely, S. M. Seitz and R. Szeliski, “Photo tourism: exploring photo collections in 3D”, ACM Transactions on Graphics (SIGGRAPH) 25(3), 835–846, 2006.
• C. Harris and M. Stephens, “A combined corner and edge detector”, Alvey Vision Conference, 1988.
• T. Lindeberg, “Feature detection with automatic scale selection”, International Journal of Computer Vision 30(2), 79–116, 1998.
• D. G. Lowe, “Distinctive image features from scale-invariant keypoints”, International Journal of Computer Vision 60(2), 91–110, 2004.
• K. Mikolajczyk and C. Schmid, “Scale & affine invariant interest point detectors”, International Journal of Computer Vision 60(1), 63–86, 2004.
• D. DeTone, T. Malisiewicz and A. Rabinovich, “SuperPoint: self-supervised interest point detection and description”, CVPR Workshops, 2018.
• M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Communications of the ACM 24(6), 381–395, 1981.
• O. Chum, J. Matas and J. Kittler, “Locally optimized RANSAC”, DAGM Symposium on Pattern Recognition, 2003.
• O. Chum and J. Matas, “Matching with PROSAC — progressive sample consensus”, CVPR, 2005.

Visual search and category-level recognition:

• J. Sivic and A. Zisserman, “Video Google: a text retrieval approach to object matching in videos”, ICCV, 2003.
• D. Nistér and H. Stewénius, “Scalable recognition with a vocabulary tree”, CVPR, 2006.
• J. Philbin, O. Chum, M. Isard, J. Sivic and A. Zisserman, “Object retrieval with large vocabularies and fast spatial matching”, CVPR, 2007.
• O. Chum, J. Philbin, J. Sivic, M. Isard and A. Zisserman, “Total recall: automatic query expansion with a generative feature model for object retrieval”, ICCV, 2007.
• H. Jégou, M. Douze and C. Schmid, “Product quantization for nearest neighbor search”, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 117–128, 2011.
• H. Jégou, M. Douze, C. Schmid and P. Pérez, “Aggregating local descriptors into a compact image representation”, CVPR, 2010.
• J. Johnson, M. Douze and H. Jégou, “Billion-scale similarity search with GPUs”, IEEE Transactions on Big Data, 2019.
• Y. A. Malkov and D. A. Yashunin, “Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs”, IEEE Transactions on Pattern Analysis and Machine Intelligence 42(4), 824–836, 2020.
• G. Csurka, C. Dance, L. Fan, J. Willamowski and C. Bray, “Visual categorization with bags of keypoints”, ECCV Workshop on Statistical Learning in Computer Vision, 2004.
• S. Lazebnik, C. Schmid and J. Ponce, “Beyond bags of features: spatial pyramid matching for recognizing natural scene categories”, CVPR, 2006.
• F. Perronnin, J. Sánchez and T. Mensink, “Improving the Fisher kernel for large-scale image classification”, ECCV, 2010.
• J. Sánchez, F. Perronnin, T. Mensink and J. Verbeek, “Image classification with the Fisher vector: theory and practice”, International Journal of Computer Vision 105(3), 222–245, 2013.
• R. Tibshirani, “Regression shrinkage and selection via the lasso”, Journal of the Royal Statistical Society, Series B 58(1), 267–288, 1996.
• B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images”, Nature 381, 607–609, 1996.
• M. Aharon, M. Elad and A. Bruckstein, “K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation”, IEEE Transactions on Signal Processing 54(11), 4311–4322, 2006.
• J. Mairal, F. Bach, J. Ponce and G. Sapiro, “Online dictionary learning for sparse coding”, ICML, 2009.
• J. Yang, K. Yu, Y. Gong and T. Huang, “Linear spatial pyramid matching using sparse coding for image classification”, CVPR, 2009.
• A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems”, SIAM Journal on Imaging Sciences 2(1), 183–202, 2009.

Neural networks and convolutional networks:

• G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals and Systems 2(4), 303–314, 1989.
• K. Hornik, “Approximation capabilities of multilayer feedforward networks”, Neural Networks 4(2), 251–257, 1991.
• M. Leshno, V. Ya. Lin, A. Pinkus and S. Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function”, Neural Networks 6(6), 861–867, 1993.
• D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization”, ICLR, 2015.
• I. Loshchilov and F. Hutter, “Decoupled weight decay regularization”, ICLR, 2019.
• L. Bottou, F. E. Curtis and J. Nocedal, “Optimization methods for large-scale machine learning”, SIAM Review 60(2), 223–311, 2018.
• K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: surpassing human-level performance on ImageNet classification”, ICCV, 2015.
• N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting”, Journal of Machine Learning Research 15, 1929–1958, 2014.
• S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift”, ICML, 2015.
• Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition”, Proceedings of the IEEE 86(11), 2278–2324, 1998.
• A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet classification with deep convolutional neural networks”, NeurIPS, 2012.
• K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition”, ICLR, 2015.
• K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition”, CVPR, 2016.
• R. Zhang, “Making convolutional networks shift-invariant again”, ICML, 2019.
• A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit and N. Houlsby, “An image is worth 16x16 words: transformers for image recognition at scale”, ICLR, 2021.
• M. Oquab, L. Bottou, I. Laptev and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks”, CVPR, 2014.
• R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR, 2014.
• S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks”, NeurIPS, 2015.
• J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You only look once: unified, real-time object detection”, CVPR, 2016.
• N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, “End-to-end object detection with transformers”, ECCV, 2020.
• J. Long, E. Shelhamer and T. Darrell, “Fully convolutional networks for semantic segmentation”, CVPR, 2015.
• K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN”, ICCV, 2017.
• M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge”, International Journal of Computer Vision 88(2), 303–338, 2010.
• T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. L. Zitnick, “Microsoft COCO: common objects in context”, ECCV, 2014.

Motion and human actions:

• B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision”, IJCAI, 1981.
• B. K. P. Horn and B. G. Schunck, “Determining optical flow”, Artificial Intelligence 17, 185–203, 1981.
• J. Shi and C. Tomasi, “Good features to track”, CVPR, 1994.
• A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. van der Smagt, D. Cremers and T. Brox, “FlowNet: learning optical flow with convolutional networks”, ICCV, 2015.
• Z. Teed and J. Deng, “RAFT: recurrent all-pairs field transforms for optical flow”, ECCV, 2020.
• D. J. Butler, J. Wulff, G. B. Stanley and M. J. Black, “A naturalistic open source movie for optical flow evaluation”, ECCV, 2012.
• I. Laptev, “On space-time interest points”, International Journal of Computer Vision 64(2–3), 107–123, 2005.
• I. Laptev, M. Marszałek, C. Schmid and B. Rozenfeld, “Learning realistic human actions from movies”, CVPR, 2008.
• H. Wang and C. Schmid, “Action recognition with improved trajectories”, ICCV, 2013.
• K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos”, NeurIPS, 2014.
• D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks”, ICCV, 2015.
• J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the Kinetics dataset”, CVPR, 2017.
• G. Varol, I. Laptev and C. Schmid, “Long-term temporal convolutions for action recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1510–1517, 2018.
• A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: a video vision transformer”, ICCV, 2021.
• K. Soomro, A. R. Zamir and M. Shah, “UCF101: a dataset of 101 human actions classes from videos in the wild”, arXiv:1212.0402, 2012.
• W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman and A. Zisserman, “The Kinetics human action video dataset”, arXiv:1705.06950, 2017.
• G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev and C. Schmid, “Learning from synthetic humans”, CVPR, 2017.

3D object recognition:

• W. Kabsch, “A solution for the best rotation to relate two sets of vectors”, Acta Crystallographica A 32, 922–923, 1976.
• S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns”, IEEE Transactions on Pattern Analysis and Machine Intelligence 13(4), 376–380, 1991.
• P. J. Besl and N. D. McKay, “A method for registration of 3-D shapes”, IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 239–256, 1992.
• V. Lepetit, F. Moreno-Noguer and P. Fua, “EPnP: an accurate O(n) solution to the PnP problem”, International Journal of Computer Vision 81(2), 155–166, 2009.
• C. R. Qi, H. Su, K. Mo and L. J. Guibas, “PointNet: deep learning on point sets for 3D classification and segmentation”, CVPR, 2017.
• H. Su, S. Maji, E. Kalogerakis and E. Learned-Miller, “Multi-view convolutional neural networks for 3D shape recognition”, ICCV, 2015.
• D. Maturana and S. Scherer, “VoxNet: a 3D convolutional neural network for real-time object recognition”, IROS, 2015.
• Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang and J. Xiao, “3D ShapeNets: a deep representation for volumetric shapes”, CVPR, 2015.
• M. Aubry, D. Maturana, A. A. Efros, B. C. Russell and J. Sivic, “Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models”, CVPR, 2014.
• Y. Labbé, J. Carpentier, M. Aubry and J. Sivic, “CosyPose: consistent multi-view multi-object 6D pose estimation”, ECCV, 2020.
• M. Loper, N. Mahmood, J. Romero, G. Pons-Moll and M. J. Black, “SMPL: a skinned multi-person linear model”, ACM Transactions on Graphics (SIGGRAPH Asia) 34(6), 2015.
• G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer, I. Laptev and C. Schmid, “BodyNet: volumetric inference of 3D human body shapes”, ECCV, 2018.

Weak, self- and language supervision:

• T. G. Dietterich, R. H. Lathrop and T. Lozano-Pérez, “Solving the multiple instance problem with axis-parallel rectangles”, Artificial Intelligence 89(1–2), 31–71, 1997.
• H. Bilen and A. Vedaldi, “Weakly supervised deep detection networks”, CVPR, 2016.
• M. Oquab, L. Bottou, I. Laptev and J. Sivic, “Is object localization for free? Weakly-supervised learning with convolutional neural networks”, CVPR, 2015.
• B. Zhou, A. Khosla, A. Lapedriza, A. Oliva and A. Torralba, “Learning deep features for discriminative localization”, CVPR, 2016.
• C. Wah, S. Branson, P. Welinder, P. Perona and S. Belongie, “The Caltech-UCSD Birds-200-2011 dataset”, Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
• A. van den Oord, Y. Li and O. Vinyals, “Representation learning with contrastive predictive coding”, arXiv:1807.03748, 2018.
• T. Chen, S. Kornblith, M. Norouzi and G. Hinton, “A simple framework for contrastive learning of visual representations”, ICML, 2020.
• A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger and I. Sutskever, “Learning transferable visual models from natural language supervision”, ICML, 2021.
• P. Bojanowski, R. Lajugie, F. Bach, I. Laptev, J. Ponce, C. Schmid and J. Sivic, “Weakly supervised action labeling in videos under ordering constraints”, ECCV, 2014.
• J.-B. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev and S. Lacoste-Julien, “Unsupervised learning from narrated instruction videos”, CVPR, 2016.
• A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev and J. Sivic, “HowTo100M: learning a text-video embedding by watching hundred million narrated video clips”, ICCV, 2019.

Research directions (ch. 11):

• M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski and A. Joulin, “Emerging properties in self-supervised vision transformers”, ICCV, 2021.
• M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “DINOv2: learning robust visual features without supervision”, Transactions on Machine Learning Research, 2024.
• K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners”, CVPR, 2022.
• A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár and R. Girshick, “Segment anything”, ICCV, 2023.
• D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations”, ICLR, 2019.
• J.-B. Alayrac, J. Donahue, P. Luc, A. Miech et al., “Flamingo: a visual language model for few-shot learning”, NeurIPS, 2022.
• A. Yang, A. Miech, J. Sivic, I. Laptev and C. Schmid, “Just ask: learning to answer questions from millions of narrated videos”, ICCV, 2021.
• B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi and R. Ng, “NeRF: representing scenes as neural radiance fields for view synthesis”, ECCV, 2020.
• B. Kerbl, G. Kopanas, T. Leimkühler and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering”, ACM Transactions on Graphics (SIGGRAPH) 42(4), 2023.
• S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii and J. Revaud, “DUSt3R: geometric 3D vision made easy”, CVPR, 2024.
• P.-E. Sarlin, D. DeTone, T. Malisiewicz and A. Rabinovich, “SuperGlue: learning feature matching with graph neural networks”, CVPR, 2020.
• J. Sun, Z. Shen, Y. Wang, H. Bao and X. Zhou, “LoFTR: detector-free local feature matching with transformers”, CVPR, 2021.
• J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited”, CVPR, 2016.