Fundamentals of Reproducible Research and Free Software

Contents

• Introduction and overview
• 0. Review: floating point, probability and graphs
• 1. Free and open-source software: introduction
• 2. Licensing
• 3. Economic models of free software projects
• 4. Case studies
• 5. Reproducible research: introduction
• 6. Numerical determinism: floating point and pseudo-random numbers
• 7. Software environments, version control and provenance
• 8. Publishing reproducible research: the editorial process
• 9. Legal aspects of the code, the article and the data
• 10. Writing a reproducible scientific article
• 11. Research directions and open problems
• Exercises
• Projects and assessment
• References

Introduction and overview

Audience: a student with a good M1 in mathematics, computer science or physics who already writes research code in Python, has met git, and will soon publish, reuse or review code, articles and data.

Empirical first: sum the same 100,000 single-precision numbers from left to right and from right to left, and you get −423.29346 and −423.29407 (code in ch. 6). Copy a function from a public repository that has no licence file, and by default you have no right to redistribute it (ch. 1). Report the best of ten training seeds, and you announce a gain of about 1.5 standard deviations that does not exist (ch. 5). This course explains these numerical, legal and statistical failures and how a reproducible publication avoids them.

The course has two halves. The first studies free software as a legal and economic object: what a licence grants, how licences combine, and who pays for the software that research depends on. The second studies reproducible research: why published computational results often cannot be regenerated, and how to write, package, publish and review an article so that they can.

Related MVA pages: the practical follow-up Open Science in the Age of AI: Reproducible Research Project, the legal frame of Regulating AI, and the reproducible IPOL experiments used in Image Denoising.

What you will be able to do

• Distinguish free software, open source, source-available and proprietary software, and state the four freedoms and the criteria of the Open Source Definition.
• Read a licence: identify its permissions, conditions and limitations, the scope of its copyleft and its patent clauses.
• Decide whether components under different licences can be combined and distributed, and write the SPDX expression of the result.
• Analyse the economic model of a free software project, and explain its under-provision with the volunteer’s dilemma.
• Evaluate a software project with a fixed grid: type of application, licences, rights holders, governance, economic model and reproducibility.
• Define reproducibility, replicability and robustness, and quantify how low power, multiple testing and seed picking produce irreproducible claims.
• Control the numerical sources of irreproducibility: summation order, floating-point tolerances, pseudo-random seeds and streams.
• Freeze and record a software environment, read version control as a tamper-evident history, and build a pipeline that recomputes exactly what changed.
• Describe the editorial process of journals and conferences, and of venues that review code, such as IPOL and ReScience C.
• Identify who owns the code, the article and the data of a research project, and which terms make a reproducibility package redistributable.
• Write an article whose pseudo-code, code, figures and numbers can be checked against each other, and audit somebody else’s.

Official objectives and syllabus (2026–2027)

The course description on master-mva.com presents a course on free software and reproducible research: how to write and publish reproducible research, the legal aspects of the code, the article and the data, and the good practices to write free software and perform reproducible research. Group discussions, debates and individual dissertations are part of the activities.

The description states that the plan covers the minimum knowledge that any PhD candidate or master’s student in computational sciences should reach to perform reliable research. It presents the MVA course “Projet de recherche reproductible” as a practical follow-up for students who want to carry out an actual reproducible-research project.

Every official topic is taught on this page. Its extensions beyond the syllabus are the review chapter 0, the quantitative models that make the debates precise (the volunteer’s dilemma, the positive predictive value of findings, summation error bounds, the complexity of dependency resolution, Merkle hashing, a model of peer-review noise), and the research chapter 11.

Where this course fits in the programme

Official prerequisites: not displayed. It is a Learning course of semester 1 (period 1), with 5 ECTS and 24 h. It is one of the two courses that carry the RR track label in the catalogue, with the semester-2 project course; the validation rules state that the RR track is open to all, and that validating one or two RR modules places a student in the track.

• Preparation — the refreshers Scientific Python, NumPy and PyTorch (the code of every chapter) and Probability and Statistics (Bayes’ formula, tests and p-values for ch. 5).
• Related courses of semester 1 — Image Denoising (online experiments with IPOL, the journal of ch. 8.3), Sub-Pixel Image Processing (continuous models turned into numerical algorithms), Deep Learning (training whose seeds and nondeterminism ch. 6 controls), Computational Statistics (simulation of random variables) and the Turing Seminar (debates on evaluating often closed-source AI systems).
• Courses of semester 2 that build on it — Open Science in the Age of AI: Reproducible Research Project, the official practical follow-up; Regulating AI; Training and Deploying Large-Scale Models (environments and determinism at scale); Responsible Machine Learning. The Research Internship is where the practices of ch. 10 are used for real.

Course outline

• Review: floating point, probability and graphs — the rounding model, Bayes’ formula and tests, directed acyclic graphs and hash functions.
• Free and open-source software: introduction — software as a copyrighted work, the four freedoms, the Open Source Definition, a short history.
• Licensing — permissions, conditions and copyleft; when licences can be combined; SPDX; French and European licences; patents.
• Economic models of free software projects — software as a public good, the volunteer’s dilemma, business models, relicensing and forks, the truck factor.
• Case studies — an analysis grid applied to the Linux kernel, scikit-learn, OpenCV and dual-licensed projects.
• Reproducible research: introduction — vocabulary, evidence of a crisis, sources of irreproducibility, and the statistics of findings that do not replicate.
• Numerical determinism — rounding errors in sums, compensated summation, sources of nondeterminism, pseudo-random generators and seeds.
• Software environments, version control and provenance — dependency resolution, git as a Merkle graph, content-addressed pipelines.
• Publishing reproducible research: the editorial process — the journey of a manuscript, the noise of peer review, venues that review code, identifiers and archives.
• Legal aspects of the code, the article and the data — ownership, open access, database rights and personal data, redistributable packages, model weights.
• Writing a reproducible scientific article — matching pseudo-code and code, tests with tolerances, reporting over seeds, packaging, auditing.
• Research directions and open problems — long-term reproducibility, reproducible arithmetic, variance in benchmarks, automated reproduction, sustainability (self-study).

Session plan

24 h in class = 10 sessions (6 × 2 h + 4 × 3 h). The volume is the Université Paris-Saclay programme table (lectures 12 h, tutorials 3 h, labs 9 h). The organisation is the one announced on master-mva.com: 6 master classes of 2 h; 3 TP sessions of 3 h, in which each group works on its final project and gives a partial presentation; and a final 3 h session of presentations with a group discussion. The order of the sessions and the split of the chapters between them are a proposal.

Level 1 = foundations (definitions, vocabulary, first examples), 2 = core results and methods, 3 = synthesis and research-level practice; the level never decreases.

Self-study, outside the 24 h (about 6.5 h in total): ch. 0 Review: floating point, probability and graphs, to read before session 1 (≈ 1.5 h) · ch. 11 Research directions and open problems (≈ 3 h, with Ex. 15–16) · running and modifying the Python blocks of the chapters on your own machine (≈ 2 h).

How to study this module

• Read ch. 0 before the first session: it reviews the M1 tools used later. Then read the chapters in order, each using only earlier ones; ch. 2 is used by ch. 4 and 9, ch. 6 by ch. 10 and 11, and ch. 7 by ch. 10.
• Choose the software project of your group before session 4, and fill one line of the grid of Definition 4.1 after each lecture.
• After each session, do the exercises of its row; the ★ ones check the definitions, the ★★ ones ask for a short proof or a computation.
• Run every Python block. They use only the standard library, NumPy, SciPy and, in Ex. 8, NetworkX; each runs on a laptop in a few seconds, with fixed seeds.
• Keep a lab notebook from the first day: the commit hash, the environment and the seed of every result you show in a partial presentation.
• These notes are not legal advice. Each statement of law names its source; for a real decision, ask your institution’s legal or technology-transfer office.

0. Review: floating point, probability and graphs

Three M1 tools return throughout the course: the rounding model of floating-point arithmetic (ch. 6 and 10), Bayes’ formula with the vocabulary of tests (ch. 5 and 8), and directed acyclic graphs with hash functions (ch. 7). This chapter states them in the form used later.

0.1 Floating-point numbers

Each operation is accurate, but associativity is lost: a sum depends on the order of its terms. Near 10^{16}, binary64 numbers are 2 apart, so adding 1 to 10^{16} does nothing.

import numpy as np

u = np.finfo(np.float64).eps / 2          # unit roundoff of binary64
print(f"u = {u:.3e}")                      # 1.110e-16
print(0.1 + 0.2 == 0.3, repr(0.1 + 0.2))   # False 0.30000000000000004
print((1e16 + 1.0) - 1e16, (1e16 - 1e16) + 1.0)  # 0.0 1.0
print(np.spacing(1e16))                    # gap between neighbours of 1e16: 2.0

0.2 Bayes’ formula and tests

For events A and B with \mathbb{P}(B) > 0, Bayes’ formula with the law of total probability reads \mathbb{P}(A \mid B) = \frac{\mathbb{P}(B \mid A)\,\mathbb{P}(A)}{\mathbb{P}(B \mid A)\,\mathbb{P}(A) + \mathbb{P}(B \mid A^c)\,\mathbb{P}(A^c)} .

0.3 Directed acyclic graphs and hash functions

1. Free and open-source software: introduction

This chapter fixes the vocabulary of the course: what software is as a legal object, who controls it by default, and what “free software” and “open source” promise.

1.1 Source code and copyright

A computer program is protected by copyright as a literary work: under the Berne Convention as applied by the TRIPS agreement (article 10), the European Directive 2009/24/EC, and, in France, article L112-2 of the Code de la propriété intellectuelle.

Protection is automatic, without registration, and lasts in the European Union until 70 years after the author’s death. It covers the expression, that is the code, but not the ideas and principles behind it (Directive 2009/24/EC, article 1(2)), which include algorithms insofar as they are ideas.

The rights holder is the author or, for software written by an employee in the course of their duties, the employer (ch. 9.2). A licence is the document by which the rights holder grants others some of these rights.

1.2 Free software and open source

The two definitions come from different philosophies, ethical for the FSF and practical for the OSI, but select almost the same licences, hence the umbrella term FOSS. Exceptions are rare: the OSI approves the NASA Open Source Agreement 1.3, which the FSF does not consider free.

1.3 A short history

• 1983–1989: Richard Stallman announces the GNU project, creates the FSF (1985) and publishes the GNU General Public License (1989).
• 1991–1992: Linus Torvalds releases Linux (1991) and places it under the GPL version 2 the following year.
• 1998: the term “open source” is adopted and the OSI founded, after Netscape announces the release of its browser’s source code.
• 2001–2007: Creative Commons is founded (2001) to license texts and images; the GPL version 3 appears (2007).

1.4 Why free software matters for research

A computational result can be checked only if its code can be run for any purpose, read and passed to a colleague: freedoms 0, 1 and 2. A proprietary dependency or a discontinued version can make a result impossible to regenerate. A free licence is necessary but not sufficient: the code must also be available, documented, runnable and faithful to the article, the subject of ch. 5–10.

2. Licensing

A licence turns the default “all rights reserved” into a list of what others may do. This chapter reads licences clause by clause, sorts them into families, and answers the question every project meets: can these components be distributed together?

2.1 Anatomy of a licence

2.2 Families and copyleft

Copyleft is triggered by distribution: private use and modification carry no obligation. Running a program as a network service is not distribution, so the GPL asks nothing of the operator; the AGPL (section 13) closes this gap by obliging the operator to offer the source to remote users. The GPL does not extend to a mere aggregation of separate programs; the FSF holds that linking a library creates a combined work, and the LGPL was written to allow linking from programs under other licences.

2.3 When can licences be combined?

The following model makes the question precise; it simplifies legal reasoning and does not replace it.

The code encodes facts from the FSF’s licence list: MIT and BSD-3-Clause are compatible with both GPL versions; Apache-2.0 only with version 3; GPL-2.0-only and version 3 are incompatible, but “version 2 or later” code can be used under version 3; the GPL and AGPL version 3 allow combination (section 13 of both).

# Toy model of Definition 2.3: a licence = (imposed conditions O, tolerated conditions A).
ANY = None  # permissive: tolerates any condition on the combined work
GPL3_OK = {"notice", "no-endorsement", "state-changes", "patent-termination",
           "source", "under-GPL-3.0", "network-source"}
LIC = {
    "MIT":              ({"notice"}, ANY),
    "BSD-3-Clause":     ({"notice", "no-endorsement"}, ANY),
    "Apache-2.0":       ({"notice", "state-changes", "patent-termination"}, ANY),
    "GPL-2.0-only":     ({"notice", "source", "under-GPL-2.0"},
                         {"notice", "no-endorsement", "source", "under-GPL-2.0"}),
    "GPL-3.0-or-later": ({"notice", "source", "under-GPL-3.0"}, GPL3_OK),
    "AGPL-3.0-only":    ({"notice", "source", "under-GPL-3.0", "network-source"}, GPL3_OK),
}

def compatible(licences):
    O = set().union(*(LIC[l][0] for l in licences))
    return all(LIC[l][1] is ANY or O <= LIC[l][1] for l in licences)

names = list(LIC)
print([(a, b) for i, a in enumerate(names) for b in names[i + 1:] if not compatible([a, b])])
print(compatible(["MIT", "Apache-2.0", "GPL-3.0-or-later"]))   # True
print(compatible(["MIT", "Apache-2.0", "GPL-2.0-only"]))       # False

def spdx_ok(expr, accepted):
    """Evaluate an SPDX expression without parentheses: AND binds tighter than OR."""
    return any(all(t.strip() in accepted for t in conj.split(" AND "))
               for conj in expr.split(" OR "))

print(spdx_ok("MIT OR Apache-2.0", {"Apache-2.0"}))                 # True
print(spdx_ok("GPL-2.0-only AND BSD-3-Clause", {"BSD-3-Clause"}))   # False

The model finds three incompatible pairs: Apache-2.0 with GPL-2.0-only, and GPL-2.0-only with each version-3 licence.

The model ignores the kind of combination (copying, linking, separate processes), licence exceptions, and the fact that only courts settle interpretations. These notes are not legal advice.

2.4 Applying a licence: SPDX and REUSE

SPDX, a Linux Foundation project standardised as ISO/IEC 5962:2021, names every licence and combines names into expressions, which spdx_ok above evaluates:

• MIT OR Apache-2.0: the recipient chooses either licence;
• GPL-2.0-only AND BSD-3-Clause: both apply, for instance to two parts of a package;
• GPL-2.0-only WITH Linux-syscall-note: a licence with an additional permission.

To license a project, put the licence text in a LICENSE file, start each source file with a copyright line and SPDX-License-Identifier: GPL-3.0-or-later, declare the expression in the package metadata (License-Expression for Python), and check the tree with the REUSE tool of the Free Software Foundation Europe.

2.5 French and European licences

• CeCILL (CEA, CNRS, Inria), in French and English, governed by French law: CeCILL-2.1 (2013) is a copyleft that explicitly allows relicensing under the GNU GPL; CeCILL-B is permissive with a strong citation obligation; CeCILL-C is a library-level copyleft, close to the LGPL.
• EUPL-1.2 (European Commission, 2017), a copyleft in the official languages of the European Union, whose appendix lists compatible licences, among them the GPL, AGPL, LGPL, MPL-2.0 and CeCILL-2.x.

2.6 Patents

Copyright protects the code; a patent protects an invention, possibly the method the code implements, whoever wrote it. A free licence does not guarantee freedom to use a method patented by a third party. The European Patent Convention (article 52) excludes programs for computers “as such”, but software-related inventions with a technical character can be patented in Europe.

3. Economic models of free software projects

Free software costs nothing to copy, yet years of work to write and maintain. This chapter explains why such a good is under-provided, how projects finance themselves, and how to measure their fragility.

3.1 Software as a public good

Copying software costs almost nothing, and a free licence gives up the right to exclude: free software is a public good. Its private provision is then insufficient, since each user prefers that another one pays (Bergstrom, Blume and Varian, 1986). The volunteer’s dilemma quantifies this.

3.2 The volunteer’s dilemma

Consider n \ge 2 users of a library. If at least one of them fixes a critical bug, each gains v > 0; fixing costs the volunteer c, with 0 < c < v.

The payoffs are:

• v - c for a volunteer;
• v for an abstainer when someone volunteers;
• 0 for everyone when nobody does.

The pure equilibria show the remedy: coordination that designates the volunteer, as maintainers paid by a company, a foundation or a research institute do.

3.3 Why people contribute

Lerner and Tirole (2002) distinguish immediate benefits, such as needing the fix oneself (the “itch” of Raymond, 1999) or being paid for it, from delayed ones: public contributions signal skill to future employers and peers. In research, code is part of the output, but the credit it receives remains weak compared with articles (ch. 8.4).

3.4 Business models

Dual licensing needs one entity able to sell a licence for the whole code, hence contributor licence agreements (CLAs) that grant it the necessary rights.

3.5 Relicensing and forks

Three recent cases follow one pattern: cloud competition, a move to a source-available licence, a community fork.

• Elasticsearch left Apache-2.0 in 2021; the OpenSearch fork kept it. Elastic added the AGPL version 3 as an option in 2024.
• Terraform moved from MPL-2.0 to the Business Source License in 2023; the OpenTofu fork is hosted by the Linux Foundation.
• Redis left BSD-3-Clause in 2024; the Valkey fork is hosted by the Linux Foundation. Redis added the AGPL version 3 as an option in 2025.

3.6 Measuring fragility: the truck factor

Avelino and coauthors estimate it greedily, removing at each step the developer who authors the most files. The code uses a variant that removes the developer whose departure orphans the most files, compares it with exhaustive search, and checks Theorem 3.2 by simulation.

import numpy as np
from itertools import combinations

# Volunteer's dilemma (Theorem 3.2): value v, cost c, n users
v, c = 10.0, 4.0
for n in [2, 5, 10, 100]:
    q = (c / v) ** (1 / (n - 1))          # probability of abstaining
    print(f"n={n:3d}  q={q:.3f}  P(nobody)={q**n:.3f}")
rng = np.random.default_rng(0)
n = 10; q = (c / v) ** (1 / (n - 1))
print("simulated, n=10:", (rng.random((200_000, n)) < q).all(axis=1).mean())

# Truck factor (Definition 3.4) on a synthetic authorship relation
rng = np.random.default_rng(1)
n_files, n_dev = 60, 8
p = 1 / np.arange(1, n_dev + 1); p /= p.sum()          # a few prolific authors
authors = [set(rng.choice(n_dev, size=rng.integers(1, 3), replace=False, p=p))
           for _ in range(n_files)]
orphaned = lambda S: sum(a <= S for a in authors)

def truck_factor_greedy():
    S = set()
    while orphaned(S) <= n_files / 2:
        S.add(max(set(range(n_dev)) - S, key=lambda d: orphaned(S | {d})))
    return len(S)

def truck_factor_exact():
    return next(k for k in range(1, n_dev + 1)
                if any(orphaned(set(S)) > n_files / 2 for S in combinations(range(n_dev), k)))

print("greedy:", truck_factor_greedy(), " exact:", truck_factor_exact())   # 3 and 3

Both give 3 here, but a greedy heuristic can overestimate the truck factor (Ex. 9).

4. Case studies

The official assessment asks each group to study a real software project. This chapter fixes a grid for that study and applies it to a kernel with thousands of rights holders, a scientific library, a computer-vision library hit by a patent, and dual-licensed products.

4.1 The Linux kernel

• Licence: GPL-2.0-only, with a note excluding from the copyleft the user programs that only use system calls (GPL-2.0-only WITH Linux-syscall-note).
• Rights holders: thousands of contributors keep their copyright, without CLA; each patch certifies the Developer Certificate of Origin (2004) by a “Signed-off-by” line. A move to the GPL version 3 would need all their consents (ch. 3.5): practically impossible.
• Economic model: most changes are written by developers employed by companies; the Linux Foundation provides neutral governance.

4.2 scikit-learn

• Type and licence: a Python machine-learning library under BSD-3-Clause, started as a Google Summer of Code project (2007) and first released publicly by Inria researchers in 2010; Pedregosa et al. (2011) is its reference article.
• Economic model: the permissive licence lets companies embed it in proprietary products, which brings users and sponsors; a consortium hosted by the Inria Foundation has funded maintainers since 2018.
• Reproducibility: the random_state parameter of randomised estimators, deprecation cycles, extensive tests, documentation for each version.

4.3 OpenCV and the SIFT patent

• Licence: BSD-3-Clause until release 4.4, Apache-2.0 from release 4.5.0 (2020), which adds a patent grant from contributors.
• Patents: SIFT and SURF lived in a non-free module enabled at build time (Example 2.6).
• Lesson: the copyright licence, third-party patents and the build options that select what is distributed are three layers to check separately.

4.4 Dual licensing: MySQL, Qt and CGAL

• MySQL: the GPL version 2 or a commercial licence. When Oracle announced in 2009 the acquisition of Sun, owner of MySQL, the original author of MySQL started the MariaDB fork.
• Qt: the GPL, the LGPL version 3, or commercial licences sold by The Qt Company.
• CGAL, the Computational Geometry Algorithms Library: most packages under GPL-3.0-or-later, the kernel under LGPL-3.0-or-later, commercial licences sold by GeometryFactory. Academic users get free software; industrial users pay.

4.5 A licence inventory of your own environment

The grid starts with an inventory. Python packages declare their licence in their metadata, as an SPDX expression or, in older packages, as a classifier or free text.

from collections import Counter
from importlib import metadata

def licence_of(dist):
    """Best-effort licence string from package metadata (SPDX expression first)."""
    m = dist.metadata
    if m.get("License-Expression"):
        return m["License-Expression"]
    classifiers = [c.split(" :: ")[-1] for c in (m.get_all("Classifier") or [])
                   if c.startswith("License ::")]
    if classifiers:
        return classifiers[0]
    text = (m.get("License") or "").strip().splitlines()
    return text[0][:40] if text else "UNKNOWN"

for name in ["numpy", "scipy", "scikit-learn", "networkx"]:
    try:
        d = metadata.distribution(name)
        print(f"{name:13s} {d.version:10s} {licence_of(d)}")
    except metadata.PackageNotFoundError:
        print(f"{name:13s} not installed")

print(Counter(licence_of(d) for d in metadata.distributions()).most_common(5))

A “BSD” project is often an AND of licences: the NumPy release used to test these notes (2.5.3) declares BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0, one licence per vendored component. A free-text “BSD License” does not say which BSD licence applies; then read the LICENSE file itself.

5. Reproducible research: introduction

A computational article makes claims that depend on code, data and an environment that the reader rarely sees. This chapter defines what reproducing such a claim means, reviews the evidence that many claims cannot be reproduced, and quantifies the statistical side of the problem.

5.1 Origins and vocabulary

In the early 1990s, Jon Claerbout’s group at Stanford arranged its books and theses so that every figure could be rebuilt by one command (Claerbout and Karrenbach, 1992). Buckheit and Donoho (1995) summarised the idea:

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

Some communities swap the two words (Barba, 2018; Plesser, 2018); the ACM swapped the meanings of its badges in 2020 to align with the definitions above. Always state yours. Peng (2011) describes a spectrum: publication only; with code; with code and data; with linked, executable code and data; full replication.

5.2 Evidence

• Psychology: the Open Science Collaboration (2015) repeated 100 studies; 97 % of the originals had significant results, against 36 % of the replications.
• All sciences: among 1,576 researchers surveyed by Nature (Baker, 2016), more than 70 % had failed to reproduce another scientist’s experiments.
• Machine learning: Raff (2019) attempted to reimplement 255 papers without their code and succeeded for about 63.5 %; Kapoor and Narayanan (2023) traced many failures to data leakage.
• Notebooks: of about 1.4 million Jupyter notebooks collected by Pimentel et al. (2019), about a quarter of those re-executed ran without error, and about 4 % reproduced their stored results.

5.3 Sources of irreproducibility

5.4 The statistics of findings that do not replicate

A perfectly reproducible computation can still support a claim that does not replicate.

Reporting the best of ten seeds inflates the result by about 1.5 standard deviations of the seed noise. Henderson et al. (2018) showed that, in deep reinforcement learning, averages over two small sets of seeds of the same algorithm can differ significantly.

import numpy as np
from scipy.stats import norm

def ppv(R, power, alpha=0.05):                     # Theorem 5.2
    return power * R / (power * R + alpha)

for R in [1.0, 0.25, 0.1]:
    print(f"R={R:4.2f}  PPV(power .8)={ppv(R, .8):.3f}  PPV(power .2)={ppv(R, .2):.3f}")

z_crit = norm.ppf(0.975)                           # Proposition 5.4
for p in [0.05, 0.01, 0.005]:
    print(f"p={p:5.3f}  P(replication significant)={norm.cdf(norm.ppf(1 - p/2) - z_crit):.3f}")

print("P(at least one of 20 tests)", round(1 - 0.95**20, 3))   # Proposition 5.5

rng = np.random.default_rng(0)                     # Proposition 5.6
for n in [2, 5, 10]:
    e_n = rng.standard_normal((100_000, n)).max(axis=1).mean()
    print(f"n={n:2d}  e_n = {e_n:.3f}")

6. Numerical determinism: floating point and pseudo-random numbers

Rerunning the same code on the same data does not guarantee the same numbers. This chapter bounds the effect of the order of floating-point operations, lists the sources of nondeterminism, and makes pseudo-random numbers reproducible.

6.1 Rounding errors in sums

For an integer m with mu < 1, write \gamma_m = \frac{m u}{1 - m u} .

The bound is a worst case; rounding errors of random sign partly cancel (Ex. 11). Compensated summation carries the rounding error of each addition into the next.

import math
import numpy as np

rng = np.random.default_rng(42)
xs = rng.standard_normal(100_000).astype(np.float32)
ref = math.fsum(xs.astype(np.float64))            # correctly rounded reference

def naive(v):                                      # left-to-right summation
    s = np.float32(0)
    for t in v:
        s = np.float32(s + t)
    return s

def kahan(v):                                      # Algorithm 6.4
    s, c = np.float32(0), np.float32(0)
    for t in v:
        y = np.float32(t - c); t2 = np.float32(s + y)
        c = np.float32((t2 - s) - y); s = t2
    return s

u32 = np.finfo(np.float32).eps / 2
n = len(xs)
bound = (n - 1) * u32 / (1 - (n - 1) * u32) * np.abs(xs).astype(np.float64).sum()
for name, val in [("left to right", naive(xs)), ("right to left", naive(xs[::-1])),
                  ("numpy pairwise", xs.sum()), ("kahan", kahan(xs))]:
    print(f"{name:15s} {float(val):.5f}  error {abs(float(val) - ref):.1e}")
print(f"worst-case bound of Theorem 6.2: {bound:.1e}")

The order or rounding of operations also changes with the number of threads of a reduction or of a BLAS library, with atomic additions on GPUs, with compiler options such as -ffast-math or fused multiply–add, and with library versions. PyTorch’s torch.use_deterministic_algorithms(True) raises an error when an operation has no deterministic implementation. Python also randomises string hashes in each process, so the iteration order of a set of strings may vary unless PYTHONHASHSEED is fixed.

6.2 Pseudo-random numbers

For m = 16: (a, c) = (5, 3) has period 16; (7, 3) fails the third condition and has period 4; (5, 2) fails the first and has period 8.

Python’s random module uses the Mersenne Twister, of period 2^{19937} - 1 (Matsumoto and Nishimura, 1998); NumPy’s default_rng uses PCG64 (O’Neill, 2014) and spawns independent streams from one master seed with SeedSequence. Four rules follow:

• choose one master seed per experiment and record it;
• derive one stream per worker, split or repetition, never seeding from the clock;
• seed every library involved: NumPy, PyTorch (torch.manual_seed), data-loader workers;
• report results over several seeds (Proposition 5.6).

import numpy as np

def period(a, c, m, x0=0):
    seen, x, k = {}, x0, 0
    while x not in seen:
        seen[x] = k
        x, k = (a * x + c) % m, k + 1
    return k - seen[x]

print([period(5, 3, 16), period(7, 3, 16), period(5, 2, 16)])   # [16, 4, 8]

def first_draws(master_seed):                       # one master seed, three streams
    children = np.random.SeedSequence(master_seed).spawn(3)
    return [float(np.random.default_rng(s).random()) for s in children]

print([round(x, 4) for x in first_draws(2026)])     # three different streams
print(first_draws(2026) == first_draws(2026))      # True: the same seed gives the same streams

7. Software environments, version control and provenance

Code that ran last year may not install today, and a figure may come from a version of the code that nobody can identify. This chapter explains why environments are hard to rebuild, how version control names every state of a project, and how a pipeline knows exactly what to recompute.

7.1 Software environments

With semantic versioning, a version MAJOR.MINOR.PATCH increases MAJOR for incompatible changes, MINOR for compatible additions and PATCH for fixes. Constraints such as numpy>=1.24,<2 leave the installer a choice, and choosing well is hard in a precise sense.

Installers therefore use SAT solvers or heuristics, whose answer depends on the solver and on the packages published since: the same requirement file can give different environments on different days. Remedies, from the weakest to the strongest:

• pin exact versions, ideally with file hashes (pip install --require-hashes);
• commit the lock file of the package manager, which records the resolved set;
• ship a container image (Docker, Apptainer), which also freezes system libraries;
• use a functional package manager (Nix, Guix), where each package is a function of its exact inputs; guix time-machine rebuilds an environment as it was at a past commit of the package collection.

7.2 Version control as a Merkle graph

Git names objects by their hash. A blob is a file’s content; a tree lists names, permissions and hashes of blobs and sub-trees; a commit holds the hash of a tree, the hashes of its parents, an author, a date and a message.

A commit hash quoted in an article thus identifies every file and the whole history behind a result. Accidental collisions are made negligible by Proposition 0.7. Deliberate collisions of SHA-1, git’s hash, became feasible in 2017 (Stevens et al.); git now detects the known attacks and supports SHA-256 repositories.

import hashlib
from graphlib import TopologicalSorter

def git_blob(data: bytes) -> str:
    """Name of a file in git: SHA-1 of a header and the content."""
    return hashlib.sha1(b"blob %d\0" % len(data) + data).hexdigest()

print(git_blob(b""))           # e69de29bb2d1d6434b8b29ae775ad8c2e48c5391, git's empty file
print(git_blob(b"hello\n"))    # equals: echo hello | git hash-object --stdin

def h(*parts: str) -> str:
    return hashlib.sha256("\x1f".join(parts).encode()).hexdigest()[:12]

# A pipeline (Definition 7.5): each step = (code, parameters, inputs)
steps = {
    "data":    ("load v1",    "seed=0",  []),
    "clean":   ("drop NaN",   "",        ["data"]),
    "train":   ("fit ridge",  "lam=0.1", ["clean"]),
    "figure2": ("plot curve", "",        ["train"]),
    "table1":  ("summary",    "",        ["clean"]),
}

def keys(steps):                               # Proposition 7.6: hash in topological order
    order = TopologicalSorter({s: set(d[2]) for s, d in steps.items()}).static_order()
    k = {}
    for s in order:
        code, params, inputs = steps[s]
        k[s] = h(code, params, *[k[i] for i in inputs])
    return k

before = keys(steps)
steps["train"] = ("fit ridge", "lam=0.2", ["clean"])      # change one parameter
after = keys(steps)
print("recompute:", sorted(s for s in steps if before[s] != after[s]))   # ['figure2', 'train']

Commit often, tag the version used for each article (git tag paper-v1), never rewrite a history others have fetched, and write the commit hash into every result file.

7.3 Pipelines and provenance

The script checks it: changing the regularisation of train recomputes train and figure2, not table1. The classical make compares modification times, a weaker criterion; DVC, Nix and Guix index outputs by hashes of their inputs. Next to each result, store a provenance record, in the spirit of the W3C PROV model (2013): commit hash, lock file or image digest, command line, seeds, data checksums, hardware and date.

8. Publishing reproducible research: the editorial process

Publishing means passing an editorial process designed before code and data became part of research. This chapter follows a manuscript through it, models the noise of peer review, and describes the venues and identifiers that make code part of the publication.

8.1 The editorial process

1. Submission, often after a preprint on arXiv or HAL.
2. Desk check by the editor: scope, format, plagiarism; possible rejection without review.
3. Peer review by two to four reviewers, single-anonymous, double-anonymous or open; conferences add area chairs, a rebuttal and sometimes public reviews on OpenReview.
4. Decision and revision: accept, revise or reject; a revision comes with a response letter and a new round.
5. Acceptance and contract: a copyright transfer or a licence to publish (ch. 9.3).
6. Production and afterwards: proofs, the version of record with a DOI, then errata or, for serious errors, retraction.

8.2 Peer review as a noisy measurement

In the 2014 NeurIPS experiment, part of the submissions went to two independent committees; Cortes and Lawrence (2021) report disagreement on a large share of each committee’s accepted papers, of the order of the case \tau = 1. Review filters for plausibility and seldom re-executes code: reproducibility must be built into the artefacts.

import numpy as np
from scipy.stats import norm, multivariate_normal

def consistency(tau, a):                    # Proposition 8.2
    rho = 1 / (1 + tau**2)
    z = norm.ppf(1 - a)
    both = multivariate_normal(mean=[0, 0], cov=[[1, rho], [rho, 1]]).cdf([-z, -z])
    return both / a

for tau in [0.25, 0.5, 1.0, 2.0]:
    print(f"tau={tau:4.2f}  rho={1/(1+tau**2):.2f}  P(B | A)={consistency(tau, 0.25):.3f}")

rng = np.random.default_rng(0)              # Monte Carlo check for tau = 1
Q = rng.standard_normal(200_000)
SA, SB = Q + rng.standard_normal(Q.size), Q + rng.standard_normal(Q.size)
accA, accB = SA > np.quantile(SA, 0.75), SB > np.quantile(SB, 0.75)
print(f"simulated: {(accA & accB).sum() / accA.sum():.3f}")

8.3 Venues that review the code

• Artifact evaluation: the ACM awards badges after reviewing artefacts: Artifacts Available, Artifacts Evaluated (functional or reusable), Results Reproduced, Results Replicated.
• Checklists: since its 2019 reproducibility programme (Pineau et al., 2021), NeurIPS asks authors for a checklist on code, data, hyperparameters and statistics.
• ReScience C (Rougier et al., 2017) publishes replications of computational results, with open review of article and code on GitHub.
• IPOL, Image Processing On Line (Limare and Morel, 2011), publishes each article with its source code and an online demonstrator, reviewed together: the reviewer checks that the code implements the algorithm described, and any reader can run it on their own data.

The object under review becomes a triple, text, code and demonstration, and the editorial process gains a step: reviewing the code against the pseudo-code, which ch. 10 prepares.

8.4 Identifiers and archives

• A DOI names articles, data and code persistently; Zenodo, run by CERN, archives GitHub releases with a DOI.
• Software Heritage archives public source code under intrinsic, git-compatible SWHIDs: the empty file is swh:1:cnt:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391.
• HAL, the French national open archive, accepts software deposits linked to Software Heritage.
• ORCID identifies authors; a CITATION.cff file says how to cite software, following the principles of Smith et al. (2016).
• The FAIR principles (Wilkinson et al., 2016) ask data and software to be findable, accessible, interoperable and reusable.

9. Legal aspects of the code, the article and the data

A reproducibility package bundles code, text and data, which the law treats differently. This chapter explains who owns each, under which terms each can be shared, and when the whole package can be redistributed.

9.1 Three objects, three regimes

9.2 The code: who owns it

• In France, the economic rights on software created by employees in the exercise of their duties, or following their employer’s instructions, belong to the employer (article L113-9 of the Code de la propriété intellectuelle): releasing research code is the institution’s decision, often delegated to the laboratory.
• Unless agreed otherwise, the author of software cannot oppose modifications that harm neither their honour nor their reputation (article L121-7).
• For students, interns and visitors, it depends on their status and hosting agreement: check it early.

9.3 The article

At acceptance, authors transfer their copyright to the publisher or grant it a licence to publish. In France, the law for a digital republic of 7 October 2016 created a right of open-access deposit, now article L533-4 of the Code de la recherche: for an article from research funded at least half by public funds, the author may make the accepted manuscript freely available after at most 6 months in science, technology and medicine, or 12 months in the humanities and social sciences, even after transferring their rights.

With the rights retention strategy of cOAlition S, authors announce at submission that CC BY applies to any resulting accepted manuscript. Creative Commons licences combine BY (attribution), SA (share alike), NC (non-commercial) and ND (no derivatives); only CC BY, CC BY-SA and CC0 allow every reuse, and Creative Commons advises against its licences, CC0 aside, for software.

9.4 The data

• Facts are not protected; an original selection or arrangement of them can be.
• Databases in the European Union have a sui generis right for 15 years when their making required a substantial investment (Directive 96/9/EC).
• Text and data mining is allowed by Directive (EU) 2019/790 for research organisations on lawfully accessed works (article 3), and for others unless rights holders opt out (article 4).
• Personal data fall under the GDPR (Regulation (EU) 2016/679), with health data as a special category (article 9) and research rules requiring safeguards such as pseudonymisation (article 89); pseudonymised data remain personal data.
• Terms of use are a contract: data released “for non-commercial research only” cannot be redistributed by their users.

Common data licences are CC0, CC BY 4.0, the Open Database License (a copyleft) and Etalab’s Licence Ouverte 2.0.

9.5 A redistributable reproducibility package

Stodden (2009) proposed a Reproducible Research Standard: text and figures under CC BY, code under a permissive licence such as the modified BSD, data in the public domain, so that the package can be reused with attribution only.

9.6 Models and weights

The legal status of trained weights is unsettled. The OSI’s Open Source AI Definition 1.0 (October 2024) requires the freedoms of Definition 1.2 for an AI system, with access to the complete code, the parameters, and sufficiently detailed information about the training data. Models whose weights are published with restrictions of use, often called open-weight, are not open source in this sense. The European AI Act (Regulation (EU) 2024/1689) lightens some obligations for models under free and open-source licences, except general-purpose models with systemic risk; see Regulating AI.

10. Writing a reproducible scientific article

This chapter turns the previous ones into a method for writing. Its central idea: the pseudo-code, the code, the numbers and the figures of an article should be checkable against each other, by the authors first and by any reader afterwards.

10.1 Anatomy of a reproducible article

• Method: definitions, then the algorithm in pseudo-code, with every parameter named and given a default value.
• Implementation notes: every deviation of the code from the pseudo-code, such as vectorisation or numerical safeguards.
• Experiments: data and their provenance, protocol, seeds, statistics over repetitions, and the command producing each figure and table.
• Limitations, with failure examples, and an availability statement with identifiers (ch. 8.4) and licences (ch. 9).

10.2 From pseudo-code to code

Otsu’s method thresholds a grey-level image into two classes. Let p_i be the proportion of pixels of level i \in \{0, \dots, 255\}.

For a threshold t, class 0 holds the levels below t, with weight \omega_0 and mean \mu_0; class 1 the others, with weight \omega_1 = 1 - \omega_0 and mean \mu_1.

Form (b) needs only cumulative sums of the histogram, so all thresholds are evaluated in one pass: a deviation from the literal loop, to be documented and tested.

import numpy as np

def otsu_reference(img):
    """Algorithm 10.2 transcribed literally."""
    best_t, best_var = 0, -1.0
    for t in range(1, 256):                                  # line 1
        low, high = img[img < t], img[img >= t]              # line 2
        if low.size == 0 or high.size == 0:
            continue
        w0, w1 = low.size / img.size, high.size / img.size
        var_b = w0 * w1 * (low.mean() - high.mean()) ** 2    # line 3
        if var_b > best_var:                                 # line 4
            best_t, best_var = t, var_b
    return best_t

def otsu_fast(img):
    """Lines 2-3 for every t at once, by Proposition 10.3 (b): a documented deviation."""
    p = np.bincount(img.ravel(), minlength=256) / img.size
    w0 = np.cumsum(p)[:-1]                                   # omega_0(t), t = 1..255
    m0 = np.cumsum(p * np.arange(256))[:-1]                  # mu(t)
    mT = m0[-1] + 255 * p[255]
    with np.errstate(divide="ignore", invalid="ignore"):
        var_b = (mT * w0 - m0) ** 2 / (w0 * (1 - w0))
    var_b[~np.isfinite(var_b)] = -1.0                        # empty classes are skipped
    return int(np.argmax(var_b)) + 1                         # first maximiser (line 4)

rng = np.random.default_rng(7)
for trial in range(3):                                       # test of Definition 10.1
    img = np.concatenate([rng.normal(70 + 20 * trial, 15, 6000), rng.normal(170, 25, 4000)])
    img = np.clip(img, 0, 255).astype(np.uint8).reshape(100, 100)
    print(otsu_reference(img), otsu_fast(img))
    assert otsu_reference(img) == otsu_fast(img), "implementation deviates from the pseudo-code"

10.3 Tests with tolerances

Floating-point tests must allow for rounding, and no more. By Corollary 6.3, a reference and an implementation summing the same n terms in two orders never differ by more than 2\gamma_{n-1} \sum_i |x_i|; a larger difference is a bug.

NumPy’s isclose(a, b) tests |a - b| \le \mathrm{atol} + \mathrm{rtol}\,|b|: it is not symmetric, and its default atol of 10^{-8} is meaningless near zero, so set both tolerances. Add regression tests comparing the article’s main numbers with stored values.

10.4 Reporting over seeds

10.5 Packaging: the minimum to reach

The “minimum” column is the minimum knowledge announced by the official objectives: met for every article, it makes each result checkable.

10.6 Auditing someone else’s article

An audit reports the level reached and each discrepancy, with whether it supports or weakens the article’s claims: the reproducibility part of the official project.

11. Research directions and open problems

This chapter is for self-study. It points to active research on the questions of the course, in a neutral voice, and ends with open problems suitable for a dissertation or an internship.

11.1 Long-term reproducibility

Code decays even when nobody touches it: compilers, libraries, operating systems and hardware change around it. The Ten Years Reproducibility Challenge organised by ReScience C asked researchers to run again the code of their own articles published before 2010; Perkel (2020) reports on its lessons, among them missing dependencies and obsolete platforms.

Two lines of work address the problem at its root.

• Archiving the source code itself. Software Heritage, described by Di Cosmo and Zacchiroli (2017), archives publicly available source code and names it by intrinsic identifiers (ch. 8.4).
• Rebuilding environments bit for bit. Functional package managers make every package a function of its exact inputs. Dolstra and coauthors introduced Nix (2004); Courtès and Wurmus (2015) showed how Guix brings user-controlled, reproducible environments to high-performance computing.

Open questions remain for hardware: GPU generations, proprietary drivers and the emulation of obsolete platforms.

11.2 Reproducible arithmetic at scale

Corollary 6.3 shows that parallel sums depend on the order of operations. Demmel and Nguyen (2013) designed summation algorithms whose result is independent of the order and of the number of processors, at a moderate cost; such ideas underlie the ReproBLAS library.

Low precision makes the question sharper. The bfloat16 format used to train large networks has precision p = 8, so u = 2^{-8}; for a dot product of length n = 4096, nu = 16 and the bound of Theorem 6.2 no longer applies. This is why accumulations are done in higher precision.

The outputs of large language model inference servers can depend on the batch in which a request is processed, when kernels change their reduction order with the batch size. Making such systems deterministic without a large loss of speed is an open engineering and research problem.

11.3 Variance and comparisons in machine learning

• Bouthillier and coauthors (2021) decomposed the variance of machine-learning benchmarks into its sources (data sampling, initialisation, data order, augmentation, hyperparameter search) and recommended randomising as many of them as possible when comparing methods.
• Dodge and coauthors (2019) proposed to report the expected validation performance as a function of the tuning budget, rather than a single best score, an answer to the bias of Proposition 5.6.
• Melis, Dyer and Blunsom (2018) found that well-tuned standard recurrent language models matched more recent architectures, and Lucic and coauthors (2018) found no evidence that the generative adversarial networks they compared consistently beat the original one under a fair budget.
• Steegen and coauthors (2016) proposed the multiverse analysis: run every reasonable variant of an analysis and report the whole distribution of results.

11.4 Automated reproduction

Benchmarks now ask AI agents to reproduce research. PaperBench (Starace and coauthors, 2025) asks agents to replicate recent machine-learning papers from scratch, graded by detailed rubrics written with the papers’ authors. CORE-Bench (Siegel and coauthors, 2024) asks them to reproduce the results of published code capsules. Both show agents far from reliable. Whether automated checks of levels L2 to L4 (Definition 10.6) could become part of the editorial process is open.

11.5 Sustainability, security and law

• Critical infrastructure maintained by few people. In 2024 a backdoor was found in the xz compression library (CVE-2024-3094), inserted by a contributor who had gained the trust of its single overloaded maintainer: a truck factor of 1 and an instance of the volunteer’s dilemma.
• Regulation. The European Cyber Resilience Act (Regulation (EU) 2024/2847) sets security obligations for products with digital elements and a lighter regime for open-source software stewards.
• Value. Hoffmann, Nagle and Zhou (2024) estimated that firms would have to spend several times more on software if open-source software did not exist.
• Law and AI. Whether training on copyleft code creates obligations, whether model weights are protected, and how licences with use restrictions should be classified are open questions, now examined by courts and regulators.

11.6 Open problems

• A measure of reproducibility that is cheap to compute, robust to gaming, and predictive of replicability.
• Deterministic training and inference of large models at a negligible cost in speed.
• Fair comparison of methods under different computational budgets.
• Durable credit for research software and for maintenance, comparable to that given to articles.
• Licences and definitions of openness adapted to models, data and their combination.

Exercises

1. ★ Free or not? Classify each case as free software (open source), source-available or proprietary, and name the freedom or criterion at stake: (a) a licence allowing use “for academic purposes only”; (b) GPL-3.0; (c) a repository on a public platform without a licence; (d) Apache-2.0 with the Commons Clause, which removes the right to sell the software; (e) CC BY-SA 4.0 for a data set.

   Solution: (a) Not free: it discriminates against a field of endeavour (criterion 6) and breaks freedom 0; source-available if the source is shown, proprietary otherwise. (b) Free and open source. (c) Proprietary by default: all rights reserved (ch. 1.1), whatever its visibility.

   (d) Source-available, not open source: the clause also forbids paid services built on the software, so it breaks freedoms 0 and 2 and the first criterion (free redistribution). (e) A free licence for a non-software work: it allows every reuse under attribution and share-alike.

2. ★ Reading a licence. For MIT and Apache-2.0, list the permissions, conditions and limitations of Definition 2.1. What must you do when you redistribute a modified Apache-2.0 file?

   Solution: MIT permits everything, under the single condition of keeping the copyright and permission notice; it excludes warranty and liability. Apache-2.0 adds an explicit patent licence (section 3), which ends for whoever sues over patents, and excludes trademark rights (section 6).

   To redistribute a modified file: give recipients a copy of the licence, mark the modified files with prominent notices, keep the copyright and attribution notices, and include the attribution notices of a NOTICE file if the project has one (section 4).

3. ★ Combining licences. With the facts of ch. 2.3, decide whether each set can be distributed as one program, and under which licence: (a) MIT, BSD-3-Clause and GPL-2.0-only; (b) Apache-2.0 and GPL-3.0-or-later; (c) GPL-2.0-only and GPL-3.0-only; (d) GPL-3.0-only and AGPL-3.0-only.

   Solution: (a) yes, as a whole under GPL-2.0-only, keeping the MIT and BSD notices; the SPDX expression of the source is MIT AND BSD-3-Clause AND GPL-2.0-only. (b) Yes, under the GPL version 3. (c) No. (d) Yes, by section 13 of both licences; the AGPL conditions continue to apply to the AGPL part.

4. ★ The volunteer’s dilemma in numbers. With v = 6 and c = 3, compute the abstention probability q, the probability that nobody volunteers, and the expected number of volunteers for n = 2 and n = 3. What is the limit as n \to \infty?

   Solution: here c/v = 0.5. For n = 2: q = 0.5, nobody with probability 0.25, expected volunteers 2 \times 0.5 = 1.

   For n = 3: q = 0.5^{1/2} \approx 0.707, nobody with probability 0.5^{3/2} \approx 0.354, expected volunteers 3(1 - q) \approx 0.879. The probability that nobody volunteers tends to c/v = 0.5.

5. ★ Positive predictive value and replication. (a) Compute the PPV for R = 0.25, \alpha = 0.05 and power 0.5. (b) A study reports p = 0.01; under the assumption of Proposition 5.4, what is the probability that an identical replication is significant? (c) What is the probability of at least one false positive among 10 independent tests at level 0.05?

   Solution: (a) 0.125 / (0.125 + 0.05) \approx 0.714. (b) \Phi(2.576 - 1.960) = \Phi(0.616) \approx 0.731. (c) 1 - 0.95^{10} \approx 0.401.

6. ★ Floating point by hand. (a) In binary64, compare (0.1 + 0.2) + 0.3 and 0.1 + (0.2 + 0.3). (b) Find the smallest positive integer N such that float32(N) + 1 equals float32(N).

   Solution: (a) Python prints 0.6000000000000001 and 0.6: each sum is correctly rounded (Proposition 0.2), but the intermediate roundings differ, so associativity fails.

   (b) N = 2^{24} = 16{,}777{,}216. Binary32 has p = 24, so integers up to 2^{24} are exact. Beyond it, consecutive floats are 2 apart, and 2^{24} + 1 is a tie, rounded to the even neighbour 2^{24}.

7. ★ Names in git (Python). Compute the git name of a file containing hello\n, then of the same file with one character changed. Build a tiny tree object from two files and check that renaming a file changes the tree’s name but not the blobs’ names.

   Solution:

   import hashlib
   
   def obj(kind: bytes, data: bytes) -> bytes:          # raw 20-byte git name
       return hashlib.sha1(kind + b" %d\0" % len(data) + data).digest()
   
   def tree(entries):                                   # entries: {filename: content}
       body = b"".join(b"100644 " + name.encode() + b"\0" + obj(b"blob", data)
                       for name, data in sorted(entries.items()))
       return obj(b"tree", body).hex()
   
   print(obj(b"blob", b"hello\n").hex())   # ce013625030ba8dba906f756967f9e9ca394464a
   print(obj(b"blob", b"hellp\n").hex())   # a completely different name
   t1 = tree({"a.py": b"x = 1\n", "b.py": b"y = 2\n"})
   t2 = tree({"a.py": b"x = 1\n", "c.py": b"y = 2\n"})  # b.py renamed to c.py
   print(t1 != t2, obj(b"blob", b"y = 2\n").hex())      # True, and the blob name is unchanged

   The tree hashes the file names together with the blob names (Definition 7.3), so a rename changes the tree, and therefore every commit that contains it.

8. ★★ Compatibility graphs (Python). In the model of Definition 2.3, prove that compatibility of two licences is reflexive and symmetric, show with ch. 2.3 that it is not transitive, and prove that the compatible sets are exactly the cliques of the compatibility graph. Compute the maximal compatible sets of the six licences of the code of ch. 2.3.

   Solution: reflexivity holds because O(\ell) \subseteq A(\ell); symmetry because the definition of a compatible set does not depend on the order. MIT is compatible with Apache-2.0 and with GPL-2.0-only, which are not compatible with each other. By Proposition 2.4, a set is compatible exactly when all its pairs are, that is, when it is a clique.

   import networkx as nx
   from itertools import combinations
   
   incompatible = {("Apache-2.0", "GPL-2.0-only"), ("GPL-2.0-only", "GPL-3.0-or-later"),
                   ("GPL-2.0-only", "AGPL-3.0-only")}
   names = ["MIT", "BSD-3-Clause", "Apache-2.0", "GPL-2.0-only", "GPL-3.0-or-later",
            "AGPL-3.0-only"]
   G = nx.Graph()
   G.add_nodes_from(names)
   G.add_edges_from(p for p in combinations(names, 2)
                    if p not in incompatible and p[::-1] not in incompatible)
   for clique in nx.find_cliques(G):
       print(sorted(clique))

   The two maximal compatible sets are {MIT, BSD-3-Clause, GPL-2.0-only} and {MIT, BSD-3-Clause, Apache-2.0, GPL-3.0-or-later, AGPL-3.0-only}.

9. ★★ When the greedy truck factor fails. A project has 10 files: 3 written by X alone, 5 written jointly by A and B, one by Y alone and one by Z alone. Compute the truck factor exactly, then run by hand the greedy variant used in the code of ch. 3.6, and the ranking by number of files authored of Avelino and coauthors.

   Solution: more than half means at least 6 orphaned files. No pair suffices: the best pairs, \{A, B\} and \{X, Y\}, orphan 5 and 4 files. The triple \{A, B, X\} orphans 8 files, so \mathrm{TF} = 3.

   The greedy variant first removes X (3 files), then Y (4 files), then Z (5 files), since removing A or B alone orphans nothing.

   It then needs both A and B, and returns 5: it misses departures that only matter together.

   The ranking by files authored removes A and B first (5 files each), then X, and returns the exact value 3 on this project.

   It fails elsewhere. Take four files written by A with four different coauthors, and three files each by X and Y alone. The ranking removes A first and returns 3, whereas \{X, Y\} gives 2. Only exhaustive search is exact.

10. ★★ The best of n seeds. Let X_1, X_2, X_3 be independent standard normal variables, with density \varphi and distribution function \Phi. (a) Show that \mathbb{E}\big[\max_i X_i\big] = 3 \int_{\mathbb{R}} x\,\varphi(x)\,\Phi(x)^2\,dx .

    2. Using \varphi'(x) = -x\,\varphi(x), deduce that e_3 = 3/(2\sqrt{\pi}) \approx 0.846, and check the value e_2 = 1/\sqrt{\pi} of Proposition 5.6 by the same method.

    Hints and expected result: (a) the maximum is X_1 with probability \Phi(X_1)^2 given X_1, and the three variables play symmetric roles. (b) An integration by parts turns \int x\,\varphi\,\Phi^2 into 2 \int \varphi^2\,\Phi.

    Then \varphi(x)^2 equals \frac{1}{2\sqrt{\pi}} times the density of \mathcal{N}(0, 1/2). If Y \sim \mathcal{N}(0, 1/2) is independent of Z \sim \mathcal{N}(0, 1), the remaining integral is \mathbb{P}(Z \le Y) = 1/2.

    Hence \int x\,\varphi\,\Phi^2 = \frac{1}{2\sqrt{\pi}} and e_3 = \frac{3}{2\sqrt{\pi}}. For two seeds, the same steps give e_2 = 2\int x\,\varphi\,\Phi\,dx = 2\int \varphi^2\,dx = \frac{1}{\sqrt{\pi}} .

11. ★★ How large are summation errors really? (Python) Sum n standard normal numbers in binary32 from left to right, in 20 random orders, for n = 10^3, 10^4, 10^5. Compare the standard deviation of the results with the bound of Corollary 6.3, and with the estimate obtained by treating the rounding errors as independent, of mean 0 and standard deviation about u/\sqrt{3}: \frac{u}{\sqrt{3}} \Big( \sum_{k=1}^{n} s_k^2 \Big)^{1/2}, \qquad s_k = x_1 + \dots + x_k .

    Solution:

    import numpy as np
    
    rng = np.random.default_rng(3)
    u = np.finfo(np.float32).eps / 2
    for n in [1_000, 10_000, 100_000]:
        x = rng.standard_normal(n).astype(np.float32)
        sums = [np.cumsum(x[rng.permutation(n)], dtype=np.float32)[-1] for _ in range(20)]
        A = np.abs(x).astype(np.float64).sum()
        bound = 2 * (n - 1) * u / (1 - (n - 1) * u) * A               # Corollary 6.3
        partial = np.cumsum(x.astype(np.float64))
        typical = u / np.sqrt(3) * np.sqrt((partial**2).sum())        # random-sign estimate
        print(f"n={n:6d}  std={np.std(sums):.1e}  estimate={typical:.1e}  bound={bound:.1e}")

    The standard deviation (about 2 \times 10^{-5}, 10^{-4} and 8 \times 10^{-4}) is within a factor 1.5 of the random-sign estimate.

    For these zero-mean data the partial sums grow like \sqrt{k}, so the typical error grows roughly like n, while the worst-case bound grows like n^2: at n = 10^5 they differ by six orders of magnitude. The bound remains the only guarantee.

12. ★★ From 3-SAT to packages (Python). Encode the formula (x_1 \lor \lnot x_2 \lor x_3) \land (\lnot x_1 \lor x_2) \land (\lnot x_3) as a repository, following the proof of Theorem 7.2. Find an installation of the target by exhaustive search, and read a satisfying assignment from it.

    Solution:

    from itertools import product
    
    clauses = [[1, -2, 3], [-1, 2], [-3]]                 # literal +i is x_i, -i is not x_i
    pkg = lambda lit: f"x{abs(lit)}{'+' if lit > 0 else '-'}"
    depends = {"t": [[f"c{j}"] for j in range(len(clauses))]}   # conjunction of disjunctions
    depends.update({f"c{j}": [[pkg(l) for l in C]] for j, C in enumerate(clauses)})
    conflicts = {(f"x{i}+", f"x{i}-") for i in (1, 2, 3)}
    
    def is_installation(S):
        deps_ok = all(any(d in S for d in disj) for p in S for disj in depends.get(p, []))
        return deps_ok and not any(a in S and b in S for a, b in conflicts)
    
    base = {"t", "c0", "c1", "c2"}
    for signs in product("+-", repeat=3):
        S = base | {f"x{i + 1}{s}" for i, s in enumerate(signs)}
        if is_installation(S):
            print(sorted(S), {f"x{i + 1}": s == "+" for i, s in enumerate(signs)})

    Two installations exist, with x_3 false and x_1 = x_2; for example x_1 = x_2 = true satisfies the three clauses.

13. ★★ Review noise and a redistributable package. (a) In the model of Definition 8.1 with a = 0.25, find numerically the noise \tau for which \mathbb{P}(B \text{ accepts} \mid A \text{ accepts}) = 0.6. (b) A package combines into one program MIT code, an Apache-2.0 module and a GPL-2.0-only module, and ships CC BY figures beside it. Can it be redistributed, and what is the smallest change that makes it redistributable?

    Hints and expected result: (a) the function consistency of ch. 8.2 decreases in \tau; a bisection between 0.5 and 1 (for instance scipy.optimize.brentq) gives \tau \approx 0.66.

    (b) No: by Proposition 9.2 the figures are fine, but the combined program contains the incompatible pair Apache-2.0 and GPL-2.0-only. The smallest change is to replace one of these two modules, or to obtain “version 2 or later” for the GPL module and distribute the program under the GPL version 3.

14. ★★ Otsu and seeds. (a) Prove the decomposition \sigma^2 = \sigma_W^2 + \sigma_B^2 of Proposition 10.3 from the definitions of the class weights and means. (b) A method’s score has seed-to-seed standard deviation 0.8 point. How many seeds give a 95 % interval of half-width 0.25 point? (c) Two methods run on the same seeds have equal standard deviations 0.8 and correlation 0.9; how many seeds are needed for the same precision on their difference?

    Solution: (a) Write \sum_i p_i (i - \mu_T)^2 as the sum over the two classes, and expand i - \mu_T = (i - \mu_k) + (\mu_k - \mu_T) in class k. The cross terms vanish by definition of \mu_k, which leaves \omega_0 \sigma_0^2 + \omega_1 \sigma_1^2 + \omega_0 (\mu_0 - \mu_T)^2 + \omega_1 (\mu_1 - \mu_T)^2 . The last two terms equal \omega_0 \omega_1 (\mu_0 - \mu_1)^2.

    (b) (1.96 \times 0.8 / 0.25)^2 \approx 39.3, so 40 seeds.

    (c) The variance of the difference is 0.64 + 0.64 - 2 \times 0.9 \times 0.64 = 0.128, a standard deviation of about 0.358. Then (1.96 \times 0.358 / 0.25)^2 \approx 7.9, so 8 seeds suffice: pairing divides the cost by five.

15. ★★★ Individual dissertation. Write a two-page dissertation on the question of ch. 9: should publicly funded research software be released under a copyleft or a permissive licence?

    Hints and expected result: define both families precisely (Definition 2.2) and state the legal facts that constrain the choice (ch. 9.2, CeCILL in ch. 2.5). Use the economic arguments of ch. 3: adoption and sponsors under permissive licences (ch. 4.2), returns and dual licensing under copyleft (ch. 4.4). Discuss compatibility with the dependencies (Proposition 2.4). Conclude with a decision rule that depends on the kind of software, such as a library, an application or a service, rather than a single answer.

16. ★★★ Audit an IPOL article. Choose an article of the journal Image Processing On Line, for instance “Anatomy of the SIFT Method” (Rey-Otero and Delbracio, 2014). Fill the grid of Definition 4.1, determine its audit level (Definition 10.6), and list every discrepancy between the pseudo-code and the code.

    Hints and expected result: IPOL articles come with their source code, so L1 is usually reached at once; L2 requires building the code in a documented environment. Use the online demonstrator for L3 on the article’s own examples. For L4, map each line of each algorithm to the code (Definition 10.1) and test the parts that differ. Report each discrepancy with its effect on the published claims; this exercise is a good rehearsal for Group project 2.

Projects and assessment

The official assessment (master-mva.com) is a project: each group of 1 to 3 students studies a software project and presents it with slides — its licences, economic model, type of application and other aspects, and its level of reproducibility, including whether the sources match the pseudo-codes of the article. The final presentation counts for 80 % and the intermediate TP presentations for 20 %; Université Paris-Saclay describes the assessment as “Evaluation Continue non Intégrale”.

Group project 1 below is the official assignment, structured by the grid of Definition 4.1 and the audit levels of Definition 10.6. The other subjects are proposals for individual work, for a research-oriented extension, or for preparing the follow-up project course. The official assessment has no written exam; the exam proposal is for self-assessment.

Solo projects

1. [Solo · ≈ 20 h] Paper study: variance in machine-learning benchmarks. Reproduce at small scale the main finding of Bouthillier et al. (2021): several sources of variance, and not only the initialisation seed, must be randomised to compare two methods.
   • Data: the digits and breast_cancer data sets bundled with scikit-learn. Models: a linear classifier and a small multilayer perceptron, both trained by stochastic gradient in PyTorch on a CPU, with separate generators for the initialisation and the data order (scikit-learn’s MLPClassifier draws both from one random_state, and its default logistic regression has no randomness at all).
   • Implement: 50 repetitions randomising the data split, the initialisation and the data order separately, then jointly; estimate each variance share and the probability that one method beats the other.
   • Success criterion: a table of variance shares with standard errors, compared with the ranking of sources reported in the paper, and a figure of that probability against the number of seeds (Proposition 5.6, Proposition 10.5).
   • Deliverables: code, a 4–6 page report, a 10-minute talk. Chapters 5, 6 and 10.
2. [Solo · ≈ 20 h] Licence and economic study of a scientific library. The individual version of the official assignment, on a library such as scikit-image, GDAL, Astropy or FEniCS.
   • Fill the seven lines of the grid of Definition 4.1 with sources for each fact.
   • Inventory the licences of the dependencies and vendored code, write the SPDX expression of a binary distribution, and check its compatibility (Proposition 2.4).
   • Compute the truck factor from the git history (git log --name-only), with the greedy heuristic and an exhaustive check on the 15 most active authors.
   • Success criterion: a complete, sourced grid and a justified SPDX expression. Deliverables: 10-minute slides and a 4-page report. Chapters 2, 3 and 4.
3. [Solo · ≈ 20 h] Bitwise-reproducible training on a CPU. Make a small PyTorch training run, a multilayer perceptron on the scikit-learn digits data, reproducible bit for bit, then measure what breaks it.
   • Factors to vary one at a time: number of threads, data-loader workers, float32 against float64, deterministic mode on and off, library version.
   • Success criterion: identical hashes of the final weights over five runs in the controlled setting, and a table of the size of the differences when one factor changes, compared with Corollary 6.3.
   • Deliverables: code with a lock file, a 4–6 page report, a 10-minute talk. Chapters 6 and 7. It prepares the Reproducible Research Project of semester 2.

Group projects

1. [Group of 2–3 · ≈ 30 h each] The official project: audit of a scientific software project and its article. Choose a project that comes with an article containing pseudo-code, for instance an IPOL article or a library with a reference paper.
   • Member 1: licences, rights holders, governance and economic model (grid lines 1–6, ch. 1–4).
   • Member 2: environment and rerun (levels L2–L3): rebuild, rerun, regenerate two published figures or tables within a stated tolerance (ch. 6–7).
   • Member 3: code against pseudo-code (level L4): line-by-line mapping, list of deviations, faithfulness tests (Definition 10.1).
   • Milestones: one per lab session (sessions 4, 7, 9), presented in 5 minutes. Success criterion: an audit level justified by evidence, with every discrepancy classified as for or against reproducibility.
   • Deliverables: slides for a 20-minute final presentation, a 6–8 page report, the scripts. Chapters 1–10.
2. [Group of 2–3 · ≈ 30 h each] Paper study: “Anatomy of the SIFT Method” (Rey-Otero and Delbracio, 2014). Audit this IPOL article and compare its implementation with OpenCV’s SIFT.
   • Member 1: map the article’s algorithms to its published code and list the deviations (Definition 10.1).
   • Member 2: run both implementations on images from sklearn.datasets.load_sample_image, rotated and rescaled, and measure keypoint repeatability with a tolerance in pixels.
   • Member 3: legal and economic analysis: the licences of both codes, the SIFT patent and its expiry (Example 2.6), the build options of OpenCV.
   • Success criterion: a table of matching steps, repeatability numbers with standard errors over 10 transformations, and an explained list of differences. Deliverables: code, a 6–8 page report, a 20-minute talk. Chapters 2, 4, 6 and 10.
3. [Group of 2–3 · ≈ 30 h each] Cross-reproduction exchange. Two groups each package a small experiment, then reproduce each other’s from the package alone, as reviewers of a venue that reviews code (ch. 8.3).
   • The experiment: ridge regression or k-means on scikit-learn data, with three figures and one table.
   • Member 1: packaging following the table of ch. 10.5 (licence, lock file, download script with checksums, CITATION.cff).
   • Member 2: a content-addressed pipeline (Proposition 7.6) and a provenance record per result.
   • Member 3: the review of the other group’s package, reporting the audit level reached.
   • Success criterion: the other group regenerates every figure and table with one command, within the stated tolerance. Deliverables: both packages, a review report, a 20-minute talk. Chapters 6, 7, 8 and 10.

Written exam (proposal)

Two hours, no documents, a non-programmable calculator allowed; four questions.

1. Question (5 points). Define copyleft and its scopes. Explain why an Apache-2.0 module and a GPL-2.0-only module cannot be distributed together in one program, whereas Apache-2.0 and the GPL version 3 can. What would a project with thousands of rights holders and no CLA need in order to move to the GPL version 3? Expected: Definition 2.2 and the table of ch. 2.2; the patent termination clause as a further restriction for the GPL version 2, Proposition 2.4; the rule of ch. 3.5 and the case of ch. 4.1: the consent of every rights holder.

2. Question (5 points). (a) In the volunteer’s dilemma with v = 8, c = 2 and n = 4, compute the abstention probability and the probability that nobody volunteers. (b) Compute the PPV of a finding for R = 0.2, power 0.6 and \alpha = 0.05.

   Expected: (a) q = 0.25^{1/3} \approx 0.630 and 0.25^{4/3} \approx 0.157 (Theorem 3.2). (b) 0.12 / 0.17 \approx 0.706 (Theorem 5.2).

3. Question (5 points). Prove Lemma 6.1 and Theorem 6.2, and deduce Corollary 6.3. Application: n = 1000 binary32 numbers with \sum_i |x_i| = 500.

   Expected: the bounds (1 \pm u)^m on the product, e^x \le 1/(1-x) and Bernoulli’s inequality; the unrolled recursion; the triangle inequality. Numerically \gamma_{999} \approx 5.95 \times 10^{-5}, so the error is at most about 0.030 and two orders differ by at most about 0.060.

4. Question (5 points). A colleague’s article: the code is on a public platform without a licence, the data set is “for non-commercial research only”, the reported score is the best of 5 seeds, and the code normalises the input while the pseudo-code does not. Give the audit level, the legal problems, and a corrective action for each issue.

   Expected: level L0 in the sense of Definition 10.6, since the code may be read but not reused (ch. 1.1); a licence chosen with the institution (ch. 9.2); a download script with checksums instead of redistributed data (Example 9.3). The best of 5 seeds overstates the mean by about 1.16\sigma (Proposition 5.6), so report the mean and standard error over enough seeds (Proposition 10.5). The normalisation must be documented as a deviation and tested (Definition 10.1).

Grading rubric (proposal)

The official weights are kept: 20 % for the intermediate presentations and 80 % for the final one. The split of the final 80 % below is a proposal.

References

Official sources

• master-mva.com, course page “Fondamentaux de la recherche reproductible et du logiciel libre / Fundamentals of reproducible research and free software”, 2026–2027: https://www.master-mva.com/cours/fondamentaux-de-la-recherche-reproductible-et-du-logiciel-libre-fundamentals-of-reproducible-research-and-free-software/. The page lists the personal page of the instructor, http://mcolom.info.
• Université Paris-Saclay, M2 Mathematics, Vision, Learning, programme table (ECTS and hourly volumes): https://www.universite-paris-saclay.fr/en/education/master/mathematics-and-applications/m2-mathematics-vision-learning.

Definitions, licences and legal texts

• Free Software Foundation, What is Free Software? and Various Licenses and Comments about Them, https://www.gnu.org/philosophy/free-sw.html and https://www.gnu.org/licenses/license-list.html.
• Open Source Initiative, The Open Source Definition, https://opensource.org/osd; The Open Source AI Definition 1.0, 2024.
• Free Software Foundation, GNU General Public License, version 2 (1991) and version 3 (2007); GNU Affero General Public License, version 3 (2007).
• Apache Software Foundation, Apache License, Version 2.0, 2004. Mozilla Foundation, Mozilla Public License 2.0, 2012.
• CEA, CNRS, Inria, CeCILL Free Software License Agreement, version 2.1, 2013. European Commission, European Union Public Licence, version 1.2, 2017.
• ISO/IEC 5962:2021, Information technology — SPDX® Specification V2.2.1.
• Directive 2009/24/EC on the legal protection of computer programs; Directive 96/9/EC on the legal protection of databases; Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market.
• Regulation (EU) 2016/679 (General Data Protection Regulation); Regulation (EU) 2024/1689 (Artificial Intelligence Act); Regulation (EU) 2024/2847 (Cyber Resilience Act).
• Code de la propriété intellectuelle, articles L112-2, L113-9 and L121-7; Code de la recherche, article L533-4, created by the loi n° 2016-1321 du 7 octobre 2016 pour une République numérique.
• ACM, Artifact Review and Badging, version 1.1, 2020.
• L. Moreau, P. Missier (eds.), PROV-DM: The PROV Data Model, W3C Recommendation, 2013.

Books

• R. M. Stallman, Free Software, Free Society: Selected Essays of Richard M. Stallman, GNU Press, 2002.
• E. S. Raymond, The Cathedral and the Bazaar, O’Reilly, 1999.
• L. Rosen, Open Source Licensing: Software Freedom and Intellectual Property Law, Prentice Hall, 2004.
• A. M. St. Laurent, Understanding Open Source and Free Software Licensing, O’Reilly, 2004.
• N. Eghbal, Working in Public: The Making and Maintenance of Open Source Software, Stripe Press, 2020.
• National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, The National Academies Press, 2019.
• The Turing Way Community, The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research, Zenodo, 2022.
• N. J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd edition, SIAM, 2002.
• D. E. Knuth, The Art of Computer Programming, Volume 2: Seminumerical Algorithms, 3rd edition, Addison-Wesley, 1997.
• S. Chacon, B. Straub, Pro Git, 2nd edition, Apress, 2014.

Economics of free software

• P. A. Samuelson, The pure theory of public expenditure, The Review of Economics and Statistics 36(4), 387–389, 1954.
• A. Diekmann, Volunteer’s dilemma, Journal of Conflict Resolution 29(4), 605–610, 1985.
• T. Bergstrom, L. Blume, H. Varian, On the private provision of public goods, Journal of Public Economics 29(1), 25–49, 1986.
• J. Lerner, J. Tirole, Some simple economics of open source, Journal of Industrial Economics 50(2), 197–234, 2002.
• G. Avelino, L. Passos, A. Hora, M. T. Valente, A novel approach for estimating truck factors, IEEE International Conference on Program Comprehension (ICPC), 2016.
• M. Hoffmann, F. Nagle, Y. Zhou, The value of open source software, Harvard Business School Working Paper 24-038, 2024.

Reproducible research

• J. F. Claerbout, M. Karrenbach, Electronic documents give reproducible research a new meaning, SEG Technical Program Expanded Abstracts, 601–604, 1992.
• J. B. Buckheit, D. L. Donoho, WaveLab and reproducible research, in Wavelets and Statistics, Lecture Notes in Statistics 103, Springer, 55–81, 1995.
• D. L. Donoho, A. Maleki, I. U. Rahman, M. Shahram, V. Stodden, Reproducible research in computational harmonic analysis, Computing in Science & Engineering 11(1), 8–18, 2009.
• V. Stodden, The legal framework for reproducible scientific research: licensing and copyright, Computing in Science & Engineering 11(1), 35–40, 2009.
• P. Vandewalle, J. Kovačević, M. Vetterli, Reproducible research in signal processing, IEEE Signal Processing Magazine 26(3), 37–47, 2009.
• R. D. Peng, Reproducible research in computational science, Science 334(6060), 1226–1227, 2011.
• G. K. Sandve, A. Nekrutenko, J. Taylor, E. Hovig, Ten simple rules for reproducible computational research, PLoS Computational Biology 9(10), e1003285, 2013.
• G. Wilson et al., Best practices for scientific computing, PLoS Biology 12(1), e1001745, 2014.
• G. Wilson, J. Bryan, K. Cranston, J. Kitzes, L. Nederbragt, T. K. Teal, Good enough practices in scientific computing, PLoS Computational Biology 13(6), e1005510, 2017.
• L. A. Barba, Terminologies for reproducible research, arXiv:1802.03311, 2018.
• H. E. Plesser, Reproducibility vs. replicability: a brief history of a confused terminology, Frontiers in Neuroinformatics 11, 76, 2018.
• Open Science Collaboration, Estimating the reproducibility of psychological science, Science 349(6251), aac4716, 2015.
• M. Baker, 1,500 scientists lift the lid on reproducibility, Nature 533, 452–454, 2016.
• J. P. A. Ioannidis, Why most published research findings are false, PLoS Medicine 2(8), e124, 2005.
• S. N. Goodman, A comment on replication, p-values and evidence, Statistics in Medicine 11(7), 875–879, 1992.
• J. F. Pimentel, L. Murta, V. Braganholo, J. Freire, A large-scale study about quality and reproducibility of Jupyter notebooks, IEEE/ACM International Conference on Mining Software Repositories (MSR), 2019.
• E. Raff, A step toward quantifying independently reproducible machine learning research, NeurIPS, 2019.
• P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, D. Meger, Deep reinforcement learning that matters, AAAI, 2018.
• S. Kapoor, A. Narayanan, Leakage and the reproducibility crisis in machine-learning-based science, Patterns 4(9), 100804, 2023.

Publishing, review and archiving

• N. Limare, J.-M. Morel, The IPOL initiative: publishing and testing algorithms on line for reproducible research in image processing, Procedia Computer Science 4, 716–725, 2011.
• M. Colom, B. Kerautret, N. Limare, P. Monasse, J.-M. Morel, IPOL: a new journal for fully reproducible research; analysis of four years development, International Conference on New Technologies, Mobility and Security (NTMS), 2015.
• N. P. Rougier et al., Sustainable computational science: the ReScience initiative, PeerJ Computer Science 3, e142, 2017.
• J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivière, A. Beygelzimer, F. d’Alché-Buc, E. Fox, H. Larochelle, Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program), Journal of Machine Learning Research 22(164), 1–20, 2021.
• C. Cortes, N. D. Lawrence, Inconsistency in conference peer review: revisiting the 2014 NeurIPS experiment, arXiv:2109.09774, 2021.
• A. M. Smith, D. S. Katz, K. E. Niemeyer, FORCE11 Software Citation Working Group, Software citation principles, PeerJ Computer Science 2, e86, 2016.
• M. D. Wilkinson et al., The FAIR guiding principles for scientific data management and stewardship, Scientific Data 3, 160018, 2016.
• R. Di Cosmo, S. Zacchiroli, Software Heritage: why and how to preserve software source code, iPRES, 2017.

Numerics, randomness, environments and algorithms

• D. Goldberg, What every computer scientist should know about floating-point arithmetic, ACM Computing Surveys 23(1), 5–48, 1991.
• W. Kahan, Further remarks on reducing truncation errors, Communications of the ACM 8(1), 40, 1965.
• J. Demmel, H. D. Nguyen, Fast reproducible floating-point summation, IEEE Symposium on Computer Arithmetic (ARITH), 2013.
• T. E. Hull, A. R. Dobell, Random number generators, SIAM Review 4(3), 230–254, 1962.
• M. Matsumoto, T. Nishimura, Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator, ACM Transactions on Modeling and Computer Simulation 8(1), 3–30, 1998.
• M. E. O’Neill, PCG: a family of simple fast space-efficient statistically good algorithms for random number generation, Technical Report HMC-CS-2014-0905, Harvey Mudd College, 2014.
• F. Mancinelli, J. Boender, R. Di Cosmo, J. Vouillon, B. Durak, X. Leroy, R. Treinen, Managing the complexity of large free and open source package-based software distributions, IEEE/ACM International Conference on Automated Software Engineering (ASE), 2006.
• R. C. Merkle, A digital signature based on a conventional encryption function, CRYPTO ’87, Lecture Notes in Computer Science 293, 369–378, 1988.
• M. Stevens, E. Bursztein, P. Karpman, A. Albertini, Y. Markov, The first collision for full SHA-1, CRYPTO 2017, Lecture Notes in Computer Science 10401, 2017.
• E. Dolstra, M. de Jonge, E. Visser, Nix: a safe and policy-free system for software deployment, USENIX LISA, 2004.
• L. Courtès, R. Wurmus, Reproducible and user-controlled software environments in HPC with Guix, Euro-Par 2015 Workshops, Lecture Notes in Computer Science 9523, 2015.
• N. Otsu, A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics 9(1), 62–66, 1979.
• D. G. Lowe, Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision 60(2), 91–110, 2004.
• I. Rey-Otero, M. Delbracio, Anatomy of the SIFT method, Image Processing On Line 4, 370–396, 2014.
• F. Pedregosa et al., Scikit-learn: machine learning in Python, Journal of Machine Learning Research 12, 2825–2830, 2011.

Research directions

• X. Bouthillier et al., Accounting for variance in machine learning benchmarks, Proceedings of Machine Learning and Systems (MLSys), 2021.
• J. Dodge, S. Gururangan, D. Card, R. Schwartz, N. A. Smith, Show your work: improved reporting of experimental results, EMNLP-IJCNLP, 2019.
• G. Melis, C. Dyer, P. Blunsom, On the state of the art of evaluation in neural language models, ICLR, 2018.
• M. Lucic, K. Kurach, M. Michalski, S. Gelly, O. Bousquet, Are GANs created equal? A large-scale study, NeurIPS, 2018.
• S. Steegen, F. Tuerlinckx, A. Gelman, W. Vanpaemel, Increasing transparency through a multiverse analysis, Perspectives on Psychological Science 11(5), 702–712, 2016.
• J. M. Perkel, Challenge to scientists: does your ten-year-old code still run?, Nature 584, 656–658, 2020.
• G. Starace et al., PaperBench: evaluating AI’s ability to replicate AI research, arXiv:2504.01848, 2025.
• Z. S. Siegel, S. Kapoor, N. Nadgir, B. Stroebl, A. Narayanan, CORE-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark, arXiv:2409.11363, 2024.