--- name: eurosys-reproducibility description: Use when hardening the reproducibility story of a EuroSys paper — recording hardware and software provenance for every number, taming performance variance with repeated runs and dispersion reporting, versioning workloads and traces, and writing an availability statement that survives both double-blind review and the sysartifacts AEC. --- # EuroSys Reproducibility Use this while experiments are still running — reproducibility retrofitted in deadline week is transcription, not engineering. The venue context: EuroSys papers live on measured performance claims, the community runs a badge-granting artifact evaluation (`sysartifacts.github.io`), and SIGOPS has publicly digested five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08). A paper whose numbers cannot be regenerated by its own authors three months later fails both review-time scrutiny and post-acceptance AE. ## The provenance ledger Keep one machine-readable record per reported number. Minimum fields: | Field | Why EuroSys reviewers care | |---|---| | Commit hash of system under test | "Which version got 2.1x?" is a real AE question | | Baseline name + version + config | Untuned-baseline suspicion is the venue's default | | Hardware: CPU/RAM/NIC/storage, topology | Systems results rarely transfer across boxes | | OS/kernel, key library versions | Kernel changes move I/O and scheduler numbers | | Workload/trace + generator seed | Trace provenance is checked, not assumed | | Repetitions, warm-up policy | Distinguishes measurement from anecdote | | Timestamp + raw-output path | Lets you rebuild any figure from raw logs | ## Variance discipline Single-run numbers are the most common silent reproducibility failure in systems evaluation: - Repeat every headline measurement enough times to see its spread; report median plus an explicit dispersion measure (stdev, IQR, or min–max), and say in the caption which one it is. - Isolate noise sources you control: pin frequencies, disable turbo where it distorts comparisons, note co-located load, randomize run order across systems so drift does not favor yours. - Tail metrics (p99 and beyond) need far more samples than means; state the sample count whenever a tail latency is claimed. - When a difference is within run-to-run spread, say so — EuroSys reviewers reward calibrated claims over uniform victory narratives. ## Automation floor The practical bar: any figure regenerates from raw data with one command. ```bash # Layout that keeps figures honest experiments/ fig7_throughput/ run.sh # executes the sweep, writes results/*.csv with metadata header plot.py # reads results/, emits fig7.pdf — no hand-edited numbers results/ # raw outputs, never overwritten, one dir per run timestamp make fig7 # the only path by which fig7.pdf ever changes ``` If a plot was ever touched manually, its provenance is broken and the AEC will find the discrepancy before you do. ## Availability statement, two audiences - **Review time (double-blind):** describe what exists — "an anonymized repository containing the system, workload generator, and run scripts accompanies the submission" — without leaking the lab's identity through URLs, paths, or commit authors. - **Camera-ready / AE time:** replace with the DOI-backed archive and the badge set being sought. Restricted traces (production data, partner NDAs) need an honest fallback: a synthetic generator calibrated to the trace's published statistics, with the calibration method described. ## Restricted evidence, stated honestly Some EuroSys evidence legitimately cannot ship — production traces under NDA, partner clusters, proprietary workloads. The honest pattern: - Name the restriction and its scope precisely ("the ingestion trace from operator X cannot be released; its summary statistics are in Table 3"). - Ship a calibrated synthetic substitute and the calibration procedure, so external readers can approximate the regime. - Keep at least one headline result on fully public inputs; a paper whose every number depends on unreleasable data asks reviewers for faith the venue does not trade in. - Never let the availability paragraph imply more openness than the AEC will find; the badge process makes overstatement visible in print. ## Pre-deadline reproducibility drill One week before the paper gate, run the drill on a machine that never ran the experiments: 1. Clean checkout, environment build from the lockfile alone — record every undocumented step it turns out to need. 2. Regenerate two figures end to end: one cheap, one expensive. 3. Diff regenerated numbers against the draft's numbers; investigate any drift beyond the reported dispersion. 4. Fix the documentation, not just the outcome — the drill's product is the README the AEC will eventually read. ## Quick self-test 1. Can a new student regenerate Figure 7 from a clean checkout in one command? 2. Does every table cell trace to a raw log file with hardware metadata? 3. Are repetition counts and dispersion visible for every performance claim? 4. Would the availability paragraph survive both anonymity and the AEC? ## Output format ```text [Repro grade] regenerable / scripted-with-gaps / manual [Ledger coverage] [Variance findings] [Workload provenance] [Availability draft] ```