--- name: mlsys-reproducibility description: Use when hardening the reproducibility of MLSys performance claims, pinning the full system layer from driver to interconnect, separating ML randomness from systems noise, choosing repetition counts and variance reporting for throughput and latency numbers, and disclosing hardware, workloads, and cost so strangers can re-measure results. --- # MLSys Reproducibility Use this while experiments are still running — reproducibility at this venue is a measurement-design property, not a packaging afterthought. An MLSys claim is typically "system A beats system B by X% on workload W on hardware H," and every one of those four variables can silently drift. The venue's culture (badge-based artifact evaluation, the MLPerf benchmark lineage published in its own proceedings) means reviewers assume performance numbers will eventually be re-measured by someone else. ## Two kinds of nondeterminism — control them separately | Source | Examples | Control | |---|---|---| | ML randomness | Init seeds, data order, dropout, sampling temperature | Fix and log seeds; report across-seed variation where accuracy matters | | Systems noise | Clock boosting/thermal state, co-tenant interference, NUMA/PCIe placement, network jitter, filesystem caches | Warmup phases, repeated trials, exclusive nodes, pinned placement, reporting distributions | Papers routinely fix seeds meticulously while leaving thermal state and placement uncontrolled — backwards for a performance paper, where systems noise usually dwarfs seed effects on latency numbers. ## The environment pin — deeper than requirements.txt A latency claim depends on layers a Python lockfile never sees. Record all of them: - Hardware: GPU/accelerator model and count, CPU, memory, interconnect (NVLink/PCIe generation, NIC), storage class. - System: driver version, CUDA/ROCm version, container image digest, kernel version. - Framework: exact framework build, compilation flags, graph/eager mode, precision (FP16/BF16/FP8/INT4), and any autotuning caches — a warm autotuner cache can fake a speedup that a fresh machine cannot reproduce. - Serving stack: batch policy, concurrency limits, admission control settings. ## Measurement harness discipline ```python import time, statistics def measure(step, warmup=20, trials=200): for _ in range(warmup): # exclude JIT, autotuning, cache-fill effects step() xs = [] for _ in range(trials): t0 = time.perf_counter() step() # synchronize accelerator inside step() xs.append(time.perf_counter() - t0) xs.sort() return { "p50": xs[len(xs)//2], "p99": xs[int(len(xs)*0.99)], "mean": statistics.fmean(xs), "stdev": statistics.stdev(xs), "trials": trials, } ``` - Never report a single run for any latency or throughput number; state trial counts and either stdev or percentile spread in every table caption. - Report tails (p95/p99) for anything serving-shaped; means alone hide the behavior systems reviewers care about most. - Synchronize accelerators before timestamps — async launch makes GPUs look infinitely fast in naive harnesses. - Interleave A/B trials (ABABAB, not AAABBB) so thermal drift and co-tenant noise hit both systems equally. - Keep raw measurement logs; tables should be generated from logs by script, so the paper, the artifact, and reality cannot diverge. ## Disclosure floor for the paper itself - Workload identity: exact models, datasets or traces, sequence-length/request-rate distributions, and how the workload was chosen (a named benchmark family beats a bespoke workload for credibility — reviewers know the MLPerf-style conventions). - Baseline versions and their tuning budget, stated symmetrically with your system's. - Total compute consumed and, where the paper argues cost efficiency, the price basis ($/GPU-hour source and date) behind any dollar figures. - Energy numbers, if claimed, with the measurement method (whole-node meter vs. software counters) — the two can disagree wildly. - What was *not* controlled, honestly: shared cluster, single hardware family, one precision mode. A scoped claim survives re-measurement; an unscoped one does not. ## Common measurement bugs this venue catches Each of these has sunk real performance claims; check for them before a reviewer does. - **Warm-cache flattery**: benchmarking after the dataset, weights, or autotuner cache is hot, while the baseline runs cold. Symmetrize or report both regimes. - **Async mirage**: timing GPU work without device synchronization, measuring launch latency instead of execution. - **Batch-mismatch comparisons**: your system at its best batch size versus the baseline at its default — a tuning-parity violation wearing a measurement disguise. - **Coordinated omission**: measuring latency only for requests the system accepted, while it sheds load; report drop/timeout rates next to every latency figure. - **Averaging across heterogeneous workloads**: a single mean over workloads of wildly different scales lets one workload buy the headline; report per-workload numbers. - **Power-state contamination**: comparing runs taken at different GPU clock or thermal states; record clocks and lock them where the platform allows. ## Pre-submission reproducibility drill 1. Fresh-clone the repo on a machine that has never run the project; follow only the README. Every undocumented step found here is a future AE failure. 2. Regenerate the two most important tables from raw logs with one command each. 3. Diff regenerated numbers against the PDF; investigate any discrepancy beyond stated variance — this drill catches stale-table bugs that reviewers cannot, but artifact evaluators will. 4. Record the wall-clock and dollar cost of the full reproduction; put it in the appendix so others can budget. ## Cycle-volatility warning Whether MLSys requires a reproducibility checklist or statement at submission time is a per-cycle decision that could not be verified for 2026 (待核实) — check the current CFP and OpenReview form fields rather than assuming either way. Artifact-evaluation badge mechanics live in `mlsys-artifact-evaluation`. ## Output format ```text [Claim under audit] [Environment pin] [Noise controls] [Disclosure gaps] [Drill result] [Fixes] ```