--- name: cvpr-reproducibility description: Use when hardening a CVPR paper's reproducibility story, covering the Compute Reporting Form's hardware and compute sections, training-recipe disclosure, benchmark protocol and split hygiene, seed and variance reporting for vision experiments, and closing the gaps reviewers probe at a benchmark-driven venue. --- # CVPR Reproducibility At CVPR, reproducibility failures rarely look like fraud; they look like a table nobody can match because one augmentation flag, one crop size, or one pretraining corpus went unstated. This skill hardens the paper against that fate, anchored in the 2026-cycle machinery (checked 2026-07-08): the Compute Reporting Form, the anonymous supplement, and reviewers trained on a decade of un-reproducible state-of-the-art claims. ## The CRF as a reproducibility floor The 2026 cycle attached a Compute Reporting Form to every submission — Section 1 (hardware specification) and Section 5 (verification) mandatory, deeper sections optional, with an explicit opt-out route for proprietary constraints. Treat the mandatory floor as the start, not the ceiling: | CRF layer | What it pins down | Why reviewers care | |---|---|---| | Hardware (mandatory) | GPU model, count, primary configuration | Grounds every "real-time" and "efficient" claim | | Verification (mandatory) | Author attestation | Somebody owns the numbers | | Task/compute (optional) | GPU-hours or FLOPs per result | Separates a 4-GPU method from a 512-GPU method | | Full logs (optional) | Run-level records | The strongest possible "we actually ran this" | If your contribution *is* efficiency, filling only the mandatory sections undercuts your own claim — report the compute and let the numbers argue. ## The recipe ledger Vision results are recipe-sensitive. Maintain one machine-readable ledger from the first experiment, and generate the paper's implementation-details paragraph *from it* instead of reconstructing details in deadline week: ```yaml # recipe-ledger.yaml — one block per reported table row table3_row2: backbone: vit-b16, pretrain: data: aug: [rrc-224, hflip, randaug-m9] optim: adamw, lr: 1.0e-4, sched: cosine, epochs: 90, batch: 1024 seed: 3407 # and whether cudnn deterministic was set hardware: 8xA100-80G # must agree with CRF Section 1 command: scripts/train.sh configs/table3_row2.yaml ``` The ledger's second job is internal: when a reviewer asks in January which schedule produced Figure 5, you answer from the file in minutes. ## Benchmark hygiene that survives scrutiny - **Split discipline.** Name the exact split files; never tune on anything downstream of the test set — including "just checking" runs that leak into architecture choices. - **Test-set frequency.** State how many times the test set was evaluated. Val-set ablations + one final test run is the defensible pattern. - **Metric provenance.** Cite the metric implementation (which mAP variant? whose FID code, which feature extractor?). Same-name metrics differ across repos by whole points. - **Pretraining disclosure.** "Initialized from X" changes the comparison class; a method beating from-scratch baselines while riding a web-scale pretrain is a different claim. - **Baseline re-runs.** Say which baseline numbers are quoted from papers versus re-run under your protocol, and match backbones and schedules when you re-run. ## Variance at vision scale Full multi-seed grids are often unaffordable at modern training budgets, and reviewers know it. The credible middle ground: multi-seed the cheap decisive experiments (small backbone, headline ablation) and report mean ± std; run the flagship once but state so explicitly; never present a 0.2-point gain as a finding when the same table shows seed-level noise of 0.4. If evaluation itself is stochastic (generation, sampling-based detection), repeat evaluation, not just training. ## Determinism knobs, documented once Bit-exact reproduction is often impossible on GPU stacks, but *stating your determinism posture* is always possible and costs three lines in the supplement: - Which seeds were fixed (Python/NumPy/framework/dataloader workers) and whether deterministic kernels were enabled — note the speed cost if they were not. - Sources of accepted nondeterminism: atomics in scatter ops, multi-GPU reduction order, augmentation pipelines keyed to worker scheduling. - The reproduction tolerance you actually observed: "re-running row 2 varies by ±0.15 mAP across identical-seed runs on different node counts" tells a reproducer whether their 0.1-point discrepancy is a bug or physics. Teams that measure this once, early, stop having the "is 78.4 vs 78.6 a failure to reproduce?" argument — with reviewers and with themselves. ## Statement-level honesty Reproducibility text is a claims surface. "Code will be released" is a promise the community tracks; "results reproducible from the supplement" is checkable in January. Write the availability paragraph to match what is genuinely packaged: what ships in the supplement now, what is released at camera-ready (datasets claimed as contributions **must** be public by then — verified 2026 policy), and what cannot be released and why. ## Held-out evaluation servers Several vision benchmarks score on withheld test sets via submission servers with rate limits. This changes reproducibility mechanics: your reported test number is a server receipt, not a rerunnable command. Record the submission ID and date in the recipe ledger, respect per-week submission caps as an ethics matter (burning entries to tune on test is the community's canonical sin), and give reproducers the exact validation-set protocol that predicts the server number. ## Reverify each cycle - CRF structure, which sections are mandatory, and award tie-ins. - Any reproducibility checklist added to the OpenReview form. - Supplement caps that constrain how much recipe/code detail ships at review time (2026 caps 待核实). ## Output format ```text [Repro grade] recipe-complete / gaps found [CRF] hardware row consistent with paper claims: yes/no; optional sections: [Ledger] rows covering all reported tables: [Benchmark hygiene] splits · test-set count · metric provenance · pretrain disclosure [Variance] multi-seeded: ; single-run flagged: [Fix list] ```