--- name: rss-reproducibility description: Use when strengthening the reproducibility of an RSS (Robotics: Science and Systems) paper whose evidence lives partly on hardware — platform disclosure ledgers, log-backed trial claims, seeds and configs for the computational half, honest replication tiers, and release plans aligned with the free open-access proceedings culture. --- # RSS Reproducibility Make an embodied result checkable. Robotics reproducibility is two different problems wearing one name: the computational half (code, seeds, configs) can be rerun by anyone; the physical half (this robot, this room, this cloth) can only be *specified* well enough for a peer lab to attempt replication. Treat the halves separately and state which tier each claim sits in. ## Replication tiers — declare one per claim | Tier | What a reader can do | Minimum you must provide | |---|---|---| | Rerunnable | Reproduce the number from the release | Code, seeds, configs, data, one entry command | | Re-collectable | Regenerate equivalent data on similar hardware | Full platform ledger + protocol document | | Auditable | Verify your numbers follow from your logs | Raw trial logs + analysis scripts | | Testimonial | Trust the authors | Nothing — and reviewers price it accordingly | A paper mixing tiers is fine; a paper *implying* Rerunnable while delivering Testimonial is the credibility failure RSS reviewers punish hardest. ## The platform ledger For the physical half, disclosure is the reproducibility. A peer lab needs: - Robot make/model and any modifications; end-effector; sensor suite with mounting and calibration procedure; control frequency and stack (firmware through policy). - Environment specifics that carry load: table friction, lighting, object sourcing (purchasable items beat lab-fabricated ones), workspace dimensions. - Human procedure: reset steps, intervention rules, operator role — written as instructions, not narrative. - Timing facts that gate feasibility: wall-clock per trial, total campaign hours, battery/thermal constraints that shaped the protocol. ## Log-backed claims The strongest robotics reproducibility artifact is the raw trial ledger: ```text trial_0142 cond=gripper_B obj=towel_3 seed=1142 result=fail cause=release_timing video=clips/0142.mp4 notes=operator_intervention:none ``` - One row per attempt, appended at collection time, never edited afterward. - Analysis scripts consume the ledger and emit every table in the paper — so PDF numbers cannot drift from evidence. - Ship the ledger (anonymized) in the supplement; it converts "trust us" trial counts into auditable ones. ## What to capture during the campaign (not after) Reproducibility in robotics is mostly a *collection-time* discipline; these cannot be reconstructed later: - Per-trial ledger rows (above) appended live, including aborted attempts. - Camera footage of every trial, named by trial ID — storage is cheaper than a re-run campaign, and it doubles as `rss-supplementary` raw material. - Calibration snapshots (camera extrinsics, force-torque zeroing) at session start, so drift between sessions is diagnosable. - Software state per session: commit hash, config checksum, simulator build. - Environmental incidentals that later become reviewer questions: object wear, lighting changes, floor surface swaps. - A daily one-paragraph campaign log; six weeks later it is the only honest answer to "why does condition B have 19 trials instead of 25?" ## Computational half: standard, therefore mandatory - Seeds for every stochastic component (policy training, sampler init, domain randomization) and the variance across them, not one lucky seed. - Exact dependency manifest, simulator version and physics timestep, GPU/CPU spec, and training wall-clock. - Config files as the single source of hyperparameters; no numbers living only in the paper text. ## Release plan and the venue's open culture RSS proceedings are free to every reader at roboticsproceedings.org; a paywalled PDF with closed evidence would be off-culture. Plan the release in two states: anonymized supplement at submission (`rss-supplementary`), public repository + ledger + footage at camera-ready (`rss-camera-ready`). If something cannot be released (proprietary platform, safety), say what and why in the paper — a stated gap reads as honesty, a silent one as concealment. ## Vignette: an honest tier map A paper couples a learned grasp ranker (trained in simulation) with hardware clutter-clearing trials. Its declared map: the simulation training curves and ranker metrics are **Rerunnable** (seeds, configs, dataset, one command); the hardware success rates are **Auditable** (ledger plus analysis scripts shipped) and **Re-collectable** for labs with a comparable arm (platform ledger and protocol document provided); the wear-dependent behavior of one deformable object is **Testimonial**, and the paper says so in one sentence. Reviewers can now disagree with the evidence, but not discover its limits by surprise — which is the entire game. ## Simulator determinism caveats - Physics engines are only conditionally deterministic: thread counts, contact solver iterations, and hardware can change trajectories under the same seed. Record engine version, timestep, solver settings — and state whether exact replay or statistical equivalence is the reproduction target. - GPU nondeterminism in training means "same seed" still needs a variance report across seeds to be meaningful. ## Output format ```text [Tier map] Rerunnable/Re-collectable/Auditable/Testimonial> [Platform ledger] complete / gaps: [Trial ledger] exists+shipped / exists / absent [Computational gaps] [Release plan] camera-ready state> [Honesty debt] ```