--- name: rss-experiments description: Use when designing or auditing the experimental campaign for an RSS (Robotics: Science and Systems) paper — hypothesis-shaped robot experiments, trial protocols and per-condition counts, mechanism-isolating ablations, simulation-versus-hardware evidence splits, and failure attribution that supports a scientific claim rather than a demo reel. --- # RSS Experiments Design experiments that *test the paper's claim*, not experiments that showcase the system. At RSS the evaluation section is where the scientific claim either becomes falsifiable or is exposed as marketing. ## Start from the claim, derive the conditions Write the claim, then derive what evidence its logical form demands: | Claim form | Evidence the form demands | |---|---| | "X causes the improvement" | Ablation removing only X; everything else frozen | | "the gain transfers" | Held-out tasks/objects/platforms named *before* running | | "method M has property P" | The condition where P would break, deliberately tested | | "T is the bottleneck" | Failure attribution showing T dominates the failure mass | | "faster/cheaper at equal quality" | Matched-quality comparison, not best-vs-default | An experiment matrix that cannot be traced back to a claim row is footage, not evidence — cut it or move it to the supplement. ## Hardware trial protocol - **Pre-register internally:** success criterion, reset procedure, object/task distribution, and stopping rule written down before the campaign. Post-hoc criteria are how demo bias enters honest labs. - **Per-condition counts in every table.** "n = 25 per object, 6 objects" is auditable; "extensive trials" is not. - **Report the denominator.** All attempts count — aborted runs, resets, operator interventions. A human silently rescuing the robot between trials is part of the system and must appear in the protocol. - **Small-n honesty:** hardware budgets cap trials, so use interval estimates suited to small samples and let the language match their width. Twenty trials support "in our setting"; they do not support "reliably in general." ## Simulation and hardware: split the ledger - Label every number sim or real; never average across the boundary. - Simulation earns its place by *scale* (sweeps, distributions) or *danger* (failure regimes hardware cannot safely visit); hardware earns the claim's headline. - If the claim mentions the physical world, at least one decisive result must be physical. A sim-only paper must scope its claim to simulation explicitly. - When sim and real disagree, that gap is a finding — report it, do not tune it away silently. ## Failure attribution, the RSS signature move A distribution over failure causes is often more scientifically valuable than the success rate above it. Build one: 1. Define 4-6 mutually exclusive failure categories before the campaign (perception, planning, timing, slip, hardware fault, other). 2. Label every failed trial at collection time, from logs — retrospective labeling from memory drifts optimistic. 3. Report the distribution per condition; a method that shifts failure mass between categories is telling you *how* it works. ## Trial-budget arithmetic (do this before the campaign) Hardware time is the binding constraint, so budget it explicitly: ```text conditions: 2 methods x 3 environments x (1 ablation + 1 transfer) = 10 trials/condition: 25 -> 250 attempts minutes/attempt (incl. reset): 6 -> 25 robot-hours overhead (faults, recalibration, re-runs): x1.5 -> ~38 robot-hours robot access: 4 h/day -> ~10 working days, before any surprise ``` If the arithmetic says the matrix cannot finish before the January freeze, shrink the *matrix*, not the per-condition counts — fewer conditions with auditable n beat a full grid of anecdotes. Reserve the final week for zero data collection. ## Reporting floor Non-negotiables for the results section, independent of subfield: - Trials per condition and total attempts, including discards, with discard rules. - Success criterion stated operationally (what sensor reading / judge decides). - Dispersion for every stochastic number — interval, quartiles, or all raw points when n is small; never a bare mean. - Failure distribution table matched to the labeled categories. - Wall-clock and compute for learned components; trial duration for hardware. - Which numbers are sim and which are real, in the table itself, not the caption. ## Baseline fairness - Give baselines the same tuning budget, sensor stream, and reset quality as the proposed method, and say so in one sentence. - Prefer the strongest *published* configuration over a reimplementation you cannot make fast; if reimplementing, report your version's score on a setting where the original published numbers exist. ## Vignette: turning a demo matrix into a claim test A draft evaluates a new whole-body controller on eight household tasks and reports success rates — a demo matrix. The claim, once written down, is "torque-limit awareness, not trajectory optimality, drives the reliability gain." The redesign: (1) an ablation running the same controller with torque-awareness disabled, same eight tasks; (2) a stress condition where torque limits bind hard (heavy objects) versus one where they never bind — the claim predicts the gain concentrates in the first; (3) failure attribution distinguishing torque-saturation failures from tracking failures. Three conditions replaced five decorative tasks, and the paper gained a falsifiable spine without new hardware. ## Output format ```text [Claim -> condition map] experiment> [Protocol status] pre-registered / drifting / post-hoc [Counts] [Sim/real ledger] clean split / mixed (fix) [Failure attribution] [Decisive missing run] ```