--- name: multiverse-analysis description: Use this skill after confound-audit and before any conclusion is drawn from the analysis. Required for medium and high-stakes analyses; recommended always. Addresses the garden-of-forking-paths problem — every analytical choice (segmentation, outlier rule, time window, transformation, model) could have plausibly gone differently, and a result that holds on only one path is fragile. Logs every choice with rationale and alternatives considered, runs alternative paths where feasible, and flags fragile results that don't hold up across the multiverse. Trigger any time conclusions are being drawn from a single analytical path. --- # Multiverse Analysis Andrew Gelman's garden of forking paths: even without conscious p-hacking, an analyst makes many small choices that could plausibly have gone the other way. The result you report is one path through a forest of possible analyses. Standard inference assumes a single pre-specified path; without that, your nominal confidence overstates the truth. For LLM-driven analytics this is worse, not better. LLMs explore many paths fast and tend to surface the one with the cleanest narrative. ## What this skill catches - **Path-laundering.** A choice that was plausibly arbitrary at the time gets presented in the writeup as obvious or pre-specified, hiding the alternative paths that would have produced different numbers. - **Fragile results.** A finding that holds only under the chosen segmentation or window, and would attenuate or flip under equally defensible alternatives — but is reported without that fragility flagged. - **Untracked post-hoc choices.** Decisions made after seeing data that look pre-specified by the time the writeup is drafted, because no one logged the timeline. - **The cleanest-narrative bias** (LLM-specific). The path that produces the most communicable story tends to be the one that gets surfaced — even when other paths would have produced an equally valid, less clean answer. ## What this skill enforces ### Path logging Every analytical choice gets logged. For each choice: - **What was chosen** — the decision and the value - **Alternatives considered** — what else was on the table - **Rationale** — why this choice - **Pre-specified vs. post-hoc** — was this in the analytical plan, or did it emerge during analysis? Choice types worth logging: population filters, time-window boundaries, outlier rules, segment definitions, missing-data treatment, transformation choices, model specifications, comparison baselines, threshold cutoffs. ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "" \ --chosen "" \ --alternatives "" "" \ --rationale "" \ --pre-specified # include this flag if the choice was in the plan; omit if post-hoc ``` ### Multiverse runs For each meaningfully ambiguous choice, run the analysis under at least one alternative and report both: - **Time window** — try a different reasonable window, compare results - **Outlier rule** — with vs. without, compare - **Segment definition** — tight vs. loose, compare - **Missing-data treatment** — complete-case vs. imputed, compare - **Threshold cutoff** — at the cutoff vs. ±1 step, compare You don't need to run the full Cartesian product. Run the alternatives that are individually plausible, focusing on the choices most likely to swing the result. When you do, mark the run on the path decision: ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "<...>" \ --chosen "" \ --rationale "multiverse run; result moved from X to Y" \ --multiverse-run ``` ### Fragility flag A result is **robust** if it holds (in direction and roughly in magnitude) across the multiverse runs. A result is **fragile** if it holds on the chosen path but flips or attenuates significantly on plausible alternatives. Fragile results get an explicit fragility flag in the writeup; they should not be presented as findings without it. ### Post-hoc choices Choices made after seeing data (post-hoc) get tagged in the path log and explicitly disclosed in the writeup. They are not forbidden — sometimes you find something genuinely worth following — but they are subject to a higher fragility threshold and cannot be presented as confirmatory. ## Logging Log this skill's activation: ```bash python3 .analytics/db.py log-skill \ --skill multiverse-analysis \ --question-slug \ --completed ``` ## Boundary cases **"All my choices were pre-specified in the plan."** Then `log-path --pre-specified` for each. Multiverse-run any that are individually arbitrary anyway — pre-specification protects the analysis from confirmation drift, but doesn't make the choice itself less arbitrary on its own merits. **"Running every alternative is too expensive."** Run the alternatives most likely to swing the result. The point is not exhaustiveness; the point is to surface fragility, and fragility usually shows up on the obvious axes (window, outlier rule, segment definition). **"The user wants a single number; the multiverse gives a range."** Report both. The single number is the headline; the range is the honesty. A 12% lift that ranges 4-15% across the multiverse is a different finding than a 12% lift that ranges 11-13% — both are real, but they call for different decisions. ## Handoff After multiverse runs and the fragility assessment is complete, hand off to `sensitivity-analysis/SKILL.md`.