--- name: analytical-self-review description: Use this skill after multiverse and sensitivity analysis are complete, before producing any writeup or summary. Final pass against known analytical pitfalls — Simpson's paradox, base-rate fallacy, leakage, p-hacking, ecological fallacy, survivorship bias, falsification. The "what would change my mind" statement from framing-the-question is now compared against the actual results. Required before any analytical output is presented as a finding. Trigger before any conclusion, writeup, dashboard, or summary is delivered. --- # Analytical Self-Review The last gate before findings reach a decision-maker. This is the pass that catches pitfalls the earlier skills don't. ## What this skill catches - **Aggregation traps.** A directionally-correct overall result that reverses when broken out by an obvious subgroup (Simpson's paradox). - **Context-stripped percentages.** "Up 50%" without the base rate, where 0.2% → 0.3% looks identical to 20% → 30%. - **Outcome leakage.** Predictor information sourced from the outcome window, common when cohort membership is defined using post-treatment events. - **Question drift.** The analysis answered a question close to but not the same as the one originally framed, and the writeup is being drafted as if it answered the original. - **Falsifying conditions that triggered and got ignored.** "What would change my mind" was written down at framing time; the actual results meet a falsifying condition; the writeup is drafting the conclusion anyway. ## The checklist ### Simpson's paradox Does the aggregate result reverse when broken out by a relevant subgroup? Always check at least one obvious segmentation (cohort tenure, customer segment, geography, time period, prior level of activity). ### Base-rate fallacy Are you reporting a relative or absolute change without context for the underlying base rate? "Conversion up 50%" from 0.2% to 0.3% is different from 20% to 30%. ### Leakage Did information from the outcome window contaminate the predictor window? Most common in cohort analyses where membership is defined using post-treatment data. ### Survivorship bias Are you comparing units that stayed to units that churned? Are dropped accounts represented? ### Ecological fallacy Are you inferring individual behavior from aggregate data, or vice versa? ### P-hacking / forking paths Has the multiverse analysis flagged this as fragile? If so, the writeup must lead with the fragility, not bury it. ### Compare against the falsifying conditions Pull up the "what would change my mind" statement from the question doc. Walk through each falsifying condition. Did any of them trigger? If yes, the conclusion needs to update. ### Falsification queries For the headline claim, write and run at least one query that would *disprove* the claim if it were false. (E.g., for "the new feature drives more conversions": run a query that would show *no* conversion difference if the feature were inert. Did it return what the conclusion predicts?) ## The drift check Re-read the original question and the analytical plan. Has the analysis answered the question that was framed, or has the question silently changed during the work? If the question changed, that is allowed — but the writeup must say so explicitly. ("During analysis, we expanded scope from new-customers-acquired-in-May to all customers active in May, because [reason].") Silent question-drift is one of the most common ways analyses end up answering the wrong thing. ## Logging Log this skill's activation: ```bash python3 .analytics/db.py log-skill \ --skill analytical-self-review \ --question-slug \ --completed ``` If the drift check, Simpson check, or falsification query surfaces a finding that changes the conclusion, log it as a path decision so the writeup reflects it: ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "self-review-finding" \ --chosen "" \ --rationale "" \ --fragility-note "conclusion updated post-results" ``` ## Boundary cases **"All checks pass cleanly."** Possible. But before signing off, ask: did I work through each check, or did I scan the list and self-assert? Run at least one falsification query — if you skipped it, the review wasn't done. **"A check failed but I have an explanation."** Document the explanation. The conclusion may still hold, but the writeup names the check that failed and the explanation that resolved it. "Simpson reverses, but the subgroup driving the reversal is too small to matter" is fine; "Simpson reverses but I didn't think it was important" is not. **"The analysis is small / ad-hoc."** Light review still applies — at minimum, run the drift check and the falsifying-condition comparison. Those are the two most common ways even small analyses go wrong. ## Handoff After self-review passes (with any updates logged), hand off to `evidence-over-claims/SKILL.md`.