--- name: predict-then-query description: KEYSTONE skill for the analytics-skills methodology. MUST trigger EVERY TIME a headline analytical query is about to be run — that is, the query whose result will appear in the writeup or inform the decision. This skill REFUSES to run the headline query until a written prediction (point estimate, range, reasoning, confidence, surprise condition) has been logged to the methodology DB via the CLI. After the query runs, this skill compares predicted vs. actual; if actual is OUT_OF_RANGE, it forces a stop-and-investigate before the result is trusted. Catches the LLM-native failure mode of hallucinated-but-plausible numbers, calibrates analyst priors over time, and creates an auditable record. Trigger any time SQL is about to be written, a headline metric is about to be computed, or "let's run the numbers" is in scope. --- # Predict Then Query This is the keystone of the methodology. The analytics analog of test-driven development's signature move: *specify the expected behavior before implementation*. ## CRITICAL: No silent skip If the data is right there and the query is easy to write, the temptation to skip prediction is highest. Resist it. This skill exists *because* the easy path produces hallucinated numbers and confirmation drift. The prediction step is the only thing that catches LLM-native failure modes that no other skill catches: 1. **Hallucinated numbers** — plausible-sounding stats emitted without an actual query, or with a query that has bugs producing reasonable-looking output. 2. **Confirmation drift** — gravitating toward results that match what the user (or your prior) seemed to expect. If you find yourself about to write SQL or read a query result without a logged prediction first: **STOP.** Go back. Write the prediction. Log it. Then run the query. ## Procedure ### 1. Write the prediction Five elements, all required: - **Point estimate** — your best single-number guess. Do not refuse to commit. - **Range** — low and high you'd be surprised to fall outside. Honest, not too wide (useless) or too narrow (always failing). - **Reasoning** — 2-4 sentences on why these numbers. Reference base rates, prior similar analyses, domain knowledge. - **Confidence** — 0 to 1, your subjective confidence that actual will be in range. - **Surprise condition** — one sentence on the result that would most update your model. ### 2. Log the prediction BEFORE the query runs ```bash python3 .analytics/db.py log-prediction \ --question-slug \ --predicted \ --range-low \ --range-high \ --reasoning "<2-4 sentences>" \ --confidence <0..1> \ --surprise "" ``` The CLI returns a JSON object including `prediction_id`. Save it. ### 3. Run the headline query Now (and only now) run the SQL. ### 4. Log the actual result ```bash python3 .analytics/db.py log-actual \ --prediction-id \ --actual \ --notes "" ``` The CLI returns: - `delta` — actual minus predicted - `in_range` — boolean - `verdict` — either `"in_range"` or `"OUT_OF_RANGE — STOP"` ### 5. Honor the verdict - **`in_range`**: Proceed. Trust calibrated. Hand off to next skill. - **`OUT_OF_RANGE — STOP`**: Do NOT interpret the result. Before drilling into query-bug-vs-model-update, run a one-pass context check — the surprise may already be explained by context the analyst didn't have at framing time. 1. **Re-scan adjacent context.** Run `python3 .analytics/db.py adjacent-questions --slug ` again. The analysis may have drifted into new domain space since framing, and a sibling question's findings — an intervention, a definitional change, a parallel result — may directly explain this surprise. Review any new hits. 2. **Ask the user three concrete questions.** Do not paraphrase, do not soften, do not assume the answer is "no": - *"Is there an active or recent intervention in this window I didn't factor in?"* - *"Is there a parallel analysis whose findings could redirect this one?"* - *"Has a data-pipeline or definitional change landed since the last time we measured this?"* The point is not to give the user an opening to rationalize the surprise — it is to surface load-bearing context the framework couldn't have reached on its own. 3. **Log the response as a path-decision regardless of whether new context surfaces.** The audit trail records that the check was performed: ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "out-of-range-context-check" \ --chosen "" \ --rationale "Triggered by prediction # OUT_OF_RANGE: predicted

, actual " ``` If the user surfaced a relevant confound, ALSO log it to the confound library via `log-confound` and update the question doc's "Confounds known up front." Then — and only then — proceed to query-bug-vs-model-update analysis. Two possibilities, in this order: 1. **Query bug** — re-examine joins, filters, grain, NULL handling, type coercion. The query is guilty until proven innocent. 2. **Model update** — your prior was wrong; this is the most informative outcome. Document what specifically updated. Do NOT collapse to (2) before ruling out (1). Re-run the prediction-and-query cycle with a fresh prediction reflecting what you've learned. **Why this gate exists.** OUT_OF_RANGE means the analyst's prior diverged from reality. The most common cause in real-world analytics is not a query bug and not a wrong model — it is a *missing piece of context* that, once surfaced, makes the actual result obvious in hindsight. The context check is cheap (one CLI call + three questions), fires only when something has already paused the flow, and produces an audit-trail entry whether or not it finds anything. ### 6. Log this skill's activation ```bash python3 .analytics/db.py log-skill \ --skill predict-then-query \ --question-slug \ --completed ``` ## Boundary cases **"I have no prior."** You always have *some* prior. If genuinely uncertain, set a wide range and low confidence (e.g., 0.1–0.2). Logging "no prior, very low confidence" is itself useful — it identifies analyses where you were flying blind. **"This query is exploratory, not headline."** Exploratory queries (data inventory, distribution checks, sanity counts) are not headline queries — they don't trigger this skill. The skill triggers when the query result will appear in the writeup or inform the decision. If unsure, ask: would the user act on this number? If yes, predict first. **"The user is watching me work in real-time."** That's the point. The prediction is *for* both of you. It makes your prior visible — exactly the transparency that builds trust. Encourage the user to predict alongside you. **"My predicted range is so wide that of course actual fell inside."** Yes, this is gameable — wide ranges always pass. The methodology DB tracks range width and confidence over time; persistent very-wide ranges with low confidence flag the analyst as miscalibrated, not rigorous. Calibration is the long-game metric. ## What the calibration data eventually gives you After enough resolved predictions, run: ```bash python3 .analytics/db.py calibration --min 5 ``` This returns hit-rate-by-confidence-bin. An analyst who has been doing this for months gets: - **Hit rate by confidence** — does your "0.8 confidence" actually pan out 80% of the time? - **Sharpness over time** — is your range tightening as domain knowledge accumulates? - **Bias direction** — systematic over- or under-prediction in certain domains - **Most informative misses** — predictions that failed by the largest delta This is the long-term payoff. An analyst (or team) running this for six months has *evidence* about their own intuitions. ## Handoff After predict-then-query completes: - If an authoritative version of this number exists somewhere → `reconciling-to-source/SKILL.md` - Otherwise → `confound-audit/SKILL.md`