--- name: framing-the-question description: Use this skill at the start of EVERY rigorous analytical request — when the user has asked for a metric, comparison, lift estimate, segmentation, dashboard, report, A/B read, attribution claim, or any number-producing analysis AND has not explicitly opted into ad-hoc mode. This skill REFUSES to proceed to data discovery or querying until the question is operationalized (specific, measurable), the decision the answer informs is named, "what would change my mind" is written down, and stakes are flagged. The question doc is logged to the methodology DB. Trigger on phrases like "how is X doing", "is Y working", "compare A and B", "what's the impact of", "did we hit", "is there a difference", "show me", or any analytical framing — even casual ones. Most analytical requests arrive vague; this skill makes them sharp before any data is touched. --- # Framing the Question The most common cause of wrong analysis is correct execution of the wrong question. This skill exists to prevent that. ## CRITICAL: No back door Earlier versions of this skill had an "if the user resists" branch that allowed bypass. **That branch has been removed in v0.2 because it was being taken without user resistance.** The rules: - **You may not unilaterally classify a request as ad-hoc.** That decision is made by `getting-started` based on explicit user signal, before this skill activates. - **By the time you reach this skill, the request has been classified as rigorous.** That classification is binding. You do not re-classify mid-skill. - **If you find yourself drafting a "this is ad-hoc, no predictions logged, no audit" disclaimer:** STOP. That disclaimer is a smell that you are about to bypass the methodology. The correct response is to engage with this skill's procedure, not to release-valve out of it. - **If the user pushes back during framing** ("just give me the number"), do NOT cave silently. Surface the choice: *"Right now we're in rigorous mode. If you'd rather an ad-hoc directional answer, say so explicitly and I'll switch — otherwise I need to keep sharpening the question."* Let the user make the choice; never make it for them. ## The four tests Before any data discovery or query, the question must satisfy four tests. Work each one with the user. One missing element = one question. ### Test 1: Operationalized The question names a specific, measurable construct. The operationalized version names: - **Population** (which users, sessions, accounts, transactions — name the unit of analysis) - **Time window** (start, end, calendar boundaries, timezone) - **Metric** (exact construct — "engagement" is not specific; "% of weekly active users who completed onboarding step 3" is) - **Comparison** (what vs. what) - **Threshold for "meaningful"** (effect size, statistical, or business-significance) If any of these is missing, ask. One question per missing element. ### Test 2: Decision-anchored Name the decision the answer will inform. "I'm curious" is not a decision. If no decision is named, ask: *"What decision will this answer inform? That helps calibrate rigor."* ### Test 3: Falsifiable — "what would change my mind?" Before running anything, you and the user write down what evidence would change the conclusion. This is the most uncomfortable step. It is also the most important — it defends against motivated reasoning. Examples: - *"If lift is <2%, I'd conclude the program isn't moving the needle."* - *"If lift is >10% but driven by 3 outliers, I'd treat it as fragile."* - *"If pre-period baseline is missing for >20% of cohort, I can't answer this question."* This statement gets logged. `analytical-self-review` checks results against it later. ### Test 4: Stakes-flagged Tag as one of: - **low** — curiosity, internal directional check; light methodology - **medium** — departmental decision, recurring report; full methodology - **high** — board, regulatory, vendor renewal, large capital; full methodology + reconciling-to-source mandatory + multiverse-analysis mandatory If user doesn't pick, default to **medium** and say so. ## Adjacent context check After the four tests pass and before locking the question doc, scan for in-flight or recent questions that touch the same domain, population, or window. The failure mode this prevents: launching an analysis whose surprising result is already explained by a sibling question's findings that nobody surfaced into framing. The missing context is often sitting in `.analytics/questions/` the whole time — the framework just didn't reach for it. Procedure: 1. Run the adjacency CLI: ```bash python3 .analytics/db.py adjacent-questions --slug ``` Scoring combines slug-token overlap, decision-keyword overlap, recency, and in-flight status. Default `--min-score 1.0` filters noise; raise it if too many hits, lower it if you suspect a relevant question is just under the threshold. 2. For each returned slug, open the question doc (`.analytics/questions/.md`) and check: does it name a confound, intervention, finding, or constraint that affects this question's population, window, or metric? 3. **For each adjacent question whose findings are load-bearing for this one**, add an entry to "Confounds known up front" citing the adjacent slug. The cited entry should name the specific mechanism (e.g., *"Concurrent intervention: the discount campaign documented in `` overlaps this question's window and is known to inflate the headline metric — must control for or exclude"*). 4. **For each adjacent question reviewed and judged not relevant**, log a path-decision so the audit trail shows the check was performed: ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "context-check-dismissed" \ --chosen "Not relevant to this framing" \ --alternatives "" \ --rationale "" ``` 5. **If `adjacent-questions` returns zero results**, log one path-decision noting the check ran and returned empty: ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "context-check-dismissed" \ --chosen "No adjacent questions surfaced" \ --rationale "Adjacency CLI returned no hits at default min-score" ``` The audit trail records the check itself, not just non-empty results. Skipping this step is what the skill exists to refuse. This check is **required** at every framing — the friction is what closes the gap. ## When framing depends on data you haven't inspected yet Some elements of the four tests can't be fully settled without seeing the data — the population definition might require knowing the table's grain, the time window might require knowing freshness, and the metric formula might require knowing whether a vendor source is pre-aggregated. This is normal. It is not a reason to skip framing. What to do: 1. **Lock in a defensible default working assumption** — the most-charitable, most-likely-correct version given what you know now. *"Population: users who held Gold tier on the campaign-start date (snapshot rule)"* is a working assumption; *"Population: Gold-tier users"* is not. 2. **Flag the dependency explicitly in the question doc.** Mark the field as pending: `Population: Gold-on-2026-04-13 — pending data-discovery confirmation that tier history is available at daily grain`. The flag is what licenses you to proceed; vague framing that hides behind "we'll figure it out in discovery" is what this skill exists to refuse. 3. **Proceed to data-discovery anyway.** Discovery is a legitimate place to learn that an assumption was wrong. The skill order is `framing → discovery`, but the *flow* allows back-edges — see "Re-framing after discovery" below. The bar for Test 1 is not "I have certainty"; it's "I have a defensible written assumption with every dependency named." A pending-flag without a default working assumption is not framing; it's a stub. ## When the data is (or might be) pre-aggregated A specific failure mode that's bigger than ordinary "I don't know yet" uncertainty: real-world warehouses often expose a metric only in **pre-aggregated cube, snapshot, or vendor-report form** — with a baked-in baseline, a baked-in window length, a baked-in cohort definition, or a baked-in segmentation. When the only available source is pre-aggregated, the framing may be asking for a number that *was never computable from the data the analyst can reach.* Examples: - `weekly_comparison_snapshot` exposes `current_period_count` (1-week distinct) and `baseline_count` (4-week distinct). Framing that assumed event-level grain and a custom YoY comparison is unreachable from this table — the multi-window plan was never available. - A vendor "marketing performance" report rolls up to channel-week with revenue already attributed. Framing that wanted user-level attribution under a different model can't be answered from this source. What to do at framing time: 1. **Ask up front whether the metric exists in cube / vendor-report / snapshot form.** Treat aggregated sources as the *most likely* shape of available data, not the exception. If you don't know, flag it as a dependency (per "When framing depends on data") AND name the specific risk. 2. **Flag any framing decision that requires raw or lower-grain data than likely exists.** A `Metric` line that needs visit-level joins, custom cohorts, or windows that don't match the cube's baked-in window is a load-bearing assumption — write it as one. 3. **Acknowledge that discovery's findings can force re-framing.** If discovery confirms only the aggregated form exists, the resolution may be to (a) accept the cube's baked-in baseline and re-frame the comparison around it, (b) seek a different source at lower grain (often via a different system entirely — different DB, different vendor portal, raw event stream), or (c) downgrade the question's specificity. All three are legitimate; silently substituting the cube's metric for the framed metric is not. The phrase to internalize: *"the choice was never available."* That's different from *"I haven't decided yet."* Pre-aggregation removes choices; framing has to either confirm the choice is still on the table or commit, in writing, to a fallback. ## Re-framing after discovery Discovery exists to surface schema, grain, freshness, hygiene, and pre-aggregation realities — and any of those can invalidate a framing choice. When that happens, **return to framing**. Do not silently mutate the analytical plan around the new constraint; the audit trail breaks and downstream skills can't tell which version of the question they're answering. Procedure: 1. Open `.analytics/questions/YYYY-MM-DD-slug.md` and append a `## Revisions` entry: a dated bullet stating what changed, why, and which discovery finding forced it. Do not delete the original framing — keep both visible. 2. Update the affected fields in place (`Population`, `Metric`, `Time window`, `Threshold`, etc.) so downstream skills read the current version. 3. Log the re-frame as a path-decision (post-hoc): ```bash python3 .analytics/db.py log-path \ --question-slug \ --decision-type "re-frame" \ --chosen "" \ --alternatives "" \ --rationale "" ``` 4. Re-run `python3 .analytics/db.py show-question --slug ` to confirm the DB and doc agree. If the revision is large enough that the question is meaningfully different — different decision, different stakes, different unit of analysis — create a new question doc with a new slug and link back to the original in its Revisions entry. Don't retrofit one slug across two distinct questions. ## Output When all four tests pass, save the question doc to `.analytics/questions/YYYY-MM-DD-slug.md`: ```markdown # Question: ## Decision: ## Stakes: ## What would change my mind: - - ## Population: ## Time window: ## Metric: ## Comparison: ## Threshold for meaningful: ## Confounds known up front: - ## Created: ## Revisions: ``` Then log to the methodology DB via the CLI: ```bash python3 .analytics/db.py log-question \ --slug \ --stakes \ --decision "" \ --falsifying "" "" ``` The CLI returns a JSON object including `question_id` — save that for downstream skills. Then log this skill's activation: ```bash python3 .analytics/db.py log-skill \ --skill framing-the-question \ --question-slug \ --completed ``` ## Verify before handoff Before handing off, run: ```bash python3 .analytics/db.py show-question --slug ``` If it returns the question, you're good. If it returns "Not found", the log failed — debug before proceeding. ## Handoff State explicitly to the user: *"Question framed and logged. Slug: ``. Moving to data discovery."* Then read `data-discovery/SKILL.md`. ## Examples **Vague:** "How is the new onboarding flow doing this month?" **Sharp:** Question: For users who started signup in May 2026 (May 1–31, UTC), is 7-day activation rate at least 3 percentage points higher than the same cohort metric in Feb 2026? Decision: Whether to roll the new flow to 100% in Q3. Stakes: medium. What would change my mind: <1pp lift = not working; >3pp but coincident with an unrelated bug-fix that landed mid-month = confounded; baseline data missing for >20% of cohort = different design needed. **Vague:** "Is the referral promo working?" **Sharp:** Question: For users who received the May referral incentive email, is 30-day repeat-purchase rate at least 5 percentage points higher than their own 90-day pre-incentive baseline? Decision: Whether to repeat the campaign in July. Stakes: low. What would change my mind: <2pp = not worth repeating; lift driven entirely by purchase frequency rather than per-order spend = different mechanism; >5 outliers dominating = fragile.