---
name: analyze-prod-issue
description: >-
  Diagnose a production issue end-to-end across Honeycomb (error shape) and the
  prod DB error_tracker tables (who + why), scope impact honestly, and optionally
  file a PII-scrubbed issue. Invoke with: /analyze-prod-issue "<symptom>"
disable-model-invocation: true
---

# Analyze Prod Issue

Take a production symptom (e.g. "parents can't book") and diagnose it end-to-end by
correlating **two** sources — Honeycomb for the error *shape*, the prod DB `error_tracker`
tables for the *who + why* — then scope impact honestly and optionally file a scrubbed issue.

**Type:** Rigid workflow. Follow steps exactly.

**Prerequisite:** read-only prod DB access must be set up (`bin/prod-db`). If it errors, see
`docs/runbooks/prod-db-access.md` first.

The exact Honeycomb tool calls and prod SQL are in `references/cookbook.md` — read it when a
step says to.

---

## Step 1 — Orient (Honeycomb)

Call `mcp__honeycomb__get_workspace_context`. Note the environment slugs; production is the
`live` environment (there is also `dev`). Use `live` in every subsequent Honeycomb call.

**Done when:** you know the `live` env slug and its dataset.

## Step 2 — Find the error surface

- `mcp__honeycomb__list_spans` over `14d` — ranks span names by count with `root_count`.
  Scan for the failing area *and* for **funnel drops** (an entry span with many mounts but
  its terminal action span near zero often is the reported symptom).
- `mcp__honeycomb__run_query` with `filter error=true`, `breakdown ["name"]`, `time_range
  "14d"` — ranks the actually-failing spans by count.

**Done when:** you have candidate failing span name(s) with counts. If there are zero
`error=true` spans, the symptom is not an exception — pivot to the funnel-drop reading.

## Step 3 — Trace the causal chain

`mcp__honeycomb__get_trace` on a failing trace (`show_events: true`). Read the waterfall:
find the span where the error is *raised* and the call chain above it (HTTP entry → worker →
context function → repo query).

**Honeycomb gap to expect:** a DB error inside a rolled-back transaction surfaces only as
`query_error` / `status_message` with **no exception text** (spans carry no event). That
absence is the signal to go to the prod DB — do not stop here.

**Done when:** the failing chain and the raising span are mapped.

## Step 4 — Pull the real reason (prod DB)

Query `error_tracker` via `bin/prod-db -c "<SQL>"` (read-only reader role). See
`references/cookbook.md` for the schema and the exact queries.

- `error_tracker_errors` — `kind`, `reason`, `source_function`, `last_occurrence_at`,
  `status`. Find the error matching the trace.
- `error_tracker_occurrences` — `context` (Oban `job.args` + the full event payload:
  `event_type`, `entity_id`, user email/name), `stacktrace`, `breadcrumbs`, and
  `job.state` / `job.attempt`.

This is where the swallowed DB error, the affected user, the triggering event, and the Oban
disposition (`retryable` vs `discard`) actually live.

**Done when:** you know the underlying error, the affected entity, the triggering event, and
the occurrence count + date range.

## Step 5 — Cross-reference schema & data

Confirm the hypothesis against real rows before believing it:

- Does the row the handler tried to create **already exist**? (duplicate vs genuinely missing)
- What are the **actual** constraint/index names? (a changeset `unique_constraint` name
  mismatch makes a violation *raise* instead of returning a changeset error)
- **Affected-vs-total counts** — how many entities are in the bad state vs the population?

**Done when:** the mechanism is proven from data, not assumed.

## Step 6 — Scope impact honestly

The point of Step 5 is to *correct the premise*, not confirm it. Produce a one-paragraph
impact statement that:

- Separates **noise vs a real user-blocker** — an erroring/discarded job is not automatically
  a blocked user (the side effect may already have succeeded on another path).
- Quantifies affected users by **count**.
- **Splits disjoint phenomena** — two symptoms with the same surface can have different
  causes and different populations; do not conflate them.

**Done when:** an honest, numeric impact statement exists — even if it contradicts the
original complaint.

## Step 7 — Confirm root cause in code

Grep the implicated modules to anchor the fix. State explicitly whether the bug exists on
`main` or the behaviour is a **live-vs-main deploy lag** (live can raise what `main` already
handles). Check the current commit on `live` if it matters.

**Done when:** the fix direction names concrete files and says whether deploying `main` fixes it.

## Step 8 — (Optional) File a scrubbed issue

If the finding warrants tracking, hand off to `/create-issue`.

**Public-repo PII rule (non-negotiable):** reference affected users by **count + opaque UUID
only**. Never put names or emails in a public issue — GitHub edit history is public, so
scrubbing after the fact requires deleting and recreating the issue. Keep the
user_id ↔ identity mapping in the DB, not the issue.

**Done when:** the issue exists with count-only references, or you and the user decide not to file.

---

## Rules

- **Two sources, not one.** Honeycomb gives the *shape* (chain, span, timing); `error_tracker`
  gives the *who + why* (reason, event payload, stacktrace, Oban state). Neither alone is enough.
- **Read-only always.** Reach prod only through `bin/prod-db` (the SELECT-only `reader` role).
  Never open a write path against production.
- **Count, never names, in public artifacts.** PII stays in the DB; issues and PRs get counts
  and UUIDs.
- **Scope honestly.** "Erroring" ≠ "user-blocked." Prove impact from data before claiming it,
  and correct the premise when the data disagrees.
- **The rollback hides the error.** When Honeycomb shows only `:rollback` / `query_error` with
  no text, the real reason is in `error_tracker_occurrences.context`, not the span.