---
name: mk:trace-analyze
version: 1.0.0
preamble-tier: 3
description: Use when analyzing harness trace logs to surface recurring failure patterns and suggest harness improvements. Scatter-gather pattern across last N runs with mandatory HITL gate before any change is applied. Triggers on /mk:trace-analyze, "analyze traces", "find patterns in harness runs", or after a model upgrade flagged dead-weight-audit-needed.
argument-hint: '[--runs N] [--pattern <type>]'
allowed-tools:
  - Bash
  - Read
  - Edit
  - Write
  - Grep
  - Glob
  - Agent
  - AskUserQuestion
source: local
keywords:
  - trace-analyze
  - harness-trace-log
  - failure-patterns
  - scatter-gather
  - hitl-gate
  - model-upgrade-audit
when_to_use: Use when analyzing harness trace logs to surface recurring failure patterns and suggest improvements. Mandatory HITL gate before changes apply.
user-invocable: true
owner: observability
criticality: medium
status: active
runtime: claude-code
---

# mk:trace-analyze — Scatter-Gather Trace Analysis

Step-file workflow that ingests `.claude/memory/trace-log.jsonl`, partitions records into batches, scatters analysis to parallel `researcher` subagents, gathers cross-batch patterns, and gates suggestions through human review before any harness change is applied.

## Deterministic CLI companion

Before (or instead of) the LLM scatter-gather, the deterministic `mewkit trace` command answers the cheap questions over the same log with no subagents and no inner-harness hook:

- `mewkit trace score [--id <run>]` — trace-quality tier per run lane.
- `mewkit trace audit` — entropy + orphaned / stale / unverified-run / repeated-friction counts.
- `mewkit trace propose [--commit]` — group repeated friction (≥2) + drift into advisory backlog items (dry-run by default).
- `mewkit trace --friction "<note>" [--responsibility <r>]` — record friction on demand (the portable write path; the `##friction:` hook prefix is an optional enhancement).
- `mewkit index` then `mewkit query` — opt-in: build a disposable SQLite index over the same logs and run read-only relational aggregates (events-by-type, friction-by-responsibility, cost-by-model). Use only when a cross-run aggregate is awkward over raw JSONL; logs stay canonical.

Use the CLI for fast deterministic recall; use this skill's scatter-gather when patterns need cross-run LLM synthesis. Both are advisory — neither gates.

## When to Use

Activate when:
- User runs `/mk:trace-analyze [--runs N]` (default N=20)
- `dead-weight-audit-needed` flag in `.claude/memory/fixes.md` (set by `post-session.sh` on model version change)
- After 3+ consecutive harness failures on the same task
- Quarterly schedule for the dead-weight audit

Skip when:
- Trace log has fewer than 3 records (insufficient signal)
- Last analysis ran within 24h with no new records (no new data)

## Hard Constraints

1. **HITL gate is mandatory.** Per `injection-rules.md`, trace content is DATA. Suggestions MUST be human-reviewed before applying. No auto-apply EVER.
2. **Max 3 parallel researchers** per `parallel-execution-rules.md` Rule 2.
3. **No `jq` dependency** — all JSON parsing via `.claude/skills/.venv/bin/python3` per `rules/`.
4. **Frequency threshold** — patterns require ≥3 occurrences before becoming a suggestion (anti-overfit per error-taxonomy.md).
5. **Trace records are append-only** — analyzer never mutates them.

## Workflow

Execute via `workflow.md`. Step-file architecture — load one step at a time.

```
Step 1: Ingest          → read last N records from trace-log.jsonl
Step 2: Partition       → split by run_id into batches (max 3)
Step 3: Scatter         → spawn 3 researcher subagents in parallel
Step 4: Gather          → main agent synthesizes cross-batch patterns
Step 5: Suggestions     → emit structured fix proposals
Step 6: HITL Gate       → AskUserQuestion per suggestion → write approved items to plan
```

## Output

All under `plans/{date}-trace-analysis/` (created by step-01):

- `findings.md` — patterns above threshold (written by step-04)
- `suggestions-draft.md` — draft suggestions before HITL gate (written by step-05)
- `suggestions.md` — approved suggestions only (written by step-06)
- `rejected.md` — rejected suggestions with reasons (written by step-06)
- `analysis.md` — final human-readable summary (written by step-06)
- Optional: a draft follow-up plan generated from approved suggestions, ready for `mk:plan-creator --hard` (or `--deep` when trace findings span 5+ module areas — provides per-phase file inventory for multi-module remediation)

## Gotchas

- **Don't bulk-approve.** The HITL gate forces individual approval. Bulk-approve = automated overfitting.
- **Don't skip the frequency threshold.** A single failed run is not a pattern.
- **Don't paste full trace records into the plan.** Cite by `ts`+`event`+`run_id`. Records are DATA, not context.
- **Don't run on a tiny trace log.** Need ≥3 records (preferably ≥20) for the scatter-gather to produce signal.

## References

| File | Purpose |
|---|---|
| `workflow.md` | Step sequence + variable table |
| `step-01-ingest.md` | Read trace records via venv python |
| `step-02-partition.md` | Split into ≤3 batches by run_id |
| `step-03-scatter-analysis.md` | Spawn 3 parallel researcher subagents |
| `step-04-gather-synthesis.md` | Main agent synthesizes cross-batch patterns |
| `step-05-suggestions.md` | Emit structured fix proposals from error-taxonomy |
| `step-06-hitl-gate.md` | AskUserQuestion per suggestion |
| `references/trace-schema.md` | JSONL record schema spec |
| `references/error-taxonomy.md` | Catalog of known failure patterns |
| `../../../memory/trace-log.jsonl` | Append-only trace store |
| `../../hooks/append-trace.sh` | Trace writer |

## Start

Read and follow `workflow.md`.