--- name: scorpio-postmortem description: Analyze a failed or degraded pipeline run to produce an evidence-backed diagnosis with root causes, affected sessions, and proposed remediation tasks. Use after a run fails, times out, or produces unexpected results and you need structured failure analysis. --- # Postmortem A structured methodology for diagnosing pipeline run failures. This skill reads run artifacts, identifies root causes with evidence, and generates remediation tasks compatible with **scorpio-execute**. ## When to Use - After a pipeline run fails or completes with errors - When `scorpio doctor` needs deeper analysis than the built-in analyzer provides - When multiple failures share a common root cause that needs investigation - When the operator wants a human-readable explanation of what went wrong ## Prerequisites You need access to a completed (or crashed) run directory: ``` .scorpio/runs// ├── state.json # Run state and phase tracking ├── PIPELINE-PLAN.json # Original plan ├── results/ # Session result JSON files │ └── session-NNN.json ├── logs/ # Pipeline and session logs │ ├── pipeline.log │ ├── session-NNN-stdout.log │ └── session-NNN-timeout.json # Timeout sidecar (if applicable) ├── sessions/ # Session metadata ├── journal/ # Event journal entries ├── postmortem.json # Existing postmortem (if already generated) └── postmortem.md # Existing markdown summary ``` If `postmortem.json` already exists from the built-in analyzer (`scripts/postmortem-analyzer.sh`), use it as a starting point and enrich the analysis. ## Input **Run directory**: Provided as an argument or resolved from the latest failed run. If invoked via `scorpio doctor`, the run directory is already resolved. Otherwise, find it: ```bash # Latest failed run ls -td .scorpio/runs/*/ | while read d; do status=$(jq -r '.status' "$d/state.json" 2>/dev/null) [[ "$status" == "failed" ]] && echo "$d" && break done ``` ## Analysis Phases ``` Phase 1: Inventory → Catalog available artifacts and run metadata Phase 2: Classify → Categorize each session outcome Phase 3: Diagnose → Identify root causes with evidence Phase 4: Correlate → Find cross-session patterns and shared causes Phase 5: Report → Write structured postmortem report Phase 6: Remediate → Generate fix tasks for scorpio-execute ``` ## Phase 1: Inventory Catalog what's available in the run directory. ### Artifact Checklist ```markdown ## Run Inventory: - [ ] state.json — run status, phase history - [ ] PIPELINE-PLAN.json — planned sessions and phases - [ ] results/session-*.json — session outcomes (count: ___) - [ ] logs/pipeline.log — orchestrator log - [ ] logs/session-*-stdout.log — session output (count: ___) - [ ] logs/session-*-timeout.json — timeout sidecars (count: ___) - [ ] journal/ — event journal entries (count: ___) - [ ] postmortem.json — existing analyzer output (yes/no) ``` Record what's present and what's missing — missing artifacts are themselves a finding. ## Phase 2: Classify Categorize each session by outcome: | Outcome | Indicators | |---------|------------| | **success** | `status: "success"` in result JSON, commits present | | **failed** | `status: "failed"`, `failure_reason` populated | | **timed_out** | `status: "failed"`, `timeout_reason` in result or timeout sidecar | | **needs_decision** | `status: "needs_decision"`, `decision_needed` populated | | **no_result** | No result file exists (session crashed before writing) | | **already_complete** | `status: "already_complete"`, no new work done | ### Session Summary Table ```markdown | Session | Skill | Status | Tasks | Notes | |---------|-------|--------|-------|-------| | session-001 | scorpio-execute | success | CORE-001, CORE-002 | 2 commits | | session-002 | scorpio-execute | failed | CORE-003 | Missing dependency | | session-003 | scorpio-qa | timed_out | - | Idle timeout at 180s | ``` ## Phase 3: Diagnose For each non-success session, identify the root cause. ### Evidence Requirements Per CONTROL-PLANE-SEMANTICS.md §6.2, every finding MUST include at least one evidence entry: ```json { "type": "log | artifact | config | session_output", "path": "relative/path/to/evidence", "line_range": [130, 135], "excerpt": "Relevant snippet" } ``` ### Root-Cause Classes | Class | Description | Typical evidence | |-------|-------------|-----------------| | **code** | Bug in task implementation or skill logic | Session output, error messages | | **config** | Misconfiguration in pipeline or environment | Config files, env vars | | **policy** | Pipeline policy caused rejection | Pipeline log, gate messages | | **environment** | External dependency or infra issue | Network errors, missing binaries | ### Severity Assignment | Severity | Criteria | |----------|----------| | **critical** | Run cannot proceed; data loss risk | | **high** | Session(s) failed; results incomplete; likely to recur | | **medium** | Degraded behavior; workaround available | | **low** | Cosmetic or informational | ### Diagnosis Template For each finding: ```markdown ### Finding: F-NNN — [Title] **Severity**: critical | high | medium | low **Root-cause class**: code | config | policy | environment **Affected sessions**: session-NNN, session-NNN **Description**: [What went wrong and why] **Evidence**: 1. `logs/session-002-stdout.log:142-148` — Error message showing... 2. `results/session-002.json` — failure_reason field confirms... **Remediation**: - Action: [What should be done] - Confidence: high | medium | low - Auto-fixable: yes | no ``` ## Phase 4: Correlate Look for patterns across findings: - **Shared root cause**: Multiple sessions failed for the same reason - **Cascade failure**: One session's failure caused downstream failures - **Configuration drift**: Config worked before but now causes issues - **Recurring pattern**: Same failure seen in previous runs ### Correlation Template ```markdown ## Cross-Session Patterns ### Pattern: [Name] - **Findings**: F-001, F-003 - **Common cause**: [Shared root cause] - **Impact**: [How many sessions affected] - **Single fix addresses both**: yes | no ``` ## Phase 5: Report Write the structured postmortem to `/postmortem.json` following the schema in CONTROL-PLANE-SEMANTICS.md §6.1. Also write a human-readable `/postmortem.md` summary: ```markdown # Postmortem: **Run status**: failed **Generated**: **Sessions**: N total, N succeeded, N failed ## Summary [2-3 sentence overview of what happened] ## Findings ### F-001 — [Title] (critical) [Description with evidence references] ### F-002 — [Title] (high) [Description with evidence references] ## Recommendations 1. [P0] [Specific action] 2. [P1] [Specific action] ## Evidence Index - `logs/pipeline.log` — referenced by F-001 - `results/session-002.json` — referenced by F-001, F-002 ``` ## Phase 6: Remediate Generate remediation tasks compatible with **scorpio-execute**. ### Task Generation Rules - One task per finding (or per correlated group if a single fix resolves multiple findings) - Task priority maps from finding severity: critical→P0, high→P1, medium→P2, low→P3 - Each task references the finding ID and evidence - Max 5 files to modify per task ### Remediation Task Template ```markdown ## REWORK-[NNN] Fix [brief description] **Type**: Bug **Priority**: [P0-P3] **Estimate**: [XS-L] **Dependencies**: None **Source**: Postmortem finding F-[NNN] from run ### Description [Root cause summary from the finding] ### Root Cause **Location**: `file:line` **Evidence**: [Reference to postmortem evidence] ### Fix Approach [Specific steps to resolve] ### Acceptance Criteria - [ ] [Behavior that should now work] - [ ] [Edge case handled] - [ ] Existing tests pass - [ ] Regression test added ### Technical Notes - Postmortem report: `/postmortem.json` - Finding: F-[NNN] ``` ## Output Artifacts | Artifact | Path | Consumer | |----------|------|----------| | Structured report | `/postmortem.json` | `scorpio doctor`, pipeline controller | | Markdown summary | `/postmortem.md` | Operator review | | Remediation tasks | `{docs.tasks}/REWORK-*.md` | scorpio-execute | ## What NOT To Do - Do not guess at root causes without evidence - Do not generate findings for sessions that succeeded - Do not create remediation tasks for low-severity informational findings - Do not modify any code during analysis — this skill is read-only investigation - Do not read the entire codebase — focus on run artifacts and referenced files - Do not conflate this skill with **retrospective** — postmortem analyzes a single run's failures; retrospective analyzes patterns across a development cycle ## Relationship to Built-in Analyzer `scripts/postmortem-analyzer.sh` is a fast, deterministic bash script that produces a first-pass `postmortem.json`. This skill provides deeper analysis: | Aspect | Built-in analyzer | Postmortem skill | |--------|-------------------|------------------| | Speed | Fast (seconds) | Thorough (minutes) | | Evidence depth | Log excerpts only | Full artifact analysis | | Root-cause quality | Pattern matching | Reasoned diagnosis | | Correlation | Per-session only | Cross-session patterns | | Remediation | Generic suggestions | Specific fix tasks | Use the built-in analyzer for automated pipeline recovery. Use this skill when an operator needs to understand *why* something failed. ## Skill Chain ``` [run fails] → scorpio doctor → postmortem-analyzer.sh (fast pass) │ ▼ postmortem (this skill) → deep analysis │ ▼ scorpio-execute → execute remediation tasks │ ▼ scorpio-qa → validate fixes ``` ## Differentiation from Other Skills | Skill | Focus | Scope | |-------|-------|-------| | **postmortem** | Why did this run fail? | Single run's artifacts | | **scorpio-bug-investigate** | Why does this code have a bug? | Codebase exploration | | **retrospective** | What patterns emerged across a cycle? | Multiple runs/tasks | | **scorpio-lessons** | What should we remember for next time? | Aggregated insights |