--- name: incident-playbook description: Generate runbooks for common AI automation failures. Covers model timeouts, hallucination detection, cost spikes, accuracy drops, data pipeline failures, and prompt injection attempts. --- # Incident Playbook — AI Failure Runbook Generator ## Purpose Generate structured runbooks for common AI automation failures. Every AI system in production will eventually encounter failures specific to the AI domain: models time out, hallucinations slip through, costs spike unexpectedly, accuracy degrades silently, data pipelines break, and adversarial inputs probe for weaknesses. This skill produces a comprehensive playbook tailored to a specific automation, covering nine failure categories with detection methods, severity classification, step-by-step response procedures, root cause investigation guides, and client communication templates. The playbook is the document your team reaches for at 2 AM when the system is misbehaving. Run this **after** the automation architecture is understood (ideally after `/ai-invert` and `/guardrail-spec`) and **before** or **during** deployment. The output complements the `/deploy-checklist` by ensuring operational readiness. ## Pre-flight: Auto-sync Check Before starting, read the root `CLAUDE.md` (if it exists). If `## How to Approach Tasks` appears AFTER `## Tech Stack` or `## Architecture`, the file has an outdated section order. Silently run `/genai-sync` to fix it before continuing with this skill. ## Input The user provides the name of the automation or system to generate a playbook for. This can be: - A specific automation name (e.g., "support-classifier", "document-extractor") - A system or pipeline name (e.g., "rag-pipeline", "lead-scoring-system") - A reference to existing `.plans/` artifacts that describe the automation ## Workflow ### Step 1: Understand the Automation Architecture **Actions:** 1. Read the automation codebase: prompts, model configurations, pipeline configs, application code, infrastructure definitions 2. Check root context files (`CLAUDE.md`, `AGENTS.md`) for architecture overview and conventions 3. Read `.plans/` for related artifacts: - `INVERT-*.md` — risk analyses identifying failure modes (especially dimensions B-H). Use the risk list to prioritize which runbooks need the most detail. - `GUARDRAILS-*.md` and `GUARDRAILS-REPORT-*.md` — guardrail specs and validation results. Reference actual detection methods and fallback behaviors in runbooks. - `COST-*.md` — cost projections and budget thresholds. Use for cost spike incident thresholds. - `BASELINE-*.md` — baseline metrics. Use for accuracy/latency regression thresholds in detection. - `EVAL-*/` — eval suites and results. Use eval thresholds as accuracy incident triggers. - `OPS-.md` — ai-ops monitoring config (if exists). **Use the actual dashboard names, alert names, and metric queries defined here** instead of generic monitoring instructions. If ai-ops has defined "alert: cost-spike-daily triggers at >150% of budget via PagerDuty to #oncall-ai," reference that exact alert in the cost spike runbook. - `DECISIONS.md` — prior decisions constraining the system - `LEARNINGS.md` — lessons from prior incidents. If a prior incident was learned from, reference the specific learning. 4. Map the system architecture: - **External dependencies**: model providers (which APIs, which models), embedding services, databases, third-party APIs - **Data pipeline**: ingestion sources, processing steps, storage, retrieval - **Monitoring**: what is currently monitored, what alerts exist, who is on-call - **Deployment**: how the system is deployed, how rollback works 5. Identify the **blast radius** of each component failure — what breaks downstream? ### Step 2: Generate Incident Runbooks For each of the nine failure categories below, generate a complete runbook section. Tailor every section to the specific automation — generic advice is not useful at 2 AM. **Every incident must include a "Time to Detect" estimate** — how long this failure would go unnoticed WITHOUT proactive monitoring. This drives monitoring investment decisions: - **Seconds-minutes:** Immediate user impact, requests fail visibly (e.g., timeouts) - **Hours:** Degraded quality accumulates, users notice after several interactions (e.g., accuracy drift) - **Days-weeks:** Silent degradation, no user-visible symptom until threshold breach (e.g., stale embeddings, cost creep) Incidents with days-weeks detection lag are the most dangerous — they need proactive monitoring most urgently. If a failure category does not apply to this automation (e.g., no RAG pipeline means "Embedding Index Stale" is irrelevant), note it as "N/A for this system" with a one-line explanation, and move on. --- #### Incident 1: Model Timeout / Rate Limit Exceeded **Detection:** - Typical signals: increased latency, 429 status codes, request queue growth - **Time to detect (without monitoring):** Seconds-minutes — requests fail immediately and visibly - **Detection commands** (tailor to THIS project's actual tools): ``` # Check error rate (replace with project's actual monitoring tool/query) ``` **Every detection section must include at least one runnable command, log query, or dashboard URL** that the on-call person can execute immediately. "Check the dashboard" is not a detection command. "Open https://grafana.internal/d/ai-pipeline and check the error-rate panel for the last 15 minutes" is. **Severity Classification:** | Severity | Symptom | Blast Radius (for THIS automation) | |----------|---------|-----------------------------------| | P1 | User-facing requests failing | | | P2 | Batch processing delayed, user-facing has fallback | | | P3 | Only non-critical background tasks affected | | **Rule:** Severity must name the specific blast radius for THIS system, not generic impact. "User-facing requests failing" becomes "Classification API returns errors → client dashboard shows 'processing' indefinitely → Slack notifications stop → 2K daily users affected." **Immediate Response:** 1. Confirm the incident — check model provider status page and internal metrics 2. Activate fallback model (if configured) — document the specific fallback path 3. Enable request queuing or retry with exponential backoff 4. If rate limited: check for runaway loops or misconfigured batch jobs consuming quota 5. Communicate to affected users if response times are visibly degraded **Root Cause Investigation:** - Check if the model provider has a known outage (status page, social media, support channels) - Check if traffic volume spiked unexpectedly (new client, batch job, retry storm) - Check if a code change increased tokens per request or request frequency - Review rate limit headers from recent responses to understand consumption patterns **Resolution:** - If provider outage: wait for resolution, keep fallback active, monitor provider status - If self-inflicted (traffic spike, runaway job): fix the root cause, then restore primary model - If persistent rate limiting: request quota increase, implement better request batching, or add caching **Prevention:** - Configure fallback model with automatic failover - Implement circuit breaker pattern for model API calls - Set up rate limit consumption alerts at 70% and 90% of quota - Cache frequent requests to reduce API call volume - Implement request prioritization (user-facing > batch) **Client Communication Template:** > We are experiencing intermittent delays in [system name] responses due to elevated demand on our AI infrastructure. Our team is actively working on this. Expected resolution: [time]. Your requests are being queued and will be processed. No data has been lost. --- #### Incident 2: Token Limit Exceeded (Context Too Large) **Detection:** - Error codes indicating context length exceeded (provider-specific error codes) - Requests failing with specific input patterns (long documents, many retrieved chunks) - Sudden increase in model API errors correlated with input size **Severity Classification:** - P1 if a class of inputs consistently fails with no fallback - P2 if only edge-case inputs are affected and most requests succeed - P3 if the system gracefully truncates and produces acceptable output **Immediate Response:** 1. Identify which inputs are triggering the limit — log the input sizes 2. Check if a recent change increased context size (more RAG chunks, longer system prompt, new few-shot examples) 3. If possible, temporarily reduce context (fewer chunks, shorter system prompt) to restore service 4. Route oversized requests to a model with a larger context window (if available) **Root Cause Investigation:** - Measure token distribution across components: system prompt + few-shot examples + retrieved context + user input + output reservation - Identify which component grew — was it a prompt change, a retrieval config change, or unusual input? - Check if token counting is accurate (tokenizer mismatch between estimation and actual API) **Resolution:** - Implement input truncation with graceful degradation (summarize long inputs, reduce chunk count) - Optimize system prompt and few-shot examples for token efficiency - Upgrade to a model with a larger context window if the use case requires it - Implement dynamic context management — adjust chunk count based on input size **Prevention:** - Add pre-flight token count validation before every model API call - Set hard limits on each context component (system prompt budget, retrieval budget, input budget) - Monitor token usage distribution over time — detect drift before it causes failures - Test with maximum-size inputs in the eval suite **Client Communication Template:** > Some requests with unusually large inputs are experiencing processing errors in [system name]. Our team has identified the issue and is implementing a fix. Requests with typical input sizes are unaffected. If you are experiencing errors, please try with a shorter input while we resolve this. --- #### Incident 3: Content Filter Triggered **Detection:** - Model provider returns content filter / safety block responses - Spike in blocked requests that should be legitimate - User complaints about requests being rejected **Severity Classification:** - P1 if the content filter is blocking a high percentage of legitimate requests - P2 if only specific input patterns are affected - P3 if the filter is working correctly but users need guidance **Immediate Response:** 1. Determine if the filter is the model provider's built-in filter or your custom guardrail 2. If provider filter: check for provider-side filter sensitivity changes (often unannounced) 3. If custom guardrail: check for recent configuration changes to thresholds or patterns 4. Review sample blocked requests to classify as true positives vs false positives 5. If false positive rate is high: temporarily relax custom filters (if safe) or route to fallback **Root Cause Investigation:** - Review blocked request samples — what pattern triggers the filter? - Check if the model provider updated their content policy or filter model - Check if input patterns changed (new use case, new client, different language) - If custom filter: review filter rules for overly broad patterns **Resolution:** - If provider filter: adjust prompts to avoid triggering patterns, contact provider support, consider alternative model - If custom filter: refine detection rules to reduce false positives while maintaining protection - Add exemptions for known-safe input patterns (with logging) - Implement a secondary path for filtered requests (human review queue) **Prevention:** - Test prompts against known content filter edge cases in eval suite - Monitor content filter trigger rate as a key metric — alert on sudden changes - Maintain a list of known false-positive input patterns and test against them - Keep custom filter thresholds configurable so they can be tuned without redeployment **Client Communication Template:** > Some requests in [system name] are being flagged by our safety filters. We are reviewing the filter configuration to reduce false positives. If your request was incorrectly blocked, please contact [support channel] and we will process it manually. --- #### Incident 4: Hallucination Detected in Production **Detection:** - User reports factual errors, fabricated citations, or invented data in outputs - Automated fact-checking or consistency checks flag discrepancies - Output audit reveals claims not supported by retrieved context - Downstream systems receive incorrect data from the automation **Severity Classification:** - P1 if the hallucination caused a client action (financial decision, published content, customer communication) - P1 if the hallucination involves fabricated legal, medical, or financial information - P2 if detected internally before impacting client actions - P3 if the hallucination is minor (formatting, low-impact phrasing) and caught by review **Immediate Response:** 1. Identify the scope — is this a one-off or a pattern? Check recent outputs for similar hallucinations 2. If client-facing: immediately notify the affected client with a correction 3. If the hallucination caused downstream actions: assess impact and initiate reversal if possible 4. If a pattern is detected: add a temporary guardrail (keyword filter, human review for affected output types) 5. Preserve the input, output, and retrieved context for root cause analysis **Root Cause Investigation:** - Was relevant context retrieved? Check retrieval results for the failing request - Was the context accurate? Check source documents for correctness and freshness - Did the model ignore the context? Compare output claims against retrieved chunks - Was the prompt encouraging fabrication? (e.g., "always provide an answer" without "say you don't know if unsure") - Did a model version change introduce new hallucination patterns? **Resolution:** - Fix the prompt to explicitly instruct "only answer based on provided context" and "say you don't know if unsure" - Improve retrieval quality (better chunking, reranking, relevance filtering) - Add output verification guardrails (fact-check against source data, consistency checks) - Update eval suite with the hallucination case as a regression test - If model version caused the issue: pin to previous version or switch models **Prevention:** - Include hallucination-specific test cases in the eval suite (questions where the answer is NOT in the context) - Implement automated output-vs-context consistency checking - Require citation for factual claims in the prompt design - Monitor hallucination rate as a key quality metric - Regular human review sampling of production outputs **Client Communication Template:** > We identified an error in a response provided by [system name] on [date]. The output contained [brief description of error]. We have corrected this and implemented additional verification to prevent recurrence. We apologize for any inconvenience. If you took action based on this output, please [specific remediation guidance]. --- #### Incident 5: Embedding Index Stale or Corrupted **Detection:** - Retrieval quality drops — model outputs become less relevant or accurate - Known documents are not being retrieved for relevant queries - Embedding index metadata shows last update timestamp is stale - Index size mismatch with expected document count **Severity Classification:** - P1 if the system is returning irrelevant or outdated information to clients - P2 if retrieval quality is degraded but outputs are still roughly correct - P3 if only newly added documents are not yet indexed (known delay) **Immediate Response:** 1. Check the last successful index update timestamp 2. Verify the embedding pipeline is running — check job logs, scheduler status 3. If index is corrupted: switch to the last known-good index snapshot (if available) 4. If stale: trigger a manual index rebuild 5. If the system is producing notably wrong outputs: add a temporary disclaimer or route to fallback **Root Cause Investigation:** - Check the embedding pipeline job logs — did a job fail silently? - Check source data availability — is the upstream data source accessible? - Check for schema changes in source documents that broke the parser - Check embedding model availability — did the embedding API have an outage? - Check storage — did the index exceed storage limits? **Resolution:** - Fix the pipeline failure and trigger a full reindex - If source data changed format: update the parser and reprocess - If embedding model changed: re-embed all documents with the new model - Verify index integrity after rebuild (document count, spot-check retrievals) **Prevention:** - Monitor index freshness as a key metric — alert when last update exceeds threshold - Implement index integrity checks (document count, embedding dimension, spot-check queries) - Maintain index snapshots for quick rollback - Run a daily retrieval quality test (known query -> expected document pairs) - Set up pipeline failure alerts with clear error messages **Client Communication Template:** > [System name] may be using slightly outdated information for responses generated between [start time] and [end time]. We have refreshed our data and responses are now current. If you received a response during this window that seems outdated, please resubmit your request. --- #### Incident 6: Cost Spike Detected **Detection:** - Cost monitoring alert triggers (daily spend exceeds threshold) - Token usage dashboard shows unexpected increase - Model provider billing alert fires - Unusual request volume or request size patterns **Severity Classification:** - P1 if spend rate will exceed monthly budget within days - P2 if spend is elevated but within manageable range - P3 if the spike is explained by legitimate usage growth **Immediate Response:** 1. Identify the source of the spike — which endpoint, which client, which model? 2. Check for runaway loops, retry storms, or misconfigured batch jobs 3. If a specific source is identified: throttle or pause it 4. If unexplained: enable request sampling to reduce volume while investigating 5. Check if a code change increased tokens per request (longer prompts, more context, removed caching) **Root Cause Investigation:** - Compare current request patterns with baseline: volume, tokens per request, model used - Check for recent deployments that changed prompt length, retrieval count, or model selection - Check for client-side changes that increased request volume - Check if caching is functioning correctly — cache hit rate vs baseline - Check for duplicate or unnecessary model calls in the pipeline **Resolution:** - Fix the root cause (runaway job, cache failure, prompt bloat, missing dedup) - Implement or restore caching for frequently repeated requests - Optimize prompts for token efficiency if they have grown - If legitimate growth: update budget projections and get approval for increased spend **Prevention:** - Set cost alerts at multiple thresholds (50%, 80%, 100% of budget) - Implement per-client and per-endpoint cost tracking - Add automatic throttling when spend rate exceeds configured threshold - Review token usage in every deployment checklist - Cache system prompt tokens where the provider supports it (prompt caching) **Client Communication Template:** > We detected unusual usage patterns in [system name] that resulted in elevated processing costs. We have identified and resolved the cause. There is no impact to your service or data. [If client-caused: We recommend reviewing your integration to ensure request patterns align with expected usage.] --- #### Incident 7: Accuracy Drop Below Threshold **Detection:** - Automated eval suite shows accuracy below threshold on scheduled run - User feedback indicates declining output quality - Downstream metrics (conversion rate, error rate, rejection rate) shift negatively - Human review sampling shows increased error rate **Severity Classification:** - P1 if accuracy drop is significant (>10% below baseline) and client-facing - P2 if accuracy drop is moderate (5-10% below baseline) or detected before client impact - P3 if accuracy drop is minor (<5%) and caught by monitoring **Immediate Response:** 1. Confirm the accuracy drop with a focused eval run (not just monitoring noise) 2. Check if the model provider made a change (model version update, behavior change) 3. Check if input distribution shifted (new type of request, different client, seasonal pattern) 4. If drop is severe and confirmed: activate fallback (previous model version, simplified prompt, human review) 5. Notify stakeholders of the quality degradation and response plan **Root Cause Investigation:** - Run evals against the previous model version — does accuracy restore? (model change) - Run evals with recent production inputs vs original eval inputs — does accuracy differ? (distribution shift) - Check for prompt changes, retrieval changes, or guardrail changes that could affect output quality - Check if the training data or knowledge base has changed - Analyze error patterns — is the accuracy drop uniform or concentrated in specific input types? **Resolution:** - If model change: pin to previous version, evaluate new version, adjust prompts for new model behavior - If distribution shift: update prompts and eval suite to handle new input patterns - If code change: revert the change and investigate - Update baseline with corrected performance metrics - Add regression test cases for the patterns that failed **Prevention:** - Run automated evals on a regular schedule (daily or weekly) - Monitor accuracy-correlated proxy metrics in real time (confidence scores, retrieval relevance) - Pin model versions and test before upgrading - Maintain a diverse eval set that covers known input variations - Track input distribution metrics to detect shift early **Client Communication Template:** > We detected a temporary reduction in [system name] output quality and have taken corrective action. Outputs generated between [start time] and [end time] may have reduced accuracy. We recommend reviewing any critical decisions made based on outputs from this period. Quality has been restored to normal levels. --- #### Incident 8: Data Pipeline Failure (Upstream Source Unavailable) **Detection:** - Pipeline health check fails — source API returns errors or timeouts - Data freshness monitoring shows last successful fetch exceeds threshold - Zero new documents processed in expected time window - Scheduled ingestion job reports failure **Severity Classification:** - P1 if the system is serving answers based on critically outdated data - P2 if data is stale but still within acceptable freshness window - P3 if the pipeline failure is detected early and a retry will resolve it **Immediate Response:** 1. Identify which upstream source is unavailable — check connectivity, authentication, source status 2. Check if the system can operate on cached/stale data with acceptable quality 3. If data freshness is critical: add a disclaimer to outputs noting the data staleness 4. Trigger a manual pipeline run once the source is available 5. Check if other downstream systems are also affected **Root Cause Investigation:** - Check upstream source status — is it a known outage, maintenance window, or permanent change? - Check authentication — did API keys expire, credentials rotate, or permissions change? - Check network — firewall changes, DNS issues, VPN requirements? - Check data format — did the source schema change, breaking the parser? - Check rate limits — is the pipeline being throttled by the source? **Resolution:** - If source outage: wait for resolution, implement retry with backoff, trigger backfill when available - If auth failure: rotate credentials, update secrets, verify permissions - If schema change: update parser, reprocess recent data, verify data integrity - If permanent source change: evaluate alternative sources or negotiate with the source provider **Prevention:** - Monitor pipeline health with heartbeat checks (not just failure alerts) - Implement graceful degradation — serve stale data with freshness indicator rather than failing - Set up credential expiration monitoring and proactive rotation - Maintain source API documentation and change notification subscriptions - Test pipeline recovery by simulating source failures in staging **Client Communication Template:** > [System name] is currently operating with data last updated on [timestamp] due to a temporary disruption in our data feed. Responses remain available but may not reflect changes after [timestamp]. We expect to restore full data freshness by [estimated time]. Critical decisions should be verified against primary sources during this period. --- #### Incident 9: Prompt Injection Attempt Detected **Detection:** - Guardrail triggers on prompt injection patterns (role switching, instruction override, encoded instructions) - Canary token detected in output (system prompt leakage) - Output suddenly deviates from expected format or content scope - Unusual input patterns from a specific user or session **Severity Classification:** - P1 if the injection succeeded — system prompt leaked, business rules bypassed, or unauthorized output produced - P2 if the injection was detected and blocked but the attempt was sophisticated - P3 if the injection was a known pattern and was blocked by standard guardrails **Immediate Response:** 1. If the injection succeeded: assess what was exposed or what action was taken 2. Block or rate-limit the offending user/session/IP 3. Review the last N outputs from the same user/session for prior successful injections 4. If system prompt was leaked: assess the sensitivity of leaked content 5. Preserve the full request/response chain for security review **Root Cause Investigation:** - How did the injection bypass existing guardrails? (new pattern, encoding trick, multi-turn attack) - Was this a targeted attack or automated probing? - What was the attacker's likely objective? (data extraction, behavior manipulation, system prompt theft) - Review guardrail coverage — which defense layer failed or was missing? - Check if similar patterns appear in other sessions **Resolution:** - Add the new injection pattern to the detection rules - If system prompt was leaked: rotate any sensitive information in the prompt - Strengthen guardrail layers — add classifier-based detection if using only pattern matching - Implement output monitoring for system prompt fragments - If a vulnerability was exploited: patch and re-test **Prevention:** - Layer injection defenses (pattern matching + classifier + input/output consistency check) - Embed canary tokens in system prompts to detect leakage - Minimize sensitive information in system prompts - Implement session-level anomaly detection - Regular adversarial testing (red-team the prompts) - Rate-limit users who trigger injection guardrails **Client Communication Template:** > Our security systems detected and blocked an unauthorized attempt to manipulate [system name]. No client data was exposed or compromised. We have strengthened our defenses against this type of attempt. [If client needs to take action: We recommend reviewing any outputs from session [ID] before relying on them.] --- ### Step 3: Write to Disk **Actions:** 1. Create `.plans/` directory if it does not exist 2. Generate a short kebab-case name from the automation (e.g., `support-classifier`, `rag-pipeline`, `document-extractor`) 3. Write the full playbook to `.plans/PLAYBOOK-.md` with this header: ```markdown # Incident Playbook: **Created:** **System:** **Architecture:** **On-Call:** **Escalation Path:** ## Incident Response Overview | # | Incident Type | Applicable? | Severity Range | |---|--------------|-------------|----------------| | 1 | Model Timeout / Rate Limit | | | | 2 | Token Limit Exceeded | | | | 3 | Content Filter Triggered | | | | 4 | Hallucination in Production | | | | 5 | Embedding Index Stale | | | | 6 | Cost Spike | | | | 7 | Accuracy Drop | | | | 8 | Data Pipeline Failure | | | | 9 | Prompt Injection | | | ## Runbooks ``` 4. Inform the user: "Incident playbook saved to `.plans/PLAYBOOK-.md`" ### Step 4: Present Summary After writing to disk, present a concise summary: ``` ## Incident Playbook: **Applicable Incidents:** **System Dependencies:** ### Escalation Chain | Time Since Detection | Action | Who | |---------------------|--------|-----| | 0 min | On-call acknowledges, starts runbook | via | | 15 min | If not resolved: escalate to team lead | via | | 30 min | If P1 and not resolved: escalate to engineering manager | via | | 1 hour | If P1 and client-facing: notify client success team | via | | 2 hours | If P1 and not resolved: executive notification | via | Adjust time thresholds based on the automation's SLA. Client-facing P1 incidents should escalate faster than internal P2s. ### Quick Reference | Incident | Detection | Severity | First Action | |----------|-----------|----------|-------------| | Model Timeout | | | | | Token Limit | | | | | ... | ... | ... | ... | Playbook saved to `.plans/PLAYBOOK-.md`. ``` ## Notes - This skill is language-agnostic — it produces operational runbooks, not code - Read actual code, configs, and architecture before writing runbooks — generic runbooks are useless at incident time - Tailor every runbook to the specific automation — name the actual model provider, actual monitoring tools, actual deployment platform - If an `/ai-invert` analysis exists, use its risk findings to prioritize which runbooks need the most detail - If a `/guardrail-spec` exists, reference its detection methods and fallback behaviors in the runbooks - Client communication templates should be professional, honest, and specific — avoid jargon but do not hide what happened - Severity classifications should be calibrated to the specific automation's impact — a P1 for an internal tool may be a P3 for a critical client-facing system - Runbooks should be executable by someone who did not build the system — include specific commands, URLs, and contact information where possible - Review and update the playbook after every real incident — lessons learned should feed back into prevention measures - The playbook complements the `/deploy-checklist` — the checklist verifies readiness, the playbook handles what goes wrong after deployment