--- name: AI Error Learner description: Automatically detect and catalog recurring errors as AI pain points in need of skills. Activates when the same error is encountered twice in a session. allowed-tools: Read, Write, Edit, Grep, Glob version: 2.2.0 --- # AI Error Learner Skill ## Purpose Automatically track errors that Claude encounters during work, and when the same error pattern is hit twice, catalog it as an "AI Pain Point" that needs a dedicated skill to handle efficiently. This creates a self-improving system where recurring problems become documented opportunities for automation. ## Auto-Activation Triggers This skill activates when: - An error/exception occurs that Claude has seen before in this or recent sessions - Claude encounters the same type of failure twice (e.g., same error pattern, same tool failure) - User mentions "you keep hitting that error" or similar - Claude recognizes a pattern it has failed on previously - Tool output contains error that matches a known error signature **and the detection passes the false-positive filters in §"Detection False-Positive Filters" below** ## CRITICAL: Detect-Decide-Act Protocol **This skill MUST run on EVERY tool result that indicates failure.** ### Mandatory Detection Points Claude MUST check for errors after EVERY: 1. **Bash command** — Exit code != 0, stderr output, error keywords (subject to false-positive filters) 2. **File operation** — File not found, permission denied, parse failures 3. **API call** — Timeout, rate limit, authentication failure, 4xx/5xx responses 4. **MCP tool** — Tool execution failure, unexpected response format 5. **Edit/Write** — Validation failure, conflict, file locked ### Real-Time Observation Pattern ``` EVERY TOOL CALL ↓ 1. DETECT: Scan result for error indicators - Apply Detection False-Positive Filters (§ below) BEFORE fingerprinting - Reject substring matches that are not real errors ↓ 2. DECIDE: Is this a catalogable error? - Generate stripped fingerprint (§ "Fingerprint Stripping") - Check error history - Determine if repeat occurrence ↓ 3. ACT: Update history or catalog pain point - First time: Log to history with status="watching" - Second time: PROMOTE — set status="pain_point", assign pain_point_id, append to ai-pain-points.md - Third+ time: Escalate priority to "high" ``` The actuator step (third stage) is **mandatory** at count==2. Failing to write to ai-pain-points.md is a skill bug, not a feature. ## Detection False-Positive Filters (NEW in v2.0) **Why this section exists**: v1 of this skill matched error keywords as raw substrings. Diagnostic snapshot taken 2026-04-29 showed an 82% false-positive rate on the high-frequency labeled set: numbers in `ls -la` byte counts being read as HTTP status codes, the word "error" appearing inside filenames (`error_log.jsonl`, `ai-error-learner/`), "FAILED" in section headers of validation reports, INFO logs from access-log batches, etc. These filters MUST run BEFORE fingerprinting. ### Filter F1 — Suppress numeric matches outside HTTP context A 3-digit number is NOT an HTTP status code if any of these hold: - stdout begins with `total \d+\n` (ls -la output) — the number is a byte count - stdout begins with `drwx` or other ls permission glyphs - stdout matches a JSON metadata structure (`{"metadata":` or `"": `) — the number is field data - stdout starts with `:\s+` (grep line-number prefix) - stdout matches git diff --stat (`\| \d+ [+\-]+`) - The number is NOT preceded within 30 characters by one of: `HTTP`, `Status`, `Response`, `code`, `curl`, `fetch`, `GET`, `POST`, `PUT`, `DELETE` If any filter triggers, drop ALL numeric error_types from the entry. ### Filter F2 — Word-boundary matching for "error" keyword The keyword `error` (case-insensitive) must NOT match if either neighbor is a word character (`[a-zA-Z0-9_-]`). Examples that should be REJECTED: - `error_log.jsonl` (filename) - `ai-error-learner/` (directory name) - `error-handler.js` (filename) - `Update error tracking logs` (commit message subject) - `proper\nerror handling` (Python docstring text) Match only when `error` (or `Error`/`ERROR`) is followed immediately by `:`, ` in `, ` while `, or end-of-token AND adjacent stdout context contains an actual error indicator (Traceback, Exception, exit code, exec failed, ImportError, panic, FATAL). ### Filter F3 — Access-log batch handling If stdout matches the uvicorn/nginx access-log pattern: ``` INFO: : - " HTTP/" ``` Do NOT batch-fingerprint. Instead: 1. Split into lines, parse status code per line. 2. Compute 5xx_rate = (5xx count) / (total log lines). 3. Suppress fingerprinting if 5xx_rate < 0.05 (5%). 4. If above threshold, fingerprint per-route (METHOD + path), not per-batch. ### Filter F4 — Section-header reports (relaxed in v2.2) Strings like `FAILED`, `ERROR`, `TIMEOUT` appearing as section headers (preceded by `=====` or `# ` or `[ ]`) in validation/audit/build reports are reporting on tests, not failing. Suppress. **v2.2 carve-out — pytest banners are NOT suppressed.** The canonical pytest section banners - `============================= test session starts ==============================` - `==================================== ERRORS ====================================` - `==================================== FAILURES ==================================` - `=========================== short test summary info ============================` DO carry real failures (timeouts, ImportErrors, assertion failures) within the visible context window or the metadata `error_types` list. F4 must therefore exempt these specific banners while still suppressing generic `VALIDATION/REPORT/AUDIT/BUILD` headers. The exemption is narrow: only the four banner phrases above; any other `=====` header still drops. When a pytest banner is admitted, the stripping rule S8 below derives a meaningful key from the observed `error_types` (`pytest-timeout`, `pytest-failed`, `pytest-error`, or `pytest-test-session`) so the entry is not collapsed into a generic `bash-*` bucket. ### Filter F5 — git output If stdout matches any of: - `^On branch \w+` - `^M \S+` or `^MM \S+` (git status -s) - `^commit [0-9a-f]{40}` - `^ \S+\s+\|\s+\d+ [+\-]+` (git diff --stat) drop ERROR/FAILED keyword matches unless an actual Traceback/Exception/exit-code line appears elsewhere in stdout. ### Filter F6 — Source-code echo If stdout starts with `#!/`, `import \w`, `from \w+ import`, or `def \w+\(`, the bash command echoed source code; suppress keyword matches. ### Filter F7 — Documentation/example/data-listing suppressor (NEW in v2.1 — precision tightening) **Why**: After F1-F6 v2.0 reached precision 0.6786 on the 50-entry labeled set, the residual 4 false-positive labels were not about the word `error` in prose (as originally hypothesized in IMP-0001). The actual residual FPs were data/listing contexts that survived F1's narrow `numeric_only + ls/grep/diff-stat` gate: file-path listings, structured-data dumps (`Total teams: 134\nKeys: [...]`), INFO-only log batches (OAuth/Loaded/uvicorn-INFO), and wrapper-extraction fallthroughs carrying grep/INFO content. F7 generalizes the "non-failing context" concept to include all of these. F7 is **precision-only**: it only fires when `REAL_ERROR_INDICATORS` (Traceback, Exception, ImportError, panic, FATAL, exec failed, prctl … failed, command not found, HTTP \d{3} on its own line, curl failed, cwebp/convert not found, GOOGLE_CLIENT_ID not set) does **not** match anywhere in the context. By construction this cannot regress recall: every currently-admitted true/partial labeled entry has at least one real-error indicator hit. A context is suppressed by F7 when ALL of: - No `REAL_ERROR_INDICATORS` match anywhere in the context window AND - ANY of the following sub-rules fires: | Sub-rule | Pattern | Targets | |---|---|---| | F7a | stdout starts with `/Users/` or `/home/` etc. AND has ≥2 absolute-path lines | `find` output read as 404 | | F7b | stdout begins with `Total \w+: \d+`, `Sample \w+:`, `Keys: [`, `Length: \d+`, `Columns: [`, or `Schema: {` | data-summary dumps | | F7c | ≥80% of non-blank lines match INFO/DEBUG/Loaded/level=info/level=debug/Connected to/Listening on/Started/OAuth: Loaded | nginx/uvicorn/app INFO batches | | F7d | wrapper-extraction failed AND inner content begins with `\d+:\s+` (grep line-number prefix) | grep output through wrapper | | F7d2 | wrapper-extraction failed AND inner content begins with OAuth/Loaded/INFO/DEBUG/level=info | INFO log through wrapper | | F7e | descriptive `error\s+(handling\|tracking\|reporting\|logging\|message\|format\|code\|type\|class)\b`, `"\w*error\w*":` schema field, or `e.g., error` / `example…error` | the originally-documented prose hypothesis | This filter targets the residual 4 false-positive labeled entries directly. F7 runs AFTER F1-F6 and only fires when those admitted the entry; combined with the no-real-error-indicator gate, it cannot regress on entries other filters already dropped. **Conservative principle**: F7 is precision-only. If any doubt about whether a context contains a real error, F7 must NOT fire. The real-error-indicator allowlist guarantees genuine failures with the suppressed surface forms (e.g., a path listing followed by a Traceback) are still admitted. ## Fingerprint Stripping (NEW in v2.0) The v1 fingerprint key embedded raw stdout. This caused semantically identical errors to receive distinct keys. v2 generates a **stripped** fingerprint that captures the failure signature, not the surrounding noise. ### Stripping Rules (apply in order, first match wins) | # | If stdout contains... | Fingerprint = | |---|---|---| | S1 | ` \| (Error\|Warning) in :` | `docker--in-` | | S2 | `^(HTTP )?\d{3}\s*$` | `curl-status-` | | S3 | `command not found:\s*(\w+)` | `command-not-found-` | | S4 | `ImportError: cannot import name '(\w+)'` | `import-error-` | | S5 | `exec failed:.*"(\w+)":` | `oci-exec-failed-` | | S6 | `Traceback (most recent call last)`...`:` | `python-` | | S8 (v2.2) | pytest banner present (`=== test session starts ===` / `=== ERRORS ===` / `=== FAILURES ===` / `=== short test summary info ===`) | `pytest-timeout` / `pytest-failed` / `pytest-error` / `pytest-test-session` (chosen by the first matching `error_types` entry) | | S7 | (default) | `-` (drop stdout from key) | The first matching rule wins. Goal: collapse the 1,774 raw fingerprints to fewer than 1,200 unique stripped keys (≥30% reduction). ## Error Fingerprinting (revised) Errors are fingerprinted by: 1. Apply Detection False-Positive Filters (§ above). If any filter fires, do NOT create a fingerprint. 2. Apply Fingerprint Stripping rules (§ above) to derive a stable key. 3. The stripped key is the canonical fingerprint. **Fingerprint Format (v2):** ``` ``` Examples: - `docker-Error-in-cpuinfo` - `curl-status-404` - `command-not-found-python` - `import-error-SessionLocal` - `oci-exec-failed-nslookup` - `python-ImportError` ## Core Workflow ### Step 1 — Error Detection Tool output arrives. Apply F1-F7 filters first. If suppressed, return immediately. ### Step 2 — Generate Stripped Fingerprint Apply S1-S7 rules in order. First match wins. ### Step 3 — Check Error History Read `.claude/pain-points/ai-error-history.json`. If the stripped fingerprint exists, increment count and update last_seen. ### Step 4 — Catalog or Update - **count == 1**: store under `status: "watching"`. No further action. - **count == 2**: PROMOTE. Set `status: "pain_point"`, assign next `AI-PAIN-XXXX` id, append a markdown entry to `ai-pain-points.md` using the template in § "AI Pain Point Format". - **count >= 3**: increment, set `priority: "high"` if not already. The promotion at count==2 is **not optional**. Skipping it is a skill failure. ## AI Pain Point Format When promoting at count==2, append the following block to `.claude/pain-points/ai-pain-points.md` and increment `Last AI Pain Point ID`: ```markdown ### [AI-PAIN-XXXX] Brief description of the recurring error - **Error Fingerprint**: `` - **Occurrence Count**: 2 - **First Seen**: - **Last Seen**: - **Impact**: - **Contexts Encountered**: 1. 2. - **Error Details**: - **Proposed Skill**: ``` All eight fields are required for the harness to mark the entry well-formed. ## Skill Proposal Generation For every promoted pain point, generate a `Proposed Skill` block: ```markdown ## Proposed Skill: **Problem**: **Trigger Conditions**: - - **Skill Workflow**: 1. 2. 3. **Required Tools**: **Expected Outcome**: **Effort Estimate**: Small | Medium | Large ``` The proposal must be *substantive* — at least 3 distinct workflow steps, a specific tool list, and a concrete trigger condition. Vague proposals ("when error happens", "do something useful") fail the harness's `proposal_nontrivial` sub-metric. ## Storage Structure ### Error History File (v2 format) Location: `.claude/pain-points/ai-error-history.json` ```json { "version": "2.0.0", "last_updated": "ISO timestamp", "errors": { "": { "count": 2, "first_seen": "ISO timestamp", "last_seen": "ISO timestamp", "tool": "Bash", "error_types": ["..."], "contexts": ["...", "..."], "pain_point_id": "AI-PAIN-XXXX", "status": "pain_point", "priority": "normal" } } } ``` ### Migration When v2 first runs, walk the v1 file and re-fingerprint each entry under the S1-S7 rules. Multiple v1 keys may collapse to one v2 key — sum their counts, take min(first_seen) and max(last_seen). ## Configuration ### Thresholds - **Catalog Threshold**: 2 occurrences (default). PROMOTE at count==2. - **Priority Escalation**: 5 occurrences → priority="high" - **False-Positive Filter**: applied unconditionally before fingerprinting ### Exclusions - All entries that fail F1-F7 filters - Transient network errors (unless persistent under stripped fingerprint) - Test-fixture errors that intentionally raise exceptions ## Best Practices 1. **Apply filters first, fingerprint second.** Reversing this re-introduces the v1 false-positive problem. 2. **Don't over-strip.** S1-S7 should preserve enough signature to distinguish two real failures (e.g., `command-not-found-python` and `command-not-found-cwebp` should remain distinct). 3. **Promote, don't stall.** count==2 means an entry leaves the `watching` state. The 4-month silence under v1 (Dec 2025 – Apr 2026, IMP-0000) was caused by skipping this step. 4. **Cite contexts in pain points.** Include both `contexts[0]` and `contexts[1]` in the markdown so the proposal is actionable, not abstract. ## Integration Points ### With Pain Point Manager - AI Pain Points use the `AI-PAIN-XXXX` id namespace. - Cross-reference with regular pain points for prioritization. ### With Memory System - Once a skill is created from a pain point, log a trajectory entry in `~/.claude-desktop/memory/domains/skill-evolution.md`. ### With Skill Improvement Log - Every skill update derived from a pain point gets an `IMP-XXXX` entry. Improvements to *this* skill (the actuator) are logged the same way. ## Skill Metadata **Version:** 2.2.0 **Created:** 2025-12-31 **Last revised:** 2026-05-04 (IMP-claudekit-aiel-0003 — F4 pytest carve-out + S8 pytest stripping rule) **Category:** Self-Improvement & Error Handling **Integration:** Pain Point Manager, Memory System **Maintenance**: Weekly error review, Monthly cleanup **Eval harness:** `.claude/skills/ai-error-learner/evals/` ## Changelog ### v2.2.0 (2026-05-04) — IMP-claudekit-aiel-0003 (applied) - F4 carve-out: pytest banners (`test session starts`, `ERRORS`, `FAILURES`, `short test summary info`) are NO LONGER suppressed by F4. Generic `VALIDATION/REPORT/AUDIT/BUILD` headers still suppressed. - New stripping rule S8: pytest-banner contexts are stripped to `pytest-timeout` / `pytest-failed` / `pytest-error` / `pytest-test-session` based on `error_types`, instead of falling through to the generic `bash-` bucket. - Recovers labeled rank 19 (pytest timeout, truth=partial). Recall 0.9048 → 0.9524 (+0.0476). Composite 0.553 → 0.5573 (+0.0043). - The companion F3 relaxation (admit per-route 5xx fingerprints when `error_types` carries a true 5xx code) was prototyped and **rejected**: under the current rubric, the partial=0.5 weight in precision combined with cluster-06's `filtered_or_per_route` grading produces a net composite drag (-0.0005 combined, -0.0057 F3-only) that exceeds the +0.0238 recall gain on rank 10. Documented for future re-evaluation if the rubric is rebalanced or if the access-log per-route splitting can produce ≥2 distinct stripped keys among cluster-06 survivors. ### v2.1.0 (2026-05-04) — IMP-claudekit-aiel-0002 (applied) - Added Detection False-Positive Filter F7 (data-listing/INFO-batch/wrapper-fallthrough/doc-prose suppressor) — paired with executor.py change to encode F7a-F7e regex set in `passes_filters()` - Eliminates the residual 4 false-positive labels under the 50-entry hand-graded set: precision 0.6786 → 0.95 (+0.27), recall preserved at 0.9048 - Composite score 0.515 → 0.553 (+0.038) - Re-frames F7's hypothesis from "documentation/example error references" (IMP-claudekit-aiel-0001 candidate) to "non-failing data/listing contexts" after empirical analysis of which labels survived F1-F6 ### v2.0.0 (2026-04-29) — IMP-0002 (Kenny approved) - Added Detection False-Positive Filters (F1-F6) — addresses 82% false-positive rate measured at baseline - Added Fingerprint Stripping rules (S1-S7) — collapses semantically-identical fingerprints - Made promotion at count==2 unambiguously mandatory; clarified the actuator step - Migration logic for v1 → v2 fingerprints ### v1.0.0 (2025-12-31) - Initial release. Detection sensor only; actuator not exercised.