--- description: Set up LLM workflow discipline for projects that use AI / LLM calls in production or eval suites. Authors an eval-cost-watcher agent that projects token cost BEFORE regression evals run, plus an AI-workflow-discipline rule covering mock-mode placement, fixture freshness, and multi-stage cost accumulation. Optionally a small eval-regression skill stub. Invoke /dotclaude:ai-workflow only when the project actually has AI workflows. --- # `/dotclaude:ai-workflow` — LLM workflow + cost discipline kit You are setting up the AI-workflow safety layer — the discipline that prevents night-long eval runs from producing four-figure surprise bills, and that catches the mock-mode-bypass class of bug at the source rather than in the invoice. The eval-cost-watcher agent is the showpiece. It reads the diff, counts fixtures, projects cost ranges, and suggests cheaper alternatives — all BEFORE the eval runs. ## Phase 1 — Read the project's AI shape Before any question: 1. **Detect AI SDK presence** — does the project actually have LLM calls? ```bash grep -rln "@anthropic-ai/sdk\|openai\|@ai-sdk\|@google/generative-ai\|gemini\|claude" \ package.json Cargo.toml pyproject.toml requirements.txt 2>/dev/null | head grep -rn "from anthropic\|from openai\|import.*@anthropic" \ --include="*.py" --include="*.ts" --include="*.js" . 2>/dev/null | head -10 ``` If NO AI dependencies, STOP. Report applicability check failed. Tell the user this skill doesn't apply. 2. **AI workflow directories** — find where prompts / configs / fixtures live: ```bash find . -type d \( -name "ai" -o -name "llm" -o -name "prompts" -o -name "evals" \ -o -name "workflows" -o -name "agents" -o -name "fixtures" \) \ -not -path "*/node_modules/*" 2>/dev/null | head find . -path ./node_modules -prune -o \ \( -name "*.prompt.*" -o -name "*system-prompt*" -o -name "*instruction*" \) \ -print 2>/dev/null | head ``` 3. **Eval suite** — find the eval entry points: ```bash cat package.json 2>/dev/null | grep -A 2 '"scripts"' | grep -E "eval|test:ai|test:llm" find . -name "*.eval.*" -o -name "*eval*.py" -o -name "*eval*.ts" \ -not -path "*/node_modules/*" 2>/dev/null | head ls evals/ tests/eval/ tests/llm/ 2>/dev/null ``` 4. **Model identifiers** — read the config: ```bash grep -rEn "MODEL_NAME|model.*=.*\"|model: ['\"]|MODEL_ID|model_id" \ --include="*.ts" --include="*.py" --include="*.json" --include="*.yaml" . \ 2>/dev/null | head -30 ``` Note which model tiers the project uses (frontier-large vs frontier-mid vs frontier-fast). Each tier swap is a 5-10× cost multiplier. 5. **Mock mode** — does one exist? ```bash grep -rn "MOCK_MODE\|MOCK_AI\|mockAi\|dryRun\|mock_response\|AI_MOCK\|FIXTURE_MODE" \ --include="*.ts" --include="*.py" --include="*.js" . 2>/dev/null | head ``` If yes, note WHERE the mock-mode check fires. Then trace whether ALL entry points to the LLM workflow pass through the mock-aware layer, or whether some bypass it (the war-story pattern at `../examples/the-test-passed-for-the-wrong-reason.md`). 6. **Fixture count + cascade depth** — gauge the cost scale: ```bash find . -path "*fixtures*" -type f 2>/dev/null | wc -l # Read 1-2 workflow files to count cascade stages find . -name "*workflow*.ts" -o -name "*pipeline*.ts" -o -name "*cascade*.ts" \ -not -path "*/node_modules/*" 2>/dev/null | head ``` 7. **Recent AI-cost-related commits**: ```bash git log --oneline --grep="cost\|token\|eval\|fixture\|model\|mock" -20 ``` Build mental model of: what AI architecture (single call / multi-stage cascade / agent loop), what models per stage, what mock-mode coverage looks like, what the eval cost scales with. ## Phase 2 — Interview Open `interview.md` (same directory). 3-4 questions. Adaptive — skip what Phase 1 already answered. - **Workflow shape** — single call / multi-stage / eval harness with fixtures? Production or dev-only? - **Cost ceiling** — what's a "this eval got too expensive" threshold per iteration? - **Models in use** — providers + tiers per stage (often discoverable from Phase 1; confirm). - **Past cost surprises** — drives the watcher's safety thresholds. ## Phase 3 — Read the principles Read these from `../../principles/` SELECTIVELY: **Always read** (this kit's core): - `ai-cost-monitoring.md` — the five-step projection methodology **Cross-reference** (link from the watcher's body, do not re-author): - `code-review.md` — mock-mode bypass is a code-review pattern; the watcher's mock-coverage check overlaps - `pre-flight.md` — pre-flight should ask about cost impact for AI-workflow changes **Read the war-story example** — load-bearing for this skill: - `../examples/the-test-passed-for-the-wrong-reason.md` — the canonical mock-mode-bypass failure. The watcher's mock-coverage check exists because of this class of bug; the AI-workflow-discipline rule references it. ## Phase 4 — Author the kit ### Agents (in `.claude-staging/agents/`) - **`eval-cost-watcher.md`** — the pre-eval cost projector - Frontmatter: `description:` — derived from `ai-cost-monitoring.md` principle, tuned to THE user's stack - Tool surface: `Read, Grep, Glob, Bash` (for `git diff`, file counting, fixture counting). NO `Edit`, NO `Write`, NO ability to actually run the eval — projection is the role, NOT execution. - Body sections: - The five-step methodology (classify-diff / read-model+temperature / count-fixtures / estimate-tokens / produce-projection) - **Project-specific model-tier map** — concrete model identifiers from Phase 1 + their tier classification (frontier-large = ``, frontier-mid = ``, frontier-fast = ``). With a "verify pricing at provider's docs — drifts" note. - **Project-specific cascade architecture** — list the stages, the passes per stage, the fixture sets per stage. If the workflow has shape "Stage A (research, 1 pass) → Stage B (synthesis, 2 passes) → Stage C (critic, 1 pass) per fixture × 20 fixtures = 80 calls per full eval," encode that. - **Mock-mode coverage map** — for each entry point to the workflow, note whether it respects mock mode. Explicitly flag any that BYPASS it (the war-story pattern). This is the most-load-bearing section: a bypass shipped to production is a four-figure surprise bill waiting to happen. - **Cost ceiling + alarm threshold** — derived from the user's interview answer to AI2. The watcher recommends NOT running the eval if the projection exceeds the ceiling without explicit override. - Projection-format template (affected stages / model changes / cost range / cheaper alternatives / recommendation) - "I do not run the eval" constraint explicit - Model: mid-tier reasoning model. The work is diff-classification + arithmetic; doesn't require the heaviest reasoning. Save top-tier for code-review / pre-flight. ### Rules (in `.claude-staging/rules/`) - **`ai-workflow-discipline.md`** (small rule; author only if helpful for the project's shape) - **Mock-mode placement** — single rule: "mock-mode flag is a property of the WORKFLOW, not the route." Cross-reference the war story. Specifically prohibit the pattern where the mock check lives only in the HTTP layer; require the workflow's entry function to check. - **Fixture freshness** — when prompts change, fixtures captured against old prompts are stale and produce false-positive eval results. Encode the convention for re-capturing fixtures (manual? scripted? what's the command?). - **Multi-stage cost accumulation** — cascade depth × passes × fixtures × output tokens. The cost-watcher does the math; the rule reminds the user that this math compounds. - **Model-swap discipline** — tier swaps (frontier-fast → frontier-large) need explicit justification + cost projection. The rule says: "Before swapping the model on any stage, run the watcher." - Skip authoring this rule entirely if the project's AI usage is minimal (single call, no cascade, no eval suite) — the watcher carries the load alone. ### Skills (in `.claude-staging/skills/`) - **`eval-regression/SKILL.md`** (stub — author only if the project has a meaningful eval suite) - User-invocable (`/eval-regression`) - Wraps the watcher + the actual eval-run command + the result-comparison step - Workflow: (1) run cost projection via watcher, (2) prompt user to confirm, (3) run eval against fixture set, (4) compare results to baseline, (5) flag regressions - This is a stub if the project doesn't yet have a structured baseline-comparison; the user fills it in as the eval suite matures. - Skip entirely if the project has no eval suite (just inline AI calls without a regression harness). ### Hooks The AI-workflow kit doesn't typically ship hooks. Edit-time hooks would have to read the diff and project cost on EVERY edit; the agent doing that on demand is the right shape. One exception: a pre-eval-run wrapper script could invoke the watcher automatically; that's the `eval-regression` skill's job, not a hook. ## Phase 5 — Stage + present + commit ### Staging Write everything to `.claude-staging/` first, organized by artifact type. ### Present Walk the user through: 1. **The kit overview** — what landed (typically 1 agent + maybe 1 rule + maybe 1 skill) 2. **Top 2-3 highlight artifacts** — concrete reasoning. NOT "I added a cost-watcher" but: "Your workflow has stages × passes × fixtures = calls per full eval. Model tier per stage: . Currently your bill for one full eval projects at $-$. The watcher will run pre-eval and bail if any single run exceeds your $ threshold." 3. **Mock-mode coverage** — call out explicitly which entry points respect mock mode and which BYPASS it. If you found a bypass, surface it: "Your dev cron at `` calls the workflow directly, bypassing the mock check at ``. The war-story pattern. The watcher will flag any change that propagates work through the bypassed path." 4. **What got SKIPPED** — and why. "Skipped `eval-regression` skill because no eval suite found — re-run this skill when you build one." 5. **Model + token-cost note** — eval-cost-watcher uses mid-tier reasoning model and is cheap to invoke (compared to the costs it projects). Worst case it costs $0.10 to prevent a $40 surprise eval — favorable asymmetry. ### Approve → commit After explicit user approval, move `.claude-staging/` → `.claude/` and commit with structured message: ``` feat(.claude): AI workflow discipline (dotclaude:ai-workflow) Authored: - agents: eval-cost-watcher - rules: [ai-workflow-discipline] - skills: [eval-regression] Workflow shape: Models in use: Mock-mode coverage: Per-eval cost projection: $-$ Alarm ceiling: $ ``` ## Non-negotiable rules for this flow 1. **Applicability gate is HARD.** If Phase 1 shows no AI SDK dependencies, this skill DOES NOT APPLY. Don't ship a phantom cost-watcher for a project with no AI workflow. Report applicability failure cleanly and stop. The user can re-invoke once AI lands in the codebase. 2. **The watcher does NOT run the eval.** Projection from source files, never from actual model calls. The whole point is to project BEFORE spending; if the watcher itself burns calls to estimate, the asymmetry collapses. Tool surface excludes anything that could invoke the workflow. 3. **Mock-mode coverage is the highest-priority finding.** A workflow that bypasses mock mode is a four-figure surprise bill waiting to fire. If Phase 1 surfaces a bypass, the watcher's body should treat it as an explicit ALWAYS-FLAG finding on any change affecting the bypassed path. Don't bury it; surface it. 4. **Pricing is in a maintained config, not hardcoded.** LLM provider prices change. The watcher's body should reference a maintained config (project-local `ai-pricing.json` or similar) with a "verify against provider's docs" note. NEVER hardcode numbers — they go stale and the watcher's projections become silently wrong. 5. **Projections are ranges, not points.** Different fixtures produce different output lengths. Different inputs produce different cache-hit behavior. The watcher reports `$low - $high` and acknowledges the gap. Single-point estimates that turn out to be 3× off train users to distrust the agent. 6. **Cheaper alternatives are mandatory output.** A projection alone says "this will cost $X." A projection + alternatives says "this will cost $X; here are three ways to get the same signal for less." Always provide the alternatives — subset of fixtures, mock mode (if applicable), skip a cascade stage, run only the affected stage. The user needs the option list to make the run/skip decision. 7. **Tier swaps are called out separately.** Swapping frontier-fast for frontier-large is a 5-10× multiplier on its own — the highest-impact category of change. Don't bury it inside a general projection; the watcher should have a dedicated "Model changes" section that flags swaps explicitly with the multiplier number. 8. **The watcher is cheap; the eval is expensive.** The watcher's own model is mid-tier and its context is bounded. If the watcher itself approaches the cost of the eval it's projecting, the whole abstraction is broken. Watch the watcher's spend during authoring — keep its prompt focused, its context tight, and its model choice modest.