---
name: ax
description: Audit agent-facing docs, hooks, skills, subagents, rules, permissions, and CLI surfaces against AX principles. Maps each finding to the right Claude Code mechanism (not just CLAUDE.md) and A/B tests high-impact doc fixes with sub-agents. Use when agent behavior is wrong due to missing/unclear docs, poor CLI ergonomics, or misconfigured automation.
argument-hint: "[problem description]"
disable-model-invocation: true
---

# AX — Agent Experience Audit & Test

Audit the full Claude Code configuration surface — documentation, hooks, skills, subagents, rules, MCP servers, permissions, and CLI tools — against 8 AX principles. For each finding, recommend the **right mechanism** to fix it. A/B test high-impact doc fixes with Haiku sub-agents. Only validated improvements make the final report.

## Arguments

- `$ARGUMENTS` — a description of the AX problem (e.g., "agents keep running the wrong build command"). If empty, run a general audit.

## Workflow

### Phase 1: AUDIT — Discover, ground-truth, and score

#### 1a. Establish ground truth

Derive the canonical workflow from automation sources:
- CI pipelines (`.github/workflows/*`, `.gitlab-ci.yml`, etc.)
- `Taskfile.yml`, `Makefile`, `justfile`
- `pyproject.toml` / `package.json` scripts
- `Dockerfile`, `docker-compose.yml`, `devcontainer.json`
- `.env.example`, lock files (`uv.lock`, `pnpm-lock.yaml`, `poetry.lock`)

Ground truth is authoritative. If docs and automation disagree, the fix aligns docs with automation.

#### 1b. Inventory agent-facing surfaces

Discover **all** Claude Code configuration, not just markdown files.

**Documentation layer:**
- `CLAUDE.md` at repo root, `.claude/CLAUDE.md`, `~/.claude/CLAUDE.md`
- Managed policy CLAUDE.md: `/Library/Application Support/ClaudeCode/CLAUDE.md` (macOS), `/etc/claude-code/CLAUDE.md` (Linux), `C:\Program Files\ClaudeCode\CLAUDE.md` (Windows)
- Check for `@path` imports in CLAUDE.md (e.g., `@README.md`, `@docs/conventions.md`) — these expand at load time
- `AGENTS.md`, `README.md`
- `REVIEW.md` at repo root (review-only guidance for automated code review — separate from CLAUDE.md)
- `.claude/rules/*.md` (check for `paths` frontmatter scoping)
- `.claude/output-styles/*.md` (custom output styles that modify the system prompt)

**Automation layer:**
- `.claude/settings.json` and `.claude/settings.local.json` — hooks, permissions, sandbox config, env vars
- `~/.claude/settings.json` — user-level hooks and permissions
- Hooks: list all configured hooks, their events, matchers, and types (command/prompt/agent/http)
- Sandbox config: check `sandbox.enabled`, `sandbox.filesystem`, `sandbox.network` settings
- `claudeMdExcludes` setting (for monorepos — excludes irrelevant CLAUDE.md files from other teams)
- Model configuration: `model`, `availableModels`, `effortLevel`, `opusplan` in settings
- Server-managed settings (enterprise): check if org uses server-delivered or endpoint-managed settings
- Managed settings files: `managed-settings.json`, macOS MDM (`com.anthropic.claudecode`), Windows registry (`HKLM\SOFTWARE\Policies\ClaudeCode`)
- `CLAUDE_CODE_SHELL_PREFIX` — command prefix for auditing/logging all Bash commands

**Skills, commands & agents layer:**
- `.claude/skills/*/SKILL.md` — project skills (check frontmatter: `context`, `disable-model-invocation`, `allowed-tools`, `agent`)
- `~/.claude/skills/*/SKILL.md` — user skills
- `.claude/commands/*.md` — project commands (legacy skill format, still works — same as skills but no directory for supporting files, no frontmatter features like `context: fork` or `disable-model-invocation`)
- `~/.claude/commands/*.md` — user commands
- `.claude/agents/*.md` — project subagents (check: `tools`, `model`, `hooks`, `memory`, `skills`, `isolation: worktree`)
- `~/.claude/agents/*.md` — user subagents
- Auto memory: `~/.claude/projects/<project>/memory/MEMORY.md` and topic files — check for staleness, contradictions with CLAUDE.md, and bloat (>200 lines in MEMORY.md)

If commands exist alongside skills, flag potential naming conflicts (a skill and command with the same name — skill takes precedence). Recommend migrating commands to skills when they would benefit from: supporting files, `context: fork`, `disable-model-invocation`, `allowed-tools`, or `paths`-scoped loading.

**Integration layer:**
- `.mcp.json` — project MCP servers
- MCP servers in `~/.claude.json`
- Installed plugins (check `.claude-plugin/plugin.json` directories)
- LSP servers provided by plugins (code intelligence — go-to-definition, diagnostics, completions)
- GitHub Actions integration: `.github/workflows/` using `anthropics/claude-code-action` (automated PR review, issue triage)
- GitLab CI/CD integration: `.gitlab-ci.yml` with Claude Code jobs (MR creation, automated implementation)
- `REVIEW.md` at repo root — review-only guidance consumed by `claude-code-action` in code review mode
- Devcontainer: `.devcontainer/devcontainer.json`, `Dockerfile`, `init-firewall.sh` — secure sandboxed environments
- Agent teams: check `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS` in settings, `teammateMode` config
- Worktrees: `.claude/worktrees/` directory, subagent `isolation: worktree` config
- Scheduled tasks: any `/loop` or cron-based recurring automation patterns

**Build/setup context:**
- `Taskfile.yml`, `Makefile`, `pyproject.toml`, `package.json`
- `scripts/*`, `.github/workflows/*`

If `$ARGUMENTS` is provided, focus on surfaces relevant to the described problem.

#### 1c. Enumerate CLI surfaces

Discover CLIs that agents interact with:
- **Sources**: commands in CLAUDE.md/AGENTS.md code blocks, Taskfile/Makefile task bodies, pyproject.toml `[project.scripts]`, package.json `scripts`, executables in `scripts/`
- **Introspection**: run `<tool> --help` recursively (cap at 3 levels depth) to build a surface map:
  - Subcommand tree, flags (required vs optional)
  - Output format flags (`--output`, `--format`, `-o json`)
  - Safety flags (`--dry-run`, `-y`, `--force`)
  - Deprecated aliases

#### 1d. Score against AX principles

Evaluate **all surfaces** (not just docs) against 8 principles. Assign **PASS**, **WARN**, or **FAIL** per principle, per surface.

**The 8 AX Principles:**

| # | Principle | FAIL when... |
|---|-----------|--------------|
| 1 | Explicitness over convention | A non-standard workflow isn't called out explicitly |
| 2 | Fail fast with clear recovery | Errors lack concrete fix commands OR success signals |
| 3 | Minimize context rot | CLAUDE.md or rules add tokens that don't earn their keep |
| 4 | Structured over unstructured | Important info is buried in prose instead of headers, tables, or code blocks |
| 5 | Consistent patterns | Naming or formatting conventions shift within or across docs |
| 6 | Complete context at point of need | Critical runnable commands are missing at the point they're needed |
| 7 | Guard rails over documentation | Says "don't do X" but X would succeed silently — a hook, permission rule, or validation would be better |
| 8 | CI parity / single source of truth | Docs diverge from CI/automation, or the same info is maintained in multiple places |

**CLI-specific checks:**

| Check | Maps to | FAIL when... |
|-------|---------|--------------|
| Machine-readable output | #4 Structured | No `--output=json` (or equivalent) on commands returning structured data |
| Long-form input path | #6 Complete context | Command accepts freeform text but has no `--file` flag |
| Deprecation surfacing | #8 CI parity | Deprecated aliases aren't surfaced in `--help`, or docs still reference them |

**Configuration-specific checks:**

| Check | Maps to | FAIL when... |
|-------|---------|--------------|
| Hook coverage | #7 Guard rails | Repeated "don't do X" in CLAUDE.md but no PreToolUse hook to enforce it |
| Permission gaps | #7 Guard rails | Dangerous commands are not denied in permissions, relying on CLAUDE.md warnings |
| Skill sprawl | #3 Context rot | Knowledge that could be a demand-loaded skill is baked into CLAUDE.md |
| Rule scoping | #3 Context rot | File-type-specific instructions in CLAUDE.md instead of path-scoped `.claude/rules/*.md` |
| Subagent opportunities | #3 Context rot | Recurring complex tasks that could benefit from isolated context are done inline |
| MCP gaps | #6 Complete context | Docs tell agents to use external services manually when an MCP server could provide direct access |
| Compaction survival | #2 Fail fast | Critical instructions are only in conversation context, not in CLAUDE.md or rules |
| Command→skill migration | #3 Context rot | `.claude/commands/` files that would benefit from skill features (supporting files, `context: fork`, `disable-model-invocation`, `allowed-tools`) |
| Sandbox gaps | #7 Guard rails | Bash commands can write outside project or access network, but no sandbox config exists |
| `@path` bloat | #3 Context rot | CLAUDE.md inlines large sections that could use `@path` imports for progressive disclosure |
| CI review gaps | #8 CI parity | No `claude-code-action` in GitHub workflows (or Claude jobs in `.gitlab-ci.yml`), or REVIEW.md missing when automated code review is active |
| Auto memory staleness | #8 CI parity | Auto memory contains outdated info that contradicts current CLAUDE.md or automation |
| Model misconfiguration | #1 Explicitness | No `availableModels` restriction in enterprise settings, or model/effort level not documented for the team |
| Managed settings gaps | #7 Guard rails | Enterprise org has no managed settings, server-managed settings, or managed CLAUDE.md for org-wide policies |
| Shell audit logging | #7 Guard rails | No `CLAUDE_CODE_SHELL_PREFIX` for command auditing in sensitive environments |

**Tension resolution rules:**
- **#3 vs #6**: prefer progressive disclosure — inline the minimum runnable commands + one-line explanation; link or import deeper detail via `@path` or skills.
- **#7 (guard rails)**: propose the strongest available mechanism. Preference order: managed settings (cannot be overridden) > sandbox (OS-level) > hook (deterministic) > permission deny rule > CLAUDE.md instruction.

### Phase 2: PROPOSE — Select mechanism and draft fixes

For each WARN or FAIL, select the **right Claude Code mechanism** and draft a concrete fix.

#### Mechanism selection decision tree

For each finding, walk this tree top-to-bottom. Use the **first mechanism that fits**.

| If the finding is... | Use this mechanism | Why |
|---|---|---|
| Block forbidden actions or validate commands | **PreToolUse hook** (parse stdin JSON, exit 2 to block) | Deterministic enforcement, zero reliance on LLM compliance |
| Dangerous command that should never run | **Permission deny rule** in `.claude/settings.json` | Simpler than a hook when the pattern is a static tool match |
| Auto-format, auto-lint, auto-test after edits or at stop | **PostToolUse/Stop hook** (command type) | Runs every time, no LLM decision needed |
| File-type-specific coding convention | **`.claude/rules/*.md`** with `paths` frontmatter | Loads only when working with matching files, saves context |
| Repeatable workflow or reference knowledge | **Skill** (`.claude/skills/<name>/SKILL.md`, `disable-model-invocation` for user-only) | Invocable on demand or auto-loaded when relevant, doesn't bloat every session |
| Complex task needing isolation (add `isolation: worktree` if editing shared files) | **Subagent** (`.claude/agents/<name>.md`) | Own context window, own tool restrictions, returns summary |
| External service interaction | **MCP server** | Direct tool access, no shell-escaping or manual API calls |
| Critical context that must survive compaction | **CLAUDE.md** or **SessionStart hook** with `compact` matcher | Re-injected after every compaction |
| Quality gate before Claude stops | **Stop hook** (prompt or agent type) | Verifies completeness before Claude declares "done" |
| Universal project convention | **CLAUDE.md** (keep it concise, <200 lines) | Loaded every session, but only for always-applicable rules |
| Restrict filesystem/network access | **Sandbox configuration** in settings | OS-level enforcement (Seatbelt/bubblewrap), cannot be bypassed by LLM |
| Code review needs different guidance than interactive use | **REVIEW.md** at repo root | Consumed by `claude-code-action` in review mode, separate from CLAUDE.md |
| Automated PR/MR review or issue triage in CI | **GitHub Action or GitLab CI/CD job** | Runs Claude in CI with optional REVIEW.md guidance |
| Large doc section only relevant sometimes | **`@path` import** in CLAUDE.md | Loads at session start but keeps CLAUDE.md itself readable and maintainable |
| Monorepo with irrelevant CLAUDE.md from other teams | **`claudeMdExcludes`** setting | Prevents loading CLAUDE.md files that add noise from other project areas |
| Complex parallel work requiring inter-agent communication | **Agent team** with `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS` | Teammates coordinate via shared task list and direct messaging |
| Recurring monitoring or polling task | **Scheduled task** (`/loop` or `CronCreate`) | Session-scoped automation for builds, deploys, PR checks |
| Audit/log all commands an agent runs | **`CLAUDE_CODE_SHELL_PREFIX`** env var | Wraps every Bash command with a logging prefix script |
| Organization-wide policy or model restrictions | **Managed settings** (server-managed or endpoint-managed, cannot be overridden) | Enforces org-wide policies like permission deny lists, model restrictions |
| Consistent secure development environment | **Devcontainer** with firewall rules | Sandboxed container with whitelisted network access |
| Bundle of skills + hooks + agents for distribution | **Plugin** | Package and share across projects via marketplace |

**Each fix must include:**
- Which principle it addresses
- The selected mechanism and why
- The exact implementation (file path + content for hooks/skills/rules, or diff for docs)
- Estimated token impact (for doc changes)
- For hooks: the JSON config to add to settings, and any script files needed

**Fix classification:**
- **Doc-side fix**: changes to CLAUDE.md, rules, README — goes through A/B testing
- **Mechanism fix**: new hook, skill, subagent, permission rule, or MCP config — static recommendation with exact implementation
- **CLI-side finding**: problem with the CLI itself — static recommendation with interim workaround

**Triage gate:**
- **High-impact doc fixes**: change what commands an agent runs, what order it follows, or error recovery. A/B tested.
- **High-impact mechanism fixes**: replace documentation-only guardrails with deterministic enforcement. Static recommendation.
- **Low-impact**: formatting, wording, structural cleanup. Bundled in report without testing.

### Phase 3: TEST — A/B test high-impact doc fixes

For each **high-impact doc-side fix**, run a docs-only A/B test.

Mechanism fixes (hooks, skills, subagents, permissions, MCP) are **not A/B tested** — they are deterministic and don't depend on LLM interpretation.

#### Test setup

1. **Derive a test task** from the problem description or the doc's subject matter.

   For doc-side CLI fixes, use invocation-correctness tasks:
   - "Delete cluster `alice-test` in namespace `training`. What exact command?"
   - "Deploy a training run with `base.yaml`. What's the exact invocation?"

2. **Prepare two doc bundles:**
   - **Agent A**: original docs with THIS fix applied
   - **Agent B**: original docs unchanged

3. **Spawn two sub-agents** using the Agent tool:
   - `subagent_type: "general-purpose"`, `model: "haiku"`, `max_turns: 5`
   - Both get: the doc bundle injected into their prompt + the same test task
   - **No repo access** — sub-agents must rely solely on the injected docs
   - Both must produce: exact commands (with flags, working directory), expected success signals, and one likely failure + recovery path
   - **Launch both agents in parallel** (two Agent calls in one message)

4. **Evaluate results** against ground truth:
   - Correctness vs ground truth (right commands, right ordering)
   - Confidence (does the agent hedge or branch unnecessarily?)
   - Completeness (success signals, failure anticipation)
   - Clear attributable difference to the doc change

5. **Verdict per fix:**
   - **VALIDATED** — Agent A clearly outperformed Agent B
   - **INCONCLUSIVE** — No measurable difference
   - **REJECTED** — Agent A performed worse or added noise

#### Integration test

After individual A/B tests, run one final A/B test with all VALIDATED doc fixes applied together vs original docs.

### Phase 4: REPORT — Output structured results

```
# AX Audit Report

**Project:** <name>  |  **Surfaces audited:** <count>  |  **Est. doc tokens:** <total>

## Ground Truth

| Step | Canonical command | Source |
|------|-------------------|--------|
| Build | <command> | <Taskfile / CI / ...> |
| Test | <command> | ... |

## Configuration Inventory

| Surface | Files found | Status |
|---------|-------------|--------|
| CLAUDE.md | <paths> | <token count, line count> |
| .claude/rules/ | <N files, paths scoping?> | <details> |
| Output styles | <N files in .claude/output-styles/> | <details> |
| Hooks | <N hooks across N events> | <details> |
| Skills | <N project, N user> | <details> |
| Commands | <N project, N user> | <migration candidates?> |
| Subagents | <N project, N user> | <details> |
| MCP servers | <N configured> | <details> |
| Permissions | <allow/deny rules> | <details> |
| Plugins | <N installed> | <details> |
| Sandbox | <enabled/disabled> | <filesystem/network rules> |
| REVIEW.md | <exists/missing> | <details> |
| GitHub Actions | <claude-code-action present?> | <details> |
| `@path` imports | <N imports in CLAUDE.md> | <details> |
| `claudeMdExcludes` | <configured/not> | <details> |
| Auto memory | <exists/not, line count> | <staleness, contradictions?> |
| Model config | <model, availableModels, effortLevel> | <details> |
| Managed settings | <server/endpoint/none> | <what's enforced> |
| Agent teams | <enabled/disabled> | <teammateMode> |
| Devcontainer | <exists/not> | <firewall config> |
| Worktrees | <.gitignore'd?> | <subagent isolation config> |

## Scorecard

| # | Principle | Rating | Detail |
|---|-----------|--------|--------|
| 1 | Explicitness over convention | PASS/WARN/FAIL | one-line explanation |
| 2 | Fail fast with clear recovery | ... | ... |
| 3 | Minimize context rot | ... | ... |
| 4 | Structured over unstructured | ... | ... |
| 5 | Consistent patterns | ... | ... |
| 6 | Complete context at point of need | ... | ... |
| 7 | Guard rails over documentation | ... | ... |
| 8 | CI parity / single source of truth | ... | ... |

## Per-Surface Findings

| Surface | Key issues | Recommended mechanism |
|---------|-----------|----------------------|
| ... | ... | ... |

## CLI Surface Inventory

| Tool | Subcommands audited | `--output=json` | `--dry-run` | Deprecations found |
|------|--------------------:|:---------------:|:-----------:|:------------------:|
| <tool> | N | yes/no | yes/no | N |

## Mechanism Recommendations

For each recommendation, use the template and mechanism-specific fields below.

#### <PREFIX><N>. <title>
- **Principle:** #N — <name>
- **Mechanism:** <type>
- <mechanism-specific fields from table>
- **Replaces:** <what this makes unnecessary>
- **Implementation:** <file path + exact content>

| Mechanism | Prefix | Required fields |
|-----------|--------|----------------|
| Hook | H | Event, Matcher, Type, Settings JSON, Script |
| Skill | S | Path, Frontmatter, SKILL.md content |
| Subagent | A | Path, Config (model/tools/hooks/memory/skills/isolation) |
| Rule | R | Path, Paths scope (globs), Content with frontmatter |
| Permission | P | Rule pattern (allow/ask/deny), Settings JSON |
| MCP server | M | Server + transport, Install command |
| Sandbox | SB | Setting key, Settings JSON |
| REVIEW.md / CI | CI | File path, Content |
| Agent team | AT | Team structure, Settings |
| Managed settings | MS | Scope, Settings JSON, Policy enforced |
| Devcontainer | DC | Files, Firewall rules |
| Plugin | PL | Name + source, What it bundles, Install command |
| @path import | AP | CLAUDE.md line, Target file, Tokens moved |
| claudeMdExcludes | CE | Exclude patterns, Settings JSON |
| Scheduled task | ST | Type (/loop or CronCreate), Schedule, Command |
| Shell prefix | SP | Env var value, Script content |

## CLI Findings (static — not A/B tested)

### CLI-side recommendations

#### C1. <title>
- **Principle:** #N — <name>
- **Current behavior:** <what happens now>
- **Recommended CLI change:** <what the tool maintainer should change>
- **Interim doc workaround:** <what to add until the CLI is fixed>

## Validated Doc Recommendations (A/B tested)

### D1. <title>
- **Principle:** #N — <name>
- **File:** <path>
- **Change:**
  ```diff
  <exact diff>
  ```
- **A/B Result:** VALIDATED — Agent A did X; Agent B failed with Y
- **Token impact:** +N / -N tokens

## Low-Impact Changes (not A/B tested)

| Change | Principle | Mechanism | Token impact |
|--------|-----------|-----------|--------------|
| <description> | #N | <doc edit / rule / etc.> | +/- N |

## Rejected / Inconclusive Doc Changes

| Change | Verdict | Why |
|--------|---------|-----|
| <description> | INCONCLUSIVE/REJECTED | <explanation> |

## Integration Test Result

<Did combined VALIDATED doc fixes hold up? Any interaction effects?>

## Token Budget

| Metric | Value |
|--------|-------|
| Current CLAUDE.md tokens | ~N |
| After recommendations | ~M |
| Content moved to skills/rules | ~K tokens |
| Delta (net) | +/- N tokens |
| CLAUDE.md density | % lines that are runnable commands, decision rules, or structured references |
```

## Important Notes

### Constraints
- This skill is **read-only** — it never modifies files, only reports recommendations
- Be conservative with VALIDATED — only mark a doc fix as validated when there is a **clear, attributable difference** between Agent A and Agent B

### Output quality
- Hooks: provide complete settings JSON, matcher, type, and scripts
- Skills: provide complete SKILL.md with frontmatter (`disable-model-invocation`, `context: fork`, `allowed-tools` as appropriate)
- Rules: always include `paths` frontmatter when file-type-specific
- Permission syntax: `allow`/`ask`/`deny` with patterns (e.g., `Bash(npm run *)`, `Read(./.env)`, `WebFetch(domain:example.com)`)

### Mechanism guidance
- **REVIEW.md** is separate from CLAUDE.md — only for code review via `claude-code-action` or `/review`
- **`claudeMdExcludes`** essential for monorepos — prevents loading irrelevant CLAUDE.md files from other teams
- **Managed policy CLAUDE.md** + managed settings cannot be overridden by local config
- **Auto memory**: audit for staleness, contradictions with CLAUDE.md, and >200 line MEMORY.md truncation
- **Agent teams**: use `TeammateIdle`/`TaskCompleted` hooks for quality gates
- **Devcontainers**: right solution when `--dangerously-skip-permissions` is needed safely
- **ConfigChange hooks**: detect runtime setting modifications in enterprise environments