--- name: audit-tests description: Audit a test suite for testing-discipline violations — tests that need live external processes or services, tests coupled to implementation details instead of observable behavior, behaviors that landed without a test, non-deterministic tests that race on timing or leak state between cases, and copy-paste test bloat that should be parametrized. Use when asked to audit tests, check test quality, verify coverage discipline, audit the test suite, or review whether a change is properly and maintainably tested. allowed-tools: Read, Grep, Glob, Bash context: fork --- Audit the test suite against the project's testing rules. The rulebook is the testing guidance in the project `CLAUDE.md` (and any doc it imports) — re-read it before judging. Read-only: report findings, do not fix. ## Scope Audit the test files. Cross-reference against the source modules to judge coverage. Run the project's test command once if you need to confirm the suite's current state, but the audit is about *discipline*, not just pass/fail. Two modes — pick from the request: - **Change audit** (default when there are uncommitted or unmerged changes): scope to the changed files (`git diff --name-only main...HEAD` plus the working tree), and focus on checks 3 and 4 (new behavior covered, tested at the right altitude). - **Full-suite audit** (when asked to audit the whole suite): run every check across all tests; in check 3 skip the `git diff` step and judge coverage behavior-by-behavior instead. ## What to check 1. **Right layer — pure logic runs without live dependencies.** Pure-logic tests must run with no live external dependency: no spawned process, no real external CLI, no service, no network. Scan tests for process spawning (`subprocess`, `Popen`, `exec`, or the language's equivalent), real-binary invocations, and network calls. If a test spawns a real process or hits a real service to exercise pure logic, the code is at the wrong layer — it should be refactored to be callable directly. If the project provides a sanctioned stand-in (mock/fixture), flag any test that reaches for the real dependency instead. 2. **Tests assert observable behavior, not implementation details.** Flag tests that reach into private attributes (`._foo`), assert mock call counts/order where a state or output assertion would do, or pin exact log strings / internal data shapes that aren't part of the contract. The question to ask: would this test break on a refactor that preserves behavior? If yes, flag it. 3. **Every behavior is covered.** In a change audit, every new behavior or code path in the diff must have an accompanying behavior test in the same diff — flag untested new behavior. In a full-suite audit, check each behavior of each source module has a test and flag the gaps. Coverage means behaviors covered, not a line percentage. 4. **Tests run at the right altitude.** Multi-step flows are driven through the real entry points against real state, not re-implemented in the test. Flag tests that duplicate production logic instead of calling it. 5. **Maintainable — no test explosion.** Test code obeys the same conventions as source: clean, DRY, intention-revealing names. Flag copy-paste test families that should collapse into one parametrized / table-driven test, and repeated arrangement that should be a shared fixture/builder. A suite several times the size of the code it covers is a smell — surface the duplication driving it. 6. **Deterministic — no timing, order, or shared-state coupling.** Tests must be fast and deterministic. Flag a real wait used to synchronize (`sleep`, `asyncio.sleep`, wall-clock polling for a condition), a test whose outcome depends on running after another, and module- or class-level mutable state that leaks between cases. Grep for `sleep`, shared globals, and mutable module/class-level collections. A test that races or only passes in a given order is a defect even when the suite is currently green. (A short bounded wait on a genuinely external async event may be unavoidable in the thin integration layer — judge whether the same logic could be driven deterministically instead.) ## Output Report findings grouped by check, highest severity first. For each: ``` [SEVERITY] — path/to/test_file:line (or src/file for a coverage gap) What: Why: Fix: ``` Severity: **HIGH** = test needs a live external process/service, spawns a real process for unit logic, new behavior shipped with no test, or a timing/order dependence that makes the suite actually flaky. **MEDIUM** = brittle implementation-detail coupling, copy-paste test explosion that should be parametrized, or a sleep/shared-state coupling that is non-deterministic but not yet flaky. **LOW** = minor / arguable. End with a one-line verdict: `CLEAN` if no HIGH/MEDIUM findings, otherwise a count per severity. Confirm each heuristic hit by reading the test before listing it.