--- name: audit-loop-run description: Use when asked to assess loop effectiveness, audit goal achievement, or detect phantom success. disable-model-invocation: true argument-hint: "[loop-name] [--tail N] [--no-rubric-audit] [--skip-issue-creation] [--auto]" model: sonnet allowed-tools: - Bash(ll-loop:*, ll-issues:*, git:*) - Read - Glob - Grep - Write - AskUserQuestion arguments: - name: loop_name description: Loop name to assess (optional — auto-selects most recent if omitted) required: false - name: tail description: Limit history events analyzed to the N most recent (default 200) required: false - name: no_rubric_audit description: Skip the LLM rubric-vs-description pass (cost gate) required: false - name: skip_issue_creation description: Skip issue creation entirely and exit cleanly after presenting proposals required: false - name: auto description: Non-interactive mode; suppress all AskUserQuestion calls and default to no for issue creation (implies --skip-issue-creation). Also activates when LL_NON_INTERACTIVE or DANGEROUSLY_SKIP_PERMISSIONS env vars are set, or when --dangerously-skip-permissions is in effect. required: false metadata: short-description: Use when asked to assess loop effectiveness, audit goal achievement, or detect p --- # Assess Loop Audit whether a configured loop's execution actually achieved its stated goal — checking artifact mutations, threshold contracts, structural defects (phantom convergence, degenerate gates, rubric drift, sub-loop verdict laundering), and producing ranked improvement proposals. --- ## Step 1: Resolve Loop Name If `loop_name` argument is provided, resolve the most recent run folder: ```bash ls -d .loops/.history/*-/ 2>/dev/null | sort | tail -1 ``` - If empty: report "No archived runs found for ``." and stop. - Otherwise: extract `LATEST_RUN_ID` (the compact timestamp prefix, e.g. `2026-03-19T204149`). Otherwise, enumerate candidate loops: ```bash ll-loop list --running --json ``` Filter to `status` one of `"running"`, `"interrupted"`, `"failed"`, `"timed_out"`, `"awaiting_continuation"`. Sort by `updated_at` descending. Note: `ll-loop list --running --json` output does **not** include `instance_id` — entries with the same `loop_name` are indistinguishable at this level. - **Zero candidates**: Report "No interrupted or running loops found. Specify a loop name explicitly." and stop. - **One candidate**: Select automatically and report. - **Two or more candidates with distinct `loop_name` values**: Use `AskUserQuestion` to let the user pick: ``` Multiple loops found. Select one to assess: [1] — — last updated [2] — — last updated ... ``` - **Two or more candidates sharing the same `loop_name`** (multiple instances): follow up with `ll-loop status --json` to retrieve per-instance detail (`instance_id`, `pid`, `log_file`, `events_file`), then use `AskUserQuestion` to present instance-level disambiguation: ``` Multiple instances of '' found. Select one to assess: [1] — — PID — last updated [2] — — PID — last updated ... ``` --- ## Step 2: Load Loop Definition and History Load the fully-materialized FSM: ```bash ll-loop show --resolved --json ``` This returns `FSMLoop.to_dict()` JSON with always-present keys `name`, `initial`, `states`, and conditionally `description`, `context` (threshold keys live here), `max_steps`, `parameters`, `commands`. Load the event history: ```bash ll-loop history [] --json --tail ``` If either command fails, report the error and stop. --- ## Step 3: Extract Success Contract From the FSM `context` flat dict, scan for threshold keys: - `target_pass_rate`, `pass_threshold`, `quality_threshold`, `readiness_threshold` - `outcome_threshold`, `reward_target`, `target_score`, `min_per_category`, `adversarial_cap` Also scan each state's `action` text and `evaluate.prompt` text for `${context.}` interpolation patterns to detect threshold references embedded in prompts. Build the **success contract**: list of `{key, value, source}` entries where `source` is `"context"`, `"action"`, or `"evaluate.prompt"`. If no contract entries are found, note: "No threshold contract detected — loop uses implicit success criteria." --- ## Step 4: Inspect Artifacts Identify artifact paths the loop touches. Look in: 1. `context.prompt_file`, `context.system_file`, `context.output_file` and similar path-like context keys 2. State `action` text for file path patterns (`prompts/`, `data/`, `.issues/`, `image.svg`, `manifest.json`, `examples.json`) For each identified artifact path, check mutation evidence: ```bash # Check if file was modified in recent git history git log --oneline -5 -- # Check current diff git diff HEAD -- ``` For issue-based loops, inspect frontmatter: ```bash ll-issues show --json ``` Also check in-memory captures in `.loops/.history/-/state.json` under `captured` dict (schema: `{state_name: {output, stderr, exit_code, duration_ms}}`). --- ## Step 5: Phase 1 — Fault Signals Re-use the history loaded in Step 2 to identify fault signals using the **fault-signal subset** of `/ll:debug-loop-run` Step 3 (the BUG-class anomalies that broke the run). Note: `/ll:debug-loop-run` Step 3 also classifies effectiveness signals (iter-1 convergence without apply, degenerate gate, stub action) — those are **out of scope for audit-loop-run Phase 1**, since this step only synthesizes fault evidence into the scorecard. Include the verbatim fault signal list in the scorecard output. Key signals to flag (fault subset only): - Action failures (`exit_code != 0`, non-intentional) - SIGKILL / FATAL_ERROR termination - Evaluate error termination (`evaluate.verdict == "error"` on the last evaluate before `loop_complete`) — single-occurrence terminating evaluator error (`eval_error_termination`); distinct from "Evaluate failures" which covers `verdict == "fail"` 3+ times - Retry floods - Evaluate failures (`verdict == "fail"`, 3+ occurrences on the same state) - Sub-loop verdict discarded - Throttle hard stop / hard transition (`throttle_stop` = loop halted; `throttle_hard` = loop redirected via `on_throttle_hard`) --- ## Step 5.5: Shallow-Iteration Check Using the history loaded in Step 2 and the artifact evidence from Step 4, run the shallow-iteration heuristic: **1. Count tool calls**: Count the number of `action_complete` events in the loaded history. Call this `TOOL_CALL_COUNT`. **2. Identify auxiliary mutations**: From the `git diff HEAD` evidence collected in Step 4, list all files that were created or modified and are **not** in the primary artifact path set (the paths identified in Step 4: `context.prompt_file`, `context.output_file`, and similar path-like context keys). Call this count `AUX_MUTATION_COUNT`. **3. Check for diff_stall corroboration**: Scan `evaluate` events in the history. For each, check whether the resolved FSM (loaded in Step 2) has `evaluate.type == "diff_stall"` for that state and the recorded `verdict` is `"stall"` or `"no"`. Call this `DIFF_STALL_PRESENT` (true/false). **4. Apply threshold** (default threshold: 30): ``` IF TOOL_CALL_COUNT > 30 AND AUX_MUTATION_COUNT == 0: IF DIFF_STALL_PRESENT: result = "corroborated" # both heuristic and diff_stall agree ELSE: result = "warning" # heuristic alone ELSE: result = "clear" ``` The default threshold of 30 `action_complete` events is intentionally conservative — most well-structured loops either produce auxiliary artifacts or converge within this budget. Loops that burn more than 30 iterations without creating helper structure are iterating without building. **5. Emit finding** when result is `"warning"` or `"corroborated"`: ``` ⚠ Shallow-iteration: action_complete events with no auxiliary file mutations outside the primary artifact path (). [Corroborated by diff_stall evaluator verdict in state ''.] Remediation: add intermediate artifact-write states; break monolithic iteration into smaller sub-tasks that each produce a named helper file. ``` Pass `result` and `TOOL_CALL_COUNT` to the scorecard in Step 6. --- ## Step 6: Goal-vs-Outcome Scorecard Determine the verdict using the terminal state from `loop_complete` event (`terminated_by`) and the artifact/contract evidence: | Verdict | Condition | |---|---| | `met` | Terminal reached AND all threshold contracts verified AND all expected artifact mutations occurred | | `phantom` | Terminal reached AND (artifacts unchanged OR threshold unverified — only model self-reported via `llm_structured` evaluator) | | `partial` | Terminal reached AND some but not all contracts satisfied | | `partial` | `terminated_by == "max_steps"` AND `max_steps_summary` event present in JSONL (summary state ran; artifact written) | | `degraded` | Loop completed but metric trended downward vs baseline captured in `state.json` | Output the structured scorecard block: ``` ### Goal-vs-Outcome Scorecard **Goal**: "" **Contract**: **Artifacts checked**: **Phase 1 signals**: **Shallow-iteration check**: `` ( tool calls, auxiliary mutations) **Verdict**: `` **Rationale**: ``` --- ## Step 7: Rubric-vs-Description Audit **Skip this step if `--no-rubric-audit` flag is set.** For each state with `evaluate.type: llm_structured`, send a judge call comparing: - The loop's top-level `description` text - The evaluator's `prompt` text Judge prompt (single call per evaluator): > "Does this evaluator prompt operationalize the loop's stated goal? Loop goal: ''. Evaluator prompt: ''. Answer YES if the evaluator directly measures progress toward the stated goal, NO if it measures something unrelated or misaligned." Flag as **rubric drift** if the judge answers NO. Include the evaluator's state name and a brief explanation. Pattern reference: `outer-loop-eval.yaml:generate_report` state uses `evaluate.type: llm_structured` with `min_confidence: 0.7`. --- ## Step 8: Sub-Loop Verdict Laundering Check For each state where `loop:` is set (sub-loop invocation), read `on_yes` and `on_no` from the FSM JSON output: ``` state.on_yes # child reached a terminal state state.on_no # child did not reach terminal ``` **Laundering defect**: `state.on_yes == state.on_no` (after any `${context.*}` interpolation). This means the parent loop treats child success and child failure identically — the child verdict is silently discarded. Flag each laundering defect with: - State name - Child loop name (`loop:` value) - The shared next state (both `on_yes` and `on_no` point to) --- ## Step 9: Ranked Improvement Proposals Emit ranked proposals from the scorecard, rubric audit, and fault signals. Order: contract-level > rubric-level > state-level > structural. For each proposal, include a concrete YAML diff where possible: ``` ### Improvement Proposals 1. [contract] Add artifact mutation verification for `prompts/test.md` Rationale: loop reached terminal without evidence of file mutation — possible phantom success YAML diff: states: optimize: + capture: optimized_prompt + capture_file: "${context.prompt_file}" 2. [rubric] Align evaluator prompt with loop goal in state `refine_answers` Rationale: evaluate.prompt checks Python syntax; description says "improve answer quality" 3. [state] Add `on_error` routing to state `check_quality` Rationale: shell evaluator with no on_error silently routes failed runs to on_no ``` ### Deduplication Before presenting proposals, check for existing issues: ```bash grep -rl "" .issues/bugs/ .issues/enhancements/ .issues/features/ .issues/epics/ 2>/dev/null ``` Mark matches as DUPLICATE. Present only NEW proposals. **Skip this step if `--skip-issue-creation` or `--auto` flag is set (or if `LL_NON_INTERACTIVE`/`DANGEROUSLY_SKIP_PERMISSIONS` env vars are set, or `--dangerously-skip-permissions` is active).** Print: `ℹ️ Issue creation skipped (--skip-issue-creation / --auto)` and stop. Use `AskUserQuestion` to ask: ``` Create issues for these proposals? [Y/n/select] Y — create all n — cancel select — choose which to create (comma-separated numbers) ``` For each approved proposal, allocate an ID (`ll-issues next-id`) and write the issue file to the appropriate category dir. Stage each written file by its explicit path (`git add ""`) — do **not** `git add .issues/`, which sweeps in unrelated untracked/modified files (BUG-1976). --- ## Final Report ``` Assessment complete for loop: Verdict: `` Rubric audit: Laundering check: Shallow-iteration check: `` ( tool calls, auxiliary mutations — or "below threshold") Issues created: ``` --- ## Usage Examples ```bash # Assess most recent interrupted loop /ll:audit-loop-run # Assess a specific loop /ll:audit-loop-run apo-textgrad # Limit history to 100 events /ll:audit-loop-run apo-textgrad --tail 100 # Skip LLM rubric audit (cost gate) /ll:audit-loop-run apo-textgrad --no-rubric-audit ```