---
name: resume-watchdog-audit
description: The watchdog-watchdog — one read-only pass that proves the resume watchdog (the n¹ layer that revives dead autonomous Claude sessions) is itself ALIVE, TICKING, and reviving PRODUCTIVE instances, versus silently STALLED. Distrusts self-report: reads the live Fleet registry ledgers (%LOCALAPPDATA%\Fleet\registry), the scheduled-task exit codes AND each task's Principal.LogonType, never the watchdog's own "I'm fine". Verifies the n² layer (is the watchdog ticking, is its backlog draining, are resumes witnessed as real transcript turns, or merely launched_unproven) and the n³ layer (is THIS audit itself scheduled/looped and orthogonal to the failure it detects — who watches the watchman's watchman). Turnkey entry point: tools\watchdog_watchdog_audit.ps1 (read-only, emits GREEN/AMBER/RED + exit 0/2/3). Catches the exact failure this repo hit on 2026-07-09: after a boot the watchdog drained the backlog, then every scheduled task with Principal.LogonType=Interactive began returning 0x800710E0 ("operator or administrator refused the request") on this RDP/headless box while S4U siblings stayed 0x0 — an 11-task outage of the whole safety net, leaving resumes launched-but-unproven. CORRECTED ROOT CAUSE: the discriminator is LogonType (Interactive dies, S4U survives), NOT the conhost launch shim (that theory was empirically disproved — the watchdog kept failing after conhost was removed). Read-only by default; the S4U migration + live drain are operator/elevation-gated. Use after a crash/reboot, when sessions look stuck, or on a /loop cadence as the standing meta-watchdog.
disable-model-invocation: false
user-invocable: true
allowed-tools: Read, Bash, PowerShell, Grep, Glob, Write
argument-hint: "[--fix] (no args = read-only audit; --fix offers the operator-gated live drain)"
---

# /resume-watchdog-audit — is the watchdog itself alive, or silently stalled?

> **The recursion.** The resume watchdog (`fleet_resume_watchdog.ps1 -Live`, task
> `FleetResumeWatchdog`) is the **n¹** layer: it revives dead autonomous Claude
> sessions across accounts. But a watchdog that dies takes the whole fleet down
> **silently** — every dead session just stays dead and nothing complains. This skill
> is the **n²** layer (watch the watchdog) and the **n³** layer (watch the watcher of
> the watchdog). It answers from artifacts — ledger mtimes, transcript turns, and
> Task Scheduler exit codes — **never** from the watchdog's own status text.

The one rule: **silence is the failure mode.** A stalled watchdog and a healthy-with-
nothing-to-do watchdog look identical from the outside. This pass distinguishes them.

## Run it (one command)

```powershell
pwsh -NoProfile -File tools/watchdog_watchdog_audit.ps1        # human-readable verdict
pwsh -NoProfile -File tools/watchdog_watchdog_audit.ps1 -Json  # machine verdict; exit 0/2/3 = GREEN/AMBER/RED
```

That script performs Layers 0–3 below in one read-only pass and prints a
GREEN/AMBER/RED verdict with the single deciding artifact and the one action. The
sections below are the manual expansion — read them to interpret the verdict, triage
a specific straggler, or when `fak`/the script isn't on the box. **This audit must
never share the failure mode it detects: run it as an agent `/loop` or an S4U task,
never as an Interactive scheduled task** (that would die exactly the way the watchdog did).

---

## Layer 0 — locate the LIVE registry (do not trust `tools/_registry`)

`resolveSweepRegDir("")` resolves in order: `$FLEET_REG_DIR` → `$FLEET_STATE_DIR\registry`
→ `%LOCALAPPDATA%\Fleet\registry` (if it exists) → `%TEMP%\Fleet\registry` → repo
`tools/_registry` (fallback only). **The repo's `tools/_registry` is usually a stale
copy** — its `sessions.json` may be fresh (the dispatcher writes it) while its
`resume_ledger.jsonl` is days old. Always audit the resolved live dir.

```powershell
$reg = @("$env:FLEET_REG_DIR","$env:FLEET_STATE_DIR\registry","$env:LOCALAPPDATA\Fleet\registry","$env:TEMP\Fleet\registry") |
  Where-Object { $_ -and (Test-Path $_) } | Select-Object -First 1
"live regDir = $reg"
Get-ChildItem $reg -Filter 'resume_*' | Select-Object Name,Length,LastWriteTime | Format-Table -AutoSize
```

## Layer 1 (n²) — the drain verdict + the stall check

**The authoritative read-only verdict** (returns exit 3 on RED — that is by design):

```powershell
& (Get-Command fak).Source resume watchdog --status --json
```

Fields that matter: `verdict` (green/amber/red), `auto_resume_depth` (queue depth),
`silent_seconds` (oldest unrecovered queued row's silence), and `mttr_sessions[]` with
`status: launched_unproven` (resumed but no real transcript turn after launch within
`--unproven-minutes`, default 10). **Caveat:** `mode` echoes YOUR invocation, so a plain
`--status` reports `DRY-RUN` and may add a spurious *"watchdog is DRY-RUN with queued
rows"* reason — ignore that one; the load-bearing reasons are `launched_unproven` and
`silent >= silent-hours`.

**The stall check the verdict alone won't give you** — has the watchdog *ticked at all*
recently? Compare ledger mtimes to now:

```powershell
$now = Get-Date
foreach ($f in 'resume_ledger.jsonl','resume_watchdog_status.jsonl','resume_plan.json') {
  $p = Join-Path $reg $f; if (Test-Path $p) {
    $m = (Get-Item $p).LastWriteTime; "{0,-28} {1}  ({2:n0} min ago)" -f $f,$m,($now-$m).TotalMinutes } }
```

> **RED if the newest of these is > ~15 min old.** The watchdog ticks on a short cron
> (default every 10 min); no write in 15+ min means it is not running. This is the
> single most important signal and the `--status` verdict does **not** encode it —
> `--status` reads whatever the last tick left behind and cannot tell "healthy + quiet"
> from "dead since 11:42".

## Layer 1b — WHY it stalled: exit code **paired with `Principal.LogonType`**

If it stalled, the cause is almost always the launch context, not the logic. Read the
task family's last result **and its LogonType together** — the LogonType is the
load-bearing column:

```powershell
Get-ScheduledTask | Where-Object { $_.TaskName -match 'Resume|Supervisor|Watchdog|Guard|Seat|Stranded|Dispatch' } | ForEach-Object {
  $i = Get-ScheduledTaskInfo -TaskName $_.TaskName -TaskPath $_.TaskPath -EA SilentlyContinue
  [pscustomobject]@{ Task=$_.TaskName; LogonType="$($_.Principal.LogonType)"; Result=('0x{0:X}' -f $i.LastTaskResult); LastRun=$i.LastRunTime } } |
  Sort-Object LogonType | Format-Table -AutoSize
```

**Known fault (2026-07-09), CORRECTED diagnosis:** `0x800710E0` = "The operator or
administrator has refused the request" appears on **every task with
`Principal.LogonType=Interactive`**, while every **S4U** sibling returns `0x0` — on this
same headless / RDP-accessed box, at the same time, under the same launcher. That split
**is** the diagnosis: **Interactive-logon tasks are refused when there is no true
interactive console** (they stay refused even while an RDP session shows `Active` in
`qwinsta`, and die outright when it disconnects); **S4U** tasks ("run whether logged on
or not"; session 0, windowless, still AS THIS USER) are immune. The clean natural
experiment: `FleetResumeWatchdog` (Interactive → `0x800710E0`) sitting next to
`FleetStrandedRecovery` (S4U → `0x0`).

> **The `conhost.exe --headless` shim is a RED HERRING — do NOT chase it.** An earlier
> version of this skill blamed conhost. It was empirically disproved: `FleetResumeWatchdog`
> kept returning `0x800710E0` *after* conhost was removed and it launched `powershell.exe`
> directly. Removing conhost does nothing; only the LogonType matters. (`ExploitGuard MDM
> policy Refresh` / `SafeguardsReconciliation` are unrelated `ServiceAccount` system tasks
> that are always `0x0` — not the culprit.)

The failing set is not one task — enumerate ALL Interactive tasks (fleet-wide) so the
remediation covers the whole outage, not just the watchdog:

```powershell
Get-ScheduledTask | Where-Object { $_.Principal.LogonType -eq 'Interactive' } |
  ForEach-Object { $i=Get-ScheduledTaskInfo -TaskName $_.TaskName -TaskPath $_.TaskPath -EA SilentlyContinue
    [pscustomobject]@{ Task=$_.TaskName; Result=('0x{0:X}' -f $i.LastTaskResult) } } |
  Where-Object { $_.Result -eq '0x800710E0' } | Format-Table -AutoSize
```

## Layer 2 — was the response PRODUCTIVE? (not just "did it launch")

Launching a resume is cheap; **witnessing a real transcript turn after it** is the proof
the session came back to life. The status ledger records both. Quantify today's work:

```powershell
$st = Join-Path $reg 'resume_watchdog_status.jsonl'; $today = Get-Content $st | Where-Object { $_ -match (Get-Date -Format 'yyyy-MM-dd') }
$depths = foreach ($l in $today) { try { $o=$l|ConvertFrom-Json; if ($o.phase -eq 'status'){ [int]$o.auto_resume_depth } } catch {} }
$prog = @($today | Where-Object { $_ -match '"phase":"progress"' })
"peak backlog depth = {0}   drained to = {1}" -f ($depths|Measure-Object -Maximum).Maximum, ($depths|Measure-Object -Minimum).Minimum
"progress-witness rows = {0}   distinct sessions revived-with-witnessed-progress = {1}" -f $prog.Count, (@($prog|% { ($_|ConvertFrom-Json).session })|Sort-Object -Unique).Count
```

A healthy post-crash run shows the peak depth **draining toward single digits** and a
large distinct-session witness count. A backlog that grows monotonically across ticks
(`--monotonic-ticks`) is a resume storm, not recovery — also RED.

## Layer 2b — triage each `launched_unproven` straggler

For every session `--status` flags unproven, decide *stuck vs. just-not-witnessed-yet*
by going to ground truth — the transcript and the process table:

```powershell
$sid = '<uuid-from-status>'
$t = Get-ChildItem "$env:USERPROFILE\.claude*\projects\*\$sid.jsonl" -EA SilentlyContinue | Sort LastWriteTime -Desc | Select -First 1
"transcript {0}  ({1:n0} min idle)  last={2}" -f $t.FullName, ((Get-Date)-$t.LastWriteTime).TotalMinutes, (Get-Content $t.FullName -Tail 1).Substring(0,60)
# a `last-prompt` tail + a dead resume pid + idle > unproven-minutes = the watchdog stalled before it could re-revive this one
```

If the tail is the injected re-entry prompt (`type:last-prompt`), the resume pid is gone,
and it's been idle past the threshold, the straggler needs another tick the stalled
watchdog can't give it — see remediation.

## Layer 3 (n³) — who watches this audit?

The n² check is worthless if it only runs when a human remembers. Verify the meta-layer
is itself standing:

- Is `resume-watchdog-audit` wired to a **/loop cadence** or a scheduled task, so it fires
  without a human? (This repo's `run-it-all-night` / `super-loop` are candidate hosts.)
- Does a RED verdict **escalate** — a `notifications.log` toast, a Slack beat
  (`FleetSlackStatus`), or an issue — rather than just printing and exiting?
- If this audit is the ONLY thing that would notice a dead watchdog, then a dead
  auditor is a silent double-fault. Note it explicitly in the verdict.

---

## Verdict rubric

| Verdict | Condition |
|---|---|
| **GREEN** | newest ledger write < 15 min ago **and** no `launched_unproven` past threshold **and** backlog draining **and** `FleetResumeWatchdog` LastResult `0x0`. |
| **AMBER** | one straggler unproven, or a single missed tick (< ~25 min silent), backlog flat. |
| **RED** | no ledger write > 15 min (**stall**) **or** any task LastResult non-zero (e.g. `0x800710E0`) **or** ≥1 `launched_unproven` with a dead pid past threshold **or** monotonic backlog growth. |

State the verdict as: layer (n²/n³), the artifact that decided it (mtime / exit code /
transcript), and the one action. Do not soften a stall into "probably fine".

## Remediation — OPERATOR-GATED (has side effects; confirm first)

The audit above is read-only. These act — they spawn real `claude --resume` processes
and consume account quota, so **surface them and get explicit go-ahead; never run them
as part of the audit**:

1. **Fix the stall's ROOT CAUSE first** — if the audit shows down tasks with
   `LogonType=Interactive / 0x800710E0`, migrate their principal to **S4U**. This
   **requires an ELEVATED shell** (setting an S4U principal is "Access is denied"
   non-elevated). The helper does the whole fleet idempotently, dry-run by default:
   ```powershell
   # from an Administrator PowerShell:
   .\tools\migrate_fleet_tasks_to_s4u.ps1                    # dry-run: review the plan
   .\tools\migrate_fleet_tasks_to_s4u.ps1 -Apply -VerifyRun  # migrate all failing + force-run each -> expect 0x0
   ```
   Do **not** re-register to `powershell.exe`-direct as a "fix" — that changes the
   launcher, not the LogonType, and leaves the task Interactive and still failing.
2. **Drain the stragglers** (one live tick; honors the per-tick cap + source governor).
   Only meaningful once the task is S4U — otherwise the scheduled task can't tick and you
   are hand-cranking it: `pwsh -NoProfile -File tools/fleet_resume_watchdog.ps1 -Live`
3. **Close the reinstall regression** — most `register_*.ps1` installers still create
   Interactive (schtasks default), so a reinstall reintroduces the bug. `register_resume_watchdog.ps1`
   and `register_issue_dispatch.ps1` are already S4U; migrate the rest.
4. Re-run this audit (`tools/watchdog_watchdog_audit.ps1`); expect GREEN (ledger fresh,
   no down/latent task, backlog draining, resumes proven).