---
name: ares-debug
description: Diagnose a stuck, slow, or broken Ares operation by triangulating across three data sources — SSM (live ares logs + Redis on EC2), Grafana Loki (historical logs via mcp__grafana__query_loki_logs), and OTEL traces in Tempo. Use when an operation is hung/wedged, a worker keeps crashing, the orchestrator stops making progress, or a task fails with no obvious local clue. Default deployment is EC2 (`kali-ares`); K8s notes included for completeness.
---

# Debugging Ares

You are debugging a running or recent Ares operation. Pick the cheapest source first; only escalate if it doesn't answer the question.

## Read this before you do anything

**Do not declare an op healthy from process liveness, NATS/Redis ping, or token-rate alone.** A wedged Ares op happily presents as `status=running`, workers `active`, Redis green, cache hit ≥80%, and tokens climbing — while making zero external progress for hours. This has happened. Don't repeat it.

**The only valid "healthy" verdict requires a comparison:**

1. Compare *this op's* objective state now vs. 60s ago — `has_domain_admin`, domain compromise count, hosts owned, creds, hashes, vulns exploited. If none changed, that's churn, not progress.
2. Compare this op to recent ops' baseline. Pull `ares ops list` and look at how long prior ops took to hit DA / 2nd domain. **If this op is more than ~2× slower to a milestone the last 3 ops hit, treat it as wedged regardless of token rate.**

Token churn is the signature of the LLM re-evaluating the same frozen state every tick; high cache-hit rate (>80%) on a slow op is a *symptom of the wedge*, not evidence of health.

**Worker per-role log mtimes are not a signal.** In steady state the orchestrator centralizes everything via NATS into `/var/log/ares/orchestrator.log`; per-role files (`recon.log`, `cracker.log`, etc.) stay near-empty. Don't read into stale mtimes.

## Before you propose a code fix

Ares timeline events (`evt-exploit-fail-*` in `ares:op:*:timeline`) and "Assistance needed" strings the LLM emits ("the tool schema does not accept X", "current toolset lacks Y", "tool requires password but only hash available") are the failing LLM agent's confabulated explanation of its own failure — **not a bug report**. The agent does not know its own tool schemas or the orchestrator's dispatch layer, and it will invent plausible-sounding gaps that don't exist.

Before recommending a fix from one:

1. Open the tool wrapper in `ares-tools/src/**/*.rs` — does the tool actually accept the arg the LLM said was missing?
2. Open the LLM-facing schema in `ares-llm/src/tool_registry/**/*.rs` — does it declare the field?
3. Open the automation dispatcher in `ares-cli/src/orchestrator/automation/*.rs` — does it inject the credential/state from Redis into the payload?

If all three already do the thing, the LLM was confabulating. The real failure is elsewhere — the tool ran and hit a Kerberos error, dispatch timed out, worker didn't have the credential in state, etc. Grep the orchestrator log for the actual dispatch record + tool stdout/stderr; those are ground truth. Timeline events are not.

## Tight-loop / wedge signatures (grep the orchestrator tail for these first)

Run Step 0, then **before drawing any conclusion** grep the tail of `orchestrator.log` for each pattern below. If any hit, that's almost certainly your wedge:

| Pattern (regex) | Means |
|---|---|
| `clearing dedup for retry` | Wrapper-level retry loop; same task being re-dispatched every tick |
| `Dispatching <same_tool> ... <same_target>` repeated ≥3× | Automation hot loop with no backoff |
| `KDC_ERR_TGT_REVOKED\|KDC_ERR_S_PRINCIPAL_UNKNOWN\|KDC_ERR_PREAUTH_FAILED\|TGT has been revoked` | Kerberos error that will not self-heal; orchestrator may be retrying anyway |
| `tool exited with code Some\(0\)` followed by stderr content | Zero-exit-with-error: wrapper treats stderr-on-zero-exit as transient and re-tries |
| Same `task_id` shape (e.g. `trust_raise_child_<hex>`) repeated with distinct hex per tick | Dedup key churning instead of blacklisting |
| `Processing real-time discoveries count=1` ticking every 5s with no other state change | Orchestrator stuck in discovery-replay loop |
| `Waiting for blue team to finish\.\.\. active_investigations=[0-9]+` ticking every 10s | **Not a wedge — red is DONE.** Op is holding open until blue investigations drain. Check `red_completed_at` / `red_completion_reason` in meta (see Step 0). |
| `Loki request error \(retryable\)` / `Retrying Loki query after transient failure` flooding the tail | Blue team's external Loki (`$LOKI_URL`) is flapping; blue investigations grind to a crawl and starve out post-red op close. Not a red bug. |
| `Tool binary not found \(spawn failed\) — removing from available tools` firing across many recon tools (nmap_scan, enumerate_users, enumerate_shares, smb_signing_check, username_as_password) in the first seconds of the op | Tool-pruning cascade — a prior spawn failure poisoned the worker's per-process `unavailable_tools` HashSet. Only a genuine worker-process restart clears it — **not** `task ec2:restart`, which never touches `ares@` units (see Step 8). Fix: `task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'`. Full mechanism + confirmation queries in Step 3.5. |

If you don't see these but the op is slow vs. baseline, escalate to Loki / Tempo for cross-tick LLM latency or tool-call stalls.

## What goes where

| Source            | Latency  | Coverage                                        | How to query                                             |
|-------------------|----------|-------------------------------------------------|----------------------------------------------------------|
| `task ec2:status` | seconds  | Worker process state, Redis ping                | Bash                                                     |
| `task ec2:runtime`| seconds  | Per-op token/cost/domain banner                 | Bash                                                     |
| Loki (Grafana)    | seconds  | Historical `/var/log/ares/*.log` + syslog/auth  | `mcp__grafana__query_loki_logs` (datasourceUid `loki`)   |
| Tempo (Grafana)   | seconds  | OTEL traces of LLM calls + tool dispatch        | `mcp__grafana__*` Tempo proxy tools                      |
| SSM `task ec2:exec` | ~5-15s | Anything on the host (redis-cli, journalctl)    | Bash, never `tail -f`                                    |
| `task ec2:logs`   | streaming| Live tail of one role's log                     | **DO NOT use in Claude** — it's an interactive SSM session |

**Rule:** never run `task ec2:logs` from an agent — it opens an interactive SSM session that won't terminate. Always use Loki (preferred) or `task ec2:exec EC2_NAME=kali-ares CMD='tail -n 200 /var/log/ares/<role>.log'`.

**AWS auth:** use whatever ambient AWS profile has SSM access to the box — do not hard-code one. Ownership of the `kali-ares` instance has moved between profiles/accounts multiple times; a stale prefix (`AWS_PROFILE=personal AWS_REGION=us-east-1`) will produce `No running instance found matching: kali-ares` even when the box is up. Verify resolution with `task ec2:ops EC2_NAME=kali-ares` first; if it fails, try flipping between `lab` and `personal` and between `us-east-1` and `us-west-2`. The command examples below run against the ambient profile — set it explicitly only if the ambient one doesn't resolve the box.

## Step 0 — mandatory baseline triage (run all in parallel, on every invocation)

Do not skip any of these. Do not respond to the user with a verdict until you've inspected each output. The point of this step is to make it impossible to declare "healthy" without the evidence.

```bash
# 0a. Current op id + status
task ec2:ops EC2_NAME=kali-ares LATEST=true

# 0b. Current op objective state + tokens
task ec2:runtime EC2_NAME=kali-ares LATEST=true

# 0c. Process / Redis / NATS health
task ec2:status EC2_NAME=kali-ares

# 0d. The single most important probe — orchestrator tail. Grep it for the wedge signatures listed above.
task ec2:exec EC2_NAME=kali-ares \
  CMD='tail -n 300 /var/log/ares/orchestrator.log'

# 0e. Historical baseline — last several ops, to compare runtime-to-milestone
ares --ec2 kali-ares ops list | head -20

# 0f. Failed tasks for the current op
ares --ec2 kali-ares ops tasks --latest --status failed | head -80
```

Pull `op-YYYYMMDD-HHMMSS` from 0a/0b and use that as `$OP` below. After collecting:

1. **Is red already done?** Before anything else, check `red_completed_at`, `red_completion_reason`, and `red_blocked_on_blue` in the op's meta. If `red_completed_at` is set, red is NOT wedged — it ended (either by success, `"all forests dominated (post-exploitation complete)"`, or by hitting `"max runtime exceeded"`). The op status will still show `running` because the operation as a whole is holding open for blue investigations to drain; that's the "Waiting for blue team to finish" pattern in the wedge table. Don't misdiagnose an ended-red as wedged. One command:

   ```bash
   task ec2:exec EC2_NAME=kali-ares CMD="sudo redis-cli hmget ares:op:$OP:meta red_completed_at red_completion_reason red_blocked_on_blue has_domain_admin has_golden_ticket"
   ```

2. Grep the 0d output for each pattern in the "Tight-loop / wedge signatures" table. **If any hits ≥3 times, you have your root cause; jump to reporting.**
3. Compare 0b's `Domains compromised` and `Vulns exploited` against the runtime banner of recent ops in 0e. If the prior 3 ops compromised more domains in less time at this point, the current op is regressed regardless of how healthy 0a/0c look.
4. Read 0f — the failure mode of the first 5-10 failed tasks usually points at the role/tool that's flailing.

Only proceed past Step 0 to deeper probes (Loki, Tempo, SSM journals) if none of the above lands a verdict.

**Two footguns in the Step 0 commands themselves — read before you file a "Redis broken" bug:**

- `ares --ec2 kali-ares ops list` (0e/0f) connects to **local** Redis on the machine you're running from, not to the box's Redis over SSM. From an agent host with no `redis-server` and no `ec2:redis:forward` running, it will exit with `Failed to connect to Redis: Connection refused`. That's not "the box is broken" — it's the CLI wanting a live connection. When you see it, fall back to `task ec2:exec EC2_NAME=kali-ares CMD='sudo redis-cli ...'` for anything you'd have asked the CLI for.
- **There is no `ares:op:<op>:creds` key.** It is `:credentials`. Any command built on `:creds` returns an empty/zero result that reads exactly like "no credentials found" — the most expensive false negative in this document's history. Verified against `ares-core/src/state/keys.rs` and the writer verbs in `ares-core/src/state/reader.rs`:

  | Key | Writer verb | TYPE | Count with | Dump with |
  |---|---|---|---|---|
  | `:meta` | `hset` | HASH | `HLEN` | `HGETALL` / `HMGET` |
  | `:credentials` | `hset_nx` | HASH | `HLEN` | `HGETALL` |
  | `:hashes` | `hset` | HASH | `HLEN` | `HGETALL` |
  | `:vulns` | `hset_nx` | HASH | `HLEN` | `HGETALL` |
  | `:completed_tasks` | `hset` | HASH | `HLEN` | `HGETALL` |
  | `:hosts` | `rpush` | LIST | `LLEN` | `LRANGE k 0 -1` |
  | `:users` | `rpush` | LIST | `LLEN` | `LRANGE k 0 -1` |
  | `:timeline` | `rpush` | LIST | `LLEN` | `LRANGE k -50 -1` |

  Wrong verb → `WRONGTYPE`, which is loud. Wrong *key name* → `0`, which is silent. When in doubt: `redis-cli type <key>`.

## Step 1 — fast triage (Loki, last hour)

Loki has every ares log line shipped from the EC2 box. Datasource UID is `loki`. Logs are JSON; the actual line is in the `message` field, with labels `app="ares"`, `deployment="alpha-operator-range-kali-ares"`, `job=<role>.log`.

Run these in parallel:

```
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |~ "(?i)error|fatal|panic|traceback|RUST_BACKTRACE"'
  limit: 30
```

```
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "WARN|ERROR"'
  limit: 30
```

Narrow by role when you know the suspect: change `job="orchestrator.log"` to one of
`recon.log`, `credential_access.log`, `cracker.log`, `acl.log`, `privesc.log`, `lateral.log`, `coercion.log`.

Narrow by op id (substring match on the log line):

```
logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "op-20260630-201500"'
```

Use `query_loki_stats` first when you're guessing the selector — it tells you whether the stream has any entries before you waste a `query_loki_logs` call.

## Step 2 — failed tasks (operation-level)

```bash
task red:multi:tasks:list LATEST=true STATUS=failed   # K8s
ares --ec2 kali-ares ops tasks --latest --status failed   # EC2
```

Failed tasks include the worker's error message and the role that failed. Cross-reference against Loki by role + timestamp.

## Step 2.5 — attribute a specific tool call to the worker that ran it

Use this when the question is "did tool X actually run for task Y, on which worker, and did it succeed?" The canonical case is verifying cross-role routing (e.g. `credential_access`-originated `password_spray` / `username_as_password` / `laps_dump` calls must land on a `recon` worker because netexec lives there — see `RECON_ROUTED_TOOLS` in `orchestrator/tool_dispatcher/mod.rs`).

**Ground truth is the OTel span line each worker emits at INFO level when it starts a tool:**

```
Executing tool tool=<T> call_id=<T>_<hex> task_id=<origin_role>_<hex>
```

The span attributes on that same line are what you actually want:

- `agent.role` = the worker that executed the tool. Cross-routing fired if this differs from the role prefix of `task_id`.
- `attack_operation_id` / `op.id` = the op — scope every grep to this to avoid conflating past ops.
- The follow-up line for the same `call_id` carries `Tool execution failed tool=<T> err=<message>` on failure.

**The three canonical failure strings** and where they come from — memorize these because they distinguish "binary missing" from "tool ran and errored":

| String | Source | Meaning |
|---|---|---|
| `failed to spawn '<binary>' — is it installed?` | `ares-tools/src/executor.rs:219` | ENOENT: the binary isn't on this worker's `$PATH` |
| `failed to spawn impacket-ntlmrelayx (is it installed?)` | `ares-tools/src/coercion.rs:586` | Same, special-cased (no single quote — do not narrow greps to require one) |
| `Tool '<T>' is not installed on this worker.` | `worker/tool_executor.rs::unavailable_tool_response` | Cached unavailability — a prior call ENOENT'd and future calls return this without re-spawning |

Everything else in `err=...` means the binary ran and the tool logic failed (timeout, KDC error, no creds, etc.).

**Query patterns.** Prefer Loki when the label narrow is easy; SSM `grep -a` when you need cross-file correlation on the box.

```
# Loki: every executor span for tool X in this op
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "Executing tool" |= "tool=<T>" |= "<OP>"'
  limit: 50
```

```bash
# SSM: same thing, plus the failure line for the same call_id
task ec2:exec EC2_NAME=kali-ares \
  CMD='sudo grep -a "<OP>" /var/log/ares/recon.log /var/log/ares/credential_access.log | grep -a "tool=<T>" | head -20'

# End-to-end trace of one call_id across every worker log
task ec2:exec EC2_NAME=kali-ares \
  CMD='sudo grep -a "<call_id>" /var/log/ares/*.log'

# Sanity: is the binary the caller expects actually on the box right now?
task ec2:exec EC2_NAME=kali-ares \
  CMD='which netexec; ls -la /usr/local/bin/netexec /usr/bin/netexec 2>/dev/null; netexec --version 2>&1 | head -3'
```

**Gotchas (do not skip):**

1. `task ec2:exec` runs `CMD` through go-task's template engine. `{{ ... }}`, backticks, and some quoting silently fail with `"CMD required"` — that means the template ate the arg, not that `CMD` was empty. Workarounds: bind `Q="…"` locally and pass `CMD="$Q"`; keep single quotes on the outside; avoid `{{`. If you see `"CMD required"`, simplify quoting before assuming the file is empty.
2. Per-role log files are **ANSI-color-coded** on disk. `grep 'tool.name="X"'` returns 0 hits even when the tool ran because the bytes are `tool.name<ESC>[0m<ESC>[2m=<ESC>[0m"X"`. Anchor on invariant plain-text substrings: `Executing tool`, `tool=<T> call_id=`, `err=failed to spawn`, `attack_operation_id="<OP>"`. `grep -a` (force text mode) is required — the escapes make grep treat these files as binary and go silent otherwise.
3. Per-role log files stay near-empty in steady state (see the intro's "worker per-role log mtimes are not a signal") — but executor OTel spans DO land there. `recon.log` and `credential_access.log` are the right files for tool-attribution greps even though they look sparse.
4. `ingest.log` is a firehose (multi-GB); do not grep it without a `--max-count` or a very narrow anchor.

**Case study — was cross-routing broken on op-20260716-181136?** credential_access called `username_as_password`, the runner pruned it after "spawn failed". The trace resolved it in three greps:

```
agent.role=recon
task_id=credential_access_de9f5fa0be53
err=failed to spawn 'netexec' — is it installed?
```

`agent.role=recon` proved routing fired (a recon worker picked up a credential_access-originated call — cross-routing correct). The `err=` matched `executor.rs:219` verbatim, pinning the root cause on netexec missing from the box at that moment. Fix was ansible provisioning drift, not code. **Without the span attributes there was no way to distinguish "routing bug" from "environment drift" — every hypothesis based on just the runner's `WARN` line would have been wrong.**

## Step 3 — wedge detection (objective state frozen)

**The canonical wedge is NOT "tokens flatlined" — tokens almost always keep climbing during a wedge because the LLM re-evaluates the same frozen state every tick.** The canonical wedge is "objective state frozen while tokens climb." Probe state, not tokens:

```bash
# Snapshot 1 — verb matches TYPE per the table in Step 0. `credentials` NOT `creds`.
task ec2:exec EC2_NAME=kali-ares \
  CMD='redis-cli hmget "ares:op:'"$OP"':meta" has_domain_admin has_golden_ticket target_ips initialized red_completed_at red_blocked_on_blue; echo ---; for k in credentials hashes vulns completed_tasks; do printf "%s=" "$k"; redis-cli hlen "ares:op:'"$OP"':$k"; done; for k in hosts users timeline; do printf "%s=" "$k"; redis-cli llen "ares:op:'"$OP"':$k"; done'
# wait 60s
# Snapshot 2 — same command. Diff the two. Identical = wedge.
```

Cross-check against tokens: pull `ec2:runtime` at both snapshots. **Tokens climbing + state identical = textbook wedge.** Tokens climbing + state changing = healthy. Tokens flatlined + state identical = orchestrator hung (rarer).

If wedged, two further probes pinpoint where:

```bash
# Outbound HTTPS from orchestrator — zero connections = LLM API stall
task ec2:exec EC2_NAME=kali-ares CMD='ORCH=$(pgrep -f "ares orchestrator" | head -1); echo "orch_pid=$ORCH"; sudo ss -tnp 2>/dev/null | grep "pid=$ORCH" | grep -v 127.0.0.1 | wc -l'
```

```
# Loki search for retry/throttle/dedup markers in the last 30 minutes
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "clearing dedup for retry|KDC_ERR_|Task deferred|throttler|stale|wedge"'
  limit: 80
```

Remedy depends on root cause:

- Hot retry loop on a tool (`clearing dedup for retry`) → fix the dedup/blacklist logic in the relevant `automation/auto_*.rs`; in the meantime `task ec2:stop-op ... LATEST=true` to stop the burn.
- LLM API stall → check the model provider's status, then restart the orchestrator with `task ec2:restart EC2_NAME=kali-ares` (that is stop+start of `ares-orchestrator.service` and infra only — it preserves Redis but leaves workers untouched; add `task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'` if the workers are the stalled party).
- State frozen but no signature → escalate to Tempo (Step 7) to find the slow span.

## Step 3.5 — tool-pruning cascade (recon suddenly does nothing)

Distinct failure class from "wedge" and "crash." Signature: the LLM issues a normal task, workers stay `active`, but every recon/credential-access tool the LLM tries is immediately marked `Tool binary not found (spawn failed) — removing from available tools` and the LLM burns through its 24-tool list in seconds without any external effect. The op then presents as slow-vs-baseline with 0 creds / 0 hashes / 0 hosts.

**Grep the LLM runner side for the pattern:**

```bash
task ec2:exec EC2_NAME=kali-ares CMD="sudo grep -aE 'Tool binary not found \(spawn failed\)' /var/log/ares/orchestrator.log | grep -a '$OP' | grep -oE 'tool=[a-z_]+' | sort | uniq -c | sort -rn"
```

If a bunch of nxc/netexec-backed tools (`nmap_scan`, `enumerate_users`, `enumerate_shares`, `smb_signing_check`, `check_rdp_reachability`, `check_winrm_reachability`, `username_as_password`, `smb_sweep`) all show up, that's the cascade.

**Mechanism** (three separate files):

1. `ares-tools/src/executor.rs:219` — real spawn failure emits `failed to spawn '<binary>' — is it installed?`.
2. `ares-cli/src/worker/tool_executor.rs:332-333` (`is_tool_unavailable_error`) — classifies that string as "unavailable" and inserts the tool name into a **per-process `unavailable_tools: HashSet<String>`**. Every subsequent call to that tool on that worker skips the spawn entirely and returns the cached `"Tool 'X' is not installed on this worker. Do not call this tool again — it failed to spawn previously."` response (`tool_executor.rs:318-328`).
3. `ares-llm/src/agent_loop/runner.rs:60` — `dispatch_one` flattens the worker's `error` field into `output` (`"Error: {err}\n\nPartial output:\n{output}"`), then `runner.rs:532` detects `output.contains("failed to spawn")` and yanks the tool from the LLM's active list for the rest of this task, plus injects a `[SYSTEM]` message telling the LLM to stop trying.

The trap: **one transient spawn failure poisons the tool for the worker's lifetime** — no TTL, no re-probe. Runs whose spawn genuinely failed (a mid-deploy race, an apt lock, an ephemeral cgroup hiccup) leave dead tool entries that persist across every subsequent op the same worker handles.

**Deploys restart workers only if their units are already `active`** (`.taskfiles/ec2/Taskfile.yaml:255-257`), so `task ec2:deploy` usually clears the poison — but silently skips any worker whose unit is inactive, printing `no ares@ worker units active — skipping restart`. When that happens the worker keeps its poisoned `unavailable_tools` set and `/proc/<worker-pid>/exe` points at the pre-deploy inode with `(deleted)` on it (see the Step 8 deploy note).

**Confirmation & fix:**

```bash
# Check worker uptime — anything > a few hours across multiple ops is suspicious
task ec2:exec EC2_NAME=kali-ares CMD='systemctl show ares@recon.service -p ActiveEnterTimestamp,MainPID; ps -o pid,etime,cmd -C ares | head -10'

# Verify the binaries actually work from the shell (rules out "genuinely uninstalled")
task ec2:exec EC2_NAME=kali-ares CMD='which netexec nxc nmap; nxc --version 2>&1 | head -1; nmap --version 2>&1 | head -1'

# If binaries work but pruning still fires → bounce the WORKER units (keeps Redis).
# `task ec2:restart` will NOT do this — it never touches ares@ units.
task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"; systemctl is-active "ares@*.service" | sort | uniq -c'
```

If the pruning cascade repeats on the very next op with **fresh** workers, the spawn failure is reproducible — probe from inside the worker's cgroup for AppArmor denials, broken Python venvs (nxc/netexec is a pipx shim; `python3 -c 'from nxc.netexec import main'` is a direct test), or `system-ares.slice` restrictions.

Note: `sprayhound`-backed tools (`password_spray`, `asrep_roast`) use a different binary and are unaffected — seeing those still `Executing tool` in the recon.log while nxc-backed tools are pruned is the signature that isolates this to the netexec side.

## Step 4 — worker crash loop

A specific role keeps respawning. Check systemd journal via SSM:

```bash
task ec2:exec EC2_NAME=kali-ares CMD='systemctl status ares@recon --no-pager | head -30'
task ec2:exec EC2_NAME=kali-ares CMD='journalctl -u ares@recon -n 100 --no-pager'
```

(Substitute `recon` with the failing role: `credential_access`, `cracker`, `acl`, `privesc`, `lateral`, `coercion`.)

If OOM-killed, check the cgroup:

```bash
task ec2:exec EC2_NAME=kali-ares CMD='dmesg -T | grep -iE "killed process|oom" | tail -20'
```

The system-ares.slice caps memory at 12G global, ~2G per worker (see `.taskfiles/ec2/scripts/setup.sh:160`). Worker OOM = a tool process (netexec, hashcat, etc.) blew up inside the worker's cgroup.

## Step 5 — Redis state introspection

```bash
task ec2:exec EC2_NAME=kali-ares CMD='redis-cli ping'
task ec2:exec EC2_NAME=kali-ares CMD='redis-cli info keyspace'
task ec2:exec EC2_NAME=kali-ares CMD='redis-cli keys "ares:operation:*" | head -20'
task ec2:exec EC2_NAME=kali-ares CMD='redis-cli get ares:operation:active'
task ec2:exec EC2_NAME=kali-ares CMD='redis-cli hgetall "ares:op:'"$OP"':meta"'
```

For loot or shared state, prefer the typed CLI over raw Redis:

```bash
task ec2:loot EC2_NAME=kali-ares LATEST=true        # users, creds, hashes, hosts
task ec2:loot EC2_NAME=kali-ares LATEST=true DIFF=true  # only what changed since last call
```

To run blue-team queries or arbitrary `ares` commands against EC2 Redis locally, port-forward:

```bash
task ec2:redis:forward EC2_NAME=kali-ares   # blocks in foreground — DO NOT run from an agent
```

If you need local access from an agent, use `ec2:exec` with `redis-cli` instead.

## Step 6 — NATS broker

NATS is the task/RPC broker. If workers are alive but no tasks dispatch:

```bash
task ec2:exec EC2_NAME=kali-ares CMD='curl -s http://127.0.0.1:8222/varz | jq ".connections, .in_msgs, .out_msgs, .slow_consumers"'
task ec2:exec EC2_NAME=kali-ares CMD='curl -s http://127.0.0.1:8222/connz | jq ".num_connections, [.connections[].name]"'
task ec2:exec EC2_NAME=kali-ares CMD='systemctl status nats-server --no-pager | head -15'
```

## Step 7 — OTEL traces (LLM + tool call timing)

OTEL traces ship to Tempo with `service.name=ares-orchestrator|ares-<role>-agent` and `deployment.environment=staging`, `attack.team=red`. Use the Grafana Tempo proxy tools — search by `service.name` and op id (op id is set as a span attribute by the orchestrator).

Useful when:

- You want to see the LLM call latency that's stalling a tick
- A specific tool call is silent in logs but you want to confirm it ran
- You need to attribute time spent across roles for a long-running op

If Tempo search returns nothing, the orchestrator may not be exporting — verify with:

```bash
task ec2:exec EC2_NAME=kali-ares CMD='grep OTEL_EXPORTER /etc/ares/env'
```

## Step 8 — verify deploy state (binary mismatch)

A common false positive: the local CLI and the EC2 binary diverge.

```bash
ares --version                                                                                                          # local
task ec2:exec EC2_NAME=kali-ares CMD='/usr/local/bin/ares --version'          # remote
task ec2:exec EC2_NAME=kali-ares CMD='stat -c "%y %s" /usr/local/bin/ares'    # mtime + size
```

If you just landed code, re-deploy before continuing to debug. Canonical "upload updated code, then run a fresh op against dreadgoad" one-liner (Apple-Silicon-safe — `DOCKER_DEFAULT_PLATFORM` forces an x86 build, the S3 bucket is the alpha-operator-range artifact store):

```bash
DOCKER_DEFAULT_PLATFORM=linux/amd64 task -y ec2:deploy EC2_NAME=kali-ares S3_BUCKET=dread-infra-alpha-operator-range-prod-us-east-1 \
  && task -y red:ec2:multi TARGET=dreadgoad EC2_NAME=kali-ares
```

(Both halves rely on the ambient AWS profile resolving `kali-ares` — see the "AWS auth" note above. Drop the `&&` and run just the first half for a deploy-only.)

**`task ec2:restart` does NOT bounce the workers.** It is `stop` + `start` (`.taskfiles/ec2/Taskfile.yaml`): `stop` stops `ares-orchestrator.service` and `pkill -f "ares orchestrator"`; `start` brings up redis-server, nats-server and postgresql. **Neither touches a single `ares@<role>.service` unit.** Any advice that says otherwise — including older revisions of this file — has you running a command that cannot fix the problem it is prescribed for.

**`task ec2:deploy` already restarts the workers, with one catch.** After installing the binary it runs (`.taskfiles/ec2/Taskfile.yaml:255-257`, and again at `:452-454`):

```sh
UNITS=$(systemctl list-units --type=service --state=active --no-legend "ares@*.service" | awk '{print $1}' | sort -u)
if [ -z "$UNITS" ]; then echo "no ares@ worker units active — skipping restart"; else systemctl restart $UNITS; fi
```

The catch is `--state=active`: a worker that crashed, or one whose unit is loaded-but-inactive, is invisible to that query and silently keeps its stale binary. **`no ares@ worker units active — skipping restart` in deploy output means nothing was restarted.** Read for that line; don't assume the restart happened.

To force every worker unit regardless of state:

```bash
task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"; systemctl is-active "ares@*.service" | sort | uniq -c'
```

Faster deploy-only when you don't need to publish to S3 (builds natively on EC2):

```bash
task ec2:deploy EC2_NAME=kali-ares BUILD_TOOL=remote
```

## Step 9 — kill, clear, retry (last resort)

Don't do this until you've captured logs and runtime — these are destructive.

```bash
task ec2:stop-op EC2_NAME=kali-ares LATEST=true   # graceful stop of one op
task ec2:stop    EC2_NAME=kali-ares                # stop all workers (keeps Redis)
task ec2:restart EC2_NAME=kali-ares                # stop+start orchestrator + infra ONLY (keeps Redis; does NOT touch ares@ workers)
task ec2:exec EC2_NAME=kali-ares CMD='systemctl restart "ares@*.service"'   # the actual worker bounce
```

To actually wipe state, use the CLI cleanup command instead of FLUSHALL:

```bash
ares --ec2 kali-ares ops cleanup --max-age-hours 0
```

## K8s deployment notes

Same triage flow, different transport:

| EC2 command                                | K8s equivalent                                  |
|--------------------------------------------|-------------------------------------------------|
| `task ec2:status`                          | `task remote:status`                            |
| `task ec2:exec CMD='...'`                  | `kubectl exec -n attack-simulation <pod> -- ...`|
| `task ec2:logs ROLE=orchestrator`          | `task remote:logs ROLE=orchestrator`            |
| `task ec2:redis:forward`                   | `kubectl port-forward -n attack-simulation svc/redis 6379:6379` |
| Loki query (same Grafana)                  | Filter on `namespace="attack-simulation"` instead of `deployment="alpha-operator-range-kali-ares"` |

## Reference: Loki labels seen on grafana.techvomit.xyz

- `app`: `ares` covers everything ares writes
- `deployment`: `alpha-operator-range-kali-ares` for the EC2 box
- `environment`: `prod` or `local`
- `job`: `orchestrator.log`, `recon.log`, `credential_access.log`, `cracker.log`, `acl.log`, `privesc.log`, `lateral.log`, `coercion.log`, `syslog`, `auth.log`, `user-data`, `ansible`
- `service_name`: `ares`, `ares-orchestrator`, `ares-<role>-agent`, also blue: `ares-blue-orchestrator`, `ares-blue-triage`, etc.
- `host`: `kali`

If a label value is missing from this list, run `mcp__grafana__list_loki_label_values` to discover what's actually shipping.

## Reporting

When you finish debugging, return a short report:

- **Op id**, current `status`, runtime, token total, **objective state** (domains compromised, hosts owned, creds, hashes).
- **Baseline comparison** in one line: how this op's progress curve compares to the last 2-3 ops at the same runtime. Skip only if no prior op exists.
- **Verdict**: `healthy / wedged / crashed / slow-vs-baseline / unknown`. **Never** say "healthy" without citing two state snapshots 60s apart that show state advancing, or fresh log lines showing tool calls succeeding in the last minute.
- **Root cause** in one sentence, with the SSM/Loki/CLI evidence that pins it (quote the log line; cite the failed-task `task_type` and `role`).
- **Next action** — restart, redeploy, inject state, file a bug — and the exact command(s) to run.

Do not narrate every probe. The user wants the answer and the command to fix it, not the journey. But do not skip probes either: if you find yourself drafting a "healthy" verdict without having grepped the orchestrator tail for the wedge signatures and pulled `ares ops list` for baseline, stop and go back to Step 0.