--- name: llm-gateway-incident-triage description: Diagnostic playbook for triaging a self-hosted LLM gateway (LiteLLM or any Postgres-backed OpenAI-compatible proxy) running via Docker Compose on a homelab host. Use when a client reports connection failures, timeouts, "provider unreachable", "model server not responding", an unexpected fallback trigger, or an auth/key error against the gateway, and you need the actual root cause rather than a guess. metadata: version: 1.0.0 --- # Triage a self-hosted LLM gateway incident For a homelab host running a Docker Compose, Postgres-backed LLM gateway (LiteLLM is the reference case, but this applies to any similar OpenAI-compatible proxy — Portkey, a custom router, etc.) fronting multiple providers behind aliases, with per-client virtual keys, bound to a private IP. A client reports a connection failure and you need to find what actually happened, not just restart something and hope. This is a specialization of root-cause-first debugging discipline for this domain: gather evidence before touching anything, form a hypothesis only once the evidence supports it, verify against ground truth (not a convenience API), then confirm the fix actually resolved it. If `superpowers:systematic-debugging` is available in your environment, treat this playbook as that skill's four-phase structure applied here; the phases below encode the same discipline inline either way. ## Files - `scripts/collect-evidence.sh` — read-only sweep that runs the host localization, container status, restart-state, and wide-net log grep from Phase 1 in one pass. Parameterized by `GATEWAY_CONTAINER` (required), `LOG_WINDOW`, `GREP_PATTERN`, `TAILSCALE_IFACE`. Used in Steps 1-3 below. ## When to use this - A client (human or another agent) reports the gateway is unreachable, timing out, or falling back to a different provider unexpectedly. - Auth/key errors that don't match what the gateway's own API says about that key. - Any "it worked, then it didn't, then it worked again" report against a self-hosted LLM proxy. ## Phase 1 — Localize and gather evidence (read-only, no fixes yet) 1. **Confirm where you're actually running before assuming you need remote access.** Run `hostname` and check for the private network interface, e.g. `ip addr show tailscale0`. If you're already on the target host, every later step is a local `docker`/`psql` command, not an SSH round-trip. Don't skip this — assuming you need to SSH in when you're already there wastes the whole investigation's setup. 2. **Find the container and confirm it's up:** ```bash docker ps --filter "name=${GATEWAY_CONTAINER}" ``` 3. **Grep logs wide, then narrow — don't start narrow.** Pull a broad time window with a broad pattern first: ```bash docker logs "${GATEWAY_CONTAINER}" --since "${LOG_WINDOW}" -t 2>&1 \ | grep -iE "error|fail|unreachable|timeout|refused|expired|APIConnectionError|fallback" ``` The first pass will pull in structural noise — a startup banner that happens to print a model alias containing "fallback", a routine tool-call payload, etc. Read what came back and progressively drop patterns that matched noise rather than assuming the first hits are the real signal. `scripts/collect-evidence.sh` runs steps 1-3 in one shot: ```bash GATEWAY_CONTAINER= LOG_WINDOW=6h ./scripts/collect-evidence.sh ``` 4. **Convert the client's reported time to the log's timezone explicitly.** If the client reports a wall-clock time in their own local offset, convert it to the log's timezone (usually UTC) and write down the conversion before matching log lines against it. Don't eyeball "something happened a few hours ago" as good enough; a burst of activity outside the actual converted window is a red herring, not evidence. ## Phase 2 — Verify against ground truth 5. **Never trust an API convenience endpoint when it looks inconsistent with a log line — go to the database directly.** If the gateway's own `/key/info` or equivalent reports a key as valid but the logs show an auth exception for it, query the underlying table: ```bash docker exec "${DB_CONTAINER}" psql -U "${DB_USER}" -d "${DB_NAME}" \ -c "SELECT * FROM ${TABLE} WHERE token = '';" ``` If a filtered query doesn't explain the log line, **pull the full table unfiltered** rather than guessing at a better filter — the row that explains the discrepancy may not be the one you assumed was relevant (e.g. an internal admin-dashboard session token, not any real client key). ## Phase 3 — Attribute cause, don't just correlate in time 6. **Match the client-reported symptom's vocabulary to the layer that could actually produce it.** "Unreachable" / "not responding" are transport-level phrases — they describe a connection that couldn't be made, not a well-formed error response. A `401` from an expired token is a real HTTP response, not "unreachable"; it cannot be the cause of a transport-level symptom even if it's in the same time window. If the container itself restarted, that's the transport-level candidate — check: ```bash docker inspect "${GATEWAY_CONTAINER}" \ --format 'StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' ``` 7. **Rule out an upstream/provider outage explicitly — don't conclude it by omission.** Grep the gateway's own router-level fallback and connection-error patterns across the full window. If nothing real turns up, say so in the incident summary as a stated, searched-for negative ("no `APIConnectionError` events against any provider in the window"), not a silent absence. 8. **If a container restart is in play, check for an unattended-restart source before writing "root cause: unknown."** A clean `ExitCode: 0` with a normal `RestartPolicy` points to a deliberate stop+start, not a crash. Check any auto-updater/watcher container (watchtower or similar) for its own logs around the same timestamp before giving up on who triggered it. ## Phase 4 — Confirm resolution, don't just explain the past 9. **Verify current health with a live request**, not just a clean theory: ```bash curl -s -o /dev/null -w '%{http_code}\n' /v1/models \ -H "Authorization: Bearer " ``` Follow with a fresh grep of the last 30 minutes of logs. Only then is the incident actually closed. ## Quick reference: real outage vs. noise | Signal | Real outage candidate | Likely noise | | --- | --- | --- | | Error type | Transport (connection refused/timeout/reset) | Well-formed HTTP error (401/403/429) | | Source of the token/session | A real client's virtual key | An internal admin/dashboard session with an empty alias | | Timing | Overlaps a container restart or provider-side `APIConnectionError` burst | Coincides with a routine credential/session rotation window | | `docker inspect` state | Non-zero `ExitCode`, `OOMKilled: true`, unexpected `RestartCount` jump | Clean `ExitCode: 0`, restart policy explains it | ## Notes / gotchas - Parameterize container names, DB credentials, and table names the way `expose-private-service` parameterizes `BACKEND_IP` — this playbook (and `scripts/collect-evidence.sh`) never hardcodes a stack's specific names, since LiteLLM, Portkey, and custom routers all differ here. - An "I didn't find X" conclusion is only useful if you show the search (the grep pattern and window you used) — state it explicitly in whatever summary you write, not as a silent gap. - This playbook is diagnostic only. It does not restart services, rotate keys, or change config — those are separate, deliberate actions once the root cause is confirmed.