--- name: log-triage description: Analyze Elixir's error and main logs for failures, recurring issues, and operational signals, then recommend concrete actions --- # Log Triage Surface what's actually going wrong (vs. noise), group recurring issues, and hand the user a short prioritized action list. The goal is to answer "what should I fix next?" — not to paraphrase the log. ## Which file to read Start with the error log. It exists so that this triage is cheap. | File | Contents | How to use it | |---|---|---| | `/Users/otto/Projects/elixir-bot/logs/elixir-error.log` | ERROR+ with tracebacks, ~6 lines/day | **Read this first, in full.** It is the list of things that are broken. | | `/Users/otto/Projects/elixir-bot/logs/elixir.log` | INFO+, the full narrative | Read *around* an error's timestamp for context, and for the health checks below. | | `/Users/otto/Projects/elixir-bot/elixir-v5.log` | launchd stdout/stderr catch-all | Only when a crash left nothing in the rotating logs — an interpreter traceback that killed the process, or third-party output. Not the record. | Both rotating files have `.1`–`.5` siblings after rotation; widen to those only if you need history beyond the current file. ## Scope Default to the last 24 hours unless the user specifies a window (e.g. "since the last check", "since yesterday's deploy"). Log lines begin with `YYYY-MM-DD HH:MM:SS,mmm [LEVEL] logger: ...`, so filter by timestamp prefix when narrowing. Read `logs/elixir-error.log` in full regardless of window — it is small by design, and knowing whether a signature is *new* requires seeing what came before it. ## What to look for ### Actionable signals (surface these) - `[ERROR]` and `[CRITICAL]` at any logger — always report. - `Traceback` / `Exception` blocks — report with the exception type and the top app-code frame (not just the Python framework frame). - `elixir_agent: validation_failure workflow=... reason=...` — schema or parse errors in the agent's JSON output. Group by `workflow` + `reason`. - `elixir: prompt_failure ... type=... stage=... workflow=...` — agent-response failures that reached the user path. Group by `workflow` + `type`. - `elixir: prompt_feedback emoji=thumbs_down ...` (WARNING) — a user reacted thumbs-down on an Elixir response in #ask-elixir. Each is a quality signal worth surfacing. Group by `workflow` + `channel` and report counts; the representative line should include `message_id` and `reactor` so the user can grep the conversation. Thumbs-up is INFO and not a triage concern unless the user asks. - `tool_call_failure`, `tool_error`, `ingest_failed`, `signal_failure`, `retry_exhausted`, `truncation`, `unexpected_error` — any custom failure tag from the app. - Discord reconnect storms (`Attempting a reconnect` repeated within minutes) — note if >3 in an hour. - APScheduler missed / misfired jobs. - New failure signatures that did not appear earlier in the log — those are the most interesting. ### Known noise (suppress by default, mention only if user asks) - `PyNaCl is not installed, voice will NOT be supported` — environmental, harmless. - `discord.gateway: Shard ID None has connected to Gateway` — normal connect. - `apscheduler.scheduler: Adding job tentatively` / `Added job ... to job store` — startup chatter. - `elixir_heartbeat: Heartbeat: N signals detected` — routine heartbeat. Only flag if heartbeats stop or if `N` spikes abnormally. ### Operational health checks After triaging failures, spot-check these even if nothing errored: - Is `engine-tick` firing roughly every 10 minutes and `awareness-loop` on its registered cadence? Long gaps mean the scheduler stalled. - Are `runtime_job_status` failures corroborated in `logs/elixir-error.log`? - Any signal that silently went to zero — e.g. no `agent_loop` entries for a workflow that usually runs hourly? ## Grouping and dedup Do not dump raw log lines. For each issue: 1. **Signature**: a short stable key (e.g. `validation_failure/channel_update/schema_error`). 2. **Count**: how many times it fired in the window. 3. **First/last seen**: timestamps of earliest and most recent occurrence in the window. 4. **Representative line**: one full log line so the user can grep for it. 5. **Context**: if an exception, the top app-code frame (`runtime/…`, `agent/…`) and the relevant `workflow=` / `channel_id=` / `author_id=` fields. A recurring signature that fires 40 times is one issue, not 40. ## Output format Write a short triage report, top-down by priority. Use this structure: ``` ## Log Triage — **Summary:** <1 sentence — e.g. "2 recurring failures, 1 new, scheduler healthy"> ### Priority issues 1. **** — occurrences, → - Representative: `` - Likely cause: - Recommended action: 2. ... ### Low priority / noise - ### Health - Heartbeats: - Scheduled updates: - New signatures this window: ``` Keep the whole report tight — under ~40 lines for a healthy day. If nothing is wrong, say so in one sentence and stop. ## Recommended actions — be specific Do not write "investigate the error." Point to the file and what to check. Examples of the level of specificity expected: - "Recurring `validation_failure workflow=channel_update reason=schema_error detail=null response is not allowed`. The model is returning bare `null` when it should return an object or allowed null sentinel. Check `_proactive_channel_system` in `agent/prompt_builders.py` — the schema instruction may be ambiguous about when null is allowed." - "Three Discord reconnects in 20 minutes around 03:29. Likely transient gateway flap; only act if this repeats. If it does, check network / token health on the host." - "Heartbeat gap from 04:12 → 08:21 (4 hours, expected ~30 min cadence). APScheduler may have stalled — check for missed-job warnings and confirm the process didn't get OOM-killed." ## When to act vs. just report Only edit code if the user explicitly asks you to fix an issue. By default, this skill is **read-only analysis** — it produces a report and stops. The user will pick which issue to dig into next. ## Arguments $ARGUMENTS