---
name: kelly-agent-observability
license: MIT
description: "Agent Fleet Observability Desk App-in-Skill — local mock dashboard visualizing a fleet of LLM agents running behind a shared AI gateway for your organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over generated mock telemetry, not a live monitoring integration."
---

# Agent Fleet Observability Desk

## Overview

Use this skill as a local operator surface over a MOCK fleet of LLM agents
running behind a shared AI gateway for a generic organization. It aggregates
per-agent call volume, latency (p50/p95), error/failure rate, and cost into one
file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent
Health table, and a Trace Detail view that highlights exactly where a chain of
tool calls broke.

Default interaction mode: App UI. Unless the user explicitly asks for chat-only
handling, seed mock telemetry if missing, start/reuse the local app with
`app/start.sh`, and give the actual local URL. Use chat-only mode only when the
user says "纯聊天", "chat only", "no UI", or similar.

This is a demo/reference dashboard: all data is generated locally by
`scripts/generate_fleet_data.ts`. There is no real gateway, no real agents, and
no external network calls anywhere in this skill.

## App UI Screenshots

<table>
  <tr>
    <td width="50%"><img src="assets/screenshots/overview.png" alt="Fleet overview"></td>
    <td width="50%"><img src="assets/screenshots/agent-health.png" alt="Agent health table"></td>
  </tr>
  <tr>
    <td><strong>Fleet Overview</strong><br>Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid.</td>
    <td><strong>Agent Health</strong><br>Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.</td>
  </tr>
  <tr>
    <td width="50%"><img src="assets/screenshots/trace-detail.png" alt="Trace detail with chain break"></td>
    <td width="50%"><img src="assets/screenshots/overview.zh-CN.png" alt="Fleet overview in Chinese"></td>
  </tr>
  <tr>
    <td><strong>Trace Detail</strong><br>Ordered step timeline for one trace; the step where the chain broke is visually flagged.</td>
    <td><strong>Chinese UI</strong><br>Full zh-CN chrome via the language toggle or <code>lang=zh</code>.</td>
  </tr>
</table>

## Boundary

- Local, read-only mock dashboard. The app reads and writes local files only.
- The only mutating endpoint is `POST /api/handoffs`, which appends a
  human-in-the-loop acknowledgement or investigation note to a local
  `app/.data/handoffs.jsonl` file. It never calls any remote system, agent
  gateway, or third-party API.
- Never reference any real company, brand, or production system. Agent names
  are generic archetypes (Booking Assistant, Support Triage, etc.) for a
  generic organization ("your organization", "the fleet").

## Local App

Start the dashboard with:

```bash
skills/kelly-agent-observability/app/start.sh
```

First run installs `hono` and `@hono/node-server`, then seeds mock telemetry
into `app/.data/fleet.json` if it doesn't already exist. The frontend is
zero-build vanilla JS/HTML/CSS. The app uses local HTTP on `127.0.0.1`,
preferring port `3100` through `4100`, or `KELLY_AGENT_OBS_UI_PORT` when set.

To regenerate mock telemetry at any time:

```bash
node skills/kelly-agent-observability/scripts/generate_fleet_data.ts
```

## Demo Mode

- `?demo=1` opens a deterministic, fully offline mock fleet (8 agents, 48h of
  hourly buckets, ~16 traces per agent) for documentation and screenshots,
  without touching `app/.data/fleet.json`.
- `?demo=agents` and `?demo=trace` select named mock scenes (agent health table,
  a broken trace).
- `lang=en` or `lang=zh` forces UI chrome language for screenshots.
- Demo API responses never read or write local files.

UI language: support English and Chinese chrome with `Auto` default.

## Data Model

Read `references/fleet-schema.md` before editing the app, scripts, or the
generator. Primary local files:

- `app/.data/fleet.json`: generated mock telemetry (agents, per-agent metrics,
  hourly buckets, traces). Regenerated by `scripts/generate_fleet_data.ts`.
- `app/.data/handoffs.jsonl`: append-only human-in-the-loop handoff log written
  by `POST /api/handoffs`.

## Views

- `#/overview`: total calls (24h), total cost (today), degraded/critical/
  healthy agent counts, and a per-agent card grid with a call-volume sparkline.
- `#/agents`: sortable Agent Health table (call volume, p50/p95 latency, error
  rate, cost, status badge).
- `#/agents/<agent_id>`: agent detail — metrics, sparkline, recent traces, and
  a handoff form (acknowledge / needs investigation).
- `#/traces/<trace_id>`: trace detail — ordered step timeline with the
  chain-break step visually flagged, and a handoff form.
- `#/handoffs`: history of recorded handoffs.
- `#/settings`: data provider and status-threshold summary.

## Safety

- Never invent a real incident or real company data; this is illustrative mock
  telemetry only.
- Keep all writes local and file-based (`app/.data/handoffs.jsonl`). No
  external network calls anywhere in this skill.
- Redact nothing sensitive is collected here — there are no credentials in
  this skill.