--- name: agent-performance-report model: sonnet description: Produces an aggregated markdown performance report from stored trace data — tool call metrics, token usage, cost, bottlenecks, and numbered recommendations. Use when: "session cost report", "what ran this week", "why did this spike", "end-of-sprint performance review", "budget alert fired". license: proprietary category: agent-telemetry inputs: - name: time_period type: string description: Time period for the report (e.g. this week, this sprint, today) required: false outputs: - name: performance_report type: string description: Aggregated markdown performance report with tool call metrics, token usage, cost, bottlenecks, and numbered recommendations --- ## I. Philosophy A performance report is an accountability artifact. It converts the raw trace data accumulated during agent operation into a human-readable narrative that answers three questions: what ran, how well did it run, and what should change. The value of this report is not comprehensiveness — it is actionability. A report that surfaces five specific recommendations is more useful than a report that lists every span in the trace store. Structure the output so a non-technical reviewer can understand cost trends and a technical reviewer can locate bottlenecks without scrolling. ## II. When to Use Use this skill: - At the end of a work session to summarize what the agent did and how much it cost. - After an incident or unexpected failure to understand which nodes failed, in what order, and why. - As part of a sprint retrospective to analyze token spend against output quality. - When tuning agent configuration — use successive reports to measure whether changes to model selection, tool ordering, or plan structure are improving efficiency. - When a budget alert fires — use this skill to identify which operations triggered the alert and whether the spend was justified. Do not use this skill for real-time monitoring during an active session. For live tool call visibility, use `tool-intercept-logger`. This skill operates on stored, completed trace data. ## III. Workflow **Step 1 — Accept time range and filters.** Accept the following inputs: - `time_range`: required. ISO 8601 start/end pair (e.g., `2026-04-01T00:00:00Z` to `2026-04-05T23:59:59Z`). If only a start is given, end defaults to now. Convenience shortcuts: `"today"`, `"last_7_days"`, `"this_month"`. - `agent_filter`: optional. Restrict to spans generated by a specific agent ID. - `user_filter`: optional. Restrict to spans for a specific user ID. - `tool_namespace`: optional. Restrict tool-level metrics to a namespace prefix (e.g., `mcp_by_dojo:*`). If no filters are provided, report covers all agents and users for the time range. **Step 2 — Query stored spans.** Query `TraceStorage` for all spans within the time range, applying any filters. Each span contains: `SpanID`, `TraceID`, `ParentID`, `Name`, `StartTime`, `EndTime`, `Inputs`, `Outputs`, `Metadata`, `Status`. Also query for orchestration lifecycle events during the same range: - `orchestration_plan_created` — provides `plan_id`, `node_count`, `estimated_cost` - `orchestration_node_end` — provides `duration_ms`, `state` per node - `orchestration_complete` — provides `total_nodes`, `success_nodes`, `failed_nodes` per plan Query `BudgetTracker` for usage snapshots at the start and end of the time range to calculate net consumption per tier (query, session, monthly). **Step 3 — Aggregate tool call metrics.** For all spans where `Name` matches `"node."`: ``` total_calls: success_rate: durations: avg_duration_ms: p50_duration_ms: p95_duration_ms: <95th percentile> calls_by_tool: call count> ``` Compute duration statistics per tool as well as globally. **Step 4 — Aggregate token and cost metrics.** From span `Inputs`, `Outputs`, and `Metadata` (which carries OTEL attributes from the TraceLogger): ``` token_usage_by_tool: {input_tokens, output_tokens}> token_usage_by_model: {input_tokens, output_tokens}> total_input_tokens: total_output_tokens: cost_by_operation_type: estimated_cost> total_cost: ``` For cost breakdown, infer operation type from tool name using the same keyword mapping as `budget-guard`: web/compute/file/memory. **Step 5 — Aggregate DAG execution stats.** From `orchestration_plan_created` and `orchestration_complete` events: ``` plans_created: avg_nodes_per_plan: total_successful_plans: total_failed_plans: 0> parallel_ratio: sequential_ratio: <1 - parallel_ratio> avg_plan_cost_estimate: ``` `parallel_ratio` requires reconstructing DAG structure from ParentID relationships in span data. If span data is insufficient to reconstruct the DAG, report this metric as `"unavailable (insufficient span data)"` rather than approximating. **Step 6 — Identify performance issues.** Apply four detection rules: 1. **Slow tools**: tools where p95 duration > 2x the median p95 duration across all tools. List each with its p95, median, and percent above threshold. 2. **Expensive operations**: top 5 operations by estimated cost. Include tool name, call count, total cost, and cost per call. 3. **Failed nodes**: for each plan with `failed_nodes > 0`, list the failing span names, their `Status` values, and the `error_message` from span Metadata. Group repeated errors by pattern rather than listing each instance. 4. **Budget utilization trend**: compare BudgetTracker usage at start vs. end of the time range for each tier. Flag any tier where utilization increased by more than 20 percentage points within the range. **Step 7 — Generate markdown report.** Structure the report as follows: ```markdown # Agent Performance Report Period: Filters: Generated: ## Summary <3-5 sentence narrative: total activity, overall cost, success rate, notable issues. Written for a non-technical reader.> ## Tool Call Metrics ## Token and Cost Breakdown ## DAG Execution ## Performance Issues ### Slow Tools ### Most Expensive Operations ### Failed Nodes ### Budget Utilization ## Recommendations ``` **Step 8 — Output.** Return the formatted markdown report. If the report exceeds 300 lines, offer a condensed version (summary + top issues + recommendations only) alongside the full version. ## IV. Best Practices - The narrative summary in the report header matters as much as the tables. A reviewer scanning under time pressure will read the summary and jump to recommendations. Make both precise and complete. - Recommendations must be specific. "Optimize slow tools" is not a recommendation. "Consider replacing `web_search` with a cached lookup for the 3 queries called more than 10 times this week" is a recommendation. - When span data is incomplete (missing OTEL attributes, partial records), say so explicitly in the affected section rather than computing metrics from partial data without caveat. Partial accuracy is more dangerous than acknowledged gaps. - Group error patterns before listing individual failures. Five instances of `"connection timeout on mcp_by_dojo:github_search"` is one pattern, not five findings. - Token usage by model is the most actionable cost metric. Different models have different cost-per-token rates; the same token count can represent significantly different spend depending on which model was used. - If the time range spans multiple calendar months, break down cost and budget utilization by month. Monthly budget resets make cross-month aggregates misleading for capacity planning. ## V. Quality Checklist Before completing this skill, verify: - [ ] Time range confirmed with user or defaults applied with explicit note - [ ] All three data sources queried: TraceStorage spans, orchestration events, BudgetTracker snapshots - [ ] Duration statistics computed per-tool (not only globally) - [ ] Cost breakdown includes both by-tool and by-operation-type views - [ ] DAG parallel_ratio reported or noted as unavailable with reason - [ ] All four performance issue types checked (slow tools, expensive ops, failed nodes, budget trend) - [ ] Recommendations are specific and numbered (3-5 items) - [ ] Report includes generated timestamp and applied filters - [ ] Incomplete span data disclosed in affected sections, not silently omitted ## Output - Markdown report returned inline (or offered as file if > 300 lines) - Sections: Summary, Tool Call Metrics table, Token and Cost Breakdown tables, DAG Execution table, Performance Issues (slow tools, expensive ops, failed nodes, budget trend), Recommendations (numbered 3-5 items) - Generated timestamp and applied filters included in report header ## Examples **Scenario 1:** "Give me a performance report for last week" → Full markdown report covering all agents for the prior 7-day window, with tool latency tables, cost breakdown by operation type, and 5 actionable recommendations. **Scenario 2:** "Why did costs spike on April 3rd?" → Report filtered to 2026-04-03, identifying the top 5 expensive operations, any failed nodes that triggered retries, and the binding budget tier. ## Edge Cases - If TraceStorage has no spans for the requested time range, report this explicitly rather than returning empty tables. - If DAG parallel_ratio cannot be computed from span data, report it as "unavailable (insufficient span data)" rather than approximating. - If time range spans multiple months, break down cost and budget utilization by month — monthly budget resets make cross-month aggregates misleading. ## Anti-Patterns - Using this skill for real-time monitoring during an active session — use `tool-intercept-logger` instead, which captures live per-call data. - Listing every individual span failure rather than grouping repeated errors by pattern — five instances of the same timeout is one finding, not five.