--- name: ring:creating-grafana-dashboards description: | Author Grafana dashboards for Lerian Go services rooted in real lib-observability telemetry (tracing, metrics, log, constants). Three phases — Sweep (telemetry inventory), Iterate (PM deliberation on SLIs/SLOs and alerts), Author (Grafonnet libsonnet → JSON in CI) — and installs a blocking CI drift gate. Use when scaffolding dashboards, building a telemetry dictionary, or auditing observability. Skip if service is non-Go, emits no telemetry, or task is just folder organization. --- # Creating Grafana Dashboards (lib-observability, PM-team) ## When to use Sweep mode: - "Create / scaffold Grafana dashboards for this service" - "Inventory telemetry / build telemetry dictionary" - "Audit observability before designing dashboards" - "Produce dashboards as code for {service}" - "PM wants visibility into {domain} — what dashboards do we need?" Reference mode: - "What's the right panel for HTTP request latency?" - "RED vs USE methodology for this metric type?" - "How do I compose Grafonnet panels?" - "Which Grafonnet template fits a counter / histogram / gauge?" ## Skip when - Service is not a Go project (lib-observability is Go-only at this skill's scope) - Service emits no telemetry (pre-instrumentation; instrument the service before dashboard authoring, then use ring:dev-implementation to verify observability checks pass) - Task is purely Grafana folder organization or dashboard import (no authoring) - Service is consumer-only sidecar with no metrics surface ## Sequence **Runs before:** ring:dev-cycle, ring:dev-cycle-frontend ## Related **Complementary:** ring:dev-implementation, ring:codebase-explorer, ring:streaming-event-mapping, ring:using-lib-observability, ring:using-tracing **Similar:** ring:using-runtime, ring:using-assert ## Prerequisites - Go service with lib-observability initialized in bootstrap (`tracing.NewTelemetry`, `metrics.NewFactory`, `zap.NewLogger`) - At least one metric, span, or structured log emission point present - docs/ directory writable - Grafonnet toolchain available in CI (jsonnet + grafonnet-lib) — installer instructions in ci-drift-check.md Orchestrates a 3-phase, 8-gate workflow to produce Grafana dashboards grounded in real telemetry. You orchestrate. Agents explore. PM iterates. You NEVER read, write, or edit source code directly during the sweep. **Announce at start:** "Using ring:creating-grafana-dashboards through 8 gates (0–7)." ## Mode Selection | Request Shape | Mode | |---|---| | "Create / scaffold dashboards" / "build telemetry dictionary" | **Sweep** (run gates 0–7) | | "Which panel for X?" / "RED vs USE?" / "Grafonnet template for Y?" | **Reference** (load `sub-files/reference.md`) | --- # SWEEP MODE ## Telemetry Architecture (lib-observability) Lerian Go services emit telemetry through `github.com/LerianStudio/lib-observability`: - **Tracing** via `lib-observability/tracing` — `tracer.Start(ctx, name, opts...)` returning `context.Context, trace.Span` - **Metrics** via `lib-observability/metrics` — fluent factory producing `meter.Int64Counter`, `meter.Float64Histogram`, `meter.Int64UpDownCounter`, `meter.Int64ObservableGauge` - **Logs** via `lib-observability/log` (interface) and `lib-observability/zap` (implementation) — structured fields, automatically correlated with active span via `trace_id`/`span_id` - **OTel attribute / metric / event names** via `lib-observability/constants` — canonical string constants; dashboards reference these for label and metric names - **Cross-cutting** — `tenant_id` propagation through context, error attribution via `span.RecordError` + `span.SetStatus` > **Deprecated shims:** `lib-commons/v5/commons/{opentelemetry,zap,log,metrics}` still compile but route through lib-observability. New emission sites MUST import lib-observability directly. The sweep detects both canonical and shim imports. **WebFetch canonical docs (lib-observability — develop branch; main has only LICENSE + README):** - Tracing: `https://raw.githubusercontent.com/LerianStudio/lib-observability/develop/tracing/doc.go` - Metrics: `https://raw.githubusercontent.com/LerianStudio/lib-observability/develop/metrics/doc.go` - Log: `https://raw.githubusercontent.com/LerianStudio/lib-observability/develop/log/doc.go` - Constants: `https://raw.githubusercontent.com/LerianStudio/lib-observability/develop/constants/doc.go` **WebFetch changelog:** `https://raw.githubusercontent.com/LerianStudio/lib-observability/develop/CHANGELOG.md` ## Authoring Format: Grafonnet (Mandatory) Dashboards are authored as **Grafonnet** (Jsonnet templating language) — compiled to JSON in CI. Raw JSON dashboards are FORBIDDEN. Reasons: - Diffable in PR review (libsonnet is code-shaped, JSON is not) - Composable via `import` and inheritance - Templated panel patterns reusable across themes - Single source of truth — JSON is a build artifact, not a checked-in source Toolchain setup: `sub-files/ci-drift-check.md`. Panel templates: `sub-files/grafonnet-templates/`. ## Theme Taxonomy **Free-form per service.** PM defines the theme directories under `docs/dashboards/{theme}/` during Gate 5. No enforced taxonomy — Lerian services are observability islands and theme naming reflects each service's domain. Common-but-not-mandatory examples: `transactions/`, `auth/`, `ledger/`, `infrastructure/`, `business-kpis/`, `sla/`. The skill SUGGESTS themes from dictionary contents in Gate 4; PM ACCEPTS, RENAMES, MERGES, or SPLITS in Gate 5. ## Drift Gate Posture CI drift detection is **BLOCKING from day 1**. Any divergence between regenerated dictionary and committed `telemetry-dictionary.md` fails the PR. This is a deliberate cold-start choice — the skill is greenfield, no installed base to retrofit, every new metric emits under the strict regime. Drift gate spec: `sub-files/ci-drift-check.md`. ## Gate Overview | Gate | Name | Agent | Cadence | |------|------|-------|---------| | 0 | Stack Detection | Orchestrator (grep + read) | Once per run | | 1 | Telemetry Sweep (7 angles) | ring:codebase-explorer × 7 parallel | Once per run | | 2 | Dictionary Assembly + Validation | Orchestrator (deterministic merge) | Once per run | | 3 | Dictionary Rendering | Orchestrator → markdown writer | Once per run | | 4 | Theme Proposal + Dashboard Plans | Orchestrator (LLM opinion via reference.md) | Once per run | | 5 | PM Iteration — NEVER SKIPPABLE | User (PM team) | Loops until APPROVED | | 6 | Grafonnet Authoring | ring:backend-engineer-golang per theme | Per approved theme | | 7 | CI Drift Gate Setup | Orchestrator | Once (idempotent) | Gates execute sequentially. Gate 1 parallelizes internally across 7 angles. Gate 6 parallelizes per approved theme. ## Gate 0: Stack Detection Orchestrator executes directly. Detect in parallel: ``` 1. Go version: grep "^go " go.mod | head -1 2. lib-observability version: grep "lib-observability" go.mod 3. lib-commons version: grep "lib-commons" go.mod 4. Tracing package present: grep -rn "lib-observability/tracing\|lib-commons/v5/commons/opentelemetry" internal/ cmd/ # canonical + deprecated shim 5. Metrics package present: grep -rn "lib-observability/metrics\|lib-commons/v5/commons/opentelemetry" internal/ cmd/ # canonical + deprecated shim 6. Meter init: grep -rn "Meter(\|NewMeter\|meter.Int64Counter\|meter.Float64Histogram" internal/ cmd/ 7. Tracer init: grep -rn "Tracer(\|NewTracer\|tracer.Start" internal/ cmd/ 8. Log emission: grep -rn "lib-observability/log\|lib-observability/zap\|lib-commons/v5/commons/log\|lib-commons/v5/commons/zap" internal/ cmd/ # canonical + deprecated shim 9. HTTP framework: grep -rn "gofiber/fiber\|labstack/echo\|gin-gonic" go.mod 10. gRPC server: grep -rn "grpc.NewServer" internal/ cmd/ 11. RabbitMQ command consumers: grep -rn "lib-commons/v5/commons/rabbitmq" internal/ cmd/ # command queues; event emission goes through lib-streaming 12. lib-streaming present: grep "lib-streaming" go.mod 13. Tenant source: grep -rn "tmcore.GetTenantIDContext\|GetTenantID" internal/ 14. Existing dictionary: test -f docs/dashboards/telemetry-dictionary.md 15. Existing dashboards: ls docs/dashboards/ 2>/dev/null 16. Grafonnet in CI: test -f .github/workflows/telemetry-drift.yml 17. Service identity: cat go.mod | grep "^module" ``` Emit `/tmp/dashboards-recon.json`: ```json { "service_name": "...", "go_version": "...", "lib_observability_version": "...", "lib_commons_version": "...", "lib_streaming_present": false, "tracing_initialized": true, "metrics_initialized": true, "metric_emission_present": true, "trace_emission_present": true, "structured_log_present": true, "deprecated_shim_imports": ["lib-commons/v5/commons/opentelemetry"], "http_framework": "fiber|echo|gin|none", "grpc_server_present": true, "rabbitmq_command_consumers_present": true, "tenant_source": "tmcore.GetTenantIDContext", "existing_dictionary": true, "existing_themes": ["transactions", "ledger"], "drift_gate_installed": false } ``` **HARD GATE:** - If not Go → STOP. - If no lib-observability tracing/metrics usage detected (canonical or deprecated shim) → STOP, surface "service is not instrumented; instrument the service before dashboard authoring, then use ring:dev-implementation to verify observability checks pass" to user. - If service has < 3 metric/trace/log emissions → STOP, surface "insufficient telemetry surface for dashboards". ## Gate 1: Telemetry Sweep (7 Parallel Angles) ### ⛔ STOP-CHECK BEFORE DISPATCH Before emitting any Task call, count the explorers you intend to launch in this turn. - Count MUST equal 7. - If count < 7 → STOP. Do not partial-dispatch. Reconcile against the 7 angles below and try again. - The 7 angles are the canonical sweep. No substitutions, no omissions. ### ⛔ MUST NOT trickle-dispatch All 7 explorers leave in the SAME TURN, before reading any explorer output. Forbidden sequences: - Dispatch explorer 1 → read result → dispatch explorer 2 - Dispatch a subset → wait → dispatch the rest - Dispatch follow-up explorers conditioned on partial output - Loop sequentially over the angle list If you find yourself about to dispatch an explorer in a turn AFTER any explorer has already returned a result → STOP. You violated parallel dispatch. Report the violation and mark the gate INCOMPLETE rather than completing the trickle. ### Self-verify after dispatch After the dispatch turn, verify all 7 Task calls were emitted in that single turn. If fewer than 7 went out, the gate did NOT execute correctly. Mark INCOMPLETE and surface the dispatch failure — do NOT silently continue with a partial pool. ### Parallel dispatch — atomic batch Emit all 7 Task calls in a SINGLE TURN, as one atomic batch. **If your runtime exposes a `multi_tool_use.parallel` wrapper**, use it to dispatch the complete pool in one wrapped invocation. This is the canonical fan-out mechanism on OpenAI-style tool envelopes and on certain Anthropic SDK consumers — naming it explicitly activates parallel emission on runtimes where trickle-dispatch is the default behavior. **If your runtime emits parallel tool_use blocks natively** (Claude Code with Claude models), `multi_tool_use.parallel` may not be needed — but naming it is harmless and serves as an enforcement anchor. The STOP-CHECK, anti-trickle, and self-verify guards above remain binding regardless of which mechanism your runtime uses. Dispatch all 7 angles in **one parallel batch**. Wait for all before Gate 2. **Per-explorer dispatch** (`subagent_type: ring:codebase-explorer`): ``` ## Target: ## Your Angle: ## Severity / Detection Patterns / Schema / Notes ## Output Write to: /tmp/dashboards-sweep-{N}-{angle-slug}.json Schema: { angle_number, angle_name, primitives: [...] } Each primitive includes file:line, name, description, labels/attributes, unit, type-specific fields. If no findings: write file with empty primitives array. ``` The 7 angles cover: 1. **Counter metrics** — `meter.Int64Counter`, `Float64Counter`, increments, labels, descriptions, units 2. **Histogram metrics** — `meter.Float64Histogram`, `Int64Histogram`, boundaries, units, labels 3. **Gauge metrics** — `meter.Int64UpDownCounter`, `Int64ObservableGauge`, callbacks 4. **Trace spans** — `tracer.Start`, span names, kind, attributes, parent-child structure, error recording 5. **Structured log fields** — `log.With`, level usage, contexts where emitted, trace correlation 6. **Cross-cutting concerns** — `tenant_id` labeling, `trace_id`/`span_id` propagation, error attribution, request correlation 7. **Framework instrumentation** — Fiber/gRPC/RabbitMQ middleware, auto-spans, manual override sites Full angle specifications: `sub-files/sweep-angles.md`. **Verification:** 7 JSON files exist, all parse, schema-valid per `sub-files/dictionary-schema.md`. **HARD GATE:** Missing/malformed file → re-dispatch ONLY failing angle. ## Gate 2: Dictionary Assembly + Validation Orchestrator merges 7 angle JSONs into `/tmp/dashboards-dictionary.json`. Validate per `sub-files/dictionary-schema.md`: - Metric names match `^[a-z][a-z0-9_]*$` (Prometheus convention) - Span names match `^[a-z][a-z0-9_.-]*$` - All primitives have `description` ≥ 30 chars - Histograms declare `unit` (seconds, bytes, count) and `boundaries` if custom - Tenant-scoped primitives have `tenant_id` in labels/attributes - No duplicate `(name, type)` pairs across angle outputs - Cross-cutting Angle 6 findings cross-reference primitives from Angles 1–4 Validation failures → re-dispatch failing angle's explorer with correction notes. Do NOT manually edit JSON. ## Gate 3: Dictionary Rendering Orchestrator writes `docs/dashboards/telemetry-dictionary.md` from validated JSON, following `sub-files/dictionary-schema.md` rendering contract: - YAML frontmatter `_meta` block: service name, generated-at timestamp, source commit SHA, lib-commons version, primitive counts - Metrics section: one `### {metric_name}` per metric, with stable YAML block (type, unit, labels, description, emission_sites) - Traces section: one `### {span_name}` per span, with stable YAML block (kind, attributes, parents, emission_sites) - Logs section: structured fields catalog with levels and emission contexts - Cross-cutting section: tenant propagation map, trace correlation map, error attribution map **Critical**: rendering MUST be deterministic — same input JSON produces byte-identical output. Order alphabetically within each section. Sort labels alphabetically within each primitive. This is what makes drift detection in Gate 7 possible. ## Gate 4: Theme Proposal + Dashboard Plans Orchestrator analyzes the dictionary and proposes themes + dashboards. This is the LLM-opinion gate — apply `sub-files/reference.md` (RED/USE methodology, panel pattern catalog) to dictionary contents. For each proposed theme, produce a **dashboard plan** stub at `/tmp/dashboards-plan-{theme}.md`: ```markdown # Theme: {theme_name} ## Audience (proposed) - Primary: - Secondary: <...> ## Dashboards ### {dashboard_1_name} - Methodology: RED | USE | hybrid - SLIs surfaced: - Time range default: <1h | 6h | 24h | 7d> - Panels: 1. {panel_name} — {panel_pattern} on metric {metric_ref} — Grafonnet template: {template} 2. ... - Alert candidates: ### {dashboard_2_name} ... ``` Themes are SUGGESTIONS only. PM may rename, merge, split, or reject in Gate 5. ## Gate 5: PM Iteration — NEVER SKIPPABLE Present `sub-files/pm-iteration-prompts.md` checklist to PM team: - Theme names — accept, rename, merge, split? - Audience per theme — correct? - Methodology choice (RED vs USE vs hybrid) — sound for this domain? - SLIs surfaced — match what stakeholders actually need? - Time range defaults — match operational cadence? - Alert thresholds vs informational — which panels need alerts attached? - Missing dashboards — anything PM expected that wasn't proposed? **Response options:** - `APPROVED: ...` → proceed to Gate 6 for listed themes - `REVISE theme {name}: ` → loops Gate 4 for that theme only - `RENAME theme {old} -> {new}` → renames in plan, loops Gate 4 light - `REJECT theme {name}` → drops from approval list - `ADD theme {name}: ` → orchestrator generates new plan, loops Gate 4 - `BLOCKED: ` → halts skill, returns with surface for triage **HARD GATE:** Must not proceed to Gate 6 without explicit `APPROVED: ...` listing at least one theme. ## Gate 6: Grafonnet Authoring (Per Approved Theme) For EACH approved theme, dispatch `ring:backend-engineer-golang` (Lerian's Go specialist; Grafonnet is jsonnet, but the engineer's discipline around code quality and reusability transfers — and they own the lib-commons mental model that makes label correctness checkable). Per-theme dispatch: ``` ## Target: docs/dashboards/{theme}/ ## Inputs: - /tmp/dashboards-plan-{theme}.md (PM-approved plan) - docs/dashboards/telemetry-dictionary.md (canonical primitive contract) - pm-team/skills/creating-grafana-dashboards/sub-files/grafonnet-templates/ (panel libsonnet templates) - pm-team/skills/creating-grafana-dashboards/sub-files/reference.md (panel pattern → template mapping) ## Task: 1. Create docs/dashboards/{theme}/ directory 2. Write {theme}.libsonnet importing relevant grafonnet-templates panels 3. Materialize each panel from the plan with concrete metric refs from the dictionary 4. Compose into a Dashboard with rows/grid per plan structure 5. Write README.md per theme explaining: audience, SLIs, intended use, alert thresholds 6. Validate: jsonnet compiles cleanly, panel queries reference only primitives present in dictionary ## Output: - docs/dashboards/{theme}/{theme}.libsonnet - docs/dashboards/{theme}/README.md - /tmp/dashboards-build-{theme}.log (compilation output) ## Constraints: - MUST NOT invent metric names — every PromQL/LogQL/TraceQL query references primitives from telemetry-dictionary.md - MUST template tenant_id as a Grafana variable when present in primitive labels - MUST follow Grafonnet conventions (no raw JSON; if a panel pattern isn't in templates/, propose a new template) ``` **Verification:** Per theme — libsonnet compiles to JSON, README present, no metric references missing from dictionary. **HARD GATE:** Compilation failure → re-dispatch with diagnostic, do not move to Gate 7. ## Gate 7: CI Drift Gate Setup Orchestrator installs (idempotent) the drift detection workflow per `sub-files/ci-drift-check.md`: 1. Write `.github/workflows/telemetry-drift.yml` (blocking PR check) 2. Write `scripts/regenerate-telemetry-dictionary.sh` (regenerates dictionary; called by CI and locally) 3. Update `Makefile`: add `make telemetry-dictionary` target invoking the regenerate script 4. Compile all theme libsonnet to JSON in `docs/dashboards/{theme}/{theme}.json` and add to .gitignore (build artifact) 5. Surface to user: workflow path, local regen command, expected first-CI-run behavior **Idempotence:** If `.github/workflows/telemetry-drift.yml` already exists, diff against canonical version. Update only if drift detected. Surface diff to user before overwriting. ## State Persistence Save to `/tmp/dashboards-state.json`: ```json { "skill": "creating-grafana-dashboards", "service_name": "", "current_gate": 0, "gates": { "0": "PENDING", "1": "PENDING", "2": "PENDING", "3": "PENDING", "4": "PENDING", "5": "PENDING_USER_APPROVAL", "6": "PENDING", "7": "PENDING" }, "metrics": { "primitives_counters": 0, "primitives_histograms": 0, "primitives_gauges": 0, "primitives_spans": 0, "primitives_log_fields": 0, "themes_proposed": 0, "themes_approved": 0, "dashboards_authored": 0 } } ``` --- # REFERENCE MODE Full reference content in `sub-files/reference.md`. Load sections relevant to the question. ## Quick Navigation | # | Section | What you'll find | |---|---|---| | 1 | RED Methodology | Rate, Errors, Duration — when each metric type fits | | 2 | USE Methodology | Utilization, Saturation, Errors — for resources | | 3 | Panel Pattern Catalog | Mapping primitive type → panel pattern → Grafonnet template | | 4 | Theme Decision Tree | How to suggest themes from dictionary contents | | 5 | Grafonnet Conventions | Naming, composition, variable conventions, tenant templating | | 6 | Alert Threshold Heuristics | When to attach alerts, default thresholds, escalation tiers | | 7 | Cross-cutting Patterns | Tenant variable, trace exemplars, log-to-trace links | | 8 | Anti-pattern Catalog | Six failure modes (vanity panels, alert noise, etc.) | Read `sub-files/reference.md` for full detail.