---
name: backend-observability
description: Instrument features so errors, traces, and logs are correlated from the first line. Use when adding logging, tracing, or monitoring; wiring Sentry, Langfuse, or structured logs; or when asked "add logging", "instrument this", "why can't I debug prod", "correlate the error to the trace", "redact PII from logs", "set up alerts/SLOs".
license: MIT
---

# Observability Instrumentation

> The build-time counterpart to your monitoring stack. The Sentry plugin installs the SDK; `debug-sentry-monitor` triages after the fact; `audit-langfuse-llm` audits LLM traces. about instrumenting **correctly while you build** so those tools have signal to work with — and so a 3am incident is debuggable.

## When this fires
Adding logging / tracing / metrics to new code, reviewing instrumentation, or fixing "we can't tell what happened in prod." Not for installing an SDK (use the Sentry/Langfuse plugins) or post-hoc triage (use the monitor/audit skills).

## The one rule that matters most: correlation
A prod incident is only debuggable if you can pivot **error ↔ trace ↔ log ↔ user** from any one of them. Make every layer share an id.

- Generate/propagate a **request id** (or OTel `trace_id`) at the entry point (HTTP middleware, edge function, job start). Put it in async context (`AsyncLocalStorage` / context var), not a parameter threaded everywhere.
- **Stamp it on everything:** every log line, Sentry `setTag("request_id", id)` / `setContext`, and the Langfuse trace (`trace.id` or metadata). On an LLM error, attach the Langfuse trace URL to the Sentry event so you jump straight from the exception to the prompt/response.
- Set user/session/tenant scope (`Sentry.setUser`, Langfuse `userId`/`sessionId`) — scrubbed (see redaction).

## Structured logging discipline
- **Structured, not string-soup.** Emit JSON with stable fields (`level`, `msg`, `request_id`, `event`, domain ids). One event per line. Use the platform logger (pino / structlog / slog), not bare `console.log`.
- **Levels mean things:** `error` = needs a human; `warn` = degraded but handled; `info` = state transitions / business events; `debug` = dev-only, off in prod. Don't log `error` for handled flow.
- **No `console.log` shipped to prod** — it's unsearchable, unleveled, and a PII leak risk. Remove or convert to a leveled structured log.
- **Log decisions, not noise:** log the branch taken + key inputs/outputs at boundaries, not every line. A log you'd never grep is cost, not signal.

## PII / secret redaction (non-negotiable)
- Never log tokens, passwords, API keys, full PANs, auth headers, or raw request bodies. Redact at the logger/transport layer (deny-list keys + pattern scrub) so it can't be bypassed per-call.
- Configure Sentry `beforeSend` / data-scrubbing and Langfuse masking to strip PII from events/traces. Assume anything you put in a span/breadcrumb may be retained.
- For LLM traces: decide explicitly whether prompts/completions may contain PII; mask or hash before sending if the jurisdiction requires it.

## Tracing / spans (what to wrap)
- Wrap **boundaries and slow/fallible work**: inbound request, outbound HTTP/DB/queue calls, LLM calls, background jobs. Not every function.
- Name spans by operation (`http.server`, `db.query`, `llm.generate`), add attributes (route, status, row count, model) — follow **OTel semantic conventions**, including the **GenAI conventions** for LLM spans (model, tokens in/out, cost, latency, temperature).
- **Sampling:** you don't need 100%. Head-sample normal traffic (e.g. 10–20%), but **always keep errors and slow outliers**. Document the rate; it's a cost/visibility dial.

## LLM-specific (Langfuse)
- Capture per generation: prompt, response, model, input/output tokens, cost, latency, and the eval/score if you run one. Group multi-step agents under one trace with nested spans.
- Link the Langfuse `trace_id` into the surrounding request id and into Sentry on failure — so an LLM error in Sentry is one click from the full trace.
- Tag traces with `userId` / `sessionId` / release so `audit-langfuse-llm` can slice quality by cohort.

## Alerts & SLOs (signal, not noise)
- Alert on **symptoms users feel** (error-rate spike, p95 latency, checkout/login failure, LLM eval-score drop), not every exception. A pager that cries wolf gets muted.
- Define a few SLOs (availability, latency, key-flow success) and alert on burn rate, not raw counts.
- Every alert names an owner and a first action. Route via `sentry-create-alert` / your channel.

## Definition of done
- [ ] A shared request/trace id is on every log line, the Sentry scope, and the Langfuse trace.
- [ ] From a prod error you can reach the trace, the logs, and the user in ≤2 clicks.
- [ ] Logs are structured + correctly leveled; no `console.log` shipped to prod.
- [ ] PII/secret redaction is enforced at the logger + Sentry `beforeSend` + Langfuse masking.
- [ ] Spans cover boundaries with OTel-conventional names/attributes; errors are never sampled out.
- [ ] LLM generations record model/tokens/cost/latency and link back to the request id.
- [ ] Alerts fire on user-felt symptoms with an owner, not on every exception.

## Composes with
- Sentry plugin (`sentry-sdk-setup`, `sentry-setup-ai-monitoring`, `sentry-create-alert`, `sentry-otel-exporter-setup`) — SDK + alert wiring.
- Langfuse plugin (`langfuse`) + `audit-langfuse-llm` — LLM trace capture + quality audit.
- `debug-sentry-monitor` / `debug-error` — post-hoc triage of what this instrumentation surfaces.
- `data-pipeline` — per-run pipeline metrics use these same correlation + logging rules.
- `workflow-spec-workflow-spec-tdd` — make "observable" part of the spec's "done when", not an afterthought.