--- name: cursor-plugin-posthog-exploring-llm-evaluations description: >- Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common fa metadata: version: "0.1.0" --- # Exploring LLM evaluations PostHog evaluations score `$ai_generation` events. Each evaluation is one of two types, both first-class: - **`hog`** — deterministic Hog code that returns `true`/`false` (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code. - **`llm_judge`** — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level. Results from both types land in ClickHouse as `$ai_evaluation` events with the same schema, so the read/query/summary workflows are identical regardless of evaluator type — the only thing that changes is whether `$ai_evaluation_reasoning` was written by Hog code or by an LLM. This skill covers the full lifecycle: list/inspect/manage evaluation configs (Hog or LLM judge), run them on specific generations, query individual results, and get an AI-generated summary of pass/fail/N/A patterns across many runs. ## Tools | Tool | Purpose | | ---------------------------------------- | -------------------------------------------------------------- | | `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) | | `posthog:llma-evaluation-get` | Get a single evaluation config by UUID | | `posthog:llma-evaluation-create` | Create a new `llm_judge` or `hog` evaluation | | `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) | | `posthog:llma-evaluation-delete` | Soft-delete an evaluation | | `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) | | `posthog:llma-evaluation-summary-create` | AI-powered summary of pass/fail/N/A patterns across runs | | `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events | | `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored | All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`. ## Event schema Every run of an evaluation emits an `$ai_evaluation` event. Key properties: | Property | Meaning | | --------------------------- | -------------------------------------------------------- | | `$ai_evaluation_id` | UUID of the evaluation config | | `$ai_evaluation_name` | Human-readable name | | `$ai_target_event_id` | UUID of the `$ai_generation` event being scored | | `$ai_trace_id` | Parent trace ID (for jumping to the trace UI) | | `$ai_evaluation_result` | `true` = pass, `false` = fail | | `$ai_evaluation_reasoning` | Free-text explanation (set by the LLM judge or Hog code) | | `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N/A | When `$ai_evaluation_applicable = false`, the run counts as N/A regardless of `$ai_evaluation_result`. For evaluations that don't support N/A, this property may be `null` — treat null as "applicable". ## Workflow: investigate why an evaluation is failing Works the same way for `llm_judge` and `hog` evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). ### Step 1 — Find the evaluation ```json posthog:llma-evaluation-list { "search": "hallucination", "enabled": true } ``` Look at the returned `id`, `name`, `evaluation_type`, and either: - `evaluation_config.prompt` for an `llm_judge` - `evaluation_config.source` for a `hog` evaluator The Hog source is the ground truth for why a hog evaluator passes or fails — read it before assuming the failure is in the generation. ### Step 2 — Get the AI-generated summary ```json posthog:llma-evaluation-summary-create { "evaluation_id": "", "filter": "fail" } ``` Returns: - `overall_assessment` — natural-language summary - `fail_patterns` — grouped patterns with `title`, `description`, `frequency`, and `example_generation_ids` - `pass_patterns` and `na_patterns` — same shape, populated when `filter` includes them - `recommendations` — actionable next steps - `statistics` — `total_analyzed`, `pass_count`, `fail_count`, `na_count` The endpoint analyses the most recent ~250 runs (`EVALUATION_SUMMARY_MAX_RUNS`). Results are cached for one hour per `(evaluation_id, filter, set_of_generation_ids)`. Pass `force_refresh: true` to recompute. **Compare filters in two calls** to spot what's distinctive about failures vs passes: ```json posthog:llma-evaluation-summary-create { "evaluation_id": "", "filter": "pass" } ``` Then diff the `pass_patterns` against the `fail_patterns` from Step 2. ### Step 3 — Drill into example failing runs Each pattern surfaces `example_generation_ids`. Pull the underlying trace for the most representative example: ```json posthog:query-llm-trace { "traceId": "", "dateRange": {"date_from": "-30d"} } ``` (If you only have a generation ID, query for it via `execute-sql` first to find the parent trace ID — see below.) ### Step 4 — Verify the pattern with raw SQL The summary is LLM-generated and should be verified. Use `execute-sql` to count and spot-check: ```sql posthog:execute-sql SELECT properties.$ai_target_event_id AS generation_id, properties.$ai_trace_id AS trace_id, properties.$ai_evaluation_reasoning AS reasoning, timestamp FROM events WHERE event = '$ai_evaluation' AND properties.$ai_evaluation_id = '' AND properties.$ai_evaluation_result = false AND ( properties.$ai_evaluation_applicable IS NULL OR properties.$ai_evaluation_applicable != false ) AND timestamp >= now() - INTERVAL 7 DAY ORDER BY timestamp DESC LIMIT 25 ``` The N/A guard (`IS NULL OR != false`) is important — it matches the same logic the backend uses to bucket runs. ## Workflow: run an evaluation against a specific generation Use this when the user pastes a trace/generation URL and asks "what would evaluation X say about this?". ```json posthog:llma-evaluation-run { "evaluationId": "", "target_event_id": "", "timestamp": "2026-04-01T19:39:20Z", "event": "$ai_generation" } ``` The `timestamp` is required for an efficient ClickHouse lookup of the target event. Pass `distinct_id` if you have it — it speeds up the lookup further. ## Workflow: build and test a new evaluator ### Hog evaluator (deterministic, code-based) Reach for this first when the criterion is rule-based — it's cheaper, faster, and reproducible. Prototype with `llma-evaluation-test-hog` (no save): ```json posthog:llma-evaluation-test-hog { "source": "return event.properties.$ai_output_choices[1].content contains 'sorry';", "sample_count": 5, "allows_na": false } ``` The handler returns the boolean result for each of the most recent N `$ai_generation` events. Iterate on the source until it behaves as expected, then promote it via `llma-evaluation-create`: ```json posthog:llma-evaluation-create { "name": "Output is valid JSON", "description": "Fails when the assistant message can't be parsed as JSON", "evaluation_type": "hog", "evaluation_config": { "source": "let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }" }, "output_type": "boolean", "enabled": true } ``` Hog evaluators have full access to the event and its properties — common patterns include schema validation, length/token limits, regex matches, and tool-call shape checks. Because they're deterministic, results are reproducible across reruns and trivially diff-able. ### LLM-judge evaluator (subjective, prompt-based) Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality, helpfulness, on-topic-ness). There's no equivalent of `llma-evaluation-test-hog` for LLM judges — the typical loop is to create the evaluator with `enabled: false`, run it manually against a handful of representative generations via `llma-evaluation-run`, inspect the results, refine the prompt with `llma-evaluation-update`, and then flip `enabled: true` when you're satisfied: ```json posthog:llma-evaluation-create { "name": "Response stays on-topic", "description": "LLM judge — fails if the assistant changes topic from the user's question", "evaluation_type": "llm_judge", "evaluation_config": { "prompt": "You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N/A if the user did not actually ask a question." }, "output_type": "boolean", "output_config": { "allows_na": true }, "model_configuration": { "provider": "openai", "model": "gpt-5-mini" }, "enabled": false } ``` Then dry-run against a known-good and a known-bad generation: ```json posthog:llma-evaluation-run { "evaluationId": "", "target_event_id": "", "timestamp": "2026-04-01T19:39:20Z" } ``` LLM judges require organisation AI data processing approval. Hog evaluators do not. ## Workflow: manage the evaluation lifecycle | Action | Tool | | -------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Add a Hog evaluator | `llma-evaluation-create` with `evaluation_type: "hog"` and `evaluation_config.source` | | Add an LLM-judge evaluator | `llma-evaluation-create` with `evaluation_type: "llm_judge"`, `evaluation_config.prompt`, and a `model_configuration` | | Tweak the source or prompt | `llma-evaluation-update` (edits `evaluation_config.source` for Hog, `evaluation_config.prompt` for LLM judge) | | Toggle N/A handling | `llma-evaluation-update` with `output_config.allows_na` | | Disable temporarily | `llma-evaluation-update` with `enabled: false` | | Remove | `llma-evaluation-delete` (soft-delete via PATCH `{deleted: true}`) | `llm_judge` evaluations require AI data processing approval at the org level (`is_ai_data_processing_approved`). The same gate applies to `llma-evaluation-summary-create`. Hog evaluations do **not** require this gate — they run as plain code on the ingestion pipeline. ## When to use Hog vs LLM judge Reach for **Hog** by default. Switch to LLM judge only when the criterion can't be expressed as code. | Use Hog when… | Use LLM judge when… | | ----------------------------------------------------- | ------------------------------------------------------- | | The check is structural (JSON parses, schema matches) | The check is about meaning (on-topic, helpful, factual) | | You need a deterministic, reproducible result | A small amount of judgement variability is acceptable | | The criterion is cheap to compute | The criterion requires reading and understanding text | | You can't get AI data processing approval | You have approval and the criterion is genuinely fuzzy | | You need to enforce a hard limit (length, cost, etc.) | You need to rate a quality dimension | | You want sub-millisecond evaluation | A few hundred milliseconds + LLM cost are acceptable | A common pattern is to **layer them**: a Hog evaluator gates obvious format/length violations cheaply, and an LLM-judge evaluator only fires on the generations that pass the Hog gate (via `conditions`). ## Investigation patterns The summarisation tool works the same way regardless of whether the evaluator is `hog` or `llm_judge` — it analyses the resulting `$ai_evaluation` events, not the evaluator itself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is identical. ### "Why is evaluation X suddenly failing more?" 1. `llma-evaluation-list` — confirm the evaluation is still enabled and unchanged (compare `evaluation_config.source` or `evaluation_config.prompt` to the version you expect) 2. `llma-evaluation-summary-create` with `filter: "fail"` — get the dominant failure patterns and example IDs 3. SQL count of fails per day to confirm the regression window: ```sql SELECT toDate(timestamp) AS day, count() AS fails FROM events WHERE event = '$ai_evaluation' AND properties.$ai_evaluation_id = '' AND properties.$ai_evaluation_result = false AND timestamp >= now() - INTERVAL 30 DAY GROUP BY day ORDER BY day ``` 4. Drill into a representative trace per pattern via `query-llm-trace` ### "Are passes and fails caused by the same root content?" 1. Generate two summaries: one with `filter: "pass"`, one with `filter: "fail"` 2. If `pass_patterns` and `fail_patterns` describe similar content: - For an `llm_judge`: the prompt or rubric is probably ambiguous — reword `evaluation_config.prompt` and use `llma-evaluation-update` - For a `hog` evaluator: the rule is probably under- or over-matching — read the source via `llma-evaluation-get`, narrow the predicate, and retest with `llma-evaluation-test-hog` before pushing the fix via `llma-evaluation-update` ### "Did a Hog evaluator regression after a code change?" Hog evaluators are reproducible — if the source hasn't changed, identical inputs should yield identical outputs. When fail rates jump for a Hog evaluator: 1. `llma-evaluation-get` — note the current source and `updated_at` 2. Spot-check the latest failing runs with the SQL query from Step 4 above 3. Re-run the source against those exact generations using `llma-evaluation-test-hog` with a modified `conditions` filter that targets them 4. If the test results match the live results, the change is in the _generations_, not the evaluator (a model upgrade, prompt change upstream, etc.) — investigate the producer 5. If they diverge, the evaluator was edited; check git history of the source field via the activity log ### "What kinds of generations does this evaluator skip as N/A?" ```json posthog:llma-evaluation-summary-create { "evaluation_id": "", "filter": "na" } ``` Inspect `na_patterns` to see whether the N/A logic is doing the right thing. If a pattern in `na_patterns` looks like something that should have been scored: - For an `llm_judge`: the applicability instruction in the prompt is too broad — narrow it - For a `hog` evaluator with `output_config.allows_na: true`: the source is returning `null` (or whatever the N/A signal is) too eagerly — tighten the precondition ### "Score this single generation right now" `llma-evaluation-run` with the trace's generation ID and timestamp. Useful for spot-checking or wiring evaluations into a larger agent loop. ## Constructing UI links - **Evaluations list**: `https://app.posthog.com/ai-evals/evaluations` - **Single evaluation**: `https://app.posthog.com/ai-evals/evaluations/` - **Underlying generation/trace**: see the `exploring-llm-traces` skill's URL conventions Always surface the relevant link so the user can verify in the UI. ## Tips - The summary tool is **rate-limited** (burst, sustained, daily) and **caches results for one hour** — repeated calls with the same `(evaluation_id, filter)` are cheap; use `force_refresh: true` only when you genuinely need fresh analysis - Pass `generation_ids: [...]` to scope a summary to a specific cohort of runs (max 250) - The `statistics` block in the summary response is computed from raw data, not the LLM — trust those counts even if a pattern's `frequency` field is qualitative - For rich filtering not supported by `llma-evaluation-list` (e.g. by author or model configuration), fall back to `execute-sql` against the `evaluations` Postgres table or the `$ai_evaluation` ClickHouse events - When showing failure patterns to the user, always include 1-2 example trace links so they can validate the pattern visually - `llma-evaluation-*` tools use `evaluation:read` for read tools and `evaluation:write` for mutating tools; `llma-evaluation-summary-create` uses `llm_analytics:write` - Hog evaluators are reproducible — if you suspect a regression, `llma-evaluation-test-hog` with the suspect source against the failing generations is the fastest way to bisect whether the change is in the evaluator or in the producer of the generations - LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with a fixed prompt and model. If you're chasing a small regression in fail rate, prefer Hog or pin a deterministic provider/seed in the `model_configuration`