--- name: partnership-angles-eval description: > LLM-as-judge quality eval for the partnership-angles artifact. Grades grounding, distinctness, capability fit, and persuasiveness. Writes a verdict YAML. Gated by partnership-angles inline QA. disable-model-invocation: true --- # Partnership Angles Eval Independent LLM-as-Judge quality evaluation of the `angles.yaml` artifact produced by `partnership-angles`. Grades across four dimensions that test whether the three grounded angles are factually safe (every beat traces to a cited research fact), genuinely distinct from each other, tied to real Connect capabilities, and persuasive enough to serve as the foundation for a prospect-facing video. Writes a verdict YAML that `opp-eval` aggregates. Gated by `partnership-angles` inline QA — if QA failed, this skill emits `verdict: incomplete` without grading. See `skills/_eval-template.md` for shared contracts (verdict YAML shape, severity rules, inflation guard, stock blocks). See `skills/eval-calibration/SKILL.md` for the calibration methodology. ## Inputs | Source | Artifact | Used for | |---|---|---| | `partnership-angles` | `ACE/partnerships//runs//angles.yaml` | Primary artifact under judgment | | `partnership-research` | `ACE/partnerships//research/deep-research.md` | Ground truth for the grounding dimension | | `partnership-research` | `ACE/partnerships//research/connect-fit.md` | Ground truth for capability_tied dimension | | Phase 1 profile | `ACE/partnerships//prospect.yaml` | Expansion thesis + target geography | | `partnership-angles` inline QA | `phases.angles.verdict` in `run_state.yaml` | QA gate — if verdict is `fail` or `incomplete`, skip eval | ## Products - `2-research/partnership-angles-eval_verdict.yaml` — verdict YAML per `skills/_eval-template.md § Verdict YAML contract` ## Process 1. **Check the QA gate.** Read `ACE/partnerships//runs//run_state.yaml` via `drive_read_file`. Inspect `phases.angles.verdict`. If the verdict is `fail` or `incomplete`, write `verdict: incomplete` immediately and halt: ```yaml skill: partnership-angles-eval target: ran_at: capture_path: angles.yaml overall_score: 0 verdict: incomplete dimensions: {} auto_surfaced: - severity: INFO-SKIPPED message: "Skipped — partnership-angles inline QA returned verdict: . Fix QA failures first." ``` 2. **Read the artifacts and prospect context.** Read from Drive via `drive_read_file`: - `ACE/partnerships//runs//angles.yaml` - `ACE/partnerships//research/deep-research.md` - `ACE/partnerships//research/connect-fit.md` - `ACE/partnerships//prospect.yaml` (for `target_geography`, `sector`, `name`) 3. **Apply the LLM-as-Judge rubric** (see `## LLM-as-Judge Rubric` below). Grade each of the four dimensions 0–10. Compute the weighted overall score. 4. **Apply the `factual_safety` hard floor.** Before computing the verdict, check for any fabricated or uncited claim in any beat of any angle. A single confirmed fabricated statistic, invented partnership history, or invented program detail in `angles.yaml` automatically: - Floors `grounded` at ≤ 3.0 (the fabrication check is the primary grounding failure mode) - Sets suite `verdict: fail` - Surfaces a `BLOCKER` auto-surfaced entry naming the fabricated claim This floor is non-negotiable: `angles.yaml` feeds directly into a prospect-facing video narration. A single fabricated claim reaching a prospect is the worst possible failure (design §8, CLAUDE.md "No inferred backstory"). 5. **Write the verdict YAML** to `2-research/partnership-angles-eval_verdict.yaml`. Resolve or create `runs//2-research/` via `drive_create_folder` with `findOrCreate: true`. Write via `drive_create_file`. Dimensions must sum to 1.0: ```yaml dimensions: grounded: { score: <0-10>, weight: 0.40 } distinct: { score: <0-10>, weight: 0.20 } capability_tied: { score: <0-10>, weight: 0.25 } persuasiveness: { score: <0-10>, weight: 0.15 } ``` 6. **Surface auto-concerns** per `skills/_eval-template.md § Auto-surfaced severity rules`. Skill-specific surfaces: - `[BLOCKER]` if `grounded` scores ≤ 3.0 — fabricated or uncited claims in a prospect-facing angle are a hard stop. - `[BLOCKER]` if any dimension scores ≤ 3.0. - `[BLOCKER]` if overall score is below 7.0. - `[BLOCKER]` for each fabricated or uncited claim detected in any angle beat. - `[WARN]` if any angle has `grounded: false` in `angles.yaml` — an ungrounded angle may still be usable (the operator can discard it), but the eval flags it for human attention. - `[WARN]` if `capability_tied` scores 4.0–6.9 — angles may not compellingly showcase real Connect capabilities. - `[WARN]` if two angles ground the same beat with the same research fact — indicates insufficient differentiation. - `[INFO]` if all three angles are `grounded: false` — the prospect research may be insufficient to produce a viable angle set; the operator should run additional research before proceeding to production. ## LLM-as-Judge Rubric Grade `angles.yaml` against `deep-research.md`, `connect-fit.md`, and `prospect.yaml`. Every dimension is a quality/fitness judgment — structural presence was already checked by inline QA. The out-of-chain fitness requirement (per `skills/_eval-template.md § The out-of-chain fitness requirement`): `grounded` grades against **independently verifiable research citations** (out-of-chain anchor), testing whether the beats could survive a prospect's fact-check. This is the primary fitness dimension that escapes the AI authoring chain. `capability_tied` grades against **real documented Connect capabilities**, not asserted from memory — a second out-of-chain anchor. ### Dimension: `grounded` (weight 0.40) Does every beat in every angle trace back to a specific, citable fact in the research artifacts? This dimension grades whether the angles are built on evidence or on plausible-sounding invention. This is the highest-stakes fitness dimension. `angles.yaml` feeds directly into a prospect-facing video narration — a fabricated fact discovered by the prospect is the worst possible outcome (design §8). **Anchors:** - 9.0–10.0: Every beat in all three angles is grounded with a specific cited fact from `deep-research.md` or `connect-fit.md`. The grounding is traceable — a reader can point to the sentence in the research that backs the beat. No unexplained hedged language or invented precision. - 7.0–8.9: Nearly all beats are grounded; ≤2 beats across the three angles are asserted without a clear research citation, but none are demonstrably false. No fabrications. - 5.0–6.9: Several beats are thinly grounded — the research fact exists but the beat text significantly inflates or distorts it. OR 3–5 beats across angles lack traceable citations. - 3.0–4.9: Systematic grounding gaps. Multiple beats read as plausible generalizations rather than cited facts. Risk that a well-researched prospect will catch incorrect or invented claims. - 0.0–2.9: Beats are fabricated or cannot be traced to the research artifacts. **Auto-surfaced as BLOCKER — halt before this angle set is used in a prospect-facing artifact.** **Hard deduction (fabrication floor):** Any confirmed fabricated claim (a specific number, a named partnership, a program history that does not appear in the research) → `grounded` ≤ 3.0 regardless of other content. Auto-surface as BLOCKER. **Hard deduction (ungrounded beats):** Each beat text containing `[UNGROUNDED` is a 0.5-point deduction from the raw score, capped at a 3.0 total deduction. Ungrounded beats are honest (the skill flags rather than fabricates); the deduction is smaller than fabrication but the signal matters. ### Dimension: `distinct` (weight 0.20) Are the three grounded angles genuinely different stories — different heroes, different emotional arcs, different research facts used, different Connect capabilities foregrounded — or do they read as paraphrases of each other? **Anchors:** - 9.0–10.0: All three angles are clearly differentiated by hero, emotional beat, AND the research facts they foreground. A reader cannot swap beats between angles without breaking the coherence of any one angle. - 7.0–8.9: Two of the three angles are clearly differentiated; the third shares some beats or research facts with another but still has a distinct overall arc. - 5.0–6.9: Two angles feel like variations on the same story — similar hero, similar facts emphasized, overlapping capability lean. A reader might struggle to choose between them on substance. - 3.0–4.9: The three angles are largely paraphrases. Different `angle_id` labels, same underlying story beat by beat. Providing these to an operator as "distinct options" would be misleading. - 0.0–2.9: The three angles are functionally identical. No meaningful choice between them. **Hard deduction:** If any two angles use the same research fact as the `problem` beat, deduct 2.0 from the raw score (the problem beat is the most differentiation-critical beat for a partnership pitch). ### Dimension: `capability_tied` (weight 0.25) Does each angle lean compellingly on a real, validated Connect capability — one that is specific to that angle's arc and traceable to the `connect-fit.md` memo? This dimension grades against the `connect-fit.md` memo as the out-of-chain anchor: capability claims must be validated there, not asserted from memory. `connect-fit.md` was built by consulting real PDDs, programs, and the documented Connect feature set — that validation work must carry through into the angles. **Anchors:** - 9.0–10.0: Each angle's `primary_capability` and `product` beat are grounded in a specific Connect capability documented in `connect-fit.md`. The capability claim explains *why* it matters for *this* prospect's expansion situation. A reader of `connect-fit.md` can confirm the capability exists. - 7.0–8.9: Most capability claims are traceable to `connect-fit.md`; one angle uses a capability that is real but stated generically without a prospect-specific fit argument. - 5.0–6.9: Capability claims are present but thin — they name Connect features in the abstract without explaining how they fit this prospect's model or expansion situation. OR one angle's `primary_capability` cannot be found in `connect-fit.md`. - 3.0–4.9: Capability claims read as a generic "Connect does payment and verification" pitch without prospect-specific grounding. The memo was not consulted or its findings were not incorporated. - 0.0–2.9: Capability claims are demonstrably absent from the documented Connect feature set, or the angles reference capabilities that Connect doesn't have. Risk of a false pitch claim. **Hard deduction:** If any angle claims a Connect capability that is demonstrably absent from the documented feature set (not in `connect-fit.md` and not in the documented Connect feature set), deduct 3.0 from the raw score. ### Dimension: `persuasiveness` (weight 0.15) Are the grounded angles compelling enough to anchor a 90-second partnership video that would move a decision-maker at the prospect org? This dimension grades the *deployability* of the angles — given that the underlying facts are cited, does the grounded story hang together emotionally and logically? **Anchors:** - 9.0–10.0: All three angles tell a coherent, emotionally resonant story arc. The `logline` accurately captures what the angle does for *this* prospect. The `problem` beat is sharply scoped to the prospect's expansion situation; the `product` beat clearly shows the Connect capability in action; the `impact` beat closes the loop on the opening claim. A video producer could work from these beats without needing significant rework. - 7.0–8.9: Most angles are persuasive; one angle has a beat that is mechanically grounded but emotionally flat (a fact cited correctly but without a compelling framing). - 5.0–6.9: Angles are factually grounded but narratively thin. The beats are checkboxes rather than a story arc. A video producer would need significant narrative enhancement to make them work. - 3.0–4.9: The angles read as bullet-point summaries of the research rather than narrative arcs. No emotional through-line; a decision-maker would not be moved. - 0.0–2.9: The angles are not usable as video narration starters. Even with correct facts, the narrative construction is too weak to serve as a partnership pitch. ### Deduction rules - Any single dimension ≤ 3.0 → suite verdict `fail`, regardless of overall mean. - `grounded` ≤ 3.0 → suite verdict `fail` + `BLOCKER` auto-surfaced (fabrication risk). - Overall score below 7.0 → suite verdict `fail` + `BLOCKER`. - Overall score 7.0–7.9 → suite verdict `warn`. - Overall score ≥ 8.0 → suite verdict `pass`. ### Calibration targets - **Detection rate:** ≥ 80% of catalogued angles issues from `eval-calibration/known-issues.md § partnership-angles` (once populated after the first two real runs). - **Inter-run variance:** ≤ 0.5 across 3 same-model runs. - **Dimension coverage:** the rubric must distinguish (a) three plausible-sounding angles written from general AI knowledge about NGO programs (factually thin, high fabrication risk) from (b) three angles where every beat is traceable to the cited `deep-research.md` report. `grounded` is the primary fitness dimension enforcing this — a genuinely-grounded set and a fabricated set must land on opposite sides of the 7.0 threshold. - **Agreement with inline self-check:** The inline QA in `partnership-angles` runs binary structural checks; this eval grades quality. A QA-passing artifact is structurally correct; `grounded` + `capability_tied` are what separate conformant-but-empty from deployable. ## MCP Tools Used See `skills/_eval-template.md § MCP Tools Used (stock)`. - Google Drive: `drive_read_file`, `drive_create_folder`, `drive_create_file` ## Mode Behavior See `skills/_eval-template.md § Mode Behavior (stock)`. - **Auto:** Grade, write verdict + auto-surfaced concerns, return overall score and disposition. - **Review:** Pause after grading to let a human eyeball the verdict before it propagates to the production phase. ## Dry-Run Behavior See `skills/_eval-template.md § Dry-Run Behavior (stock)`. When `--dry-run` is active: - Read inputs normally (read-only operations are safe in dry-run). - Write the verdict YAML to Drive normally (human-facing artifact; safe to write in dry-run). - State tracks as `dry-run-success`. ## Change Log | Date | Change | Author | |------|--------|--------| | 2026-06-06 | Initial version. Four dimensions: grounded (0.40), distinct (0.20), capability_tied (0.25), persuasiveness (0.15). Hard BLOCKER on grounded ≤ 3 given prospect-facing fabrication risk. Gated by partnership-angles inline QA. | ACE team |