---
name: extract-trap-task
description: >-
  Turn an approved Azure DevOps pull request into hard CentralGauge benchmark
  trap-tasks. Use when authoring new AL tasks from real merged PRs / work items,
  when a task should probe BC runtime semantics (not just syntax), or when the
  existing suite is too easy. Mines the non-obvious, comment-justified decisions
  in a PR (the traps a syntax-only model gets wrong), distills each into a
  self-contained or base-app-faithful task, and ships it ONLY after a
  discrimination probe proves a correct solution passes and a naive one fails on
  a real container.
---

# Extract Trap-Task from an ADO PR

## Core insight

In a well-reviewed PR, **every code comment that justifies a non-obvious decision
marks a latent trap.** Real examples from the seed PRs (Continia `Document Output
- Extensions`, PRs 50590 / 50285):

- "We can't use a TryFunction: TryFunctions can't contain database writes on-prem."
- "BindSubscription is bound to the current call-stack frame and does NOT reliably
  propagate into Codeunit.Run's isolated frame."
- "Gate on 'Change Log Activated' FIRST — IsAlwaysLoggedTable() returns true for
  posted sales tables regardless of the global flag."
- "Read all matching rows into a List first — you can't insert into a table while
  iterating it."

Each justification is one thing a model that only memorized AL syntax gets wrong.
Harvest those justifications; each becomes a candidate task.

## The hard gate (non-negotiable)

**Ship a task ONLY when, on a real Cronus container, a correct reference solution
PASSES its oracle and a naive-trap reference solution genuinely FAILS the same
oracle.** The naive is a throwaway dev artifact — never committed. If the test
passes for both, or fails for both, the oracle does not measure the trap: redesign
or drop. This is the entire point; do not skip it to save a container run.

## Six stages

### 1. SELECT
Pick an approved/completed PR + its linked work item. The work item gives business
*intent* ("why"); the diff gives *mechanism* ("how"). Tools:
`mcp__azureDevOps__list_pull_requests` (with `pullRequestId`) →
`mcp__azureDevOps__get_pull_request_changes` +
`mcp__azureDevOps__get_work_item`.

### 2. EXTRACT
Pull the diff + work item description. Flag every comment or decision that
justifies a non-obvious AL choice. Output a candidate list, each as: "naive model
would do X; correct is Y because Z."

### 3. DISTILL
Per trap: choose containment (see `reference/containment-policy.md`), strip
project-specific objects (no Continia objects — they aren't in the vanilla
container), restate as a generic problem. The prompt describes *what* to build,
never *how*, and carries **no guiding notes** (project rule — hints that help the
model dodge the trap destroy the signal). Specifying required *behavior* is
allowed; explaining AL mechanics or naming the mechanism (`BindSubscription`,
`Permissions`, `Codeunit.Run`, `trigger OnRun`) is not.

### 4. PREMISE-GATE (do this BEFORE full authoring — the most important lesson)
Many source-PR traps lean on version- or context-specific BC behavior and **do
not reproduce** in a vanilla container. Verify the trap actually discriminates on
THIS BC build with a minimal throwaway probe first. Confirmed non-reproducing
traps from the seed batch (BC 28.0.46665.50383):
- `IsAlwaysLoggedTable("Sales Invoice Header")` = FALSE (the always-logged set is
  permission/security tables, e.g. `User` = TRUE — retarget, don't assume).
- Indirect `Permissions = tabledata X = RIMD` is NOT honored under
  `TestPermissions = Restrictive`/`NonRestrictive` (only `Disabled`) — the
  permission trap is untestable in the AL test harness here.
- A caller-bound subscription DOES propagate into a `Codeunit.Run` frame, so the
  "bind must be inside OnRun" frame-isolation trap does not discriminate.
- `StartSession` does NOT run under the default SOAP `TestIsolation = Codeunit`;
  a background-session trap needs `RequiredTestIsolation = Disabled` on the test
  codeunit + harness runner routing (see "Test isolation" below).
- An AL **Query object's child DataItem defaults to `SqlJoinType = LeftOuterJoin`,
  NOT InnerJoin** (BC 28). A "childless parents get dropped" trap does NOT
  reproduce by omitting the property — the default already keeps them; only an
  UNMOTIVATED explicit `SqlJoinType = InnerJoin` drops them, which a no-hints
  model never writes. (Query objects DO run fine from a test codeunit via
  `.Open`/`.Read`/`.Close`.)
If the premise fails, drop or retarget — never ship a non-discriminating task.
**A scout's "STRONG" rating is not a verified premise.** Triage scouts rank
candidates from the PR diff without running them on the container; batch-4's
Sustainability picks (rated STRONG) blocked 2-of-3 on premise (a backwards
Query-join default, a forcing gap). ALWAYS premise-gate on Cronus28 before
authoring, regardless of how confident the scout was.

### 5. AUTHOR + DISCRIMINATION PROBE
Write the task YAML + oracle test + prereq app (if self-contained). At dev time
only, write a **correct** and a **naive** reference solution under `scratch/`
(gitignored). Probe both with the harness (below). Ship only on the hard gate.

### 6. VALIDATE + REGISTER
`validate-tasks` (Zod load + no-hints scan), local dry-run, then
`sync-catalog --apply` and the `refresh-task-taxonomy` skill. (Registration is
often deferred; keep local until deliberately publishing.)

## The probe harness

`scripts/trap-probe.ts` runs a solution through the real compile→publish→test path
and classifies the outcome:

```
deno run -A scripts/trap-probe.ts --task <id> --solution <dir> --expect pass|fail --container Cronus28
```

- **Three outcomes, not two:** `pass` (exit 0), `fail`/MISMATCH (exit 1),
  **`inconclusive`** (exit 3 = a thrown infra error, e.g. zero-tests-after-publish;
  just re-run). A naive must produce a GENUINE `fail` (a test-assertion failure or
  a compile error), never an `inconclusive` — otherwise the discrimination is
  spurious.
- **Cronus28 only.** Container credentials are wired only for `Cronus28`; other
  containers 401. Do not fall back to Cronus281+.

## Shallow-oracle guard (learned the hard way, twice)

A two-state oracle (empty vs full) routinely leaves a hole: a plausible wrong
implementation passes both. Examples that slipped a 2-case oracle:
- A whole-table short-circuit (`if not Target.IsEmpty() then exit(0)`) passing an
  idempotent-copy test.
- An `Activated AND explicit-config` detector passing an always-logged test where
  the active case set both conditions true.

**Add an INTERMEDIATE-state case** that only the genuinely-correct implementation
passes (a partial overlap; an active-but-not-explicitly-configured state), and
prove it with a third "wrong-C" reference that passes the other cases but fails
the new one. Defend against *plausible* naive implementations, not against
adversarial cheats (a model that directly writes the observable table it was never
told exists is not a realistic failure mode).

**Codeunit-wraps-table bypass (batch-5, X024).** When the trap lives in a TABLE
FIELD (a type/property choice) but the model exposes a `codeunit` CRUD API over
that table, asserting ONLY through the API (`Register`/`GetRef`) lets a model that
caches state in a codeunit variable / `Dictionary` (never round-tripping the
table) pass regardless of the field — bypassing the whole trap. "Add an internal
cache" is a plausible LLM habit, not an adversarial cheat, whenever there's an
API indirection layer over the trapped field. The oracle must READ THE TABLE ROW
DIRECTLY (`Rec.Get(pk); Assert.AreEqual(expected, Rec."Field")`) so the value must
actually persist through the trapped field. Prove closure with a THIRD
cache-bypass reference (Text field but codeunit-`Dictionary` storage) that now
fails the direct-table assertion.

## Marking convention

Every new task: id `CG-AL-X###` (the `X` cohort namespace, regex already allows
it), `metadata.cohort: ado-trap-2026`, `metadata.source_pr: <id>`. Legacy tasks are
`CG-AL-[EMH]*`; the cohort is glob-selectable as `tasks/**/CG-AL-X*.yml`.
Difficulty comes from the directory + `metadata.difficulty`, not the id letter.

## AL / harness gotchas (save a probe cycle)

- `domains` uses the `DomainSchema` vocabulary (`src/tasks/domains.ts`:
  `codeunits`, `events`, `error-handling`, `permissions`, ...) — a DIFFERENT list
  than the 9-group `metadata.category` taxonomy. Classify both; `domains` powers
  per-domain leaderboard scores.
- `Record.Get()` on a single-row setup table needs the PK arg: `Get('')`.
- Reserved words can't be variable names: `Key`, `User`, `Protected`, ...
- A `[TryFunction]` containing a DB write fails at **runtime** (a genuine per-test
  failure → classified `fail`, good), not at compile time.
- Prereq app UUIDs must be valid hex — use a hex-safe suffix (`0a01`, not `x001`).
- Test codeunit shape: `Subtype = Test; TestPermissions = Disabled;` and
  `Assert: Codeunit Assert` (base-app Assert, not "Library Assert").
- Object-ID bands: prereq `69000-69999`, generated/model `70000-79999`, tests
  `80000-89999`. Collision-check before assigning.

## Test isolation & shared-container hygiene (batch-2 learnings)

The oracle runs in a container REUSED across tasks and candidates. Tasks that
create persistent state must not poison later runs:

- **No `[TestCleanup]` hook exists in BC 28 AL** — only rollback-based
  `TestIsolation`/`RequiredTestIsolation`. Do teardown as **start-of-test
  self-heal**: delete-before-insert all state the test touches, so a prior
  candidate's failure (the EXPECTED outcome for a hard trap) can't leave state
  that collides.
- **`Commit()` defeats the runner's rollback.** A trap that needs `Commit`
  (`ChangeCompany` cross-company reads, `StartSession` visibility) loses the
  automatic per-test rollback, so committed rows persist on the shared container.
  Without self-heal, the NEXT candidate hits a PK collision at its `[GIVEN]` seed
  and is scored a FALSE FAILURE (not a recognized infra signature → no
  auto-reroute). Make seeding idempotent; prove it by running naive-then-correct
  consecutively (correct must still pass).
- **`StartSession` needs `RequiredTestIsolation = Disabled` + harness routing.**
  The default SOAP `TestIsolation = Codeunit` blocks it. Route opt-in codeunits
  to the platform isolation-disabled runner (`infra/cg-test-harness/` +
  `HARNESS_APP_VERSION` bump in `bc-container-provider.ts`), gated so only tasks
  declaring the property are affected. `Disabled` also removes auto-rollback →
  self-heal applies.
- **The observable must REQUIRE the mechanism.** If a trap's observable is
  trivially computable without the mechanism (e.g. "return the sum" when the trap
  is how the sum is produced across sessions), a plausible model bypasses the
  whole mechanism and passes. Make the expected value NON-derivable from the
  visible inputs (an opaque worker formula, a per-run committed factor) and
  assert proof-of-execution (the mechanism's own side effects), so passing
  requires actually running it. Verify with a third "bypass" reference.

## What the harness can't test — scouting filters (batch-3 learnings)

Some trap classes are structurally untestable in this harness. Skip them during
SELECT, or premise-gate hard:

- **External-caller / cross-module identity is untestable.** The harness bundles
  the test codeunit AND the candidate into ONE fixed app
  (`BENCHMARK_APP_ID` in `mcp/al-tools-server.ts`), so the test is never external
  to the candidate's module. Traps that depend on `GetCallerModuleInfo`, caller
  app-identity, or a genuinely external invoker (e.g. fail-open-guard bugs) have
  no external caller to observe — both correct and naive return the same wrong
  answer. Needs a companion-app-published-after-candidate mechanism the harness
  doesn't have. Skip these.
- **Mark/MarkedOnly and page-navigation-scope traps resist discrimination.** A
  temp-record `Page.Run(0, TempItem)` (a `temporary` record backs a page fine) is
  an idiomatic AL bypass that navigates the same set without `Mark`/`MarkedOnly` —
  so the observable doesn't force the mechanism. Forcing it needs a write-back /
  persistence oracle (edit via the page, verify the REAL table changed) on the
  legacy TestPage path, which is fragile. Usually not worth it.
- **Refactoring-shape bugs are hard to force in fresh code-gen.** A trap that only
  manifests from a DRY refactor (e.g. hoisting a call one stack frame too deep) is
  rarely reproduced by a model authoring fresh code — the natural single-procedure
  solution is correct. If the only failing variant is an unnatural structure a
  real model wouldn't write, the task doesn't discriminate a plausible model.
- **Install→upgrade lifecycle is untestable.** `BcContainerProvider.prepareCandidateApp`
  uninstalls + unpublishes any prior same-name app before every publish, then
  publishes fresh with `-install`. `OnInstall` fires every task but `OnUpgrade`'s
  precondition (a pre-existing older installed version) never exists — so
  upgrade-cycle traps (Upgrade Tags re-running a migration, data-upgrade
  clobbering) cannot be triggered. The `Codeunit "Upgrade Tag"` API is callable,
  but a tag-guarded idempotent migration is just the idempotency lesson —
  already covered by shipped `H039`/`H040`/`X004`.

**Proactive anti-cheat pays off.** Apply the opaque-value + proof-of-execution
guard UP FRONT (not just after a review catches it): make the expected value
non-derivable from the visible inputs (an opaque computed formula) and assert the
mechanism's own side effects. Doing this from the first draft kept every
self-contained task in the third batch clean on first review.

## Making a task frontier-HARD (the dual-frontier finding)

**Passing the discrimination probe does NOT mean the task catches a frontier
model.** The probe only proves the task beats the naive YOU wrote. Frontier models
(Opus/Fable-class) have every well-known AL gotcha memorized, so a SINGLE obscure
semantic — however clever — they solve first-try. Proven on a dual-frontier bench
(Opus 4.8 + Fable 5): `X033` (TransferFields matches by number) and `X034` (enum
`Format` returns caption not name) were BOTH aced 100% first-try by BOTH models —
as were all 22 medium X-tasks. A one-gotcha task ranks weaker models; it does not
move a frontier.

**The ONLY pattern that catches a frontier is COMPOSITIONAL / fix-has-its-own-trap:**
several interacting traps where the naive fails AND each obvious fix a frontier
reaches for ALSO fails, for a DIFFERENT reason — so only the deep-correct sequence
passes. Proven catchers:
- **X035 "poisoned rescue"** — insert a row → run a black-box engine
  error-tolerantly. TryFunction → write blocked; `if Codeunit.Run` without
  `Commit` → write-transaction error; Run-first → engine needs the row; only
  insert→`Commit`→`if Codeunit.Run` passes. **Failed Fable 0/2, caught Opus on
  attempt 1.**
- **X037 "inner commit"** — a black-box engine `Commit`s internally before
  erroring, so `if Codeunit.Run` returning false does NOT mean rollback cleaned
  up; the model must run a FILTERED compensating purge (not a blanket `DeleteAll`,
  which wipes a decoy row). **Failed Opus BOTH attempts (0/2), Fable attempt-1.**
- **X019 "stale relocate"** — a hidden helper renames the PK (an innocuous verb
  hides the mutation) AND the obvious re-read (`Find('=')`/`Get`) uses the stale
  key. Caught Opus both attempts.

**Premise-fragility of compositional traps:** the deepest layers often don't
reproduce on the target build — the phantom-buffer premise (X036) collapsed
(`Codeunit.Run(id, Rec)` copies the record out only on SUCCESS), and a mid-loop
re-key hazard (X038) HUNG the SOAP runner (infinite revisit → inconclusive, not a
clean fail → dropped: a naive must fail, never hang/timeout). Budget ~3
compositional candidates per shippable frontier-catcher, and hard-gate every
layer on-container before authoring.

Recipe (Fable's meta-lesson, confirmed on the bench): a frontier-hard task needs
BOTH (a) an innocuous verb HIDING a state mutation, AND (b) a boobytrapped
first-instinct fix. **To verify hardness you MUST dual-frontier bench** (Opus + a
second frontier via `run-xbench.ps1 -Model "...,..."`) — the probe is necessary
but not sufficient. Single-gotcha tasks are still worth shipping for field
coverage; only compositional ones make a frontier fail.

### Repair-resistance tiers (round-3 finding, benched)

Attempt-1 catch and attempt-2 catch are DIFFERENT properties. Round-3 tasks in
new territory (X045 validate-order, X046 SystemId+staleness, X047 dimension
sets) each caught a frontier on attempt-1 (X046 caught BOTH) — but ALL were
repaired by attempt-2, because their failure messages are LEGIBLE (a SystemId
mismatch or wrong tier value points straight at the mistake). The
Commit/transaction family (X037/X040/X041) resisted BOTH attempts: its failure
signatures don't reveal where the trap is, and the obvious repair has its own
trap. So:

- **Attempt-2-resistant (hardest tier):** make the failure signature OPAQUE —
  wrong values with no hint of the mechanism, and boobytrap the natural repair.
  Transaction-boundary semantics do this naturally.
- **Attempt-1 discriminators:** still valuable — the leaderboard headline is
  AUC@2 (second-attempt-only solve scores 0.5), so these genuinely de-saturate
  the metric even though pass@2 stays 100%.
- When designing for the hardest tier, ask: "does the test failure OUTPUT leak
  the fix?" If the assert message names the field/mechanism that went wrong, a
  frontier will repair it on attempt 2.

**Round-4 amendment (benched, honest): signature engineering is NOT the lever.**
Round 4 built deliberate opacity (contract-only assert messages, digest oracles)
and even byte-identical failure collisions across wrong shapes incl. the
simulated attempt-2 repair (X050, X051) — and a controlled A/B (X052 = X045's
exact trap with the oracle collapsed to one opaque Total). Result: the A/B came
out IDENTICAL to legible X045 (Opus solved both first-try), and collisions did
not stop attempt-2 repairs. Models don't gradient-follow failure output — they
re-derive from scratch on attempt 2, so if the model KNOWS the domain, any
failure signal at all triggers a successful rewrite. The Commit-family's
attempt-2 resistance comes from KNOWLEDGE-GAP DEPTH (the entire repair space is
semantically trapped — every alternative the model can think of also fails),
not from what the failure output says. Design consequence: to resist attempt 2,
make the CORRECT solution require knowledge the model lacks, with all
plausible-looking alternatives wrong — opacity/digest oracles are fine hygiene
but buy no measurable resistance on their own.

**Bench-noise caveat:** "Model returned empty response" scores as a failed
attempt-1 and pollutes PR1/AUC@2 (observed 3/6 Fable attempts in one run).
Before reading a matrix, check failureReasons — empty-response attempts carry
zero trap signal.

## Files

| Path | Role |
|---|---|
| `scripts/trap-probe.ts` | The discrimination-probe driver (pass/fail/inconclusive) |
| `reference/containment-policy.md` | Self-contained vs base-app-faithful decision |
| `docs/superpowers/specs/2026-06-30-ado-pr-trap-tasks-design.md` | The originating design spec |
| `.claude/commands/create-task.md` | Task-authoring conventions (IDs, no-hints) |
| `.claude/commands/validate-tasks.md` | Guidance + schema validation pass |