--- name: optimise-skill description: Use when tuning/optimising/"training" a skill's judge-shaped rubric (a single text document that makes a recurring, scorable decision) against outcome-labelled evidence — e.g. "train the triage rubric", "optimise the reviewer-severity rubric", or the command "/optimise-skill ". Runs on the Claude Max subscription via the CLI. Behavioural-only (it does NOT tune a skill's description for triggering — that stays with skill-creator). The home for skill-optimisation methodology. --- # optimise-skill A **decision-policy optimiser**. It tunes ONE judge-shaped document (a rubric that makes a recurring, scorable decision) against an outcome-labelled corpus, accepting an edited rubric only when a held-out score strictly improves and adversarial gates pass. It adopts `microsoft/SkillOpt` (on OPRO foundations) as its `optimiser_engine`, vendored unedited under `vendor/skillopt/`; everything skill-specific lives in the rails outside it. **Announce at start:** "I'm using optimise-skill to tune `` against its corpus (autonomy: ``)." It is **behavioural-only**. Tuning a skill's `description:` so Claude *invokes* it is a different job (different artifact, signal, lifecycle) and stays with the official `skill-creator`. See `CHARTER.md` for the full design and `dev/papers.md` for the SkillOpt/OPRO lineage. ## Requirements Run these from the `optimise-skill/` directory: - **Python 3.10+** (tested on 3.13) with **PyYAML** — `pip install -r requirements.txt`. PyYAML is the only runtime dependency; the vendored SkillOpt's heavier backends (openai, azure, ray, …) are optional and unused on the default path. - **Live tuning** picks a provider via the manifest's `model:` field: - `model: claude` (default) — needs the **`claude` CLI** + a **Claude Max subscription** (shells out to `claude -p`; billed against Max, not the metered API). - `model: openai` — for users **without Max**: `pip install openai` and set `OPENAI_API_KEY` (pay-as-you-go OpenAI billing; Azure OpenAI works similarly via SkillOpt's backend). Note: there is no Anthropic-HTTP-API backend, so using *Claude* here means the CLI/subscription, not an Anthropic API key. - **Tests:** `pip install pytest && python3 -m pytest tests/` — runs with stub judges, no LLM required. ## When to use — the durable decision rule Apply this to any signal, any skill, before making it a target: > *Is it a recurring decision made by ONE tunable text document, where each decision > is scorable and you can counterfactually imagine the edited document scoring better, > with enough volume to hold out?* > > **Yes → an optimise-skill target.** > **No → human-triage (one-off fixes, via a judge-shaped edit-queue) or monitoring (metrics).** A complex orchestration skill has *several* recurring decision surfaces. Factor each into its own judge-shaped rubric and it becomes its own target — **same process, N targets, each earning its place**. Targets are tuned **independently**, never pooled into one corpus (a held-out gate over mixed signals is meaningless). ## The three signals — only the first feeds the optimiser | Signal | Optimiser input? | Where it goes instead | |---|---|---| | **(a) per-decision policy labels** | **YES** | projected into the standardised markdown corpus | | **(b) one-off playbook fixes** (multi-file, sparse, non-counterfactual) | **NO** | a human + a judge-shaped edit-queue | | **(c) run-health metrics** (aggregates, not a decision) | **NO** | a dashboard / regression alarm; optionally a weak prior into (a) | This skill consumes **only grain (a)**. That is what keeps the optimiser sound: no unbounded, ungated, badly-attributed edits. ## What a consumer supplies Per target, an `optimisation/` directory: ``` optimisation/ manifest.yaml # the contract (see references/contract.md) rubric.md # the ONE tunable document (the manifest's `target`) corpus/{train,val,test}/*.md # standardised-markdown corpus (grain (a)) holdout/*.md # adversarial score-gate rows the trainer never sees (incidents / reversals) guards.py # OPTIONAL — consumer anti-degeneracy code (e.g. an escalation-rate cap) scorer.py # OPTIONAL — omit → default agreement-with-label scorer (auto-loaded if declared) ``` `holdout/*.md` feeds the POST-GATE score gate (it must be non-empty for a candidate to promote). Pre-existing fixtures are **referenced where they already exist, never duplicated** — and where they're not markdown (e.g. a consumer's JSON incident fixtures), a consumer either projects them into holdout rows or reads them directly in `guards.py`. ## How it runs (the rails, around the unchanged trainer) `scripts/optimise.py`: 1. **PRE-FLIGHT** — `validate_manifest.py` checks the contract; `md_corpus.py` loads the markdown corpus into the vendored engine's items; the baseline rubric is run through the guards as a sanity check. 2. **TRAIN** — `ReflACTTrainer(cfg, adapter).train()` **unchanged**: its own Rollout → Reflect → Edit → Gate loop with an internal strict-improve `val` gate. The `CorpusJudgeAdapter` re-runs the *candidate* rubric (the judge) on each recorded context and scores its verdict against the stored label — this is what gives a gradient (pure label-replay would score every candidate identically). 3. **POST-GATE** — an adversarial `holdout/` the trainer never saw + the consumer `guards.py` + an engine-owned incident **meta-test** (enforces every holdout row flagged `incident: true`; a consumer may also carry must-catch enforcement in its own `guards.py`, as an automated-loop or reviewer-severity consumer does) → the **autonomy map** decides promote-vs-propose → human gate(s) if the policy declares any → write a versioned rubric + provenance. ## Running as a command — `/optimise-skill ` When invoked as a command (or asked conversationally to "train/optimise the rubric"), resolve the argument to a manifest and run it: 1. **Resolve the target.** A consumer name → `~/.claude/skills//optimisation/manifest.yaml`. A path ending in `manifest.yaml` → use as-is. No arg → ask which consumer. 2. **Run it** (subscription; bounded for an interactive run): ```bash python3 ~/.claude/skills/optimise-skill/scripts/run.py --max-steps 1 ``` Drop `--max-steps` for a full run once real outcome labels exist. 3. **Report** the JSON summary: the `action` (propose/auto-promote/reject), baseline vs candidate score, any guard/meta violations, and the written `candidate_path` — then offer to show the proposed rubric diff. A run needs a logged-in Claude Code (this is the Max subscription). It takes a couple of minutes (it fans many `claude` calls across the corpus + holdout). ## Triggering a run (conversational or scheduled) One entrypoint — `scripts/run.py` — is what both a conversational "train the triage rubric" and a scheduled routine call: ```bash python3 ~/.claude/skills/optimise-skill/scripts/run.py \ ~/.claude/skills//optimisation/manifest.yaml --change-category safe-code ``` It runs PRE-FLIGHT → train → POST-GATE and prints a JSON summary (decision, scores, violations, the written candidate + provenance paths). `--max-steps N` bounds the run (use `1` for a cheap smoke / short scheduled run). To run it continuously, point a scheduled agent (`/schedule`) or a cron at the same command on whatever cadence fits. **Execution model — the Max subscription, via the Claude CLI.** With `model: claude` (the default both consumers use), every model call goes through the **Claude CLI** = your **Max subscription**, for BOTH roles: the judge/rollout is the CLI judge (`scripts/claude_code_judge.py`), and the optimiser is SkillOpt's `claude_chat` backend — which in this vendored SkillOpt **is the CLI** (`vendor/skillopt/model/claude_backend.py` shells out to `claude -p`), not the HTTP API. There is **no Anthropic-API backend** in this engine at all. The run also scrubs `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` / Bedrock / Vertex from its environment so a stray key can't divert the CLI to metered billing. **Cost model — effective 15 Jun 2026:** programmatic `claude -p` (every call this engine makes) no longer draws the high-capacity *interactive* subscription; it draws a **separate, capped, non-rollover Agent-SDK credit pool** (Pro $20 / Max-5x $100 / Max-20x $200 per month) at API rates, and *pauses* when exhausted (overflow stays off — the env-scrub above now also prevents silent pay-as-you-go). Architecture is unchanged; only the cost model moved — budget bounded/scheduled runs against the monthly credit. Prerequisite: a logged-in Claude Code on the machine (or a `CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token` for headless/scheduled runs). The only non-subscription path is a different provider (`model: openai` → `openai_chat`), which needs its own key and an injected judge — not used here. For tests/shadow the judge is injected as a deterministic stub (no LLM). Contract details are in `references/contract.md`. *Validated 31 May 2026:* a bounded live run against a real consumer completed entirely on the subscription — the optimiser produced a real rubric edit and POST-GATE returned a `propose` decision with provenance written (see the rollout-parallelism note: POST-GATE fans the judge out across the holdout, default 6 workers). > **Continuous improvement caveat:** the *machinery and trigger* run now on the > subscription, but genuine improvement needs an outcome-labelled corpus. Today the > corpora are authored/seed (shadow → propose only). The loop becomes load-bearing as the > extractors fill `corpus/` from live decisions (e.g. an automated loop's per-decision > corpus, or a reviewer-severity ledger) with `outcome-derived` labels over time. ## Logging & version control of the skill Two layers record how a skill evolves: 1. **`optimisation/ledger.md`** (committed, append-only) — `run.py` writes one row per run: date, model, baseline→candidate score (Δ), action, promotable, violation counts, the candidate's content hash, and the candidate filename. This logs **every** run, including shadow proposals that were never promoted — the "what the optimiser tried and why" history. 2. **The target rubric's git history** — when a candidate is *promoted*, that is a commit to the rubric file itself, so `git log` on (e.g.) a judge-shaped skill's `rubric.md` is the canonical record of *what the skill is now*. Ledger + rubric git history together = the full evolution. Heavy per-run artifacts (trainer state, predictions, `candidate_vNNNN.md`) land in `optimisation/runs/` and are **gitignored**; the ledger is the durable, committed summary. ## Long runs & timeouts A real run (many steps over a real corpus) makes hundreds of slow `claude` calls and will exceed any single interactive timeout. Handle it by **bounding + detaching + going incremental**, not one long blocking run: - `--max-steps N` bounds the work (POST-GATE rollout is already parallelised, default 6 workers). - **Run detached** so it isn't tied to an interactive turn, leaving a trail in the ledger + a log: ```bash nohup python3 ~/.claude/skills/optimise-skill/scripts/run.py \ ~/.claude/skills//optimisation/manifest.yaml --max-steps 2 \ > /tmp/optimise-run.log 2>&1 & ``` - **Incremental on a schedule** — short bounded runs (a couple of steps) on a routine, accumulating in the ledger, rather than one marathon. This is both the timeout fix and the continuous-improvement cadence. (Defer turning the schedule on until real outcome labels exist.) ## Autonomy (presets are sugar over a per-category action map) `shadow` = all propose-only (the zero-risk phase that mints fresh agreement labels); `assisted` = `safe-*` auto-act, rest propose-only; `autonomous` = all gated auto-act. **Engine invariants:** an `unknown` category is always propose-only; `permanent_gates` never auto-flip; `shadow` always proposes. Human gates are optional per-skill policy (0..N) — an automated-loop skill might keep `financial`/`irreversible` permanently gated; a low-stakes skill declares none. > **User-facing standard — two words, owned by `upskill`** (to keep the choice low-cognitive-load): **ask-me-first** (`= shadow`, the default) / **just-do-it** (`= autonomous`), set per decision-type. The three internal presets stay as the mechanism; operators only ever pick the two. The graduation rule is **not** an approval count — a decision-type goes just-do-it only when it is **undoable** (reversibility is the line, reusing a merge-gating allow-list + git-revertibility rule), its **review provably catches that type's incident fixtures** (the calibration gate — same idea as "enable autonomous-financial only after the panel escalates every incident-regression fixture"), **and a human flips it on**. Un-undoable actions (money, statutory submissions, deletion, real sends, `config`/`secrets`, unreverted schema) **never** graduate. See `upskill/SKILL.md` § Autonomy for the full plain-language model; this engine supplies the `decide_action` map underneath it. ## Onboarding a new consumer `python3 scripts/setup.py /optimisation --apply` scaffolds the dir (idempotent; dry-run by default; `--check` reports readiness; `--force` backs up to `*.bak`), mirroring the conventional `optimisation/` layout. `scripts/seed.py` generates synthetic cold-start seeds (positives + manufactured negatives, `source: synthetic`) over an injectable runner. Every generated draft carries `PENDING_REVIEW`; a run refuses until a human sharpens the drafts and deletes the sentinels. See `references/contract.md` §9–§10. Capture of live decisions is via `scripts/decision_record.py --corpus-dir ` (§9), wired additively into the producing skill (e.g. an automated loop). ## Status Engine + rails built and RED-verified against fixtures. The "build now" slice is implemented (enriched decision-capture, the thin capture path, extractor enrichment, the promotion-gate source guard, median-of-N + margin + trust-stratified split, the setup scaffolder + synthetic seeding, the process-emitting judge) — all in `scripts/`, `vendor/` untouched. Two consumers exist (one tuning an automated loop's triage rubric, one a reviewer-severity rubric), so `schema_version: 1` is frozen.