Holdout Evaluator

Category: Dev Tools · Sub-category: tools-misc

ai:llmtype:review

Validate agent work output against hidden holdout scenarios using LLM-as-Judge evaluation, producing mapped feedback (referencing visible criteria only) and telemetry records saved to $HOME/.ai-first-kit/. Cross-references the agent's self-review evidence table against actual files to detect claims without evidence. Use when the user says 'validate holdouts', 'test gates against holdouts', 'run holdout evaluation', 'check gate effectiveness', or when invoked as a sub-agent by org-gate-review during inline gate validation. Also use when the user reports gates missing failures, gates blocking good work, or concerns that agents are gaming gate criteria — even if they don't use the word 'holdout'. This skill MUST be consulted because it operationalizes holdout validation with structured LLM-as-Judge evaluation; a conversational answer cannot systematically test holdout scenarios or produce telemetry data.

What this skill does

Holdout Evaluator is a community-contributed Claude Code skill in the tools-misc sub-category. It ships as a SKILL.md file that Claude Code auto-discovers under ~/.claude/skills/holdout-evaluator/ and loads when your prompt matches the skill's trigger.

When to invoke it: Use when the user says 'validate holdouts', 'test gates against holdouts', 'run holdout evaluation', 'check gate effectiveness', or when invoked as a sub-agent by org-gate-review during inline gate validation. Also use when the user reports gates missing failures, gates blocking good work, or concerns that agents are gaming gate criteria — even if they don't use the word 'holdout'.

Who uses this skill

The Holdout Evaluator skill is built for developers, power users, and teams automating repetitive workflows and improving developer experience. It is part of the open ClaudSkills registry, a community-curated catalog of 56,000+ capabilities you can install for Claude Code — the Claude CLI agent.

How to install

Free

Manual install (2 steps)

mkdir -p ~/.claude/skills/holdout-evaluator
curl -L https://claudskills.com/skills/holdout-evaluator/SKILL.md \
  -o ~/.claude/skills/holdout-evaluator/SKILL.md

Or just download SKILL.md directly and drop it into ~/.claude/skills/holdout-evaluator/. Claude Code auto-discovers it on next session.

Skills live at ~/.claude/skills/holdout-evaluator/SKILL.md on macOS/Linux, or %USERPROFILE%\.claude\skills\holdout-evaluator\SKILL.md on Windows. See the full install guide for step-by-step instructions.

Pro

One-click install via the desktop app

The ClaudSkills desktop app installs any skill directly into ~/.claude/skills/ with one click — no terminal required. Pro starts at $9/mo or $149 lifetime.

Pro

For the full experience including quality scoring and one-click install features for each skill — upgrade to Pro.

See pricing → Download desktop app

More Dev Tools skills

Browse all Dev Tools skills in the ClaudSkills registry, or explore these other picks from the same category:

Browse all Dev Tools skills → Top 100 skills

Part of ClaudSkills — the open registry for Claude Code skills. · What's New · Install guide · About · llms.txt

Part of Acreator Store — Adam Lankamer's AI tools: GifPerfect · AspectPerfect · SlomoPerfect · Ucaption · UTagger · AutoXPoster · TestYourSkills