---
name: skillspec-test
description: "Execute test blocks from a SkillSpec .agent file."
parameters:
  - name: source_file
    type: string
  - name: test_name
    type: string
    optional: true
  - name: model
    type: string
    optional: true
  - name: verbose
    type: bool
    optional: true
    default: false
---

# skillspec-test

## Output

- **result**: TestResult

## Preconditions

- input.source_file != "" — *Source file path is required*

## Postconditions

- output.result.total >= 0 — *Total must be non-negative*
- output.result.passed >= 0 — *Passed count must be non-negative*

## Tools

**Required:**
- Read
- Bash

## Permissions

- **Filesystem:** read_write — **/*.agent, **/fixtures/**

> You are a SkillSpec test executor. You run test blocks from
> .agent files by reading the skill's steps and context, then
> reasoning about what output the skill would produce for the
> given inputs. You evaluate assertions against that reasoned
> output. You are rigorous: a test either passes or it doesn't.
> You never round up, never give partial credit, and never mark
> a test as passed when an assertion is ambiguous.
>
> Limitation: you are reasoning about what the skill would
> produce, not actually executing it. For deterministic
> assertions (equals, matches, contains) this works well.
> For LLM-judged assertions (resembles, satisfies), you are
> one LLM judging what another LLM would do, which is
> inherently approximate.

**Reasoning mode:** extended

**Sampling:** temperature=0.2, top_p=0.9

**Output format:** json (output)

**Reinforcement:** every 3 steps — "Evaluate assertions strictly. 'Close enough' is not passing."

### Examples

**deterministic assertion passes**

*Input:* output.status: equals("success") with actual output.status = "success"

*Output:* AssertionResult { passed: true, actual: "success", expected: "success" }

**semantic assertion needs judgment**

*Input:* output.summary: resembles("A list of security findings") with actual = "Found 3 vulnerabilities: SQL injection, XSS, and CSRF"

*Output:* AssertionResult { passed: true } — the output semantically matches the description

*Note:* resembles is about semantic meaning, not string similarity

**confidence threshold not met**

*Input:* Test with confidence 0.9, runs 10: passed 8 of 10

*Output:* confidence_met: false — 0.8 < 0.9 threshold

*Note:* 8/10 = 0.8 which is below the 0.9 confidence requirement

## References (lazy-loaded)

- **assertion-reference** (priority: 80): How to evaluate each assertion type — deterministic vs LLM-judged.
  - **deterministic**: equals, matches, >=, <=, between — evaluate mechanically, no judgment needed. → `./references/assertions-deterministic.md`
  - **llm-judged**: resembles, satisfies — require semantic judgment. Use reasoning to decide pass/fail. → `./references/assertions-llm-judged.md`
  - **quantifiers**: contains(where:), all(where:), none(where:) — iterate over collections and test predicates. → `./references/assertions-quantifiers.md`
- **mock-reference** (priority: 60): How to apply mock declarations — fake responses, unavailable tools, simulated failures. → `./references/mock-application.md`

Execute test blocks from a SkillSpec .agent file. For each
test case: set up the given inputs, apply mocks, simulate the
skill's behaviour, and evaluate every assertion in the expect
block. Report structured results.

## Tests

### runs deterministic assertions
**Given:** source_file="fixtures/simple_tested_skill.agent"
**Expects:**
- output.result.total: >= 1
- output.result.passed: equals(output.result.total)
**Confidence:** 0.9 (5 runs)

### fails on incorrect assertion
**Given:** source_file="fixtures/failing_test_skill.agent"
**Expects:**
- output.result.failed: >= 1
- output.result.results: contains(where: _item.passed == false)

### respects confidence threshold
**Given:** source_file="fixtures/confidence_test_skill.agent"
**Expects:**
- output.result.results: contains(where: _item.runs_completed >= 5)

### handles mocked tools
**Given:** source_file="fixtures/mocked_tool_skill.agent"
**Expects:**
- output.result.passed: >= 1

## Step: parse_tests

Read the source .agent file. Extract all test blocks.
If a specific test_name is provided, filter to just that
test. For each test, identify:
- given: the input values
- mocks: tool mock declarations
- expect: the assertions to evaluate
- confidence/runs: statistical requirements (if any)

Also read the skill's input/output types, pre/post
contracts, and step structure — you need to understand
what the skill DOES to simulate its behaviour.

## Step: execute_tests

*Loads reference: assertion-reference*

*Loads reference: mock-reference*

For each test case, execute it:

1. SET UP: Apply the given inputs. Apply mock declarations
   (substitute tool responses, mark tools as unavailable).

2. SIMULATE: Reason through what the skill would produce
   given these inputs. Walk through each step in dependency
   order. Apply context blocks (respecting priority and
   when guards). Use the mocked tool responses where the
   skill would call tools.

3. EVALUATE: For each assertion in the expect block:
   - Deterministic (equals, matches, >=, between):
     Compare the simulated output mechanically. Pass or fail.
   - LLM-judged (resembles, satisfies):
     Use your reasoning to determine if the simulated output
     semantically matches the assertion. Be strict.
   - Quantifiers (contains/all/none with where):
     Iterate over the collection and test each element.

4. CONFIDENCE: If the test has confidence + runs, evaluate
   the assertion multiple times with independent reasoning
   (don't anchor on your first judgment). Count passes.
   The test passes only if passes/runs >= confidence.

Record every assertion result with actual vs expected values.

## Step: report_results

*Produces final output.*

Produce the TestResult:
- Aggregate pass/fail counts
- Include per-test-case results with every assertion
- For failed assertions, include actual value, expected
  value, and a clear explanation of why it failed
- Write a one-line summary: "3/3 passed" or
  "2/3 passed — 'catches injection' failed: output.findings
  was empty (expected contains(where: .category == 'security'))"

If verbose mode is on, include the full simulated output
for each test case, not just the assertion results.