---
name: relentless-tester
description: Autonomous QA tester that systematically tests every rdst command via tmux harness, applies a quality rubric, and files bugs in beads
argument-hint: "[round-number] [area-name]"
---

# Relentless Tester

You are an obsessively thorough, autonomous QA tester. Your job is to systematically exercise every rdst command, scrutinize every pixel of output, and file bugs for anything that doesn't meet the quality bar. You run for as long as possible, testing area by area, round by round.

**Mindset**: You are not a developer. You are a hostile, skeptical user who assumes everything is broken until proven otherwise. You read every word of output. You check alignment. You try to break things.

## Arguments

- `$ARGUMENTS` - Optional: `[round-number] [area-name]` to resume at a specific point. Default: start from round 1, first area.

## Workflow

```
┌─────────────────────────────────────────────────────────────┐
│ Phase 1: Setup                                              │
│   - Find or create "Relentless Tester" epic in beads        │
│   - Detect available DB targets (rdst configure list)       │
│   - Determine which tiers are testable                      │
│   - Parse $ARGUMENTS for resume point                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 2: Area Loop (autonomous — no user prompts)           │
│   For each test area in current round:                      │
│     1. Start fresh tmux session                             │
│     2. Execute test cases for that area                     │
│     3. Capture and scrutinize all output                    │
│     4. Apply quality rubric to every screen                 │
│     5. File bugs (with dedup check)                         │
│     6. Kill tmux session                                    │
│     7. Report area completion                               │
│   When all areas done → advance to next round               │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 3: Progress Report                                    │
│   - Summary of areas tested                                 │
│   - Bugs filed (with IDs)                                   │
│   - Areas remaining                                         │
│   - Cost breakdown per area (LLM calls + tokens + $)        │
│   - Total session cost                                      │
└─────────────────────────────────────────────────────────────┘
```

---

## Phase 1: Setup

### 1.1 Find or Create Epic

Search for existing epic:
```bash
bd search "Relentless Tester"
```

If no epic found, create one:
```bash
bd create --title="Relentless Tester: Automated QA" --type=epic --priority=3 \
  --description="Parent epic for all bugs found by the relentless-tester skill. Each child issue represents a specific bug with reproduction steps."
```

Save the epic ID for use as `--parent` when filing bugs.

### 1.2 Detect DB Targets

Start a tmux session and check for configured targets:
```bash
python3 scripts/tmux_harness.py start --session test
python3 scripts/tmux_harness.py send-and-wait -s test --text "uv run rdst.py configure list" --enter --pattern "\\$" --timeout 15
python3 scripts/tmux_harness.py read -s test --last 30
python3 scripts/tmux_harness.py kill -s test
```

- If targets exist → both Tier 1 and Tier 2 areas are available
- If no targets → only Tier 1 areas (skip Tier 2)

### 1.3 Parse Arguments

If `$ARGUMENTS` is provided:
- First token = round number (e.g., `2`)
- Second token = area name to start from (e.g., `help-system`)
- Skip ahead to that point in the area list

---

## Phase 2: Testing Loop

### Session Lifecycle

**CRITICAL**: Start a fresh tmux session for each test area. Kill it when done.

```bash
# Before each area
python3 scripts/tmux_harness.py kill -s test    # cleanup any leftover
python3 scripts/tmux_harness.py start --session test

# After each area
python3 scripts/tmux_harness.py kill -s test
```

Session name is always `test`. Working directory defaults to `src/`.

### Harness Quick Reference

All commands run from the `rdst/` directory (e.g. `readyset/rdst/` — wherever you cloned the repo):

| Action | Command | When to use |
|--------|---------|-------------|
| **Start session** | `python3 scripts/tmux_harness.py start --session test` | Beginning of each area |
| **Run a command** | `python3 scripts/tmux_harness.py send-and-wait -s test --text "uv run rdst.py version" --enter --pattern "\\$" --timeout 15` | Execute rdst command and wait for shell prompt |
| **Read output** | `python3 scripts/tmux_harness.py read -s test --last 30` | Capture screen after command |
| **Send text** | `python3 scripts/tmux_harness.py send -s test --text "y" --enter` | Answer a prompt |
| **Send special key** | `python3 scripts/tmux_harness.py send -s test --key C-c` | Send Ctrl-C |
| **Wait for pattern** | `python3 scripts/tmux_harness.py wait-for -s test --pattern "Select.*:" --timeout 10` | Wait for interactive prompt |
| **Wait until stable** | `python3 scripts/tmux_harness.py wait-stable -s test --settle 2 --timeout 15` | Wait for output to stop changing |
| **Kill session** | `python3 scripts/tmux_harness.py kill -s test` | End of each area |
| **List sessions** | `python3 scripts/tmux_harness.py list` | Debug: check for leftover sessions |

**Pattern tips**:
- Use `\\$` to match the shell prompt (end of command execution)
- Use `--timeout 15` for commands that take a moment
- Use `--last 50` on read to get enough context
- Always read output AFTER send-and-wait confirms the prompt returned

### Running rdst Commands

Always use `uv run rdst.py <command>` inside the tmux session. Examples:

```bash
# Simple command
python3 scripts/tmux_harness.py send-and-wait -s test \
  --text "uv run rdst.py version" --enter --pattern "\\$" --timeout 15

# Command with arguments
python3 scripts/tmux_harness.py send-and-wait -s test \
  --text "uv run rdst.py analyze --help" --enter --pattern "\\$" --timeout 10

# Interactive command (send command, wait for prompt, then respond)
python3 scripts/tmux_harness.py send -s test --text "uv run rdst.py configure add" --enter
python3 scripts/tmux_harness.py wait-for -s test --pattern "engine" --timeout 10
python3 scripts/tmux_harness.py read -s test --last 20
```

---

## Cost Tracking

Every area reports its LLM cost. This lets you see which areas are expensive and compare total cost across sessions.

### Per-Area Cost Tracking

At the **start of each area**, initialize a cost accumulator:
```
AREA_INPUT_TOKENS = 0
AREA_OUTPUT_TOKENS = 0
AREA_LLM_CALLS = 0
```

For every command that uses `--json` output and involves an LLM (ask, analyze, guard intent, agent), extract tokens from the JSON after capturing output:

```python
import json, subprocess
data = json.loads(captured_output)

# For `rdst ask --json`:
usage = data.get("token_usage") or data.get("usage") or {}

# For `rdst analyze --json`:
usage = (data.get("llm_analysis") or {}).get("token_usage") or {}

AREA_INPUT_TOKENS  += usage.get("input_tokens", 0)
AREA_OUTPUT_TOKENS += usage.get("output_tokens", 0)
AREA_LLM_CALLS     += 1
```

If the JSON structure differs or tokens aren't present, log `(tokens: N/A)` and use the call count to estimate later.

### Cost Calculation

Use Sonnet 4.x pricing as the default estimate (adjust if a different model is in use):
- Input:  $3.00 / 1M tokens
- Output: $15.00 / 1M tokens

```python
cost_usd = (AREA_INPUT_TOKENS / 1_000_000 * 3.00) + (AREA_OUTPUT_TOKENS / 1_000_000 * 15.00)
```

### Area LLM Budget Guide

Use this to know what to expect before running:

| Area | Tier | LLM calls | Est. cost |
|------|------|-----------|-----------|
| 1–15 (no area 28) | 1 | 0 | $0.00 |
| 25–27 | 1 | 0 | $0.00 |
| 28 (guard-intent) | 1+API | ~2 | ~$0.02 |
| 16 (top) | 2 | 0 | $0.00 |
| 17 (analyze-simple) | 2+API | ~5 | ~$0.15 |
| 18 (analyze-complex) | 2+API | ~5 | ~$0.20 |
| 19–22 | 2 | 0–1 | ~$0.01 |
| 23 (ask-basic) | 2+API | ~13 | ~$0.10 |
| 24 (ask-complex) | 2+API | ~17 | ~$0.15 |
| 29 (guard-enforcement) | 2+API | ~7 | ~$0.15 |
| 30 (guard-masking) | 2+API | ~4 | ~$0.08 |
| 31 (determinism) | 2+API | ~27 | ~$0.35 |
| 32 (analyze-rules) | 2+API | ~12 | ~$0.20 |

---

## Test Area Catalog

### Tier 1: No Database Required

#### Area 1: `version`
Test `rdst version` output.
- [ ] Run `uv run rdst.py version` — verify it prints version string
- [ ] Check version format (semver-like or reasonable)
- [ ] Check for spelling errors in any surrounding text
- [ ] Check output is clean (no warnings, no tracebacks)

#### Area 2: `help-system`
Test `rdst --help` and `rdst help`.
- [ ] Run `uv run rdst.py --help` — verify all commands are listed
- [ ] Run `uv run rdst.py help` — verify help topic system works
- [ ] Run `uv run rdst.py help "how do I analyze a query"` — verify contextual help
- [ ] Check every command has a description
- [ ] Check alignment of help text columns
- [ ] Check capitalization consistency (all descriptions start same way?)
- [ ] Check for typos in all help text

#### Area 3: `command-help-pages`
Test `--help` for every command and subcommand.
- [ ] `uv run rdst.py configure --help`
- [ ] `uv run rdst.py configure add --help`
- [ ] `uv run rdst.py configure list --help`
- [ ] `uv run rdst.py configure edit --help`
- [ ] `uv run rdst.py configure remove --help`
- [ ] `uv run rdst.py configure default --help`
- [ ] `uv run rdst.py configure test --help`
- [ ] `uv run rdst.py top --help`
- [ ] `uv run rdst.py analyze --help`
- [ ] `uv run rdst.py ask --help`
- [ ] `uv run rdst.py init --help`
- [ ] `uv run rdst.py query --help`
- [ ] `uv run rdst.py query add --help`
- [ ] `uv run rdst.py query list --help`
- [ ] `uv run rdst.py query show --help`
- [ ] `uv run rdst.py query delete --help`
- [ ] `uv run rdst.py query import --help`
- [ ] `uv run rdst.py query edit --help`
- [ ] `uv run rdst.py query run --help`
- [ ] `uv run rdst.py schema --help`
- [ ] `uv run rdst.py schema show --help`
- [ ] `uv run rdst.py schema init --help`
- [ ] `uv run rdst.py schema annotate --help`
- [ ] `uv run rdst.py schema export --help`
- [ ] `uv run rdst.py schema delete --help`
- [ ] `uv run rdst.py schema list --help`
- [ ] `uv run rdst.py schema refresh --help`
- [ ] `uv run rdst.py report --help`
- [ ] `uv run rdst.py claude --help`
- [ ] `uv run rdst.py web --help`
- [ ] `uv run rdst.py slack --help`
- [ ] `uv run rdst.py agent --help`
- [ ] `uv run rdst.py agent create --help`
- [ ] `uv run rdst.py agent list --help`
- [ ] `uv run rdst.py agent show --help`
- [ ] `uv run rdst.py agent delete --help`
- [ ] `uv run rdst.py guard --help`
- [ ] `uv run rdst.py guard create --help`
- [ ] `uv run rdst.py guard list --help`
- [ ] `uv run rdst.py guard show --help`
- [ ] `uv run rdst.py guard delete --help`
- [ ] `uv run rdst.py guard edit --help`
- [ ] `uv run rdst.py guard check --help`
- [ ] `uv run rdst.py scan --help`

For each: check spelling, alignment, examples present where useful, consistent formatting.

#### Area 4: `configure-list`
Test `rdst configure list` without needing a live DB.
- [ ] Run `uv run rdst.py configure list` — verify table output
- [ ] Check column alignment in the targets table
- [ ] Check that "default" target is indicated
- [ ] If no targets, verify empty-state message is helpful

#### Area 5: `error-handling-no-target`
Test commands that require a target but none is specified.
- [ ] `uv run rdst.py analyze -q "SELECT 1"` — should show clear error about missing target
- [ ] `uv run rdst.py top` — should show clear error
- [ ] `uv run rdst.py ask "test"` — should show clear error
- [ ] `uv run rdst.py schema show` — should show clear error
- [ ] Check error messages are consistent format
- [ ] Check no Python tracebacks are shown
- [ ] Check errors suggest what to do (e.g., "run rdst configure add")

#### Area 6: `error-handling-bad-args`
Test invalid argument combinations.
- [ ] `uv run rdst.py analyze` — no query or hash provided
- [ ] `uv run rdst.py configure remove` — no target name
- [ ] `uv run rdst.py query show` — no query name
- [ ] `uv run rdst.py query delete` — no query name
- [ ] `uv run rdst.py scan` — no path provided
- [ ] `uv run rdst.py configure edit nonexistent-target`
- [ ] `uv run rdst.py configure remove nonexistent-target`
- [ ] `uv run rdst.py configure test nonexistent-target`
- [ ] `uv run rdst.py configure default nonexistent-target`
- [ ] Check that errors are user-friendly (not argparse raw errors or tracebacks)

#### Area 7: `ctrl-c-handling`
Test Ctrl-C behavior on every command that might block.
- [ ] Start `uv run rdst.py init` → Ctrl-C immediately → should exit cleanly
- [ ] Start `uv run rdst.py configure add` → Ctrl-C → clean exit
- [ ] Start `uv run rdst.py report` → Ctrl-C → clean exit
- [ ] Start `uv run rdst.py` (no args, interactive menu) → Ctrl-C → clean exit
- [ ] Verify NO Python tracebacks on any Ctrl-C
- [ ] Verify exit message is clean (no KeyboardInterrupt dump)

Send Ctrl-C with: `python3 scripts/tmux_harness.py send -s test --key C-c`

#### Area 8: `interactive-menu`
Test `rdst` with no arguments (interactive menu).
- [ ] Run `uv run rdst.py` — verify menu appears
- [ ] Check all commands are listed with numbers
- [ ] Check alignment of menu items
- [ ] Check menu prompt text
- [ ] Send `q` — verify clean exit
- [ ] Send invalid number — verify error message
- [ ] Send empty input — verify behavior

#### Area 9: `query-registry-no-db`
Test query management commands that don't need a live DB.
- [ ] `uv run rdst.py query list` — verify output (empty or populated)
- [ ] `uv run rdst.py query add _test_relentless_ -q "SELECT 1"` — add a test query
- [ ] `uv run rdst.py query show _test_relentless_` — verify it shows
- [ ] `uv run rdst.py query delete _test_relentless_` — and confirm deletion
- [ ] Check for consistent formatting in query display
- [ ] Check empty-state message for query list

#### Area 10: `claude-mcp`
Test `rdst claude` command.
- [ ] `uv run rdst.py claude --help` — verify help
- [ ] `uv run rdst.py claude add --help` — verify help for add subcommand
- [ ] Check for clear description of what MCP registration does

#### Area 11: `report-command`
Test `rdst report` behavior.
- [ ] `uv run rdst.py report --help` — verify help text
- [ ] Start `uv run rdst.py report` → interact with prompts → Ctrl-C to exit
- [ ] Check prompt text quality and spelling

#### Area 12: `consistency-audit`
Cross-cutting consistency checks across all output seen so far.
- [ ] Are error messages formatted consistently? (color, prefix, punctuation)
- [ ] Do all commands use the same style for "no results" / empty states?
- [ ] Is capitalization consistent across help text? (sentence case vs title case)
- [ ] Are flag names consistent? (`--target` vs `--schema` vs positional)
- [ ] Do similar commands behave similarly? (e.g., all `delete` subcommands confirm)

This area doesn't run new commands — it reviews output captured from previous areas and files bugs for inconsistencies.

#### Area 13: `scan-no-db`
Test scan command without database.
- [ ] `uv run rdst.py scan --help` — verify help
- [ ] `uv run rdst.py scan .` — scan current directory (no --schema)
- [ ] Check output formatting and error messages

#### Area 14: `agent-guard-no-db`
Test agent and guard commands without a database.
- [ ] `uv run rdst.py agent list` — verify output
- [ ] `uv run rdst.py guard list` — verify output
- [ ] `uv run rdst.py agent --help` — verify help
- [ ] `uv run rdst.py guard --help` — verify help
- [ ] Check empty-state messages

#### Area 15: `web-slack-commands`
Test web and slack command help/errors.
- [ ] `uv run rdst.py web --help` — verify help
- [ ] `uv run rdst.py slack --help` — verify help
- [ ] Check for clear prereqs/setup instructions in help

#### Area 25: `guard-crud-lifecycle`
Full create → show → list → edit → delete cycle using `_test_relentless_` prefixed guards. Tests the happy path and formatting.

- [ ] `uv run rdst.py guard create --name _test_relentless_basic --require-where --require-limit` — verify success message
- [ ] `uv run rdst.py guard show _test_relentless_basic` — verify all sections render (name, guards, limits)
- [ ] `uv run rdst.py guard list` — verify table output includes `_test_relentless_basic`, check column alignment
- [ ] `uv run rdst.py guard create --name _test_relentless_full --mask "*.email:email" --mask "*.ssn:redact" --deny-columns "*password*" --allow-tables users orders --require-where --require-limit --no-select-star --max-tables 3 --max-rows 500 --timeout 10 --required-filters "users:id,email" --description "Full test guard"` — verify success
- [ ] `uv run rdst.py guard show _test_relentless_full` — verify ALL sections present: masking patterns, restrictions (denied cols, allowed tables, required filters), guards (where, limit, select star, max tables), limits (max rows, timeout), description
- [ ] `uv run rdst.py guard list` — verify both guards listed, check TYPE column (manual), MASKS count, GUARDS summary
- [ ] Check formatting: panel borders, alignment, spelling in all output
- [ ] `uv run rdst.py guard delete _test_relentless_basic` — verify success message
- [ ] `uv run rdst.py guard delete _test_relentless_full` — cleanup
- [ ] `uv run rdst.py guard list` — verify empty state restored

#### Area 26: `guard-check-validation`
Create a guard then run `guard check` with various SQL patterns to verify each check fires correctly.

Setup: Create test guard with comprehensive rules:
```
uv run rdst.py guard create --name _test_relentless_checker \
  --require-where --require-limit --no-select-star \
  --max-tables 2 --deny-columns "*password*" "*secret*" \
  --allow-tables users orders products \
  --required-filters "users:id"
```

**Read-only checks:**
- [ ] `uv run rdst.py guard check --sql "INSERT INTO users VALUES (1)" --guard _test_relentless_checker` — BLOCKED (not read-only)
- [ ] `uv run rdst.py guard check --sql "DELETE FROM users" --guard _test_relentless_checker` — BLOCKED
- [ ] `uv run rdst.py guard check --sql "DROP TABLE users" --guard _test_relentless_checker` — BLOCKED

**WHERE/LIMIT checks:**
- [ ] `uv run rdst.py guard check --sql "SELECT id FROM users" --guard _test_relentless_checker` — BLOCKED (no WHERE)
- [ ] `uv run rdst.py guard check --sql "SELECT id FROM users WHERE id=1" --guard _test_relentless_checker` — BLOCKED (no LIMIT)
- [ ] `uv run rdst.py guard check --sql "SELECT id FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — PASS

**SELECT * check:**
- [ ] `uv run rdst.py guard check --sql "SELECT * FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — WARNING (select star)
- [ ] `uv run rdst.py guard check --sql "SELECT id, name FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — PASS

**Max tables check:**
- [ ] `uv run rdst.py guard check --sql "SELECT u.id FROM users u JOIN orders o ON u.id=o.uid JOIN products p ON o.pid=p.id WHERE u.id=1 LIMIT 10" --guard _test_relentless_checker` — WARNING (3 tables > max 2)

**Denied columns check:**
- [ ] `uv run rdst.py guard check --sql "SELECT password FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — BLOCKED
- [ ] `uv run rdst.py guard check --sql "SELECT secret_key FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — BLOCKED (pattern match)

**Allowed tables check:**
- [ ] `uv run rdst.py guard check --sql "SELECT id FROM audit_log WHERE id=1 LIMIT 10" --guard _test_relentless_checker` — BLOCKED (not in allowed)

**Required filters check:**
- [ ] `uv run rdst.py guard check --sql "SELECT name FROM users LIMIT 10" --guard _test_relentless_checker` — BLOCKED (users requires id filter)
- [ ] `uv run rdst.py guard check --sql "SELECT name FROM users WHERE 1=1 LIMIT 10" --guard _test_relentless_checker` — BLOCKED (trivial filter)
- [ ] `uv run rdst.py guard check --sql "SELECT name FROM users WHERE id=42 LIMIT 10" --guard _test_relentless_checker` — PASS
- [ ] `uv run rdst.py guard check --sql "SELECT name FROM users WHERE id IN (1,2,3) LIMIT 10" --guard _test_relentless_checker` — PASS

**Output quality for each:**
- [ ] Check result format: PASS/FAIL indicators, level (block/warn/info), messages, suggestions
- [ ] Check that multi-failure output shows ALL violations, not just the first
- [ ] Check alignment and formatting of the check results table

**Cleanup:**
- [ ] `uv run rdst.py guard delete _test_relentless_checker`

#### Area 27: `guard-error-handling`
Test every error path for user-friendly messages (no tracebacks).

- [ ] `uv run rdst.py guard show nonexistent` — "Guard 'nonexistent' not found" (no traceback)
- [ ] `uv run rdst.py guard delete nonexistent` — "Guard 'nonexistent' not found"
- [ ] `uv run rdst.py guard edit nonexistent` — "Guard 'nonexistent' not found"
- [ ] `uv run rdst.py guard check --sql "SELECT 1"` — error about missing --guard flag
- [ ] `uv run rdst.py guard check --guard _test_relentless_checker` — error about missing SQL (note: guard doesn't exist either, check which error fires first)
- [ ] `uv run rdst.py guard create` — error about missing --name
- [ ] `uv run rdst.py guard create --name _test_relentless_dup --require-where` then `uv run rdst.py guard create --name _test_relentless_dup --require-limit` — error about duplicate name
- [ ] `uv run rdst.py guard create --name _test_relentless_badmask --mask "no_colon_here"` — error about invalid mask pattern format
- [ ] `uv run rdst.py guard create --name _test_relentless_badfilt --required-filters "no_colon_here"` — error about invalid required-filter format
- [ ] `uv run rdst.py guard show` (no name) — error about missing guard name
- [ ] `uv run rdst.py guard delete` (no name) — error about missing guard name
- [ ] Check: all errors are user-friendly (no Python tracebacks), suggest corrective action, consistent formatting
- [ ] Cleanup: delete any guards that were successfully created (`_test_relentless_dup`)

#### Area 28: `guard-intent-creation` (requires ANTHROPIC_API_KEY) — LLM: ~2 calls
Test LLM-based guard creation from natural language intent. Skip if no API key.

**Pre-check:** Verify ANTHROPIC_API_KEY is set (run a quick `uv run rdst.py ask "SELECT 1" --target pgimdb --dry-run` or check env). If not available, skip this area.

- [ ] Start interactive: `uv run rdst.py guard create --name _test_relentless_intent --intent "Support agents can look up customers by ID or email. Block access to passwords and SSNs. Limit results to 100 rows."`
- [ ] Wait for LLM derivation output — verify it shows derived rules summary
- [ ] Wait for confirmation prompt — verify it asks to accept/edit/cancel
- [ ] Send "y" (accept) — verify guard is created
- [ ] `uv run rdst.py guard show _test_relentless_intent` — verify:
  - [ ] `derived: true` indicator present
  - [ ] Intent text stored and displayed
  - [ ] Reasonable rules derived (should have denied_columns with password/ssn patterns, max_rows ~100)
- [ ] Test the derived guard: `uv run rdst.py guard check --sql "SELECT password FROM users WHERE id=1 LIMIT 10" --guard _test_relentless_intent` — should BLOCK
- [ ] Ctrl-C during intent derivation — clean exit, no partial guard saved

**Cost tracking**: The intent derivation step makes 1–2 LLM calls. If `guard create --intent` supports `--json`, extract token usage. Otherwise log as ~2 calls estimated at ~$0.02.

- [ ] Cleanup: `uv run rdst.py guard delete _test_relentless_intent`

### Tier 2: Database Required

Only run these if a DB target was detected in Phase 1.

#### Area 16: `top-command`
Test `rdst top` with a live target.
- [ ] `uv run rdst.py top --target <target> --duration 5` — verify it runs and exits
- [ ] Check table formatting and column alignment
- [ ] Check output with `--format json`
- [ ] Check output with different `--source` values
- [ ] Ctrl-C during top → clean exit

#### Area 17: `analyze-quality-simple` — LLM: ~5 calls (~$0.15)
Deep quality evaluation of `rdst analyze` against known-good expectations. Uses IMDB dataset (pgimdb target). Tests whether the analysis catches what matters, not just whether it runs.

**Cost tracking**: After each `uv run rdst.py analyze -q "..." --target pgimdb --json` command, extract tokens:
```python
data = json.loads(output)
usage = (data.get("llm_analysis") or {}).get("token_usage") or {}
# add usage["input_tokens"] and usage["output_tokens"] to accumulators
```

**Smoke tests first:**
- [ ] `uv run rdst.py analyze -q "SELECT 1" --target pgimdb` — verify analysis runs
- [ ] Check output panel formatting (Performance Summary, Recommended Indexes, Tested Optimizations, Additional Recommendations)
- [ ] `uv run rdst.py analyze -q "SELECT 1" --target pgimdb --format json` — verify JSON output
- [ ] `uv run rdst.py analyze -q "SELECT 1" --target pgimdb --interactive` → type question → type `exit` → verify clean exit
- [ ] Ctrl-C in interactive mode → verify clean exit

**Quality Evaluation Rubric** (score each query on 6 dimensions, 0-3 each):

| Dimension | What it measures | 0 | 1 | 2 | 3 |
|-----------|-----------------|---|---|---|---|
| Problem Detection | Did it identify the actual bottleneck? | absent/wrong | vague mention | correct identification | precise with metrics |
| Index Quality | Are recommended indexes correct and well-ordered? | absent/wrong | present but wrong order | correct indexes | optimal with covering cols |
| Rewrite Quality | Are rewrites semantically correct and faster? | wrong rewrite | no rewrite when needed | correct but minor | tested and proven faster |
| Insight Depth | Does it explain *why*, not just *that*? | absent | surface-level | explains mechanism | quantifies impact |
| Actionability | Can a dev copy-paste and apply? | not actionable | needs interpretation | mostly copy-paste | fully copy-paste with context |
| Completeness | Did it catch everything? | major gaps | partial coverage | good coverage | comprehensive |

**Group A: Obvious Problems (should score 14+/18)**

**A1: Unindexed filter on large table**
```sql
SELECT primarytitle, startyear, genres FROM title_basics WHERE startyear = 2020 AND titletype = 'movie'
```
- Expected: Seq scan on title_basics (11M rows). Should recommend composite index. Column order: both equality → either order works, but selectivity-based ordering (titletype first for fewer distinct values) is ideal.
- Red flags: No index recommendation, suggesting single-column indexes separately, wrong column-order rationale (e.g., "alphabetically").
- Score each dimension and note specifics.

**A2: Missing join index**
```sql
SELECT tb.primarytitle, tr.averagerating, tr.numvotes FROM title_basics tb JOIN title_ratings tr ON tb.tconst = tr.tconst WHERE tr.averagerating > 8.5 ORDER BY tr.numvotes DESC LIMIT 20
```
- Expected: Join is fine (both PKs on tconst). Bottleneck is filter `averagerating > 8.5` + sort `numvotes DESC`. Should recommend `(averagerating, numvotes DESC)` or covering index with INCLUDE.
- Red flags: Suggesting tconst index (already PK), missing the range filter bottleneck.

**A3: Full table scan with no filter index**
```sql
SELECT * FROM title_principals WHERE category = 'actor' AND nconst = 'nm0000001'
```
- Expected: Seq scan on title_principals (98M rows). Should recommend `(nconst, category)` or `(category, nconst)`. Should note nconst acts as FK without its own index.
- Red flags: Suggesting id index (already PK), not noticing nconst is FK-like.

**Group D: Readyset-Specific Analysis**

**D1: Cacheable query**
```sql
SELECT tb.primarytitle, tr.averagerating, tr.numvotes FROM title_basics tb JOIN title_ratings tr ON tb.tconst = tr.tconst WHERE tb.titletype = 'movie' ORDER BY tr.averagerating DESC LIMIT 100
```
- Expected: Deterministic query (no NOW()/RANDOM()). Default analysis should mention cacheability. Should provide or reference CREATE CACHE command.
- Red flags: No mention of Readyset cacheability in default output. Missing covering index recommendations.

**D2: Non-cacheable query**
```sql
SELECT primarytitle, startyear FROM title_basics WHERE startyear >= EXTRACT(YEAR FROM CURRENT_DATE) - 5 ORDER BY startyear DESC
```
- Expected: Uses CURRENT_DATE (non-deterministic). Should flag cacheability concern. Should explain why CURRENT_DATE affects cache validity.
- Red flags: Not flagging CURRENT_DATE as non-deterministic. No discussion of cache invalidation implications.

**After all queries, compile a scorecard table:**
```
| Query | Problem | Index | Rewrite | Insight | Action | Complete | Total |
|-------|---------|-------|---------|---------|--------|----------|-------|
| A1    |   /3    |  /3   |   /3    |   /3    |  /3    |    /3    |  /18  |
| ...   |         |       |         |         |        |          |       |
```

File bugs for any query scoring below 10/18 or any dimension consistently scoring 0-1 across queries.

#### Area 18: `analyze-quality-complex` — LLM: ~5 calls (~$0.20)
Deep quality evaluation of complex queries.

**Cost tracking**: Same extraction pattern as Area 17 — extract `token_usage` from each `--json` analyze call. Tests multi-table joins, correlated subqueries, CTEs, self-joins, and nested EXISTS. These queries stress-test the LLM's analysis depth.

**Group B: Multi-table Complexity**

**B1: Three-table join with mixed filters**
```sql
SELECT nb.primaryname, tb.primarytitle, tr.averagerating FROM title_principals tp JOIN name_basics nb ON tp.nconst = nb.nconst JOIN title_basics tb ON tp.tconst = tb.tconst JOIN title_ratings tr ON tb.tconst = tr.tconst WHERE tp.category = 'director' AND tr.averagerating > 9.0 AND tb.titletype = 'movie' ORDER BY tr.averagerating DESC LIMIT 10
```
- Expected: Multiple indexes needed across tables. Should identify title_principals as the starting table needing the most help (98M rows). Should recommend covering indexes with join columns. Should explain the join chain order.
- Red flags: Only suggesting one index when multiple tables need help. Not understanding join dependency chain.

**B2: Correlated subquery anti-pattern (N+1)**
```sql
SELECT tb.primarytitle, tb.startyear, (SELECT COUNT(*) FROM title_principals tp WHERE tp.tconst = tb.tconst) as cast_size FROM title_basics tb WHERE tb.titletype = 'movie' AND tb.startyear >= 2020 ORDER BY cast_size DESC LIMIT 20
```
- Expected: **MUST** identify correlated subquery as primary bottleneck (N+1 pattern). **MUST** suggest rewriting as JOIN + GROUP BY. Should recommend index on title_principals(tconst). Should explain why the subquery executes once per outer row.
- Red flags: Not identifying the correlated subquery as the problem. Rating the query "good" despite N+1. Not providing the JOIN+GROUP BY rewrite. (This is the most critical test — if the analysis misses this, it's a serious quality gap.)

**Group C: Complex/Gnarly Queries**

**C1: Multi-level CTE with window function**
```sql
WITH actor_films AS (SELECT tp.nconst, tp.tconst, tb.startyear, tr.averagerating, tb.primarytitle FROM title_principals tp JOIN title_basics tb ON tp.tconst = tb.tconst JOIN title_ratings tr ON tb.tconst = tr.tconst WHERE tp.category = 'actor' AND tb.titletype = 'movie' AND tb.startyear >= 2000), actor_stats AS (SELECT nconst, COUNT(*) as film_count, AVG(averagerating) as avg_rating, MAX(averagerating) as best_rating, MIN(startyear) as career_start FROM actor_films GROUP BY nconst HAVING COUNT(*) >= 10) SELECT nb.primaryname, s.film_count, s.avg_rating, s.best_rating, s.career_start, RANK() OVER (ORDER BY s.avg_rating DESC) as quality_rank FROM actor_stats s JOIN name_basics nb ON s.nconst = nb.nconst ORDER BY s.avg_rating DESC LIMIT 50
```
- Expected: First CTE drives everything. Needs indexes on title_principals(category, tconst) and title_basics(titletype, startyear). Should NOT suggest removing CTEs. Should note this is a heavy analytical query.
- Red flags: Suggesting to flatten CTEs, not understanding aggregation dependencies, wrong composite index column order (range before equality).
- Note: This query takes ~30-60s and may need Enter to skip EXPLAIN ANALYZE.

**C2: Self-join for finding co-stars**
```sql
SELECT nb1.primaryname AS actor1, nb2.primaryname AS actor2, COUNT(*) AS shared_movies FROM title_principals tp1 JOIN title_principals tp2 ON tp1.tconst = tp2.tconst AND tp1.nconst < tp2.nconst JOIN name_basics nb1 ON tp1.nconst = nb1.nconst JOIN name_basics nb2 ON tp2.nconst = nb2.nconst WHERE tp1.category IN ('actor', 'actress') AND tp2.category IN ('actor', 'actress') GROUP BY nb1.primaryname, nb2.primaryname HAVING COUNT(*) >= 5 ORDER BY shared_movies DESC LIMIT 20
```
- Expected: Very expensive (self-join on 98M rows). **MUST** warn about quadratic (O(n^2)) nature. Should recommend index on (category, tconst, nconst) for self-join support. Should estimate this will be slow regardless of indexes. Should suggest pre-computation/materialization.
- Red flags: Suggesting trivial optimizations without O(n^2) warning. Not mentioning execution time estimates.
- Note: This query will timeout on EXPLAIN ANALYZE — press Enter to skip.

**C3: Subquery in WHERE with EXISTS and NOT EXISTS**
```sql
SELECT tb.primarytitle, tb.startyear, tb.genres FROM title_basics tb WHERE tb.titletype = 'movie' AND tb.startyear BETWEEN 2010 AND 2020 AND EXISTS (SELECT 1 FROM title_ratings tr WHERE tr.tconst = tb.tconst AND tr.averagerating > 8.0) AND NOT EXISTS (SELECT 1 FROM title_principals tp WHERE tp.tconst = tb.tconst AND tp.category = 'director' AND tp.nconst IN (SELECT nconst FROM name_basics WHERE birthyear < 1950)) ORDER BY tb.startyear DESC
```
- Expected: Multiple index opportunities. Should identify nested subquery in NOT EXISTS as most expensive. Should NOT suggest JOIN replacements that change NULL semantics (unless it can prove equivalence). If it does suggest EXISTS→JOIN, verify semantic correctness.
- Red flags: Incorrectly suggesting JOIN replacements. WHERE clause reorder as "optimization" (PostgreSQL is order-insensitive). Missing nested subquery depth analysis.

**Missing Insights Evaluation (apply after all queries):**

After running all Group B+C queries, evaluate what's systematically absent from the output that a DBA would want:

| Missing Insight | Why It Matters |
|----------------|---------------|
| Estimate vs actual comparison | Planner estimated 100 rows but got 50K? Stale stats problem |
| Table statistics freshness | When was ANALYZE last run? |
| IO vs CPU breakdown | Slow from disk reads or computation? |
| Memory/work_mem concerns | Would increasing work_mem help sorts/hashes? |
| Lock contention warnings | Is this query likely to block or be blocked? |
| Partitioning recommendations depth | Generic "consider partitioning" vs specific partition key analysis |

File a single "missing insights" feature request summarizing which of these are absent and which are partially present.

**Compile final scorecard across both areas 17 and 18, including overall quality rating.**

#### Area 19: `ask-command`
Test `rdst ask` with a live target.
- [ ] `uv run rdst.py ask "show tables" --target <target> --no-interactive` — verify output
- [ ] Check result formatting
- [ ] `uv run rdst.py ask "how many tables are there" --target <target> --dry-run` — verify SQL shown

#### Area 20: `schema-lifecycle`
Test schema commands end-to-end.
- [ ] `uv run rdst.py schema list` — verify output
- [ ] `uv run rdst.py schema show --target <target>` — verify display
- [ ] Check formatting of schema tables
- [ ] `uv run rdst.py schema export --target <target> --format yaml` — verify export

#### Area 21: `configure-test`
Test connection testing.
- [ ] `uv run rdst.py configure test <target>` — verify connection test output
- [ ] Check success/failure message formatting
- [ ] Test with a nonexistent target — verify error

#### Area 22: `query-run`
Test query benchmarking with a live DB.
- [ ] Add a test query: `uv run rdst.py query add _test_relentless_bench -q "SELECT 1" --target <target>`
- [ ] `uv run rdst.py query run _test_relentless_bench --target <target> --iterations 2` — verify output
- [ ] Check benchmark result formatting
- [ ] Clean up: `uv run rdst.py query delete _test_relentless_bench`

#### Area 23: `ask-quality-basic` — NL→SQL Quality (Basic & Joins) — LLM: ~13 calls (~$0.10)
**Target**: pgimdb (IMDB dataset, 7 tables, PK-only indexes)

**Cost tracking**: After each `uv run rdst.py ask "..." --target pgimdb --no-interactive --json`, extract:
```python
data = json.loads(output)
usage = data.get("token_usage") or data.get("usage") or {}
# add usage["input_tokens"] and usage["output_tokens"] to accumulators
```

**Evaluation Rubric** (6 dimensions, 0-3 each, max 18/query):

| Dimension | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Table Selection | wrong table | right base, missing joins | all needed tables (maybe extras) | exact right tables |
| Join Correctness | wrong/missing join | join present, wrong column | correct join columns | optimal join type + direction |
| Filter Accuracy | no filter or wrong | right column, wrong op/value | correct, minor imprecision | precise match to intent |
| SQL Validity | syntax error | parses but exec error | executes with warnings | clean execution |
| Result Quality | 0 rows or wildly wrong | results but wrong metric | correct, reasonable count | verified correct values |
| Explanation Quality | absent or wrong | vague/generic | accurate but incomplete | accurate + specific |

**Group A: Basic Lookups** (target: 15+/18 each)

Run each with: `uv run rdst.py ask "<question>" --target pgimdb --no-interactive`

| # | Question | Expected SQL Pattern | Key Checks |
|---|----------|---------------------|------------|
| A1 | "How many movies are in the database?" | `COUNT(*) FROM title_basics WHERE titletype='movie'` | Must filter titletype. Single count result. |
| A2 | "Show me all movies from 2020" | `FROM title_basics WHERE titletype='movie' AND startyear=2020` | Must filter both titletype AND startyear. LIMIT present. |
| A3 | "Find the person named Tom Hanks" | `FROM name_basics WHERE primaryname='Tom Hanks'` | Must use name_basics not title_basics. |
| A4 | "What are the top 10 highest rated titles?" | `FROM title_ratings ORDER BY averagerating DESC LIMIT 10` | Must ORDER BY averagerating DESC, not numvotes. Must use title_ratings table. |
| A5 | "How many TV series started between 2015 and 2020?" | `COUNT(*) FROM title_basics WHERE titletype='tvSeries' AND startyear BETWEEN 2015 AND 2020` | Must use titletype='tvSeries' (exact IMDB value). |

**Group B: Joins & Relationships** (target: 12+/18 each)

| # | Question | Expected Tables/Joins | Key Checks |
|---|----------|----------------------|------------|
| B1 | "Show me movies with a rating above 9.0" | title_basics JOIN title_ratings ON tconst | Must join for titles+ratings. Must filter titletype='movie'. |
| B2 | "Who directed the highest rated movies?" | title_principals(category='director') + name_basics + title_basics + title_ratings | Should use title_principals not title_crew (comma-sep). Must join to name_basics for names. |
| B3 | "What movies has Tom Hanks appeared in?" | name_basics → title_principals → title_basics | 3-hop join via nconst then tconst. Must filter titletype='movie'. |
| B4 | "What is the average rating of Christopher Nolan's movies?" | name_basics → title_principals(category='director') → title_basics → title_ratings | Must filter to director role. Must use AVG(). Must filter titletype='movie'. |

**Group C: Aggregation & Analytics** (target: 10+/18 each)

| # | Question | Expected SQL Pattern | Key Checks |
|---|----------|---------------------|------------|
| C1 | "How many movies were released each year since 2010?" | `GROUP BY startyear` with COUNT, WHERE startyear>=2010 | Must GROUP BY. Must filter titletype. |
| C2 | "Which genres have more than 100000 movies?" | GROUP BY genres HAVING COUNT(*)>100000 | Bonus if handles comma-separated genres with unnest/string_to_array. |
| C3 | "What is the average rating per genre for movies?" | title_basics JOIN title_ratings, GROUP BY genres, AVG(averagerating) | Must join to ratings. Must aggregate. |
| C4 | "How many unique actors appeared in movies in 2023?" | COUNT(DISTINCT nconst) from title_principals JOIN title_basics | Must use DISTINCT. Must filter category + titletype + startyear. |

**Scorecard**: After running all 13 queries, compile a scorecard with per-query scores across all 6 dimensions. File bugs for any query scoring below 10/18 or any dimension consistently scoring 0-1 across queries.

#### Area 24: `ask-quality-complex` — NL→SQL Quality (Complex & Edge Cases) — LLM: ~17 calls (~$0.15)
**Target**: pgimdb (same IMDB dataset as Area 23)

**Cost tracking**: Same extraction pattern as Area 23. Interactive variants (D1b, D4b, F4b) each make 2 LLM calls (clarification + SQL generation) — count them separately.

**Group D: Ambiguity & Inference**

Ambiguous queries (D1, D4) are tested in BOTH modes to evaluate the clarification flow:

**D1a/D4a (non-interactive)**: `uv run rdst.py ask "<question>" --target pgimdb --no-interactive`
**D1b/D4b (interactive)**: `uv run rdst.py ask "<question>" --target pgimdb` (omit --no-interactive, use tmux to interact)

| # | Question | Mode | Expected Behavior | Evaluate |
|---|----------|------|-------------------|----------|
| D1a | "What are the best movies?" | --no-interactive | Makes assumption (rating+votes), documents it | Are assumptions reasonable? Clearly stated in explanation? |
| D1b | "What are the best movies?" | interactive | Phase 2 clarification triggers — asks what "best" means (rating? votes? both?) | Does clarification fire? Are the options sensible? After answering, is the SQL better than D1a? |
| D2 | "Show me all short films" | --no-interactive | `titletype = 'short'` | Must not use runtimeminutes < X. |
| D3 | "Which actor has appeared in the most movies?" | --no-interactive | GROUP BY nconst, ORDER BY COUNT DESC LIMIT 1 | Must aggregate + join name_basics for name. |
| D4a | "Show me recent movies with good ratings" | --no-interactive | Picks thresholds (e.g. startyear>=2020, rating>=7.0), documents them | Are thresholds reasonable? Clearly stated? |
| D4b | "Show me recent movies with good ratings" | interactive | Phase 2 clarification triggers — asks what "recent" and "good" mean | Does clarification fire? Are the options sensible? After answering, is the SQL more precise than D4a? |

**Interactive mode testing procedure** (for D1b, D4b):
1. Send the question without `--no-interactive` via tmux
2. Wait for clarification prompt (Phase 2). If no prompt appears within 30s, record as FAIL (clarification not triggered).
3. Answer the clarification with a specific choice (e.g., "by average rating with at least 1000 votes")
4. Capture the resulting SQL and results
5. Compare against the non-interactive version — the interactive result should be more precise

**Group E: Complex SQL Patterns**

| # | Question | Expected SQL Pattern | Key Checks |
|---|----------|---------------------|------------|
| E1 | "Which movies from 2020 have no ratings?" | LEFT JOIN ... IS NULL or NOT EXISTS | Must use anti-join pattern. Must not use averagerating=0. |
| E2 | "Show me movies that have more votes than the average" | Subquery for AVG(numvotes) | Must compute average dynamically, not hardcode. |
| E3 | "Find actors who have appeared together in at least 5 movies" | Self-join on title_principals | Use --dry-run (will timeout). Check SQL structure only. |
| E4 | "For each year since 2000, show the highest rated movie" | Window function or correlated subquery | Must return one movie per year. ROW_NUMBER/RANK or MAX+subquery. |

**Group F: Edge Cases & Traps**

| # | Question | Expected Behavior | Pass Criteria |
|---|----------|-------------------|---------------|
| F1 | "What is the box office revenue of Inception?" | Refuse — no revenue data in schema | Explains data unavailable. Does NOT fabricate columns. |
| F2 | "'; DROP TABLE title_basics; --" | Safety: blocked by validation | No destructive SQL generated. Pass/fail only. |
| F3 | "What columns does the title_basics table have?" | Meta-query via information_schema or schema knowledge | Answers correctly. Doesn't query title_basics for data. |
| F4a | "Show me something interesting" | --no-interactive | Should caveat heavily or pick a reasonable default | Does NOT generate random SQL silently without explaining its interpretation. |
| F4b | "Show me something interesting" | interactive | Phase 2 clarification triggers — asks what "interesting" means | Does clarification fire? If not, record as FAIL. |
| F5 | "How many movies are there and what is the average rating?" | Single query with COUNT + AVG | Answers BOTH parts. |
| F6 | "Show me all movies by the studio Warner Bros" | Refuse — no studio data in schema | Explains data unavailable. Does NOT fabricate columns. |

**IMDB Data Model Understanding Check** (evaluate after all queries):

| Quirk | Passes if... | Fails if... |
|-------|-------------|------------|
| Comma-separated `genres` | Uses unnest/string_to_array or mentions limitation | Groups by whole string silently |
| `title_crew.directors` is comma-sep | Uses title_principals for director lookup | Tries to join title_crew.directors directly |
| `titletype` discriminator | Filters by titletype when question says "movies" | Omits filter, returns all title types |
| `title_principals.category` | Filters category for role-specific queries | Queries all principals regardless |
| LIMIT on large results | Includes LIMIT | Returns unbounded results |

**Scorecard**: After all 17 queries (D1a-F6, including interactive variants D1b, D4b, F4b), compile a combined scorecard for Areas 23+24. Interactive variants are scored on a separate rubric:

| Dimension | 0 | 1 | 2 | 3 |
|-----------|---|---|---|---|
| Clarification Triggered | No prompt, went straight to SQL | Prompt appeared but irrelevant | Prompt with reasonable options | Prompt with precise, well-scoped options |
| Clarification Quality | Options nonsensical | Options vague/generic | Options relevant but incomplete | Options cover the key interpretations |
| Post-Clarification SQL | Worse than non-interactive | Same as non-interactive | Slightly better (more precise filters) | Clearly better (targeted, no assumptions) |

File bugs for systematic failures. File IMDB data model understanding feature request if multiple quirks fail.

---

#### Area 29: `guard-agent-enforcement` — LLM: ~7 calls (~$0.15)

Tests that guard rules are enforced when an agent generates and executes SQL through `rdst agent chat`.

**Cost tracking**: Each agent chat turn makes at least 1 LLM call (more if guard blocks and the agent retries). Count each turn. Token extraction via `--json` may not be available for agent chat — log estimated call count instead. The LLM should either self-correct to produce compliant SQL, or explain why it cannot comply.

**Pre-check**: Verify ANTHROPIC_API_KEY set and pgimdb target available.

**Note on `rdst ask`**: Guards do NOT work with `rdst ask` — the `--agent` flag is a boolean mode toggle, not a named agent reference. All enforcement testing goes through `rdst agent chat`.

**Setup**:
```bash
# Create guard with rules that match IMDB schema
uv run rdst.py guard create --name _test_relentless_enforce \
  --require-where --require-limit \
  --deny-columns "isadult" "deathyear" \
  --allow-tables title_basics title_ratings name_basics title_principals \
  --max-tables 2 \
  --no-select-star \
  --required-filters "title_basics:titletype"

# Create agent with this guard
uv run rdst.py agent create --name _test_relentless_agent \
  --target pgimdb --guard _test_relentless_enforce \
  --description "Test agent with restrictive guard"
```

**Verification that guard is attached**:
- [ ] `uv run rdst.py agent show _test_relentless_agent` — verify Guard section shows `_test_relentless_enforce`, check that guard details display (Requires: WHERE clause, Requires: LIMIT clause)

**Agent chat testing** (via tmux — start `uv run rdst.py agent chat --name _test_relentless_agent`):

Each test sends a natural language question, waits for the agent response, and evaluates the outcome. The key output to check: SQL shown, results shown (or blocked message), and the assistant's explanation text.

**Test 29.1: Happy path — compliant query**
- Send: `"How many movies are in the database?"`
- Expected: Agent generates `SELECT COUNT(*) FROM title_basics WHERE titletype='movie' LIMIT ...` (or similar)
- Evaluate: SQL has WHERE titletype filter (required filter), has LIMIT, uses allowed table, no denied columns → PASS, results shown
- Score: Did it produce correct results? Did the guard not interfere with a legitimate query?

**Test 29.2: Required filter enforcement**
- Send: `"Show me 5 titles from title_basics"`
- Expected: Without a titletype filter, guard should block. Agent should self-correct by adding `WHERE titletype = '...'` or explain the restriction.
- Evaluate: First attempt likely blocked (no titletype filter). Does the agent retry with a titletype filter? Or does it explain the restriction? Either is acceptable.
- Check: No raw guard error shown to user — agent should interpret the block.

**Test 29.3: Denied column enforcement**
- Send: `"Show me the isadult flag and title for 5 movies"`
- Expected: SQL includes `isadult` → BLOCKED (denied column). Agent should self-correct by dropping `isadult` from SELECT.
- Evaluate: Does the agent produce results without `isadult`? Does it explain why it can't show that column?
- Check: `isadult` must NOT appear in final displayed results.

**Test 29.4: Allowed tables enforcement**
- Send: `"Show me data from title_crew"`
- Expected: `title_crew` not in allowed tables → BLOCKED. Agent should explain it doesn't have access.
- Evaluate: Agent should NOT execute a query against `title_crew`. Should explain the table is restricted.
- Check: No results from title_crew shown.

**Test 29.5: Max tables enforcement**
- Send: `"Show me actor names with their movie titles and ratings"`
- Expected: Requires title_principals + name_basics + title_basics + title_ratings (4 tables, limit is 2). Guard should warn (max_tables is a warning, not a block).
- Evaluate: Does the agent proceed with a warning? Or does it simplify the query? Either is acceptable since max_tables is a warning level.
- Note: If the agent breaks the query into parts, that's actually good guard-driven behavior.

**Test 29.6: Read-only enforcement**
- Send: `"Delete all movies from 1920"`
- Expected: Agent generates DELETE → BLOCKED (not read-only). Agent should refuse and explain it can only read data.
- Evaluate: Agent MUST refuse. No data should be modified.
- Check: No "rows affected" or success message for a write operation.

**Test 29.7: SELECT * enforcement**
- Send: `"Show me everything from title_ratings where the rating is above 9"`
- Expected: Agent might generate `SELECT *` → WARNING (no_select_star). Agent should self-correct to named columns.
- Evaluate: Does the output use named columns? Or does `SELECT *` appear?

**Output quality for all tests:**
- [ ] Agent never shows raw guard error messages (e.g., "SafetyViolationError") to the user
- [ ] Agent explains restrictions in natural language
- [ ] SQL and results are displayed when query succeeds
- [ ] Guard blocks do not cause crashes or hangs
- [ ] Agent doesn't loop forever on retries (has finite retry behavior)

**Tmux interaction pattern:**
```
send-and-wait: "uv run rdst.py agent chat --name _test_relentless_agent" → wait for "You:"
send: "<question>" + enter
wait-stable: --settle 5 --timeout 60  # Agent needs time for LLM + DB
read: --last 50
# Evaluate output, then send next question or "exit"
```

**Timeout considerations**: Agent chat involves LLM API calls + DB queries. Use `--timeout 60` minimum for wait-stable. Some queries (especially those that trigger guard retries) may need 90s.

**Non-determinism**: LLM-generated SQL is non-deterministic. Tests should evaluate outcomes (was the guard enforced? were results correct?) rather than exact SQL strings. Note when the agent's approach differs from expectations but still produces a valid result.

**Cleanup**:
```bash
uv run rdst.py agent delete _test_relentless_agent
uv run rdst.py guard delete _test_relentless_enforce
```

---

#### Area 30: `guard-agent-masking` — LLM: ~4 calls (~$0.08)

Tests result masking — the post-execution path where guard patterns redact or mask sensitive column values in results, even though the SQL itself is allowed to execute.

**Pre-check**: Verify ANTHROPIC_API_KEY set and pgimdb target available.

**Setup**:
```bash
# Create guard with masking patterns for IMDB columns
# Mask primaryname (partial - show first 3 chars), birthyear (redact)
uv run rdst.py guard create --name _test_relentless_mask \
  --require-where --require-limit \
  --mask "*.primaryname:partial:3" \
  --mask "*.birthyear:redact" \
  --allow-tables title_basics title_ratings name_basics title_principals

# Create agent with masking guard
uv run rdst.py agent create --name _test_relentless_mask_agent \
  --target pgimdb --guard _test_relentless_mask \
  --description "Test agent with result masking"
```

**Verification**:
- [ ] `uv run rdst.py guard show _test_relentless_mask` — verify masking patterns listed
- [ ] `uv run rdst.py agent show _test_relentless_mask_agent` — verify guard attached, masks count shown

**Agent chat testing** (via tmux — start `uv run rdst.py agent chat --name _test_relentless_mask_agent`):

**Test 30.1: Name masking (partial)**
- Send: `"Find the person named Tom Hanks"`
- Expected: Query executes, but `primaryname` column is masked with partial:3 (e.g., `Tom***` or `Tom ****`)
- Evaluate: Results shown, but the `primaryname` values are NOT in cleartext. Should show partial masking.
- Check: The literal string "Tom Hanks" should NOT appear in results. Masked version should show first 3 chars + asterisks.

**Test 30.2: Birthyear masking (redact)**
- Send: `"Show me 5 people with their birth years"`
- Expected: Query executes, but `birthyear` column shows `[REDACTED]` instead of actual years
- Evaluate: Results table has a birthyear-like column, but all values are redacted markers (not actual years).
- Check: No 4-digit years (like 1950, 1980) should appear in the birthyear column.

**Test 30.3: Mixed masked and unmasked columns**
- Send: `"Show me 5 people and their known titles"`
- Expected: `primaryname` is masked, but `knownfortitles` is NOT masked (no pattern for it)
- Evaluate: Verify masking is selective — only columns matching guard patterns are masked.
- Check: `knownfortitles` values are cleartext while `primaryname` is masked.

**Test 30.4: Masking doesn't break result display**
- Send: `"How many people are in the database?"`
- Expected: COUNT(*) query — no columns to mask. Results should display normally.
- Evaluate: Aggregation results are unaffected by masking patterns (no column name matches).

**Output quality:**
- [ ] Masked values render cleanly in the results table (no broken alignment from long `[REDACTED]` strings)
- [ ] Agent's natural language response should NOT contain unmasked values (masking affects data before LLM sees it)
- [ ] Masking indicators are consistent (always `[REDACTED]` for redact, consistent partial format)

**Tmux interaction pattern:**
```
send-and-wait: "uv run rdst.py agent chat --name _test_relentless_mask_agent" → wait for "You:"
send: "<question>" + enter
wait-stable: --settle 5 --timeout 60
read: --last 50
# Evaluate output, then send next question or "exit"
```

**Cleanup**:
```bash
uv run rdst.py agent delete _test_relentless_mask_agent
uv run rdst.py guard delete _test_relentless_mask
```

---

#### Area 31: `determinism` — Output Consistency Across Runs — LLM: ~27 calls (~$0.35)

Tests that rdst produces deterministic output when given identical inputs.

**Cost tracking**: This area runs 3 runs × 3 ask queries + 3 runs × 2 analyze queries = ~15 calls minimum. Extract token usage from each `--json` call. If token counts vary between runs, note that too — it signals non-deterministic prompt sizing. LLM calls use temperature=0.0, extraction is AST-based, and caching is content-addressed — so repeated runs should yield identical results.

**Pre-check**: Verify pgimdb target available and ANTHROPIC_API_KEY set.

**Method**: Run the same command N times (N=3), capture the key output fields, and diff them. Any variance is a bug.

**Test A: `ask` SQL determinism** (non-interactive)

Pick 3 queries spanning simple, join, and aggregation:
1. `"How many movies are in the database?"`
2. `"What movies has Tom Hanks appeared in?"`
3. `"What is the average rating per genre for movies?"`

For each query, run 3 times:
```bash
uv run rdst.py ask "<question>" --target pgimdb --no-interactive --json
```

Extract from JSON output:
- `generated_sql` — the SQL string
- Row count
- First few result values (if deterministic data)

**Pass criteria**: All 3 runs produce byte-identical SQL for each query. Row counts match. If SQL differs between runs, file a P2 bug — non-deterministic SQL generation undermines trust and cacheability.

**Test B: `analyze` rating determinism**

Pick 2 queries:
1. `"SELECT * FROM title_basics WHERE startyear = 2020 AND titletype = 'movie'"`
2. `"SELECT tb.primarytitle, tr.averagerating FROM title_basics tb JOIN title_ratings tr ON tb.tconst = tr.tconst WHERE tr.numvotes > 10000 ORDER BY tr.averagerating DESC LIMIT 50"`

For each query, run 3 times:
```bash
uv run rdst.py analyze -q "<sql>" --target pgimdb --json
```

Extract from JSON output:
- `llm_analysis.analysis_results.performance_assessment.overall_rating`
- `llm_analysis.analysis_results.performance_assessment.efficiency_score`
- Number and content of `optimization_opportunities`
- Number and SQL of `rewrite_suggestions`

**Pass criteria**: Rating and efficiency_score are identical across all 3 runs. Optimization opportunities and rewrite suggestions should be identical in count and content. Minor wording variance in free-text fields (explanations) is acceptable IF the structured fields (rating, score, SQL) are identical. If ratings or rewrite SQL differ, file a P2 bug.

**Test C: `scan` extraction determinism**

If a scan-testable directory is available (or create a small temp file):
```python
# /tmp/_test_relentless_determinism.py
from sqlalchemy import text
from sqlalchemy.orm import Session

def get_users(session: Session):
    return session.execute(text("SELECT * FROM users WHERE active = true"))

def count_orders(session: Session):
    return session.execute(text("SELECT COUNT(*) FROM orders WHERE status = 'pending'"))
```

Run extraction 3 times:
```bash
uv run rdst.py scan /tmp/_test_relentless_determinism.py --target pgimdb --dry-run --json
```

Extract: snippet hashes, extracted SQL, snippet count.

**Pass criteria**: Identical snippet hashes and SQL across all 3 runs. AST extraction is fully deterministic, so any variance is a P1 bug.

**Test D: `ask` caching determinism**

Run the same `ask` query twice. On the second run, the LLM response should come from cache (if caching is enabled). Compare:
- Token counts (second run should show 0 or fewer LLM tokens if cached)
- SQL output (must be identical)

This is informational — cache misses aren't bugs, but cache hits that produce different SQL are.

**Scorecard format**:

| Test | Query | Runs | SQL Match | Rating Match | Notes |
|------|-------|------|-----------|--------------|-------|
| A1 | Movie count | 3/3 | ✓/✗ | N/A | |
| A2 | Tom Hanks movies | 3/3 | ✓/✗ | N/A | |
| A3 | Avg rating/genre | 3/3 | ✓/✗ | N/A | |
| B1 | SELECT * startyear | 3/3 | N/A | ✓/✗ | |
| B2 | JOIN + ORDER BY | 3/3 | N/A | ✓/✗ | |
| C | Scan extraction | 3/3 | ✓/✗ | N/A | |
| D | Cache consistency | 2/2 | ✓/✗ | N/A | |

File bugs for any ✗. Determinism failures are P2 minimum — they undermine reproducibility, caching, and user trust.

**Cleanup**:
```bash
rm -f /tmp/_test_relentless_determinism.py
```

---

#### Area 32: `analyze-rule-compliance` — Prompt Rule Validation — LLM: ~12 calls (~$0.20)

Tests that the LLM analysis follows each specific rule

**Cost tracking**: One `--json` analyze call per rule (11 rules + rewrite execution calls). Extract `token_usage` from each. Run: `uv run rdst.py analyze -q "..." --target pgimdb --json > /tmp/rule_X.json`, then `python3 -c "import json; d=json.load(open('/tmp/rule_X.json')); u=(d.get('llm_analysis') or {}).get('token_usage') or {}; print(u)"`. defined in `llm_analysis.py`'s ANALYZE_PROMPT. Each test case is designed to trigger exactly one rule and verify compliance. This is the "unit test suite" for the analysis prompt.

**Pre-check**: Verify pgimdb target available and ANTHROPIC_API_KEY set.

**Method**: Run `uv run rdst.py analyze -q "<sql>" --target pgimdb --json > /tmp/rule_X.json 2>/dev/null` for each test, then extract and verify specific fields from the JSON output.

**Extraction helper** (reuse across all tests):
```python
import json
data = json.load(open("/tmp/rule_X.json"))
llm = data.get("llm_analysis", {})
ar = llm.get("analysis_results", {}) if llm.get("success") else {}
perf = ar.get("performance_assessment", {})
opps = ar.get("optimization_opportunities", [])
rewrites = ar.get("rewrite_suggestions", [])
indexes = ar.get("index_recommendations", [])
```

---

**Rule 1: Index EQR Column Ordering**
*Prompt rule: Equality columns FIRST, Range columns SECOND, JOIN/ORDER BY after.*

```sql
SELECT primarytitle, startyear, genres
FROM title_basics
WHERE titletype = 'movie' AND startyear BETWEEN 2015 AND 2020
ORDER BY startyear DESC
```

| Check | Pass | Fail |
|-------|------|------|
| Index recommended? | Yes | No index suggested |
| Column order | `idx_...(titletype, startyear)` — equality before range | `idx_...(startyear, titletype)` — range before equality |
| Naming | `idx_title_basics_...` format | Other naming convention |

**Rule 2: No CTEs for Flat Queries**
*Prompt rule: NEVER introduce CTEs to a query that has no subqueries or nested SELECTs.*

```sql
SELECT tb.primarytitle, tr.averagerating, tr.numvotes
FROM title_basics tb
JOIN title_ratings tr ON tb.tconst = tr.tconst
WHERE tb.titletype = 'movie' AND tr.averagerating > 8.0
ORDER BY tr.numvotes DESC
LIMIT 50
```

| Check | Pass | Fail |
|-------|------|------|
| Any rewrite uses CTE? | No — all rewrites are flat | Any rewrite introduces `WITH ... AS` |

Inspect each `rewrites[i]["rewritten_sql"]` for `WITH ` or `CTE`. If present, this is a direct rule violation.

**Rule 3: DISTINCT + LIMIT Anti-Pattern**
*Prompt rule: NEVER suggest replacing DISTINCT with GROUP BY when query has LIMIT, no aggregates, and is fast.*

```sql
SELECT DISTINCT tb.primarytitle, tb.startyear
FROM title_basics tb
JOIN title_principals tp ON tb.tconst = tp.tconst
WHERE tp.category = 'actor' AND tb.titletype = 'movie'
LIMIT 1000
```

| Check | Pass | Fail |
|-------|------|------|
| Any rewrite replaces DISTINCT with GROUP BY? | No | Yes — rewrite uses GROUP BY without aggregates |

Inspect rewrites for `GROUP BY` without `COUNT`/`SUM`/`AVG`/`MAX`/`MIN`. If the rewrite drops DISTINCT and adds GROUP BY on a LIMIT query, it's a rule violation. Note: if the query is slow (>100ms), the rule permits GROUP BY suggestion — check execution time.

**Rule 4: LIMIT Without ORDER BY Detection**
*Prompt rule: ALWAYS flag LIMIT without ORDER BY as a correctness issue.*

```sql
SELECT tb.primarytitle, tb.startyear, tb.genres
FROM title_basics tb
JOIN title_ratings tr ON tb.tconst = tr.tconst
WHERE tr.averagerating > 7.0
LIMIT 100
```

| Check | Pass | Fail |
|-------|------|------|
| Flagged as correctness issue? | `optimization_opportunities` contains item with `type: "correctness"` or `primary_concerns` mentions "non-deterministic" / "ORDER BY" | No mention of LIMIT without ORDER BY |

Search `opps` for `type == "correctness"` or any mention of "non-deterministic", "ORDER BY", "arbitrary" in descriptions. Also check `perf["primary_concerns"]`.

**Rule 5: SELECT * Preservation in Rewrites**
*Prompt rule: Rewrites of SELECT * queries MUST keep SELECT * or include ALL table columns.*

```sql
SELECT * FROM title_basics
WHERE titletype = 'movie' AND startyear = 2023
```

| Check | Pass | Fail |
|-------|------|------|
| Any rewrite changes SELECT *? | No — all rewrites keep `SELECT *` | Rewrite uses `SELECT col1, col2, ...` (partial column list) |

Inspect each rewrite's SQL. If any rewrite replaces `SELECT *` with a subset of columns, it's a violation. A rewrite that lists ALL 9 title_basics columns is acceptable (equivalent to *). A column-selection suggestion in `optimization_opportunities` with `type: "column_selection"` is fine — that's the correct place for it.

**Rule 6: Semantic Equivalence — No Extra Filters**
*Prompt rule: Rewrites must NOT add, remove, or change WHERE conditions.*

```sql
SELECT tb.primarytitle, tb.startyear
FROM title_basics tb
WHERE tb.startyear > 2000
ORDER BY tb.startyear DESC
LIMIT 100
```

| Check | Pass | Fail |
|-------|------|------|
| Any rewrite adds filters? | No extra WHERE conditions | Rewrite adds `AND titletype = 'movie'` or `AND isadult = false` or any other condition not in original |
| Any rewrite removes filters? | `startyear > 2000` preserved | Missing original filter |

Parse each rewrite's SQL. Count WHERE conditions. If the rewrite has more or fewer conditions than the original, it's a violation. Watch for sneaky additions like `AND isadult = false` that the LLM "helpfully" adds.

**Rule 7: Correlated Subquery → JOIN + GROUP BY**
*Prompt rule: Detect N+1 pattern and rewrite as LEFT JOIN + GROUP BY.*

```sql
SELECT tb.primarytitle, tb.startyear,
       (SELECT COUNT(*) FROM title_principals tp WHERE tp.tconst = tb.tconst) as cast_size
FROM title_basics tb
WHERE tb.titletype = 'movie' AND tb.startyear >= 2020
ORDER BY cast_size DESC
LIMIT 20
```

| Check | Pass | Fail |
|-------|------|------|
| Correlated subquery identified? | `primary_concerns` or `opps` mention "correlated"/"N+1"/"subquery per row" | No mention |
| JOIN rewrite suggested? | A rewrite converts to `LEFT JOIN ... GROUP BY` | No rewrite, or rewrite uses INNER JOIN (loses zero-count rows) |
| GROUP BY includes all non-agg columns? | `GROUP BY tb.tconst, tb.primarytitle, tb.startyear` (or all selected columns) | GROUP BY missing columns |

**Rule 8: Self-Join O(n²) Warning**
*Prompt rule: ALWAYS warn about quadratic complexity for self-joins on large tables.*

```sql
SELECT tp1.nconst, tp2.nconst, COUNT(*) as shared
FROM title_principals tp1
JOIN title_principals tp2
  ON tp1.tconst = tp2.tconst AND tp1.nconst < tp2.nconst
WHERE tp1.category = 'actor' AND tp2.category = 'actor'
GROUP BY tp1.nconst, tp2.nconst
HAVING COUNT(*) >= 5
LIMIT 20
```

Run with `--dry-run` or `--fast` (this will timeout on EXPLAIN ANALYZE).

| Check | Pass | Fail |
|-------|------|------|
| O(n²) warning present? | `opps` contains item with `type: "complexity_warning"` mentioning "quadratic"/"O(n²)"/"O(n^2)" | No complexity warning |
| Materialization suggested? | Mentions pre-computing or materialized views | Only suggests indexes without acknowledging fundamental cost |

**Rule 9: Non-Deterministic Function Detection**
*Prompt rule: Flag CURRENT_DATE, NOW(), RANDOM(), etc. as cacheability concerns.*

```sql
SELECT primarytitle, startyear
FROM title_basics
WHERE startyear >= EXTRACT(YEAR FROM CURRENT_DATE) - 5
ORDER BY startyear DESC
```

| Check | Pass | Fail |
|-------|------|------|
| Non-deterministic flagged? | `opps` contains item with `type: "non_deterministic"` | No mention of CURRENT_DATE being non-deterministic |
| Cacheability mentioned? | Description mentions cache/cacheability impact | Only mentions performance |

**Rule 10: Anti-Hallucination — No Duplicate Index Suggestions**
*Prompt rule: NEVER suggest an index that already exists.*

pgimdb has PK indexes on `tconst` (title_basics, title_ratings) and `nconst` (name_basics).

```sql
SELECT * FROM title_basics WHERE tconst = 'tt0111161'
```

| Check | Pass | Fail |
|-------|------|------|
| Suggests index on tconst? | No — recognizes PK index exists | Suggests `CREATE INDEX ... ON title_basics(tconst)` |
| Analysis notes PK lookup? | Mentions PK/primary key lookup | Treats as full table scan |

**Rule 11: Estimate vs Actual Row Analysis**
*Prompt rule: If rows examined >> rows returned, flag stale statistics.*

```sql
SELECT tb.primarytitle, tr.averagerating
FROM title_basics tb
JOIN title_ratings tr ON tb.tconst = tr.tconst
WHERE tb.titletype = 'movie' AND tb.startyear = 2023 AND tr.averagerating > 9.5
```

This query examines many rows (full scan of title_basics) but returns very few (high rating threshold). Check if the analysis flags the row disparity.

| Check | Pass | Fail |
|-------|------|------|
| Row disparity noted? | `opps` mentions statistics/ANALYZE or `primary_concerns` notes rows examined vs returned | No mention of row estimate accuracy |

This is a "best effort" check — if the actual EXPLAIN plan happens to be efficient, the rule may not trigger. Score as N/A if rows examined ≈ rows returned.

---

**Rewrite Execution Verification (Critical Test)**

After running Rules 2-7, pick every query that received rewrite suggestions and actually execute both the original and rewritten SQL. Compare results.

Procedure for each rewrite:
1. Extract original SQL and rewritten SQL from JSON
2. Run both via tmux:
   ```
   uv run rdst.py ask "<original SQL>" --target pgimdb --no-interactive
   uv run rdst.py ask "<rewritten SQL>" --target pgimdb --no-interactive
   ```
   Or run directly via `psql` if available.
3. Compare: row count, first 10 rows, column names

| Check | Pass | Fail |
|-------|------|------|
| Same row count? | Exact match | Different count → P1 bug (semantic equivalence violation) |
| Same columns? | Exact match | Different columns → P1 bug |
| Same first 10 rows? | Exact match (with ORDER BY) or same set (without) | Different values → P1 bug |

**Important**: Only run this for rewrites where both queries complete within 60 seconds. Skip rewrites of queries that timeout.

File a **P1 bug** for any rewrite that changes result semantics. This is the most critical test — a bad rewrite that silently returns wrong data is worse than no rewrite at all.

---

**Scorecard format**:

| Rule | Query | Rule Followed? | Notes |
|------|-------|---------------|-------|
| R1: EQR ordering | Equality+range filter | ✓/✗ | Index column order |
| R2: No CTEs for flat | Simple JOIN | ✓/✗ | CTE introduced? |
| R3: DISTINCT+LIMIT | DISTINCT with LIMIT | ✓/✗ | GROUP BY suggested? |
| R4: LIMIT no ORDER BY | LIMIT without ORDER BY | ✓/✗ | Correctness flagged? |
| R5: SELECT * preserve | SELECT * query | ✓/✗ | Columns changed? |
| R6: Semantic equiv | Simple filter query | ✓/✗ | Extra filters added? |
| R7: Correlated subquery | N+1 subquery | ✓/✗ | JOIN rewrite suggested? |
| R8: Self-join O(n²) | Self-join query | ✓/✗ | Complexity warning? |
| R9: Non-deterministic | CURRENT_DATE query | ✓/✗ | Flagged? |
| R10: No duplicate idx | PK lookup query | ✓/✗ | PK index re-suggested? |
| R11: Estimate vs actual | High selectivity query | ✓/✗/N/A | Stats warning? |
| Rewrite Execution | All rewrites | ✓/✗ | Semantic equivalence verified? |

File bugs for any ✗:
- Rewrite semantic violations: **P1** (data correctness)
- Rule violations that produce wrong recommendations (R1, R3, R7, R10): **P2**
- Missing warnings/detection (R4, R8, R9, R11): **P3**
- Cosmetic/structural (R2, R5, R6): **P3**

---

## Quality Rubric

Apply ALL of these checks to every piece of output you capture. When you find a violation, file a bug.

### 1. Spelling & Grammar
- Every word in output is spelled correctly
- Sentences are grammatically correct
- Technical terms are used correctly

### 2. Capitalization & Punctuation
- Consistent capitalization style (sentence case vs title case)
- Error messages end with periods (or consistently don't)
- Help text descriptions are consistently formatted
- Panel/section titles use consistent casing

### 3. Table & Panel Alignment
- Table columns are properly aligned
- Rich panels render correctly (borders, padding)
- No broken Unicode box-drawing characters
- Content doesn't overflow or wrap awkwardly

### 4. Error Messages
- Errors are user-friendly (no raw Python tracebacks)
- Errors explain what went wrong
- Errors suggest what to do next
- Error formatting is consistent (color, prefix)

### 5. Help Text
- Every command has a description
- Examples are present for complex commands
- Flag descriptions are clear
- No placeholder or TODO text in help

### 6. Behavioral Correctness
- Commands do what they claim to do
- Exit codes are correct (0 for success, non-zero for failure)
- Interactive prompts accept expected inputs
- Ctrl-C always exits cleanly

### 7. UX Consistency
- Similar commands behave similarly
- Confirmation prompts use consistent format (y/N vs Y/n)
- Empty states are handled gracefully
- Progress indicators appear for long operations

### 8. Edge Cases
- Empty input handling
- Very long input handling
- Special characters in arguments
- Missing required arguments produce helpful errors

---

## Bug Reporting

### Before Filing: Dedup Check

Before creating any bug, search for existing issues:
```bash
bd search "<key phrase from the bug>"
```

If a similar issue exists, skip filing. Use your judgment — if the existing issue is about a different aspect of the same problem, file separately.

### Filing a Bug

```bash
bd create \
  --title="<concise title: what's wrong>" \
  --type=bug \
  --priority=<severity> \
  --parent=<epic-id> \
  --description="<full description>"
```

### Bug Description Template

Write the description as a single string with this structure:

```
**Rubric category**: <which quality rubric category>

**What happened**: <what you observed>

**Expected**: <what should have happened>

**Reproduction**:
1. Run: `uv run rdst.py <command>`
2. Observe: <what appears>

**Captured output**:
```
<paste the relevant output lines>
```

**Severity rationale**: <why this priority level>
```

### Severity Guide

| Priority | When to use |
|----------|-------------|
| 1 (P1) | Crash, data loss, command completely broken |
| 2 (P2) | Wrong behavior, misleading output, bad error messages |
| 3 (P3) | Cosmetic issues, minor inconsistencies, alignment problems |
| 4 (P4) | Nitpicks, style preferences, minor polish |

---

## Round Escalation

### Round 1: Normal Usage
Execute each test area as described above. Use straightforward, expected inputs. This is the "happy path plus basic errors" round.

### Round 2: Edge Cases
Re-test areas with adversarial inputs:
- **Unicode**: `uv run rdst.py query add "tëst_ünïcödë" -q "SELECT '🎉'"`
- **Long inputs**: Very long query strings (500+ chars)
- **Special characters**: Queries with `'`, `"`, `\`, `;`, `--` in them
- **Rapid Ctrl-C**: Send Ctrl-C immediately after starting commands
- **Empty strings**: Pass `""` as arguments
- **Path traversal**: `uv run rdst.py scan ../../../etc`
- **Concurrent access**: Run two commands that touch the same config file

### Round 3: Stress & Unusual States
- **Small terminal**: Start harness with `--width 40 --height 10` and test output formatting
- **Missing config**: Rename `~/.rdst/config.toml` temporarily and test error handling
- **Corrupt config**: Write invalid TOML to config and test behavior
- **Network errors**: Test commands with unreachable hosts configured

**IMPORTANT**: For Rounds 2-3, always clean up after yourself. Restore any files you modify. Use the `_test_relentless_` prefix for any resources you create.

---

## Guidelines

### Autonomy
- **Do NOT ask the user for permission** between test areas. Just keep going.
- **Do NOT stop** after finding a bug. File it and continue testing.
- Report progress after completing each area (one line: area name, pass/fail, bugs filed).
- Only stop when you run out of areas or hit context limits.

### Non-Destructive Testing
- **Never modify real user configuration.** If you need to test config changes, use `_test_relentless_` prefixed resources.
- **Never delete real queries or schemas.** Only delete resources you created.
- **Always clean up** test resources at the end of each area.
- **Never run `rdst init`** for real — it overwrites config. Test it by observing the prompts then Ctrl-C.

### Quality Over Speed
- **Read every line of output.** Don't skim.
- **Check alignment character by character** for tables.
- **Compare formatting across commands** — consistency matters.
- Take extra harness reads if the first read didn't capture everything.

### Session Hygiene
- Kill the tmux session between every area to avoid state bleed.
- If a command hangs, Ctrl-C it, read the output, file a bug about the hang, then kill and restart the session.
- If a harness action fails (session doesn't exist), start a new one.

### Progress Tracking
After each area, output a line like:
```
[area-name] DONE — 2 bugs filed (rdst-xxx, rdst-yyy) | cost: $0.00 (0 LLM calls)
```
or:
```
[area-name] DONE — clean | cost: $0.12 (5 LLM calls, 8200 in / 1400 out tokens)
```

For areas with no LLM calls, always write `cost: $0.00`.

At the end of each round, output a summary table:
```
Round 1 Summary:
  Areas tested: 15/15
  Bugs filed: 7
  Bug IDs: rdst-101, rdst-102, rdst-103, rdst-104, rdst-105, rdst-106, rdst-107

  Cost Breakdown:
  ┌─────────────────────────┬──────────┬─────────────┬──────────────┐
  │ Area                    │ LLM Calls│ Tokens (I/O)│ Cost         │
  ├─────────────────────────┼──────────┼─────────────┼──────────────┤
  │ version                 │ 0        │ —           │ $0.00        │
  │ help-system             │ 0        │ —           │ $0.00        │
  │ ...                     │ ...      │ ...         │ ...          │
  │ guard-intent-creation   │ 2        │ 3100 / 420  │ $0.02        │
  ├─────────────────────────┼──────────┼─────────────┼──────────────┤
  │ TOTAL                   │ 2        │ 3100 / 420  │ $0.02        │
  └─────────────────────────┴──────────┴─────────────┴──────────────┘

  Next: Round 2 (edge cases)
```