---
name: distributed-audit
description: >
  Correctness audit for distributed systems, concurrency bugs, replication correctness,
  failover safety, split-brain, WAL ordering, cross-shard atomicity, consistency violations,
  race conditions, deadlocks, lost wakeups, linearizability, epoch monotonicity, fencing,
  VLL lock ordering, channel backpressure, snapshot consistency. Trigger on: "audit",
  "is this safe", "could this lose data", "does this break linearizability",
  "correctness check", "race condition", "split-brain", "replication lag",
  "review for correctness", "distributed systems review", "concurrency review",
  "is this correct under failure", "what happens if this crashes".
---

# FrogDB Distributed Systems Auditor

Structured correctness audits for changes touching distributed or concurrent subsystems.
Covers multi-node concerns (replication, consensus, failover, partitions) AND single-node
concerns (races, lock ordering, atomicity, channel safety).

## When to Use

- After implementing changes to cluster, replication, persistence, or concurrency code
- During PR review of changes that touch distributed subsystems
- When designing new features that affect consistency or durability
- When investigating production anomalies (data loss, inconsistency, deadlock)

## Audit Tiers

Calibrate depth to the change scope:

| Tier | When | Output |
|------|------|--------|
| **Quick Assessment** | Single-file change, bug fix, new command handler | 3-5 sentence risk assessment + test recommendation |
| **Focused Audit** | Cross-crate change, new feature touching cluster/replication/persistence | Risk matrix + invariant checks + testing plan |
| **Full Report** | Architectural change, new subsystem, protocol change, consensus changes | Full structured report with fault analysis, invariant proofs, test matrix |

**Tier Selection:**
- If the change is within a single crate and doesn't touch data flow → Quick Assessment
- If the change crosses crate boundaries OR touches persistence/replication/cluster → Focused Audit
- If the change modifies protocols, adds subsystems, or changes consensus behavior → Full Report

---

## Audit Workflow

### Step 1: Identify Change Scope

Read the diff (or design doc). Map changed files to crates using the crate-map in
`~/.claude/skills/db-architect/references/crate-map.md`. Identify:

- Which crates are affected
- Whether the change crosses crate boundaries
- Whether the change touches hot paths (shard worker loop, connection handler, WAL write)

### Step 2: Classify Risk Domains

For each changed file, classify which risk domains are affected:

| Risk Domain | Trigger Files/Paths |
|-------------|---------------------|
| Shard-local concurrency | `crates/core/src/shard/`, `crates/core/src/store/` |
| Cross-shard atomicity | `crates/vll/src/`, scatter-gather in `crates/server/src/connection/` |
| Persistence / WAL | `crates/persistence/src/`, `crates/core/src/persistence/` |
| Replication | `crates/replication/src/`, `crates/server/src/commands/replication.rs` |
| Cluster topology | `crates/cluster/src/`, `crates/server/src/commands/cluster/` |
| Failover | `crates/cluster/src/state.rs`, promotion/demotion paths |
| Client-observable consistency | `crates/server/src/connection/`, routing, dispatch |
| Pub/sub routing | `crates/server/src/connection/handlers/pubsub.rs`, `crates/core/src/pubsub/` |

### Step 3: Check Invariants

Read `references/invariants.md` for the full invariant catalog. For each affected risk domain,
check whether the change PRESERVES, puts AT RISK, or VIOLATES the relevant invariants.

**For each invariant, state one of:**
- **PRESERVED** — Change does not affect this invariant (explain briefly)
- **AT RISK** — Change could violate under certain conditions (explain conditions + mitigations)
- **VIOLATED** — Change breaks this invariant (explain how and what to fix)

### Step 4: Analyze Fault Scenarios

Read `references/fault-taxonomy.md`. For each affected risk domain, identify which fault
scenarios apply. Focus on:

1. What happens if a crash occurs mid-operation?
2. What happens under network partition?
3. What happens under resource exhaustion?
4. Are there new race conditions or ordering violations?
5. Can the change cause a previously-impossible state?

### Step 5: Recommend Testing

Read `references/testing-patterns.md`. Based on risk domains and fault scenarios, recommend:

1. Which existing test suites to run
2. Whether new tests are needed (and which type)
3. Specific Jepsen tests if the change affects cluster/replication/failover

### Step 6: Produce Output

Use the appropriate output template (see Output Templates section below).

---

## Change-to-Risk Mapping

Core lookup table — "if you changed X, check Y":

### `crates/core/src/shard/`
- **Risk:** Shard-local concurrency, message ordering
- **Check:** Per-shard linearizability (Invariant 1), shared-nothing (Invariant 7)
- **Watch for:** Mutable references across `.await` points, blocking calls in async context,
  new shared state between shard workers
- **Faults:** Lost wakeups, message reordering, shard worker panic

### `crates/core/src/store/`
- **Risk:** Data structure correctness, expiry handling
- **Check:** Per-shard linearizability (Invariant 1), total order per key
- **Watch for:** TOCTOU between key existence check and modification, missing WAL writes

### `crates/vll/src/`
- **Risk:** Cross-shard atomicity, deadlock freedom
- **Check:** VLL sorted lock order (Invariant 2), deadlock freedom
- **Watch for:** Lock acquisition NOT in sorted shard order, missing lock releases on error
  paths, new lock modes that break ordering guarantees
- **Faults:** Deadlock (if ordering violated), livelock, starvation, timeout handling

### `crates/persistence/src/`
- **Risk:** Durability, WAL sequence ordering, snapshot consistency
- **Check:** WAL sequence ordering (Invariant 3), recovery consistency (Invariant 6)
- **Watch for:** WAL writes without sequence numbers, non-atomic WriteBatch operations,
  snapshot COW semantics violations, corruption recovery edge cases
- **Faults:** Mid-write crash, mid-snapshot crash, disk full, WAL corruption, sequence gaps

### `crates/persistence/src/wal.rs`
- **Risk:** All persistence risks, plus replication ordering
- **Check:** WAL sequence ordering (Invariant 3), replication consistency (Invariant 4)
- **Critical:** WAL is the source of truth for replication. Any ordering violation here
  propagates to all replicas.

### `crates/replication/src/`
- **Risk:** Replication consistency, PSYNC protocol correctness
- **Check:** Replication consistency (Invariant 4), failover correctness (Invariant 5)
- **Watch for:** Replication ID handling errors, sequence number gaps, FULLRESYNC checkpoint
  integrity, frame checksum validation, partial sync edge cases
- **Faults:** Replica streaming stall, corruption detection, OOM during checkpoint,
  asymmetric network failure, CPU starvation of replication thread

### `crates/replication/src/primary.rs`
- **Additional risk:** WAL streaming order, backpressure propagation
- **Check:** WAL entries sent in sequence order, TCP backpressure not causing data loss

### `crates/replication/src/replica.rs`
- **Additional risk:** Stale reads, promotion correctness
- **Check:** Replica applies WAL in order, secondary replication ID preserved on promotion

### `crates/cluster/src/`
- **Risk:** Raft state machine correctness, epoch monotonicity, topology consistency
- **Check:** Epoch monotonicity (Invariant 5), failover correctness (Invariant 5)
- **Watch for:** Epoch comparisons using `<` instead of `<=`, topology updates not gated
  on promotion completion, slot assignment gaps or overlaps
- **Faults:** Split-brain window, stale epoch, Raft divergence, orchestrator isolation

### `crates/cluster/src/state.rs`
- **Critical:** Raft state transitions, leader election
- **Check:** State machine transitions are valid, no state reached without proper predecessor

### `crates/server/src/connection/`
- **Risk:** Routing correctness, pipeline ordering, client-observable consistency
- **Check:** Connection ordering (Invariant 10), pub/sub routing (Invariant 8)
- **Watch for:** MOVED/ASK redirect loops, scatter-gather partial results leaking,
  pipeline response ordering violations, transaction state leaks between connections
- **Faults:** Connection exhaustion, scatter-gather timeout, MOVED/ASK loops

### `crates/server/src/connection/handlers/pubsub.rs`
- **Risk:** Pub/sub message ordering, broadcast routing
- **Check:** Pub/sub routing via shard 0 (Invariant 8)
- **Watch for:** Broadcast pub/sub fanning out to ALL shards (causes N× message delivery),
  sharded pub/sub using wrong routing

### `crates/server/src/connection/dispatch.rs`
- **Risk:** Command routing, execution strategy selection
- **Check:** Correct ExecutionStrategy for each command, CROSSSLOT validation
- **Watch for:** Read commands routed as writes, missing CROSSSLOT checks for new
  multi-key commands

---

## FrogDB Invariants (Summary)

Read `references/invariants.md` for full details. Brief listing:

1. **Per-shard linearizability** — Within a single shard, operations are linearizable
2. **Cross-shard atomicity (VLL)** — Multi-shard ops use sorted lock order, preventing deadlocks
3. **WAL sequence ordering** — WAL entries have monotonically increasing sequence numbers
4. **Replication consistency** — WAL streaming preserves operation order to replicas
5. **Failover correctness** — Epoch monotonicity + fencing prevent stale-primary writes
6. **Recovery consistency** — WAL replay + snapshot produces correct state
7. **Shared-nothing** — No shared mutable state between shard workers
8. **Pub/sub routing** — Broadcast pub/sub coordinated via shard 0 only
9. **Channel backpressure** — Bounded channels provide natural flow control
10. **Connection ordering** — Pipeline responses match request order per connection

---

## Fault Taxonomy (Summary)

Read `references/fault-taxonomy.md` for the full catalog. Categories:

- **Timing faults** — Race conditions, TOCTOU, message reordering, lost wakeups
- **Crash faults** — Mid-write, mid-snapshot, shard panic, OOM kill, crash during replication
- **Network faults** — Symmetric/asymmetric partition, split-brain window, slow network
- **Resource exhaustion** — Memory OOM, disk full, connection exhaustion, channel backpressure
- **Protocol faults** — Stale epoch, replication ID mismatch, WAL sequence gap, MOVED/ASK loops
- **Concurrency faults** — Deadlock (prevented by VLL), livelock, mutable ref across `.await`

---

## Testing Strategy

Decision tree mapping risk domains to testing tools. Read `references/testing-patterns.md`
for full templates and guidance.

| Risk Domain | Primary Tool | Secondary Tool | When to Add Tests |
|-------------|-------------|----------------|-------------------|
| Shard-local concurrency | Shuttle | Unit tests | Any change to shard worker loop or store |
| Cross-shard atomicity | Shuttle + Turmoil | Integration tests | Any change to VLL or scatter-gather |
| Network partition | Turmoil | Jepsen | Changes to replication or cluster |
| WAL / recovery | Integration tests | Jepsen crash suite | Changes to persistence or WAL |
| Replication | Jepsen replication suite | Integration tests | Changes to replication crate |
| Failover | Jepsen raft suite | Turmoil | Changes to cluster state or promotion |
| Linearizability | Jepsen register + WGL | Shuttle | Any change affecting operation ordering |
| Pub/sub | Integration tests | — | Changes to pub/sub routing |

**Existing test suites to check:**
- `just test` — All unit and integration tests
- `just concurrency` — Shuttle + Turmoil concurrency tests
- `just jepsen-suite single` — Single-node baseline
- `just jepsen-suite crash` — Crash recovery
- `just jepsen-suite replication` — Replication correctness
- `just jepsen-suite raft` — Raft cluster correctness

---

## Output Templates

### Quick Assessment

```
## Quick Assessment: [Change Description]

**Risk level:** Low / Medium / High
**Risk domains:** [list]

[3-5 sentences explaining what could go wrong, or confirming safety]

**Invariants:** All preserved / [list any at-risk]
**Recommended tests:** [specific test commands]
```

### Focused Audit

```
## Focused Audit: [Change Description]

### Change Scope
| Crate | Files | Change Type |
|-------|-------|-------------|

### Risk Matrix
| Risk Domain | Level | Concern |
|-------------|-------|---------|

### Invariant Check
| # | Invariant | Status | Notes |
|---|-----------|--------|-------|
| 1 | Per-shard linearizability | PRESERVED / AT RISK / VIOLATED | |
| 2 | Cross-shard atomicity | ... | |
| ... | ... | ... | |

### Fault Scenarios
[For each applicable fault scenario from the taxonomy:]
- **[Fault name]:** [What happens, whether the change handles it correctly]

### Testing Plan
| Test Type | Command | Purpose |
|-----------|---------|---------|
| Existing | `just test <crate>` | Verify basic correctness |
| Existing | `just concurrency` | Verify concurrency safety |
| New | [description] | [what it verifies] |

### Recommendations
[Numbered list of specific actions]
```

### Full Report

```
## Full Distributed Systems Audit: [Change Description]

### Executive Summary
[2-3 sentences: what changed, overall risk assessment, key finding]

### Change Scope
| Crate | Files | Change Type | Risk Domain |
|-------|-------|-------------|-------------|

### Architectural Impact
[How this change affects the overall system architecture and data flow]

### Invariant Analysis
| # | Invariant | Status | Evidence |
|---|-----------|--------|----------|
[All 10 invariants checked with evidence for each status]

### Fault Analysis
[For each fault category from the taxonomy:]

#### Timing Faults
[Applicable scenarios, analysis, mitigations]

#### Crash Faults
[Applicable scenarios, analysis, mitigations]

#### Network Faults
[Applicable scenarios, analysis, mitigations]

#### Resource Exhaustion
[Applicable scenarios, analysis, mitigations]

#### Protocol Faults
[Applicable scenarios, analysis, mitigations]

#### Concurrency Faults
[Applicable scenarios, analysis, mitigations]

### Testing Matrix
| Risk Domain | Existing Coverage | Gaps | Recommended New Tests |
|-------------|-------------------|------|----------------------|

### Jepsen Coverage
| Test | Relevance | Run? | Expected Result |
|------|-----------|------|-----------------|

### Recommendations
[Ordered by priority]
1. **Critical:** [Must fix before merge]
2. **Important:** [Should fix before merge]
3. **Advisory:** [Consider for follow-up]

### Open Questions
[Unresolved concerns requiring team discussion]
```

---

## Design Review Mode

When auditing a design (not code), adjust the workflow:

1. **Skip** Step 1 (no diff to read). Instead, read the design doc or plan.
2. **Focus** Step 3 on invariant preservation *arguments* — does the design explain
   how each affected invariant is maintained?
3. **Focus** Step 4 on proposed data flows — trace message sequences under failure.
4. **Check** fault scenario coverage — does the design address what happens when
   each relevant component fails?
5. **Check** that the design includes test strategy covering concurrency and failure modes.

**Key questions for design review:**
- Under which failure modes does this design lose data?
- Under which failure modes does this design violate linearizability?
- Is there a state that can be reached but never left (deadlock, stuck state)?
- Are all timeouts bounded? What happens when they fire?
- Does this design introduce new shared mutable state?
- Does this change the consistency model visible to clients?

---

## Reference File Pointers

| When | Read |
|------|------|
| Checking specific invariants | `references/invariants.md` |
| Analyzing fault scenarios | `references/fault-taxonomy.md` |
| Choosing tests or writing new ones | `references/testing-patterns.md` |
| Mapping files to crates | `~/.claude/skills/db-architect/references/crate-map.md` |
| Understanding request flows | `website/src/content/docs/architecture/request-flows.md` |
| Jepsen test catalog | `~/.claude/skills/jepsen-testing/references/test-catalog.md` |
| Architectural design | Use `/db-architect` skill |
| Running Jepsen tests | Use `/jepsen-testing` skill |
| Running builds/tests | Use `/check` skill |

---

## When to Defer

| Situation | Skill |
|-----------|-------|
| Designing a new feature (not auditing) | `/db-architect` |
| Running Jepsen tests | `/jepsen-testing` |
| Running build/lint/test | `/check` |
| Performance testing | `/benchmark` |
| Refactoring code structure | `/refactorer` |