--- name: state-machine-workflow-patterns description: State machine discipline, saga compensation, event-sourced persistence, workflow orchestration. Use when state transitions, workflow steps, saga, compensation, event sourcing, or state machine testing. keywords: state machine, FSM, XState, state transition, saga, compensation, event sourcing, workflow orchestration, choreography, transition guard, append-only, replay, discriminated union, optimistic locking, heartbeat, timeout, model-based testing related: error-handling-patterns, async-flow-control, process-lifecycle-patterns, event-driven-messaging-patterns, domain-driven-design --- # State Machine & Workflow Patterns Rules for building state machines and multi-step workflows that survive crashes, prevent invalid transitions, and compensate on failure. **Research:** `docs/deepresearch/reports/State Machine and Workflow Patterns.md` ## Non-Negotiable Rules | # | Rule | Violation Example | Correct Pattern | |---|------|-------------------|-----------------| | 1 | **Explicit machine definition** | Ad-hoc `if (status === 'x')` scattered in controllers | Single machine definition (XState, map, or guard table) | | 2 | **Reject invalid transitions at state layer** | `pending` -> `completed` bypass (missing intermediate) | Machine only accepts defined transitions | | 3 | **Discriminated unions for state types** | `{ status?: string; startedAt?: Date; completedAt?: Date }` | `{ status: 'completed'; startedAt: Date; completedAt: Date }` | | 4 | **Persist transition history** | Single `status` column, no history | `state_transitions` table or event log | | 5 | **Saga compensation on multi-step** | `chargePayment(); createOrder()` with no rollback | Explicit compensation list, reverse-order execution | | 6 | **Timeout + heartbeat on every activity** | Workflow step with no timeout -> zombie | `startToCloseTimeout` + `heartbeatTimeout` on activities | | 7 | **Optimistic lock or single-writer** | Read-modify-write on state without version guard | `WHERE version = $v` or per-entity queue | | 8 | **Test every transition + every invalid transition** | Only test happy path | Generate tests from machine definition | | 9 | **Declarative workflows for >5 steps** | Imperative glue code across services | Temporal, Step Functions, or declarative definition | | 10 | **State metrics per status** | No visibility into stuck workflows | `gauge:workflows_in_state{state="x"}` for monitoring | ## State Transition Validation ```typescript // WRONG: Ad-hoc status checks scattered across controllers async function shipOrder(orderId: string) { const order = await db.orders.findUnique({ where: { id: orderId } }); if (order.status === 'completed') throw new Error('Already shipped'); // No check: pending -> completed bypasses 'processing' await db.orders.update({ data: { status: 'completed' } }); } // CORRECT: Single machine definition, transitions validated centrally const ORDER_TRANSITIONS: Record> = { pending: { PROCESS: 'processing' }, processing: { SHIP: 'completed', CANCEL: 'cancelled' }, completed: {}, // final state cancelled: {}, // final state }; function transition(current: string, event: string): string { const next = ORDER_TRANSITIONS[current]?.[event]; if (!next) throw new Error(`Invalid transition: ${current} + ${event}`); return next; } // Usage: validate BEFORE any DB write const nextStatus = transition(order.status, 'SHIP'); await db.orders.update({ data: { status: nextStatus, shippedAt: new Date() } }); ``` **Rule:** Never scatter `if (status === ...)` checks across multiple files. Define the machine once. All status changes go through it. ## Invalid State Prevention Through Types ```typescript // WRONG: Bag of optionals - compiles, crashes at runtime type Order = { status?: 'pending' | 'processing' | 'completed'; startedAt?: Date; completedAt?: Date; }; // order.completedAt accessed when status is 'pending' -> undefined // CORRECT: Discriminated unions make impossible states unrepresentable type PendingOrder = { status: 'pending' }; type ProcessingOrder = { status: 'processing'; startedAt: Date }; type CompletedOrder = { status: 'completed'; startedAt: Date; completedAt: Date }; type CancelledOrder = { status: 'cancelled'; cancelledAt: Date; reason: string }; type OrderState = PendingOrder | ProcessingOrder | CompletedOrder | CancelledOrder; // Compiler enforces: completedAt only accessible when status is 'completed' function getCompletionTime(order: OrderState): Date | null { if (order.status === 'completed') return order.completedAt; // type-safe return null; } ``` **Rule:** Every multi-state entity must use discriminated unions. The compiler must reject access to state-specific fields in wrong states. ## State Persistence & Recovery ```typescript // WRONG: Single status column, crash loses in-flight work await db.orders.update({ data: { status: 'processing' } }); // Pod crashes here -> no record of what happened // CORRECT: Append-only transition log + derived current state await db.execute(sql` INSERT INTO state_transitions (entity_id, entity_type, from_state, to_state, event, metadata, created_at) VALUES (${orderId}, 'order', ${currentState}, ${nextState}, ${event}, ${sql.json(meta)}, NOW()) `); await db.execute(sql` UPDATE orders SET status = ${nextState}, updated_at = NOW(), version = version + 1 WHERE id = ${orderId} AND version = ${currentVersion} `); // Recovery: replay transitions to rebuild current state const transitions = await db.execute(sql` SELECT * FROM state_transitions WHERE entity_id = ${orderId} AND entity_type = 'order' ORDER BY created_at ASC `); ``` **Schema:** ```sql CREATE TABLE state_transitions ( id UUID DEFAULT gen_random_uuid() PRIMARY KEY, entity_id UUID NOT NULL, entity_type TEXT NOT NULL, from_state TEXT NOT NULL, to_state TEXT NOT NULL, event TEXT NOT NULL, metadata JSONB DEFAULT '{}', created_at TIMESTAMPTZ DEFAULT NOW() ); CREATE INDEX idx_transitions_entity ON state_transitions (entity_id, entity_type, created_at); ``` **Rule:** For any entity that lives longer than a single request, persist every state transition. Current state = derived from transition history. This enables replay, audit, and crash recovery. ## Saga Pattern (Distributed Compensation) ```typescript // WRONG: Sequential calls with no compensation -> partial failure leaves inconsistent state await chargePayment(paymentId); await createOrder(orderId); // Fails -> money taken, no order await reserveInventory(itemId); // Never reached // CORRECT: Explicit compensations, reverse-order execution on failure class Saga { private compensations: Array<() => Promise> = []; addCompensation(fn: () => Promise): void { this.compensations.push(fn); } async compensate(): Promise { // Execute compensations in reverse order (LIFO) for (const comp of [...this.compensations].reverse()) { try { await comp(); } catch (err) { logger.error({ err }, 'Compensation failed - manual intervention needed'); // Log but continue: best-effort compensation } } } } // Usage const saga = new Saga(); const payment = await chargePayment(paymentId); saga.addCompensation(() => refundPayment(payment.transactionId)); try { const order = await createOrder(orderId); saga.addCompensation(() => cancelOrder(order.id)); await reserveInventory(itemId); saga.addCompensation(() => releaseInventory(itemId)); } catch (err) { await saga.compensate(); // Undo everything in reverse throw new Error('Order workflow failed', { cause: err }); } ``` **Rule:** Every multi-service operation must register a compensation for each completed step. On failure, compensations run in reverse order. Log compensation failures but continue (best-effort cleanup). ## Timeouts & Heartbeats ```typescript // WRONG: No timeout -> zombie workflow step runs forever await processLargeDataset(datasetId); // CORRECT: Every activity has timeout + heartbeat async function processWithTimeout( datasetId: string, signal?: AbortSignal, ): Promise { const timeout = AbortSignal.timeout(5 * 60 * 1000); // 5 min max const combined = AbortSignal.any([timeout, ...(signal ? [signal] : [])]); for await (const batch of getBatches(datasetId)) { combined.throwIfAborted(); // Heartbeat check between batches await processBatch(batch); } } // In workflow orchestrator: track last heartbeat async function monitorActivity(activityId: string, heartbeatInterval: number) { const activity = await getActivity(activityId); const elapsed = Date.now() - activity.lastHeartbeat.getTime(); if (elapsed > heartbeatInterval * 3) { logger.warn({ activityId, elapsed }, 'Activity missed heartbeat'); await timeoutActivity(activityId); } } ``` **Rule:** Every workflow activity must have `startToCloseTimeout` and `heartbeatTimeout`. Activities that process data in loops must check abort/heartbeat between iterations. ## Workflow Orchestration vs Choreography | Aspect | Orchestration | Choreography | |--------|--------------|--------------| | Control flow | Central coordinator | Events between services | | Visibility | Full workflow state in one place | Scattered across services | | Debugging | Easy: query orchestrator state | Hard: correlate events across services | | Coupling | Services depend on orchestrator | Services depend on event contracts | | Compensation | Natural: orchestrator runs compensations | Hard: who owns rollback? | | Best for | Workflows >5 steps, need audit/visibility | Loosely coupled, simple fanout | **Rule:** For workflows that need visibility, audit, or compensation, use orchestration (Temporal, Step Functions, or custom orchestrator). Use choreography only for simple event fanout where partial failure is acceptable. ## State Machine Testing ```typescript // WRONG: Only test happy path test('order completes', () => { expect(transition('processing', 'SHIP')).toBe('completed'); }); // CORRECT: Test every valid transition AND every invalid transition describe('order state machine', () => { const VALID_TRANSITIONS = [ ['pending', 'PROCESS', 'processing'], ['processing', 'SHIP', 'completed'], ['processing', 'CANCEL', 'cancelled'], ] as const; const INVALID_TRANSITIONS = [ ['pending', 'SHIP'], // Can't ship without processing ['pending', 'CANCEL'], // Must process before cancel (if that's the rule) ['completed', 'PROCESS'], // Final states accept nothing ['completed', 'SHIP'], ['cancelled', 'PROCESS'], ] as const; test.each(VALID_TRANSITIONS)('%s + %s -> %s', (from, event, to) => { expect(transition(from, event)).toBe(to); }); test.each(INVALID_TRANSITIONS)('%s + %s -> throws', (from, event) => { expect(() => transition(from, event)).toThrow('Invalid transition'); }); }); ``` **Rule:** When defining a state machine, immediately generate tests covering every valid transition and every invalid transition. Use `test.each` with the transition table. If you add a state, add its valid/invalid transition tests in the same commit. ## Concurrent State Mutation Guard ```typescript // WRONG: Read-modify-write with no guard -> lost updates const order = await db.orders.findUnique({ where: { id: orderId } }); order.status = 'completed'; await db.orders.update({ data: { status: order.status } }); // CORRECT: Optimistic locking with version column const result = await db.execute(sql` UPDATE orders SET status = ${nextStatus}, version = version + 1, updated_at = NOW() WHERE id = ${orderId} AND version = ${expectedVersion} RETURNING * `); if (result.length === 0) { throw new Error('Concurrent modification detected - retry'); } // CORRECT: Single-writer per entity (queue + dedicated processor) await redis.xadd(`order:${orderId}:events`, '*', 'event', JSON.stringify({ type: 'SHIP', timestamp: Date.now(), })); // Single consumer per stream key ensures serial processing ``` **Rule:** Never do read-modify-write on state without either optimistic locking (`WHERE version = $v`), pessimistic locking (`FOR UPDATE`), or single-writer guarantee (per-entity queue). See race-conditions-atomicity for DB-level patterns. ## Common Anti-Patterns Checklist Before submitting workflow/state-machine code, verify: - [ ] No raw string/enum status without a machine definition - [ ] No bag-of-optional-properties for multi-state entities (use discriminated unions) - [ ] No single `status` column without transition history for long-lived entities - [ ] No multi-service calls without compensation (saga pattern) - [ ] No workflow activity without timeout + heartbeat - [ ] No read-modify-write on state without concurrency guard - [ ] No state machine without tests for every valid AND invalid transition - [ ] No imperative glue code for workflows >5 steps (use declarative orchestration) - [ ] No workflow without state metrics/observability ## See Also [[error-handling-patterns]] - Error classification, discriminated union results, cause chaining race-conditions-atomicity - Optimistic locking, SKIP LOCKED, distributed locks, idempotency resilience-patterns - Circuit breakers, TaskReaper, auto-healing, multi-layer recovery [[async-flow-control]] - AbortController cancellation, Promise combinators, graceful shutdown [[event-driven-messaging-patterns]] - Outbox pattern, consumer idempotency, DLQ [[domain-driven-design]] - Aggregate boundaries, value objects, domain events