---
name: message-queue-jobs
description: Builds async job and message-queue workflows (producers/consumers, idempotency, retries with backoff, dead-letter queues, exactly-once semantics) when offloading work or decoupling services.
when_to_use: User wants background jobs, a queue/worker, event-driven processing, or to fix duplicate processing, lost messages, retry storms, or poison messages. Covers Redis/SQS/Kafka/RabbitMQ/Celery/BullMQ-style systems.
---

---
name: message-queue-jobs
description: Builds async job and message-queue workflows (producers/consumers, idempotency, retries with backoff, dead-letter queues, exactly-once semantics) when offloading work or decoupling services.
when_to_use: User wants background jobs, a queue/worker, event-driven processing, or to fix duplicate processing, lost messages, retry storms, or poison messages. Covers Redis/SQS/Kafka/RabbitMQ/Celery/BullMQ-style systems.
---

# Message Queue & Async Jobs

## When to Use

Reach for this skill when the task is one of:

- **Offload slow work** — email, image/video processing, report generation, webhooks out → move it off the request path so the API responds fast.
- **Decouple services** — service A emits an event, service B reacts, without a synchronous call chain.
- **Fan-out / scheduled / batch** — one event triggers N workers, or work runs on a timer.
- **Fix a broken queue** — symptoms map to root causes:

| Symptom | Likely root cause | Section |
|---|---|---|
| Same side effect happens twice (double charge, double email) | At-least-once delivery + non-idempotent handler | Steps 1, 3 |
| Messages vanish, work never runs | Ack-before-process, or no DLQ for failures | Steps 2, 4 |
| Worker hammers a downstream into outage | Fixed-interval retries, no jitter, no backpressure | Steps 4, 5 |
| One bad message blocks the whole partition/queue | No poison-message handling / no DLQ | Step 4 |
| Queue depth climbs forever | Consumers slower than producers, no throttle/autoscale | Steps 5, 6 |

If the work is fast, in-process, and never needs a retry, **do not add a queue** — it's complexity you'll pay for forever. Say so.

## Steps

### 1. Pick broker + delivery semantics first (this decides everything else)

- Match broker to existing infra — don't add a new system if one is already running:
  - **Redis-backed lib** (BullMQ / Celery+Redis / Sidekiq): simplest, low-latency jobs, single region. Good default for app-level background jobs.
  - **Managed queue** (SQS-style): zero ops, built-in DLQ + visibility timeout. Default when on that cloud.
  - **Log/stream** (Kafka-style): ordered, replayable, high-throughput, multi-consumer fan-out. Use for event sourcing / analytics, **not** for simple job offload.
  - **AMQP broker** (RabbitMQ-style): rich routing (topic/fanout exchanges), per-message TTL.
- **Assume at-least-once. Build for it.** True exactly-once delivery does not exist across a network. What you actually deliver is **at-least-once delivery + idempotent processing = effectively-once.** Kafka's "exactly-once" only holds for read→process→write *inside one Kafka transaction*; the moment a handler touches an external API or DB outside that transaction, you're back to at-least-once. Design idempotency (Step 3) regardless.
- Decide ordering need now: most jobs don't need it. If they do, you need a partition/group key and it caps your parallelism (one in-flight per key).

### 2. Design producer, consumer, and a versioned message schema

- **Message = a typed envelope, not a blob.** Required fields:
  ```
  { id, type, version, occurred_at, idempotency_key, payload, attempts }
  ```
- Put a **stable `idempotency_key`** in the message at produce time (e.g. derived from the business event: `order:1234:charge`). Do **not** use the broker's auto-generated message id — it changes on redelivery.
- **Keep payloads small.** Send IDs/references, not large rows or files. Store blobs in object storage and pass the key. Big payloads blow past broker size limits (SQS 256KB, etc.) and slow every consumer.
- **Schema versioning:** consumers must tolerate unknown fields and handle old `version` values during rolling deploys. Never reuse a field name with a new meaning — add a new field and bump `version`.
- **Producer flush:** confirm the broker accepted the message (await the publish ack) before you report success upstream. Fire-and-forget producers silently drop messages on broker hiccups.

### 3. Make handlers idempotent — this is the load-bearing step

Retries are guaranteed, so a handler that runs twice must produce the same result as running once.

- **Dedup table / dedup key (preferred for side-effects):** before doing work, `INSERT idempotency_key` into a table with a UNIQUE constraint. If the insert conflicts, the message was already processed → ack and skip. Do the insert and the side-effect in the **same transaction** where possible.
- **Natural idempotency:** prefer operations that are safe to repeat — `UPSERT` / `SET x=v` over `INSERT` / `x = x + 1`. Use conditional writes (`UPDATE ... WHERE status='pending'`).
- **External calls:** pass the `idempotency_key` to downstream APIs that support it (payment providers do). Wrap non-idempotent calls in your own dedup check.
- **Set a TTL on dedup keys** longer than your max retry window (e.g. retries span 1h → keep keys ≥ 24h), then expire them so the table doesn't grow unbounded.

### 4. Retries with backoff + a dead-letter queue

- **Exponential backoff WITH jitter.** Never fixed-interval, never un-jittered — synchronized retries cause a thundering herd that takes the downstream out a second time.
  ```
  delay = min(cap, base * 2 ** attempt)
  delay = random_between(0, delay)   # full jitter
  ```
- **Bounded max attempts** (typically 3–6). On exceeding, route the message to a **dead-letter queue (DLQ)** — never drop it, never retry forever.
- **Classify failures before retrying:**
  - *Transient* (timeout, 5xx, throttle) → retry with backoff.
  - *Permanent* (validation error, 4xx, malformed payload = **poison message**) → DLQ immediately, attempts wasted otherwise.
- **Poison messages:** one un-handleable message must not block the queue/partition behind it. With a DLQ this is automatic; with ordered Kafka partitions you must skip-and-DLQ explicitly or the partition stalls forever.
- **Have a DLQ drain plan:** an alarm on DLQ count + a documented redrive (fix bug → replay DLQ back to main queue). A DLQ nobody watches is a silent data-loss bucket.

### 5. Backpressure so consumers never overwhelm anything

- **Bounded concurrency** per worker (`prefetch` / `concurrency` / `maxInFlight`). Default low (e.g. 5–10) and raise with evidence. Unbounded concurrency = OOM and downstream meltdown.
- **Visibility timeout / ack deadline > p99 processing time.** Too short → the message redelivers *while you're still processing it* → duplicate work. Too long → slow recovery after a crash. Set it, and for long jobs heartbeat/extend it.
- **Throttle on queue depth:** when depth or consumer lag exceeds a threshold, slow producers or scale consumers — don't just pile on.
- **Rate-limit the downstream**, not just the queue (token bucket on the external API), so a sudden backlog drain doesn't exceed third-party quotas.

### 6. Observability — you can't operate a queue you can't see

Emit and alarm on, at minimum:

- **Queue depth** (backlog size) and **trend** — rising = consumers losing.
- **Oldest-message age / consumer lag** — the real "are we behind?" signal (depth alone lies when message size varies).
- **DLQ count** — should be ~0; any nonzero needs eyes.
- **Processing latency** (p50/p99) and **throughput** (msgs/sec).
- **Retry rate** — a spike means a downstream is degrading; alarm before it becomes a DLQ flood.
- **Trace context:** propagate a `trace_id` from producer through consumer so one logical operation is followable across the async boundary.

### 7. Test failure injection before you ship

Happy-path passing proves nothing here. Add tests that prove correctness under failure:

1. **Duplicate delivery** — send the same message twice → assert the side-effect happens exactly once (proves Step 3).
2. **Out-of-order / delayed** — deliver messages out of order → assert correctness if you claimed ordering, or assert order-independence if you didn't.
3. **Poison message** — feed a malformed payload → assert it lands in the DLQ and the next good message still processes.
4. **Crash mid-process** — kill the worker after the side-effect but **before ack** → restart → assert no duplicate (visibility timeout redelivers; idempotency must absorb it).
5. **Retry exhaustion** — force a permanent failure → assert it stops at max attempts and DLQs, no infinite loop.

## Common Errors

- **Ack-before-process.** Acking on receive (or `auto-ack`/`enable.auto.commit=true` with default timing) loses the message if the worker dies mid-job. **Always ack/commit only after the work + its side-effects are durably committed.**
- **Using the broker's message id as the idempotency key.** It changes on redelivery, so dedup never matches and you process duplicates anyway. Use a business-derived key set at produce time (Step 2).
- **Believing "exactly-once delivery" exists.** Vendors sell at-least-once or at-most-once. "Exactly-once" is a *processing* property you build with idempotency, not a delivery guarantee you buy.
- **Fixed-interval or un-jittered retries.** Turns one downstream blip into a synchronized retry storm. Always exponential + full jitter (Step 4).
- **Visibility timeout shorter than processing time.** The message reappears and a second worker starts the same job → guaranteed duplicates that look like a "random" bug. Measure p99, set timeout above it, extend for long jobs.
- **No DLQ.** Failures either retry forever (resource burn, head-of-line block) or get dropped (silent data loss). There is no good third option without a DLQ.
- **Unbounded concurrency / prefetch.** A backlog drain spins up unlimited in-flight work → OOM or a downstream outage. Always cap (Step 5).
- **Giant payloads in the message.** Hits broker size limits and bloats every consumer's memory. Send a reference; store the blob elsewhere.
- **Producer doesn't await the publish ack.** Fire-and-forget drops messages on transient broker errors and you never know. Confirm acceptance before reporting success.
- **Non-idempotent handler "fixed" by hoping retries won't happen.** They will. Idempotency is mandatory, not optional, under at-least-once.

## Verify

Before declaring done, confirm each — with evidence, not assertion:

- [ ] Idempotency test passes: same message delivered 2x → side-effect occurs once (show the test + run output).
- [ ] DLQ exists and is wired: poison message → lands in DLQ, queue keeps draining (show it).
- [ ] Retries use exponential backoff **with jitter** and a finite max-attempts (show the config/code).
- [ ] Ack/commit happens **after** the side-effect commits, not before (point to the line).
- [ ] Visibility timeout / ack deadline > measured p99 processing time (state both numbers).
- [ ] Concurrency/prefetch is bounded (show the value).
- [ ] Metrics emitted for queue depth, oldest-message age, DLQ count, retry rate; alarm on DLQ > 0.
- [ ] Crash-mid-process test: kill before ack → restart → no duplicate (show run).
- [ ] Producer awaits publish ack; payload carries a stable `idempotency_key` and `version`.

If any box can't be checked with a real run/config, it is **not** done — fix the root cause, don't loosen the test.

## Related

- `update-config` — to install a queue-health check or DLQ alarm as a recurring hook/job.