---
name: pipeline-architecture
description: >-
  Design, set up, and validate Buildkite pipelines. Use when writing or
  reviewing a .buildkite/pipeline.yml, building dynamic pipelines that upload
  steps at runtime, structuring step DAGs with depends_on/key, using matrix
  builds, group/wait/block/input/trigger steps, retries, soft_fail,
  concurrency, parallelism, conditionals, or agent queue targeting. Also covers
  validating pipelines with `buildkite-agent pipeline upload --dry-run`.
  Triggers on "Buildkite pipeline", "pipeline.yml", "dynamic pipeline", or
  "pipeline steps".
allowed-tools: Read, Write, Edit, Grep, Glob, Bash
---

# Buildkite pipeline architecture

You are a principal platform engineer designing scalable, secure Buildkite
pipelines. Favor **configuration-as-code** and **dynamic uploads** over giant
static files, and an explicit **DAG** over implicit ordering.

> Verify version-sensitive syntax via WebSearch on `buildkite.com`:
> - Defining steps: https://buildkite.com/docs/pipelines/configure/defining-steps
> - Dynamic pipelines: https://buildkite.com/docs/pipelines/configure/dynamic-pipelines
> - depends_on: https://buildkite.com/docs/pipelines/configure/depends-on
> - Matrix: https://buildkite.com/docs/pipelines/build-matrix
> - Command step: https://buildkite.com/docs/pipelines/configure/step-types/command-step
> - pipeline CLI: https://buildkite.com/docs/agent/cli/reference/pipeline

## Core principles

1. **Lean entrypoint, dynamic upload.** Keep `.buildkite/pipeline.yml` tiny; let
   it generate and upload the real steps. This keeps logic in code and lets you
   compute steps from the diff (monorepos), branch, or env.
   ```yaml
   steps:
     - label: ":pipeline: upload"
       command: ".buildkite/pipeline.sh | buildkite-agent pipeline upload"
   ```
   The script prints YAML/JSON to stdout; `pipeline upload` (no file arg) reads
   stdin. For static fragments, `buildkite-agent pipeline upload path/to.yml`.
2. **Explicit DAG with `key` + `depends_on`.** Give steps a stable `key`; depend
   on them by key. This parallelizes safely and documents intent. Avoid relying
   on top-to-bottom order or unnecessary `wait` barriers.
   - `depends_on: "build"` or a list. Use `allow_dependency_failure: true` to
     run even if a dependency failed; pair with `soft_fail` thoughtfully.
3. **Target agents explicitly.** Put `agents: { queue: "..." }` on steps so work
   lands on the right cluster/queue. Don't assume a default queue.
4. **Fail loud, retry smart.** Use `retry.automatic` for known-transient exit
   codes only; `retry.manual` for human-gated reruns. Use `soft_fail` for
   non-blocking checks, not to paper over real failures.
5. **Validate before you ship.** `buildkite-agent pipeline upload --dry-run`
   parses and renders the pipeline without uploading. See `reference/validation.md`.

## Step types (quick map)
- **command** — runs commands on an agent. The workhorse. Supports `matrix`,
  `parallelism`, `retry`, `soft_fail`, `concurrency`, `artifact_paths`, `plugins`.
- **wait** — barrier; waits for all prior steps. Prefer `depends_on` over `wait`
  for fine-grained graphs. `wait: { continue_on_failure: true }` to proceed on
  failure.
- **block** / **input** — pause for manual approval / collect fields.
- **trigger** — start a build in another pipeline (fan-out across pipelines).
- **group** — visually + logically group steps under one label.

See `reference/step-patterns.md` for copy-paste syntax of each, plus matrix,
concurrency, and conditionals.

## Designing a pipeline (workflow)

1. **Clarify topology.** Single pipeline or fan-out via `trigger`? Monorepo
   (compute steps from changed paths) or single app?
2. **Lay out the DAG.** List jobs, give each a `key`, draw dependencies. Group
   related jobs. Put long/independent jobs first so they parallelize.
3. **Pick queues.** Map each job to a `queue` (e.g. `default`, `docker`, `gpu`).
4. **Add safety.** Retries for transient steps, `concurrency_group` for
   shared/exclusive resources (deploys), `timeout_in_minutes` to cap runaways.
5. **Handle secrets** with the `pipeline-security` skill — never inline secrets.
6. **Validate** with `--dry-run`, then commit.

## Setting up from scratch (secure by default)
For a new pipeline, follow `reference/secure-setup-guide.md` — a 10-step,
secure-and-optimised setup walkthrough (lean dynamic upload → DAG → queues →
pinned plugins → OIDC secrets → fork-PR safety → caching/matrix → deploy
protection → agent hardening → validate+scan in CI), with a checklist. Start from
`templates/secure-starter-pipeline.yml`.

## Templates
- `templates/secure-starter-pipeline.yml` — secure-by-default starter (pinned,
  queue-isolated, OIDC deploy, approval + concurrency); passes `scan-pipeline.sh`.
- `templates/static-pipeline.yml` — annotated DAG with groups, matrix, retries,
  concurrency, conditionals, queues.
- `templates/dynamic-pipeline.sh` — generate steps at runtime and upload.
- `templates/trigger-fanout.yml` — fan out to downstream pipelines.

## Related
- `pipeline-security` — **audit & scan** the pipeline/agents (secrets, OIDC,
  plugin pinning, agent-hook controls, untrusted PRs). Run before shipping.
- `buildkite-mcp-debugging` — inspect real failing builds/logs live.