---
name: distributed-training-scaling
description: Scale training from one GPU to many by moving through data parallel and sharded FSDP modes, overlapping communication with compute, and reading the efficiency curve to find the ceiling. Use when adding GPUs stops making training proportionally faster and you need to locate where the scaling leaks.
---

# Distributed training scaling

Adding GPUs is supposed to make training faster in proportion, and for a while it
does, until an all-reduce or a memory wall eats the gain and the eighth GPU buys
almost nothing. Scaling well means picking the right parallel mode for what does
not fit, hiding communication behind computation, and reading the efficiency
curve so you stop before you are paying for idle silicon.

## Method

1. **Begin with DistributedDataParallel and know its bill.** DDP replicates the
   model and all-reduces gradients each step; a ring all-reduce moves about
   2 x (N-1)/N times the gradient bytes per GPU. Bucketing gradients
   (`bucket_cap_mb` near 25) lets that traffic overlap the backward pass instead
   of waiting until it finishes.
2. **Shard state with FSDP once the replica stops fitting.** When parameters,
   gradients, and optimizer state no longer fit per GPU, move to sharding. ZeRO-1
   shards optimizer state, ZeRO-2 adds gradients, and ZeRO-3 or PyTorch FSDP
   shards parameters too, cutting per-GPU memory toward one over the world size at
   the cost of gathering weights on the fly.
3. **Overlap the gathers with compute, do not serialize them.** Turn on FSDP
   `forward_prefetch` and `backward_prefetch=BACKWARD_PRE` so the next layer's
   all-gather issues while the current layer computes. Without prefetch the GPU
   stalls on communication and the sharding saving turns into a slowdown.
4. **Raise the compute-to-communication ratio.** Communication scales with model
   size while compute scales with tokens per step, so grow the local batch or use
   gradient accumulation to do more math between syncs. Watch the global batch
   size, since enlarging it changes convergence and usually needs learning-rate
   scaling and warmup.
5. **Fit a bigger local batch with BF16 and activation checkpointing.** Mixed
   precision in BF16 halves activation and gradient bytes with a stable exponent
   range, and activation checkpointing trades recompute for memory. Both free room
   to push the local batch up, which refills the compute you use to hide comms.
6. **Read the scaling efficiency curve, not just total throughput.** Plot tokens
   per second per GPU against GPU count. Ideal is flat; a slide toward 0.8 or 0.7
   of the single-GPU rate means you have gone communication-bound. Chase the
   largest scale that still holds roughly 90 percent efficiency.
7. **Tune NCCL and confirm the fabric.** Run a bandwidth test, check that NVLink
   and InfiniBand deliver expected GB/s, and set NCCL knobs (`NCCL_ALGO`, buffer
   and channel counts) when the default underperforms. An all-reduce silently
   routed over PCIe is a common reason the curve sags early.

## Signals

- Does tokens/sec/GPU stay near 90 percent of the single-GPU rate at target scale?
- Is gradient or parameter traffic overlapping compute, not serialized after it?
- Did you switch to FSDP only after the replicated model actually stopped fitting?
- Was the learning rate rescaled when the global batch grew, so convergence held?

## Boundaries

This scales a model that fits on one GPU out to many. Splitting a model too large
for a single card with tensor, pipeline, or expert parallelism is
model-parallelism; surviving node failures and resuming long runs is
checkpointing-large-training and fault-tolerant-training. The numerically safe use
of BF16 and FP8 belongs to mixed-precision-deployment.