Distributed training

Distributed training is splitting the training of one model across many GPUs because it’s too slow or too big for a single one. The simplest form is data parallelism: every GPU holds a full copy of the model, each trains on a different slice of the batch, and they average their gradients after every step. When the model itself won’t fit in one GPU’s memory, you split the model instead. Tensor parallelism cuts each layer’s large matrix math across several GPUs, which needs very fast links between them. Pipeline parallelism gives each GPU a different group of layers and passes work along like an assembly line. Big models combine all three, and the hard part is keeping communication from becoming the bottleneck.