Glossary3 sections1 min read
What is Model and tensor parallelism?
Model parallelism splits one model across several GPUs. Tensor parallelism splits the matrices of each layer across GPUs; pipeline parallelism assigns different layers to different GPUs.
Why it matters when renting a GPU
When weights exceed one GPU's memory, you must split the model, and the speed of the links between GPUs becomes critical.
Tensor parallelism communicates at every layer, so it works best within one NVLink-connected server.
Worked example
A 70B model in FP16 needs about 140 GB of weights. Split with tensor parallelism across 2 H100 SXM GPUs, each holds about 70 GB of its 80 GB; across 4, about 35 GB, leaving more room for the KV cache.
Frequently asked questions
- Is one big GPU better than two smaller ones?
- Often yes for inference: one GPU avoids communication overhead. Splitting is needed only when the model does not fit.