Glossary3 sections1 min read
What is Memory bandwidth?
Memory bandwidth is the rate, in gigabytes per second, at which a GPU can read from and write to its own memory. It caps how fast memory-bound workloads such as LLM token generation can run.
Why it matters when renting a GPU
When a model generates text one token at a time, each step reads most of its weights from memory. The GPU often waits on memory rather than on compute.
For inference, compare bandwidth alongside VRAM; for training at large batch sizes, Tensor Core throughput matters more.
Worked example
The H200 reaches 4,800 GB/s and the A100 SXM 80GB 2,039 GB/s, so the H200 can stream its memory about 2.4 times faster.
Frequently asked questions
- Is bandwidth or TFLOPS more important for inference?
- For generating tokens one at a time, bandwidth usually dominates. For processing long prompts or large batches, compute throughput matters more.