Glossary3 sections1 min read
What is HBM (High Bandwidth Memory)?
HBM (High Bandwidth Memory) is DRAM stacked in layers and mounted beside the GPU die on the same package, connected through a very wide bus. It delivers much more memory bandwidth per GPU than the GDDR memory used on graphics cards.
Why it matters when renting a GPU
Large language model inference and training spend much of their time moving weights and activations between memory and compute units. More bandwidth usually means more tokens or samples per second on memory-bound work.
HBM generations differ (HBM2, HBM2e, HBM3, HBM3e). When you compare offers, read the memory type and bandwidth on the GPU page rather than the capacity alone.
Worked example
The H100 SXM pairs 80 GB of HBM3 with 3,350 GB/s of bandwidth, while the RTX 4090 has 24 GB of GDDR6X at 1,008 GB/s: about 3.3 times more bandwidth on the data center card.
Frequently asked questions
- Is HBM better than GDDR?
- For bandwidth per GPU, yes: HBM sits next to the die on a very wide bus. GDDR is cheaper and common on graphics and workstation cards, which can be enough for smaller models and rendering.
- Which GPUs use HBM?
- Data center GPUs such as the A100, H100, H200, B200 and AMD MI300X use HBM. GeForce and most RTX workstation cards use GDDR.