Glossary3 sections1 min read
What is VRAM (GPU memory)?
VRAM is the memory physically attached to a GPU. A model, its activations and its cache must fit in VRAM (or be split across several GPUs) for the GPU to run it efficiently.
Why it matters when renting a GPU
Memory is usually the first filter when choosing a GPU to rent: if the weights and working memory do not fit, speed does not matter.
Weight memory is easy to estimate: parameters multiplied by bytes per parameter. Add room for the KV cache, activations and the framework itself.
Worked example
A 7-billion-parameter model in FP16 needs about 7 × 2 = 14 GB for its weights alone, which fits on a 24 GB RTX 4090. A 70-billion-parameter model needs about 70 × 2 = 140 GB, more than the 80 GB of one H100 SXM but less than the 141 GB of one H200.
Frequently asked questions
- How much VRAM do I need for a 70B model?
- About 140 GB for the weights in FP16, 70 GB in 8-bit and 35 GB in 4-bit, plus room for the KV cache and runtime.
- Is VRAM the same as system RAM?
- No. VRAM is on the GPU; system RAM belongs to the host. Moving data between them over PCIe is much slower than reading VRAM.