Guide4 sections2 min read
The short answer
The model weights need about the number of parameters multiplied by the bytes per parameter: 2 bytes in FP16 or BF16, 1 byte in INT8 or FP8, and half a byte in INT4. A 70B model therefore needs about 140 GB for its weights in FP16, 70 GB in INT8 and 35 GB in INT4.
The formula
Weight memory (GB) = parameters (billions) × bytes per parameter. The values in the table use decimal gigabytes and count the weights only.
Add memory for the KV cache, which grows with context length, batch size and the number of layers, plus activations and framework overhead. Leave room for them: a model whose weights exactly fill a GPU will not run on it.
Reading the table
The last column names the smallest single GPU, among common rental options, whose memory exceeds the FP16 weights. It is a starting point, not a guarantee: check the KV cache for your context length, or use several GPUs.
Quantization to INT8 or INT4 cuts weight memory by two or four, with a possible quality trade-off that you should measure on your own prompts.
Training needs much more
Training and full fine-tuning also store gradients and optimizer states, which multiply the memory needed well beyond the weights. Parameter-efficient methods such as LoRA and QLoRA keep most weights frozen and need far less.
Weight memory by model size and precision, and the smallest single GPU that holds the weights
| Model size | Weights in FP16 / BF16 | Weights in INT8 / FP8 | Weights in INT4 | Smallest single GPU (FP16) |
|---|---|---|---|---|
| 7B parameters | 14 GB | 7 GB | 3.5 GB | NVIDIA GeForce RTX 4090 24GB |
| 13B parameters | 26 GB | 13 GB | 6.5 GB | NVIDIA GeForce RTX 5090 32GB |
| 34B parameters | 68 GB | 34 GB | 17 GB | NVIDIA A100 SXM 80GB |
| 70B parameters | 140 GB | 70 GB | 35 GB | NVIDIA H200 141GB |
| 405B parameters | 810 GB | 405 GB | 202.5 GB | Several GPUs |
Frequently asked questions
- How much VRAM do I need to run a 70B model?
- About 140 GB for the weights in FP16, 70 GB in INT8 and 35 GB in INT4, plus memory for the KV cache and overhead.
- Can a 7B model run on a 24 GB GPU?
- Its weights need about 14 GB in FP16, so it fits on a 24 GB GPU such as the RTX 4090 with room left for the KV cache at moderate context lengths.
- Does the KV cache matter?
- Yes. It grows with context length and batch size and can take a large share of memory for long contexts or many concurrent requests.