How is the VRAM of an LLM calculated?

Weights take billions of parameters × bytes per parameter (2 in BF16, 1 in FP8 or INT8, 0.5 in INT4). The KV cache adds 2 × layers × KV dimension × context tokens × batch × 2 bytes. A margin covers activations and runtime buffers.

How much VRAM does a 70B model need?

Its weights take 140 GB in BF16, 70 GB in FP8 or INT8 and 35 GB in INT4, before the KV cache and runtime overhead. In BF16 it needs several GPUs or a single GPU with more than 140 GB of memory.

Is the estimate exact?

No. Real usage depends on the model architecture, the inference engine, quantization metadata and memory fragmentation. Treat the result as a planning estimate and test on the target GPU before committing to a long rental.

LLM VRAM calculator: how much GPU memory you need

Estimate the GPU memory an LLM needs from its size, precision, context and batch, and see which GPUs hold it alone or in 2, 4 or 8 cards.

This calculator estimates how much GPU memory a large language model needs for inference, from its size, weight precision, context length and batch size, and lists the GPUs that can hold it.

Estimate the VRAM of a model

Model sizeWeight precisionContext length (tokens)Batch size (concurrent sequences)Parameters (billions)LayersKV dimension per layerRuntime overhead (%)

Presets use a typical architecture for their size (layers and KV dimension). Choose Custom to enter the values of a specific model from its configuration file.

Weights: 140 GB
KV cache: 2.7 GB
Overhead: 14.3 GB
Estimated total: 157 GB

GPUs that can hold it

Several GPUs assume the model is split across them (tensor or pipeline parallelism); interconnect and framework overhead are not counted.

How the estimate is calculated

Weights (GB) = billions of parameters × bytes per parameter: 4 in FP32, 2 in BF16 or FP16, 1 in FP8 or INT8, 0.5 in INT4.
KV cache (GB) = 2 × layers × KV dimension × context tokens × batch × 2 bytes ÷ 10⁹, with keys and values stored in BF16 or FP16. The KV dimension is the number of key/value heads × the size of one head.
Total (GB) = (weights + KV cache) × (1 + overhead %), the overhead covering activations, CUDA context and runtime buffers.
A model fits on n GPUs when n × the memory of one GPU ≥ the total, assuming it is split across them.

Frequently asked questions

How is the VRAM of an LLM calculated?: Weights take billions of parameters × bytes per parameter (2 in BF16, 1 in FP8 or INT8, 0.5 in INT4). The KV cache adds 2 × layers × KV dimension × context tokens × batch × 2 bytes. A margin covers activations and runtime buffers.
How much VRAM does a 70B model need?: Its weights take 140 GB in BF16, 70 GB in FP8 or INT8 and 35 GB in INT4, before the KV cache and runtime overhead. In BF16 it needs several GPUs or a single GPU with more than 140 GB of memory.
Is the estimate exact?: No. Real usage depends on the model architecture, the inference engine, quantization metadata and memory fragmentation. Treat the result as a planning estimate and test on the target GPU before committing to a long rental.

Be ready on launch day

Create your account now: you can top up your balance and launch a GPU as soon as the market opens.

Get priority access Explore the live market