Glossary3 sections1 min read
What is KV cache?
The KV cache stores the attention keys and values computed for previous tokens so a language model does not recompute them at each new token. Its size grows with context length and batch size.
Why it matters when renting a GPU
Long contexts and many concurrent users can make the KV cache as large as the model weights. Leave VRAM for it when sizing a GPU.
Grouped-query attention, KV cache quantization and paged attention reduce this memory.
Worked example
For an illustrative model with 80 layers, 8 KV heads of dimension 128 and 2-byte values, each token needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes. An 8,192-token context therefore needs about 2.68 GB per sequence.
Frequently asked questions
- Does the KV cache depend on batch size?
- Yes. Each concurrent sequence has its own cache, so memory grows with batch size times context length.