Glossary3 sections1 min read
What is Batch size?
Batch size is the number of samples (in training) or concurrent requests (in inference) processed together in one pass on the GPU.
Why it matters when renting a GPU
Larger batches use the GPU more efficiently, until memory runs out. In inference, each extra request adds its own KV cache.
Tune batch size to your latency target: bigger batches raise throughput but can slow each individual response.
Worked example
With the KV cache example of the glossary (about 2.68 GB per 8,192-token sequence), a batch of 16 such sequences needs about 42.9 GB of cache on top of the weights.
Frequently asked questions
- What batch size should I use?
- The largest that fits in memory while meeting your latency target; measure with representative prompts.