Glossary3 sections1 min read
What is Tokens per second?
Tokens per second is the rate at which a language model generates output tokens. It is measured per user stream (latency) or summed over all concurrent streams (throughput).
Why it matters when renting a GPU
Single-stream decoding is usually memory-bound: each new token reads most of the weights once. Bandwidth sets a ceiling.
Batching many requests raises total throughput because one weight read serves several streams.
Worked example
A rough upper bound for one stream is bandwidth divided by weight size. On an H100 SXM (3,350 GB/s), a 7B FP16 model (14 GB) tops out near 239 tokens per second (3,350 ÷ 14); real numbers are lower because of overheads.
Frequently asked questions
- Why is my measured speed below the estimate?
- Kernel launch overhead, attention computation, the KV cache and framework efficiency all reduce real throughput below the bandwidth ceiling.