GPUs for LLM inference
Serving a language model has two limits: the weights and the KV cache must fit in GPU memory, and each generated token reads the weights from memory again. Memory size decides which models you can serve; memory bandwidth largely decides how fast they answer.
GPUs that fit LLM inference
Swipe sideways to see every column.
| GPU | Memory | Bandwidth | FP8 tensor | Inference, 16-bit | Inference, 4-bit | Price/h |
|---|---|---|---|---|---|---|
| L4Ada Lovelace | 24 GB | 300 GB/s | 242 TFLOPS | ≈ 9B | ≈ 30B | from $0.44 |
| RTX 4090Ada Lovelace | 24 GB | 1,008 GB/s | Not published | ≈ 9B | ≈ 30B | from $0.33 |
| RTX 5090Blackwell | 32 GB | 1,792 GB/s | Not published | ≈ 10B | ≈ 40B | from $0.69 |
| L40SAda Lovelace | 48 GB | 864 GB/s | 733 TFLOPS | ≈ 15B | ≈ 60B | from $0.79 |
| RTX PRO 6000 BlackwellBlackwell | 96 GB | 1,792 GB/s | Not published | ≈ 35B | ≈ 120B | from $1.69 |
| A100 SXMAmpere | 80 GB | 2,039 GB/s | Not published | ≈ 30B | ≈ 100B | from $1.79 |
| H100 SXMHopper | 80 GB | 3,350 GB/s | 1,979 TFLOPS | ≈ 30B | ≈ 100B | from $2.88 |
| H200Hopper | 141 GB | 4,800 GB/s | 1,979 TFLOPS | ≈ 55B | ≈ 180B | from $4.19 |
| MI300XCDNA 3 | 192 GB | 5,300 GB/s | 2,614 TFLOPS | ≈ 75B | ≈ 250B | from $4.16 |
Manufacturer specifications, per GPU; tensor throughput without sparsity. Model sizes are cautious estimates in billions of parameters, counting 80% of memory; context length, batch size and framework change the real limit. Prices are the best live offer for one GPU per hour, excluding tax.
All GPU modelsWhat matters for LLM inference
Fit the model, then the cache
A model needs about 2 bytes per parameter in 16-bit and about half a byte in 4-bit quantization. The KV cache grows with context length and the number of concurrent requests, which is why the estimates keep 20% of memory free.
Bandwidth drives tokens per second
At small batch sizes, token generation is bound by memory bandwidth, not compute: a GPU with twice the bandwidth generates tokens roughly twice as fast on the same model. Compare the bandwidth column first for chat and agent workloads.
Throughput and FP8
For batch jobs and high concurrency, tensor throughput matters more. Hopper, Ada Lovelace and newer GPUs support FP8, which halves the memory of the weights compared with 16-bit and speeds up serving engines that support it.
Compare GPUs for LLM inference
GPUs for LLM inference: frequently asked questions
Which GPU do I need to run a 70B model?
In 4-bit quantization, a 70-billion-parameter model needs roughly 40 GB for the weights alone: a single 48 GB GPU is tight, an 80 GB GPU leaves room for the cache. In 16-bit it needs about 140 GB for the weights, so at least two 80 GB GPUs.
Is an RTX 4090 good for inference?
For models up to about 30 billion parameters in 4-bit, yes: it has 24 GB of memory and about 1 TB/s of bandwidth. Larger models and long contexts need more memory per GPU.
Should I quantize my model?
Quantization to 8-bit or 4-bit lets a larger model fit and usually speeds up generation, with a small loss of quality that depends on the model and method. Test the quality on your own prompts before serving.
How do I rent a GPU for LLM inference on EverXCloud?
Create an account, top up your prepaid balance in euros, US dollars or pounds, then buy GPU hours on the spot exchange at the best offer, or place a request at the price you are willing to pay. Rentals are billed by the hour. Prices are shown excluding tax and EverXCloud is the merchant of record for every rental.
Plan your GPU rental
- GPU rental: how to rent a cloud GPU by the hour
- How to rent your first GPU
- How to choose a GPU for LLM inference
- How to estimate a GPU rental session
- H100 price per hour: what sets the rate
- How a GPU spot exchange works
- H200 price per hour: cloud pricing explained
- B200 price per hour: B200, B300 and GB200
- A100 price per hour: rent an NVIDIA A100
- RTX 4090 price per hour: rental guide
- RTX 5090 price per hour: rental guide
- L40S price per hour: rental guide
- MI300X price per hour: rental guide
- How much VRAM do you need for an LLM?
- Best GPU for Stable Diffusion and Flux
- Cheapest way to rent a GPU for AI
- Best GPU for scientific computing (FP64)
Check the workload, complete configuration and billing conditions before choosing a live offer.
Be ready on launch day
Create your account now: you can top up your balance and rent the GPU that fits your workload as soon as the market opens.
Have GPU servers?
Businesses can join the EverXCloud Network and sell hours of their GPU servers on the spot exchange.
Other workloads
- GPUs for training large language models
Pre-training and full fine-tuning on data center GPUs with HBM memory.
- GPUs for fine-tuning open models
LoRA and QLoRA adapters on a single GPU, from 24 GB up.
- GPUs for image and video generation
Diffusion models for images and video, from 24 GB cards to 80 GB GPUs.
- GPUs for 3D rendering and visual effects
Ray-traced rendering, visual effects and real-time engines.
- GPUs for scientific computing and simulation
Simulation and HPC codes on data center GPUs with HBM memory.