Glossary3 sections1 min read
What is Inference vs training?
Training computes and updates a model's weights from data; inference runs a trained model to produce outputs. Training is compute- and memory-heavy over long runs; inference is latency- and bandwidth-sensitive.
Why it matters when renting a GPU
For inference, size VRAM for weights plus KV cache and favor bandwidth. For training, favor Tensor Core throughput, memory for optimizer states and fast GPU-to-GPU links.
A GPU that is cost-effective for serving may be a poor choice for training, and the reverse.
Worked example
Serving a 7B model in FP16 needs about 14 GB of weights; fully fine-tuning it with Adam needs about 112 GB: 8 times more memory for the same model.
Frequently asked questions
- Can I use consumer GPUs for inference?
- Yes for models that fit in their memory. Data center GPUs add more memory, bandwidth and features such as MIG.