Glossary3 sections1 min read
What is Quantization (INT8, INT4)?
Quantization stores a model's weights (and sometimes activations) in fewer bits, such as 8-bit or 4-bit integers with scale factors, instead of 16-bit floating point. It cuts memory and often speeds up inference.
Why it matters when renting a GPU
Quantization is the fastest way to fit a larger model on a smaller, cheaper GPU.
Quality can drop, especially at 4 bits; evaluate on your own prompts before switching production traffic.
Worked example
A 70B model needs about 140 GB of weights in FP16, 70 GB in INT8 and 35 GB in INT4. In INT4 it fits on a single 48 GB L40S for its weights alone.
Frequently asked questions
- Does quantization reduce quality?
- Slightly at 8 bits for most models, more at 4 bits. Measure on your own evaluation set.