Glossary3 sections1 min read
What is FP64 (double precision)?
FP64, or double precision, stores each number in 64 bits (8 bytes). Scientific and engineering simulations often need it for numerical accuracy; deep learning almost never does.
Why it matters when renting a GPU
Data center GPUs such as the A100 and H100 have strong FP64 throughput; consumer and many workstation GPUs run FP64 at a small fraction of their FP32 rate.
If your code is FP64-heavy, check the manufacturer's FP64 figure before choosing a GPU, not the AI TFLOPS figure.
Worked example
An FP64 value takes 8 bytes against 2 bytes in FP16, so the same 1-billion-value array needs 8 GB in FP64 and 2 GB in FP16: four times the memory and four times the bandwidth per pass.
Frequently asked questions
- Do LLMs need FP64?
- No. LLM training and inference run in BF16, FP16, FP8 or lower precisions.