Glossary3 sections1 min read
What is BF16 (bfloat16)?
BF16, or bfloat16, is a 16-bit floating-point format with an 8-bit exponent (the same range as FP32) and a 7-bit mantissa. It is the usual precision for training and serving modern language models.
Why it matters when renting a GPU
BF16 halves memory and bandwidth compared with FP32 and rarely needs the loss scaling that FP16 training requires.
Check that the GPU has BF16 Tensor Core support; the A100 and newer data center GPUs do.
Worked example
The H100 SXM is rated at 989 dense BF16 TFLOPS, against 312 for the A100 SXM 80GB: about 3.2 times more.
Frequently asked questions
- BF16 or FP16?
- BF16 has a wider range and is easier for training; FP16 has more precision in a narrower range. Both use 2 bytes per value.