Glossary3 sections1 min read
What is FP32 and TF32?
FP32 is standard 32-bit (4-byte) floating point. TF32 is a Tensor Core mode introduced with the A100 that keeps FP32's 8-bit exponent but only a 10-bit mantissa, so FP32 code runs much faster with little accuracy loss.
Why it matters when renting a GPU
Frameworks can enable TF32 automatically for matrix math, which speeds up older FP32 training code without changes.
For new work, BF16 or FP16 mixed precision usually goes further, halving memory as well.
Worked example
FP32 weights take 4 bytes per parameter, so a 7B model needs 7 × 4 = 28 GB in FP32 against 14 GB in BF16 or FP16.
Frequently asked questions
- Is TF32 the same as FP32?
- No. TF32 is a Tensor Core math mode with FP32 range but reduced mantissa precision; inputs and outputs remain FP32 in memory.