Bits for range, bits for detail
A floating point number spends its bits on three jobs: a sign, an exponent that sets how large or small the value can get, and a mantissa that sets how finely it's resolved. The common formats split them like this:
- FP32: 1 sign bit, 8 exponent bits, 23 mantissa bits (4 bytes)
- FP16: 1 sign bit, 5 exponent bits, 10 mantissa bits (2 bytes)
- BF16: 1 sign bit, 8 exponent bits, 7 mantissa bits (2 bytes)
BF16 (bfloat16) keeps FP32's 8 exponent bits, so it covers the same enormous range, up to about 3.39e38. It pays for that with only 7 mantissa bits, so each value is rounded more coarsely than in FP16. FP16 has the finer steps but a far narrower range. Google Brain developed bfloat16 for its TPUs, and because it's FP32 with the low mantissa bits dropped, converting from FP32 is easy.
What each format costs in memory
Bytes per weight set the memory. Weights take 4 bytes each at FP32, 2 at FP16 or BF16 and 1 at FP8. For an 8B model that's 32 GB, 16 GB and 8 GB. FP16 and BF16 take identical space, so the choice between them is about numeric behavior, not memory.
A 70B model at 16 bits needs 140 GB for the weights alone. That's past a single 80 GB H100, so it takes two or more cards. Training adds gradients and optimizer states on top of the weights, which is why fine tuning needs far more memory than running the same model.
Which one you'll meet
Meta's Llama 3.1 8B config lists its data type as bfloat16, so the published weights are 16 bits each. The LLM VRAM Calculator lists FP16 and BF16 as separate options and counts the same 2 bytes for both. Run a model in the format its authors published unless you have a reason to change it.
Below 16 bits you're into quantization. FP8 takes one byte per weight, and integer schemes like Q8_0 and Q4_K_M go lower still, which is how a 70B model ends up on hardware that could never hold it at 16 bits.