The Difference Between BF16 and FP16

Range and precision

FP32

8 exponent bits · 23 fraction bits · bias 127

Exponent range

Stored exponent: 1 to 254

Bias

2^{8 - 1} - 1 = 127

Minimum exponent

1 - 127 = - 126

Maximum exponent

254 - 127 = 127

Minimum positive value

Choose a positive sign, the first usable exponent pattern, and an all-zero fraction. This gives the smallest value in the exponent range.

Minimization criterion

0 | 00000001 | 00000000000000000000000

sign | exponent | fraction

2^{1 - 127} = 2^{- 126}

≈ 1.1754943508222875 × 10⁻³⁸

Largest finite value

Keep the sign positive. Use the largest exponent before the reserved all-ones pattern, then set every fraction bit to 1.

Maximization criterion

0 | 11111110 | 11111111111111111111111

sign | exponent | fraction

Stored exponent 254 becomes 127 after subtracting the bias: 254 − 127 = 127.

(2 - 2^{- 23}) \times 2^{127}

≈ 3.4028234663852886 × 10³⁸

Precision near 1

Encode 1 with a zero fraction, then increase only the final fraction bit. The gap between these adjacent values is the precision near 1.

1.0

0 | 01111111 | 00000000000000000000000

sign | exponent | fraction

Next value after 1.0

0 | 01111111 | 00000000000000000000001

sign | exponent | fraction

ε = 2^{- 23}

= 0.00000011920928955078125

BF16

8 exponent bits · 7 fraction bits · bias 127

Exponent range

Stored exponent: 1 to 254

Bias

2^{8 - 1} - 1 = 127

Minimum exponent

1 - 127 = - 126

Maximum exponent

254 - 127 = 127

Minimum positive value

Choose a positive sign, the first usable exponent pattern, and an all-zero fraction. This gives the smallest value in the exponent range.

Minimization criterion

0 | 00000001 | 0000000

sign | exponent | fraction

2^{1 - 127} = 2^{- 126}

≈ 1.1754943508222875 × 10⁻³⁸

Largest finite value

Keep the sign positive. Use the largest exponent before the reserved all-ones pattern, then set every fraction bit to 1.

Maximization criterion

0 | 11111110 | 1111111

sign | exponent | fraction

Stored exponent 254 becomes 127 after subtracting the bias: 254 − 127 = 127.

(2 - 2^{- 7}) \times 2^{127}

≈ 3.3895313892515355 × 10³⁸

Precision near 1

Encode 1 with a zero fraction, then increase only the final fraction bit. The gap between these adjacent values is the precision near 1.

1.0

0 | 01111111 | 0000000

sign | exponent | fraction

Next value after 1.0

0 | 01111111 | 0000001

sign | exponent | fraction

ε = 2^{- 7}

= 0.0078125

FP16

5 exponent bits · 10 fraction bits · bias 15

Exponent range

Stored exponent: 1 to 30

Bias

2^{5 - 1} - 1 = 15

Minimum exponent

1 - 15 = - 14

Maximum exponent

30 - 15 = 15

Minimum positive value

Choose a positive sign, the first usable exponent pattern, and an all-zero fraction. This gives the smallest value in the exponent range.

Minimization criterion

0 | 00001 | 0000000000

sign | exponent | fraction

2^{1 - 15} = 2^{- 14}

≈ 6.103515625 × 10⁻⁵

Largest finite value

Keep the sign positive. Use the largest exponent before the reserved all-ones pattern, then set every fraction bit to 1.

Maximization criterion

0 | 11110 | 1111111111

sign | exponent | fraction

Stored exponent 30 becomes 15 after subtracting the bias: 30 − 15 = 15.

(2 - 2^{- 10}) \times 2^{15}

≈ 65,504

Precision near 1

Encode 1 with a zero fraction, then increase only the final fraction bit. The gap between these adjacent values is the precision near 1.

1.0

0 | 01111 | 0000000000

sign | exponent | fraction

Next value after 1.0

0 | 01111 | 0000000001

sign | exponent | fraction

ε = 2^{- 10}

= 0.0009765625

Why BF16 exists

During deep neural network training, gradients and intermediate values can become too small for FP16 and underflow toward zero. Other values can exceed its maximum and overflow. FP32 avoids much of this, but using FP32 everywhere is expensive.

Why not use FP32 everywhere?

FP32 gives both wide range and high precision, but each value uses 32 bits. A 16-bit format halves the storage and memory traffic per value, and modern accelerators can usually process 16-bit matrix operations much faster.

Does FP16 have higher precision than BF16?

Yes. FP16 has 10 fraction bits while BF16 has 7, so FP16 keeps more detail around a value. The tradeoff is range: FP16 spends only 5 bits on the exponent, while BF16 uses the same 8 exponent bits as FP32. BF16 therefore keeps almost the FP32 range while still using only 16 bits.

Precision: FP32 > FP16 > BF16

Range: FP32 ≈ BF16 >> FP16

Is BF16 precision loss a problem?

It can be, so training does not blindly perform every operation in BF16. Mixed-precision training commonly uses BF16 for large matrix operations and stored tensors, while accumulating results, updating optimizer state, and running numerically sensitive operations in FP32. This keeps most of the speed and memory benefit without giving up precision where it matters most.