Floating-Point Numbers

For integers, representation is all or nothing: $200$ can be written in eight bits, $300$ cannot. Fractional numbers are more troubling still. There are infinitely many real numbers between $0$ and $1$; a finite set of bits can hold only a finite subset of them. The rest cannot be represented and are rounded to the nearest representable value.

This lesson asks how that subset is chosen. The choice feeds directly into computation results: by the end of this lesson, the fact that $0.1 + 0.2$ does not come out to exactly $0.3$ will look like expected behavior rather than a quirk.

Fixed Point and Its Limit

The first solution that comes to mind is placing the decimal separator at a fixed position. The upper bits of an eight-bit value could be assigned to the integer part and the lower six to the fractional part; the positional values of the fractional bits then run $2^{-1}$, $2^{-2}$, $2^{-3}$, and so on:

$$\texttt{0101{.}1010}_2 = 4 + 1 + 0.5 + 0.125 = 5.625$$

This is fixed-point representation, and it is still used in specific domains. Its problem is that range and resolution are locked together: every bit assigned to the fraction halves the largest representable number. If a single computation needs to carry both interplanetary distances and atomic radii, a fixed separator position cannot serve both needs at once.

The Binary Counterpart of Scientific Notation

The solution is to store the separator’s position alongside the value itself. In scientific notation, writing $6.022 \times 10^{23}$ carries two pieces of information: significant digits and order of magnitude. The binary counterpart has the same structure:

$$(-1)^{s} \times 1.m \times 2^{e}$$

Here $s$ is the sign, $m$ is the fractional part (the mantissa), and $e$ is the exponent. The separator “floats” as the exponent changes — which is where the name of the representation comes from.

One detail saves a bit: every nonzero number can be written in the form $1.\dots$ by choosing the exponent appropriately. Since the leading $1$ is always present, it does not need to be stored; hardware treats it as implicit. The entire mantissa field is then given over to the fractional part, and one bit is gained for free.

The IEEE 754 Field Layout

The bit-level details of floating-point representation are defined in the IEEE 754 standard. Two common widths:

Format	Total	Sign	Exponent	Mantissa	Exponent bias
binary32 (single precision)	32 bit	1	8	23	127
binary64 (double precision)	64 bit	1	11	52	1023

The exponent field is not stored as a signed number but biased: the true exponent is found by subtracting a fixed bias from the value stored in the field. In binary32, if the field holds $130$, the true exponent is $130 - 127 = 3$. The reason for this arrangement is ordering: a biased exponent lets two positive floating-point numbers be sorted by comparing their bit patterns as if they were unsigned integers.

The two extreme values of the exponent field are reserved for special meanings:

• All bits $0$: the value is zero (if the mantissa is also zero) or a subnormal number. Subnormal numbers sustain resolution very close to zero by abandoning the implicit-$1$ rule.
• All bits $1$: the value is infinity if the mantissa is zero, and not a number (NaN) if it is nonzero.

Infinity and NaN carry the result of an overflowing or undefined operation without halting the program. $1/0$ gives infinity, $0/0$ gives NaN. NaN’s distinguishing property is that it is not even equal to itself; this is the standard way to test whether a value is NaN.

The Floating-Point Interpretation of the Shared Example

The course’s shared example, 0x41424344, splits into the following fields when interpreted as binary32:

$$\underbrace{0}_{s}\ \underbrace{1000\,0010}_{\text{exponent field}}\ \underbrace{100\,0010\,0100\,0011\,0100\,0100}_{\text{mantissa}}$$

• The sign bit is $0$, so the number is positive.
• The exponent field is $1000\,0010_2 = 130$; the true exponent is $130 - 127 = 3$.
• The mantissa field is $4{,}342{,}596$; its fractional value is $4{,}342{,}596 / 2^{23} = 0.517677783966064453125$. Adding the implicit $1$ gives a significand of $1.517677783966064453125$.

The value:

$$1.517677783966064453125 \times 2^{3} = 12.141422271728515625$$

The same 32 bits read as $1{,}094{,}861{,}636$ as an unsigned integer and as $12.141422271728515625$ as a floating-point number. The bit pattern has not changed; what changes is the interpretation rule applied to it. A variable’s type in a programming language exists precisely to select that rule.

Which Numbers Are Exactly Represented

The mantissa is a binary fraction. The values represented exactly are therefore fractions whose denominator is a power of two: $0.5$, $0.25$, $0.75$, $3.125$, and the like. Such numbers are called dyadic rationals.

$0.1$ is not one of them. This number, written with a single digit in the decimal system, has a repeating expansion in the binary system:

$$0.1_{10} = 0.0001100110011\overline{0011}\dots_2$$

A repeating expansion does not fit a finite mantissa; it is rounded to the nearest representable value. What binary64 stores is not $0.1$ but another number very close to it. The same holds for $0.2$ and $0.3$.

This is not a flaw of the language or the hardware: it is exactly as natural that $1/10$ cannot be written with finitely many binary digits as it is that $1/3$ cannot be written with finitely many decimal digits. What changes is which denominators are “lucky.”

The Result of Addition

When two rounded values are added, the result is rounded as well, and errors accumulate:

print(0.1 + 0.2)              # 0.30000000000000004
print(0.1 + 0.2 == 0.3)       # False
print(f"{0.1:.20f}")          # 0.10000000000000000555

import math
print(math.isclose(0.1 + 0.2, 0.3))   # True

import struct
(value,) = struct.unpack(">f", bytes.fromhex("41424344"))
print(value)                  # 12.141422271728516

The comparison 0.1 + 0.2 == 0.3 giving a false result is not a bug — it follows directly from the representation. Floating-point values are therefore never compared for equality; instead, one checks whether the difference falls below an acceptable threshold. Functions such as isclose perform this comparison using a combination of relative and absolute tolerance.

The struct module interprets four bytes of raw data according to a given format; the format string ">f" means “big-endian, single-precision floating point.” What the byte-order marker here means is the subject of this topic’s final lesson.

The Limit of Precision

Mantissa width determines how many significant digits can be carried. binary32 carries about 7 significant decimal digits, binary64 about 15 to 17. This limit has two practical consequences.

First: when a very large and a very small value are added, the smaller one can vanish. If a value below the mantissa’s resolution is added to a large number, the result does not change.

Second: computation order affects the result. Floating-point addition does not carry the associative property — $(a + b) + c$ and $a + (b + c)$ can give different results. This means algebraic properties familiar from integers do not carry over here, and it is one of the reasons numerical methods form a separate discipline.

In domains where decimal precision is a matter of contract, such as monetary amounts, floating-point types are not used; fixed-precision decimal types, or integers storing the smallest unit (for example, cents), are preferred instead.

Summary

• Finite bits hold only a finite subset of the real numbers; the rest are rounded to the nearest representable value.
• Floating-point representation stores a value as $(-1)^s \times 1.m \times 2^{e}$; the leading $1$ is implicit and gains one bit.
• In IEEE 754 binary32 the fields are 1 / 8 / 23 bits, and the exponent is stored with a bias of $127$; the extreme values of the exponent field are reserved for zero, subnormal numbers, infinity, and NaN.
• Exactly represented fractions are those whose denominator is a power of two; $0.1$ is not among them.
• Floating-point values are compared with tolerance, not equality; addition does not carry the associative property.
• The same 0x41424344 pattern reads as $1{,}094{,}861{,}636$ as an integer and as $12.141422271728515625$ as binary32.

Next Step

Up to this point, bit patterns have always been interpreted as a whole. Programs, however, frequently work with individual bits: turning on a flag, extracting a field, doubling a value. The next lesson defines the bit-level operators that do this; the >> and & operators used in previous lessons will also be given their rationale there.