Memory Usage

The previous topic built the dataframe abstraction over eight lessons and wrote down the default of every call. The closing of that topic pointed to a single condition: the convenience of the dataframe holds only as long as the dataset fits in memory. This topic starts where that condition lifts, and its first question is the most concrete one: how many bytes does a column hold in memory.

The answer depends not on the number of values but on the type those values are stored in. The type decision is also usually made without anyone making it: the call that reads the dataset picks a default, and the column settles into memory with that default. The unit of measure of this lesson is allocated bytes. The second number next to it is the decision quietly taken to win those bytes: which values can no longer be told apart, and by how many units which number was recorded wrongly.

• SP1. The dataset is fictional and gives the long-format reading table prepared by M26/K03: 3,199 rows, 1,329 subscribers, three periods. Instead of a real dataset, a generator that produces the same structure was written; the seed is visible and the output is identical from run to run. It is a model.
• SP2. Measurement is done with sys.getsizeof. This function returns an object’s own body and does not follow the objects it points to; the total is therefore built by hand.
• SP3. The figures were read from a runtime with eight-byte pointers and depend on the runtime. What does not change is the ratios; even if the pointer width changes, the multiple between representations stays the same.
• SP4. Readings are reported with one decimal. For two totals to count as equal the threshold is 0.05 m³, because a difference smaller than that can come only from rounding.
• SP5. The tariff tiers set up three tiers with limits of 10 and 25 m³, and the largest reading the generator can produce is 37.0 m³; the width of the integer representations is chosen according to this upper bound.

The Columns of the Dataset

# source.py — fictional long-format reading table from the regional measurement network.
# MODEL: a generator that produces the same structure was written in place of a real
# dataset; the seed is visible and the output is identical from run to run. The dataset
# prepared by M26/K03: 3199 rows, 1329 subscribers, three periods, tariff limits 10 and 25 m3.
from collections import Counter
SEED = 1246
REGION = ("north", "east", "center", "south", "west")
PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835))
DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1)   # center-weighted measurement network


def rng(seed: int):
    s = seed & 0xFFFFFFFF

    def next_value() -> float:
        nonlocal s
        s = (1103515245 * s + 12345) & 0xFFFFFFFF
        return s / 4294967296
    return next_value


def tier(reading: float) -> str:
    return "0-10" if reading <= 10 else "10-25" if reading <= 25 else "25-40"


def records() -> list[tuple]:
    r, out = rng(SEED), []
    for period, size in PERIODS:
        for i in range(size):
            o = round(5 + r() * 32.0, 1)
            out.append((10000 + i, f"S-{4000 + i}", period,
                        REGION[DENSITY[i % 10]], o, tier(o)))
    return out


r0 = records()
count = lambda j: ", ".join(f"{a} {n}" for a, n in Counter(x[j] for x in r0).most_common())
print(f"seed {SEED}; rows {len(r0)}, distinct subscribers {len({x[0] for x in r0})}, "
      f"periods {len(PERIODS)}")
print("region:", count(3))
print("tariff:", count(5))
print(f"total {sum(x[4] for x in r0):.1f} m3, smallest {min(x[4] for x in r0)}, "
      f"largest {max(x[4] for x in r0)}")

seed 1246; rows 3199, distinct subscribers 1329, periods 3
region: center 1601, east 639, north 321, south 319, west 319
tariff: 10-25 1504, 25-40 1187, 0-10 508
total 66667.9 m3, smallest 5.0, largest 37.0

The dataset carries six columns, and two of them produce all of this lesson’s measurements. The region column holds 3,199 values but contains only five distinct ones; center alone holds half of the rows. The reading column holds 3,199 numbers, all between 5.0 and 37.0, and each has one decimal. Both have the same number of rows, but in terms of what they cost in memory they look nothing alike.

One Column in Three Types

# memory.py — the same region column is held in three types and the allocated bytes are counted.
# MODEL: the column is first obtained as if read from a file, because the parser
# produces a new string object for every row and the strings are not shared
# the way they would be from a fixed list.
import csv, io, sys
from array import array

buffer = io.StringIO()
csv.writer(buffer, lineterminator="\n").writerows(r0)
rows = list(csv.reader(io.StringIO(buffer.getvalue())))
region_col = [s[3] for s in rows]
print(f"column has {len(region_col)} values, {len({id(s) for s in region_col})} distinct objects, "
      f"{len(set(region_col))} distinct values")

container = sys.getsizeof(region_col)                 # pointers only
objects = sum(sys.getsizeof(s) for s in {id(s): s for s in region_col}.values())
shared = sum(sys.getsizeof(s) for s in set(region_col))

longest = max(len(a) for a in set(region_col))        # fixed-width text
fixed = bytearray(len(region_col) * longest)
for j, a in enumerate(region_col):
    fixed[j * longest:(j + 1) * longest] = a.encode("ascii").ljust(longest)

vocab = sorted(set(region_col))                       # categorical dtype
codes = array("B", [vocab.index(a) for a in region_col])
cat_vocab = sys.getsizeof(vocab) + sum(sys.getsizeof(s) for s in vocab)

fmt = lambda h: "".join(str(v).rjust(14) if j else str(v).ljust(26)
                        for j, v in enumerate(h))
print(fmt(["region column type", "list bytes", "object bytes", "total", "bytes/row"]))
for label, cb, ob in [("N1 object list", container, objects),
                      ("N1 if counted as shared", container, shared),
                      ("N2 fixed-width text", sys.getsizeof(fixed), 0),
                      ("N3 categorical code", sys.getsizeof(codes), cat_vocab)]:
    print(fmt([label, cb, ob, cb + ob, round((cb + ob) / len(region_col), 1)]))
print(f"N1 -> N3 gain {round((container + objects) / (sys.getsizeof(codes) + cat_vocab), 1)}x; "
      f"ratio between two measurements of the same column "
      f"{round((container + objects) / (container + shared), 1)}x")

column has 3199 values, 3199 distinct objects, 5 distinct values
region column type            list bytes  object bytes         total     bytes/row
N1 object list                     26040        147797        173837          54.3
N1 if counted as shared            26040           229         26269           8.2
N2 fixed-width text                19251             0         19251           6.0
N3 categorical code                 3279           349          3628           1.1
N1 -> N3 gain 47.9x; ratio between two measurements of the same column 6.6x

The gain lies between the two ends of the table: the same five-valued column takes 173,837 bytes as an object list and 3,628 bytes as a categorical code, which is 47.9 times. The reason for the drop from 54.3 bytes per row to 1.1 bytes is a single observation: the column has 3,199 values but only five distinct ones. The categorical dtype turns that gap into structure; the column becomes an array of one-byte codes, and the five strings sit once in a vocabulary.

The second number sits in the middle of the table and concerns the measurement itself. The first row and the second row measure the same column, and there are 6.6 times between them. The distinction is this: in a column read from a file, each of the 3,199 values is a separate object, because the parser produces a new string for every row. Had the same column been built from a fixed list, five objects would have been shared and the object share would have been 229 bytes instead of 147,797 bytes. The measuring function counts the pointer and does not follow the object it points to; which number is correct depends on where the column came from, and both are defensible. If a memory report does not state which rule was used, that report is a choice.

Fixed-width text stands in the middle and carries its own decision: every row occupies as much space as the longest region name. Six bytes were chosen for the longest name; if a seven-letter region name enters the column, either the column is rebuilt or the name is truncated. The categorical dtype moves this decision into the vocabulary and removes the width problem.

The Number That Width Choice Moves

# width.py — the reading column is held in four representations; for each one the
# allocated bytes, the total and the number of rows whose tariff tier changed are printed side by side.
THRESHOLD = 0.05   # readings are reported with one decimal, a difference below this comes from rounding
reading_col = [float(s[4]) for s in rows]
f64 = array("d", reading_col)                            # 8-byte float
u16 = array("H", [round(o * 10) for o in reading_col])   # 2-byte deci-m3
u8 = array("B", [round(o) for o in reading_col])         # 1-byte whole m3
as_objects = sys.getsizeof(reading_col) + sum(sys.getsizeof(o) for o in reading_col)
exact = round(sum(reading_col), 1)

G = (22, 13, 12, 12, 17)
fmt2 = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0])
                         for j, v in enumerate(h))
print(fmt2(["reading column type", "total bytes", "bytes/row", "total m3", "tier changed"]))
for label, size, series in [("object list", as_objects, reading_col),
                            ("8-byte float", sys.getsizeof(f64), list(f64)),
                            ("2-byte deci-m3", sys.getsizeof(u16), [v / 10 for v in u16]),
                            ("1-byte whole m3", sys.getsizeof(u8), [float(v) for v in u8])]:
    changed = sum(1 for a, b in zip(reading_col, series) if tier(a) != tier(b))
    print(fmt2([label, size, round(size / len(reading_col), 1), round(sum(series), 1), changed]))
drift = abs(sum(u8) - exact)
print(f"the 1-byte choice shrinks memory {round(as_objects / sys.getsizeof(u8), 1)}x; "
      f"total drifts {drift:.1f} m3, {round(drift / THRESHOLD)}x the threshold")

# The same transformation finds 1329 distinct values in the meter_id column: the code grows
# to two bytes and the vocabulary becomes larger than the column itself.
meter_col = [s[1] for s in rows]
m_vocab = sorted(set(meter_col))
m_codes = array("H", [m_vocab.index(a) for a in meter_col])
m_categorical = sys.getsizeof(m_codes) + sys.getsizeof(m_vocab) + sum(
    sys.getsizeof(a) for a in m_vocab)
m_fixed = sys.getsizeof(bytearray(len(meter_col) * 6))
r_fixed, r_categorical = sys.getsizeof(fixed), sys.getsizeof(codes) + cat_vocab
print(f"region  : {len(vocab):5d} distinct values, fixed-width {r_fixed} bytes, "
      f"categorical {r_categorical} bytes -> {round(r_fixed / r_categorical, 1)}x gain")
print(f"meter_id: {len(m_vocab):5d} distinct values, fixed-width {m_fixed} bytes, "
      f"categorical {m_categorical} bytes -> {round(m_categorical / m_fixed, 1)}x loss")

# The whole dataset in two layouts: row records and narrow types chosen per column.
row_layout = sys.getsizeof(rows) + sum(
    sys.getsizeof(s) + sum(sys.getsizeof(a) for a in s) for s in rows)
p_vocab, t_vocab = sorted({s[2] for s in rows}), sorted({s[5] for s in rows})
columns = (array("I", [int(s[0]) for s in rows]), m_codes,
           array("B", [p_vocab.index(s[2]) for s in rows]), codes, u16,
           array("B", [t_vocab.index(s[5]) for s in rows]))
column_layout = sum(sys.getsizeof(v) for v in columns) + sum(
    sys.getsizeof(sz) + sum(sys.getsizeof(a) for a in sz)
    for sz in (m_vocab, p_vocab, vocab, t_vocab))
print(f"row layout {row_layout} bytes, column layout {column_layout} bytes -> "
      f"{round(row_layout / column_layout, 1)}x; {round(column_layout / len(rows))} bytes "
      f"per row instead of {round(row_layout / len(rows))} bytes")

reading column type     total bytes   bytes/row    total m3     tier changed
object list                  102816        32.1     66667.9                0
8-byte float                  25672         8.0     66667.9                0
2-byte deci-m3                 6478         2.0     66667.9                0
1-byte whole m3                3279         1.0     66661.0              108
the 1-byte choice shrinks memory 31.4x; total drifts 6.9 m3, 138x the threshold
region  :     5 distinct values, fixed-width 19251 bytes, categorical 3628 bytes -> 5.3x gain
meter_id:  1329 distinct values, fixed-width 19251 bytes, categorical 79653 bytes -> 4.1x loss
row layout 1298875 bytes, column layout 109714 bytes -> 11.8x; 34 bytes per row instead of 406 bytes

The first three rows are free of charge. Going from the object list to an array of eight-byte numbers shrinks memory 4 times, and going to the two-byte deci-m³ representation shrinks it 15.9 times, and the total does not change: readings are reported with one decimal, one decimal becomes an integer when multiplied by ten, and two bytes comfortably carry the value 370. Up to this point the gain of the abstraction really is free.

The fourth row shows the cost. The one-byte whole m³ representation shrinks memory 31.4 times relative to the object list and moves the total by only 6.9 m³, and that difference is about one ten-thousandth of the total; looking at the report, it seems harmless. But the last column gives another number: 108 rows change tariff tier. Rounding cancels out in the total, but at the boundary it does not; a meter that reads 10.4 m³ drops into the first tier when rounded to an integer, and that subscriber’s bill changes. Comparing the 6.9 m³ drift in the total with the threshold leads to the same place: the difference is 138 times the threshold, so it cannot be counted as coming from rounding. This is the hidden default: narrowing the width looks like a memory decision, but it is a measurement decision that determines which row falls into which tier.

The last two rows draw the limit of the categorical dtype. The same transformation gains 5.3 times on the region column relative to fixed-width text and loses 4.1 times on the meter number column. The reason is a single number: 1,329 distinct values do not fit in one byte, the code grows to two bytes and the vocabulary becomes larger than the column itself. “Switch the text column to the categorical dtype” is not a rule; it is a decision that depends on the number of distinct values, and it cannot be given without writing that number down.

The final row gives the sum of the decisions. The same 3,199 records take 1,298,875 bytes as row records and 109,714 bytes with narrow types chosen per column: 34 bytes per row instead of 406 bytes, 11.8 times. This number is the composition of six separate type decisions, and none of them comes by itself; which width and which representation was chosen for each column was written down one by one. A call that reads the dataset gives all of these decisions by default and does not report the result as a single number.

Summary

• The region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code; the gain is 47.9 times, a drop from 54.3 bytes per row to 1.1 bytes.
• Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer; a column read from a file holds 3,199 separate objects, a column built from a fixed list holds five.
• In the reading column, going from eight bytes down to two bytes does not change the total; going down to one byte shrinks memory 31.4 times relative to the object list but moves the total by 6.9 m³, and that difference is 138 times the threshold.
• The same rounding changes the tariff tier of 108 rows; the cost that does not show in the total appears at the boundary.
• The categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values; the gain depends not on the number of rows but on the number of distinct values.

Next Step

This lesson narrowed the memory cost of a column and found a difference of up to 31 times between four representations. But narrowing has a floor: 3,199 rows fit in memory with every type, and 320 million rows fit with none. At that point the question is no longer choosing a type but giving up on holding the whole dataset in memory at once. The next lesson reads the dataset in chunks and measures the balance that chunk size strikes between the memory peak and the number of passes; it counts which operations can be computed chunk by chunk and which cannot, and shows the wrong average that chunking silently produces.