---
title: 'Memory Usage'
source: 'https://academia.sh/en/courses/data-analysis-with-python/memory-usage'
course: 'Data Analysis with Python'
language: en
updated: '2026-10-07T04:31:11+00:00'
license: 'CC BY-SA 4.0'
objectives:
    - 'Measures the bytes the same column allocates as an object list, as fixed-width text and as a categorical code, and writes the gain as a multiple'
    - 'Shows that the measurement itself is a decision: depending on whether shared objects are counted once or once per row, the same column gives two different byte counts'
    - 'Counts how many rows narrowing the width of a numeric column moves in the total and in the tariff tier'
    - 'Compares how the gain of the categorical dtype depends on the number of distinct values and at what threshold it reverses'
---

# Memory Usage

In the 3,199-row dataset of the measurement network, the number of bytes a single column holds in memory depends on the type that column is stored in: the region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code, a gain of 47.9 times. Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer and does not follow the object it points to. When the reading column is cut from an object list to 1 byte, memory shrinks 31.4 times but the total drifts by 6.9 m³ and 108 rows change tariff tier; the categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values.

The previous topic built the dataframe abstraction over eight lessons and wrote down the default of every call.
The closing of that topic pointed to a single condition: the convenience of the dataframe holds only as long as
the dataset fits in memory. This topic starts where that condition lifts, and its first question is the most
concrete one: how many bytes does a column hold in memory.

The answer depends not on the number of values but on the type those values are stored in. The type decision is
also usually made without anyone making it: the call that reads the dataset picks a default, and the column settles
into memory with that default. The unit of measure of this lesson is **allocated bytes**. The second number next to
it is the decision quietly taken to win those bytes: which values can no longer be told apart, and by how many
units which number was recorded wrongly.

- **SP1.** The dataset is fictional and gives the long-format reading table prepared by M26/K03: 3,199 rows,
  1,329 subscribers, three periods. Instead of a real dataset, a generator that produces the same structure was
  written; the seed is visible and the output is identical from run to run. **It is a model.**
- **SP2.** Measurement is done with `sys.getsizeof`. This function returns an object's **own body** and does not
  follow the objects it points to; the total is therefore built by hand.
- **SP3.** The figures were read from a runtime with eight-byte pointers and depend on the runtime. What does not
  change is the **ratios**; even if the pointer width changes, the multiple between representations stays the same.
- **SP4.** Readings are reported with one decimal. For two totals to count as equal the threshold is 0.05 m³,
  because a difference smaller than that can come only from rounding.
- **SP5.** The tariff tiers set up three tiers with limits of 10 and 25 m³, and the largest reading the generator
  can produce is 37.0 m³; the width of the integer representations is chosen according to this upper bound.

## The Columns of the Dataset

```python
# source.py — fictional long-format reading table from the regional measurement network.
# MODEL: a generator that produces the same structure was written in place of a real
# dataset; the seed is visible and the output is identical from run to run. The dataset
# prepared by M26/K03: 3199 rows, 1329 subscribers, three periods, tariff limits 10 and 25 m3.
from collections import Counter
SEED = 1246
REGION = ("north", "east", "center", "south", "west")
PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835))
DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1)   # center-weighted measurement network


def rng(seed: int):
    s = seed & 0xFFFFFFFF

    def next_value() -> float:
        nonlocal s
        s = (1103515245 * s + 12345) & 0xFFFFFFFF
        return s / 4294967296
    return next_value


def tier(reading: float) -> str:
    return "0-10" if reading <= 10 else "10-25" if reading <= 25 else "25-40"


def records() -> list[tuple]:
    r, out = rng(SEED), []
    for period, size in PERIODS:
        for i in range(size):
            o = round(5 + r() * 32.0, 1)
            out.append((10000 + i, f"S-{4000 + i}", period,
                        REGION[DENSITY[i % 10]], o, tier(o)))
    return out


r0 = records()
count = lambda j: ", ".join(f"{a} {n}" for a, n in Counter(x[j] for x in r0).most_common())
print(f"seed {SEED}; rows {len(r0)}, distinct subscribers {len({x[0] for x in r0})}, "
      f"periods {len(PERIODS)}")
print("region:", count(3))
print("tariff:", count(5))
print(f"total {sum(x[4] for x in r0):.1f} m3, smallest {min(x[4] for x in r0)}, "
      f"largest {max(x[4] for x in r0)}")
```

```
seed 1246; rows 3199, distinct subscribers 1329, periods 3
region: center 1601, east 639, north 321, south 319, west 319
tariff: 10-25 1504, 25-40 1187, 0-10 508
total 66667.9 m3, smallest 5.0, largest 37.0
```

The dataset carries six columns, and two of them produce all of this lesson's measurements. The **region** column
holds 3,199 values but contains only five distinct ones; center alone holds half of the rows. The **reading**
column holds 3,199 numbers, all between 5.0 and 37.0, and each has one decimal. Both have the same number of rows,
but in terms of what they cost in memory they look nothing alike.

## One Column in Three Types

```python
# memory.py — the same region column is held in three types and the allocated bytes are counted.
# MODEL: the column is first obtained as if read from a file, because the parser
# produces a new string object for every row and the strings are not shared
# the way they would be from a fixed list.
import csv, io, sys
from array import array

buffer = io.StringIO()
csv.writer(buffer, lineterminator="\n").writerows(r0)
rows = list(csv.reader(io.StringIO(buffer.getvalue())))
region_col = [s[3] for s in rows]
print(f"column has {len(region_col)} values, {len({id(s) for s in region_col})} distinct objects, "
      f"{len(set(region_col))} distinct values")

container = sys.getsizeof(region_col)                 # pointers only
objects = sum(sys.getsizeof(s) for s in {id(s): s for s in region_col}.values())
shared = sum(sys.getsizeof(s) for s in set(region_col))

longest = max(len(a) for a in set(region_col))        # fixed-width text
fixed = bytearray(len(region_col) * longest)
for j, a in enumerate(region_col):
    fixed[j * longest:(j + 1) * longest] = a.encode("ascii").ljust(longest)

vocab = sorted(set(region_col))                       # categorical dtype
codes = array("B", [vocab.index(a) for a in region_col])
cat_vocab = sys.getsizeof(vocab) + sum(sys.getsizeof(s) for s in vocab)

fmt = lambda h: "".join(str(v).rjust(14) if j else str(v).ljust(26)
                        for j, v in enumerate(h))
print(fmt(["region column type", "list bytes", "object bytes", "total", "bytes/row"]))
for label, cb, ob in [("N1 object list", container, objects),
                      ("N1 if counted as shared", container, shared),
                      ("N2 fixed-width text", sys.getsizeof(fixed), 0),
                      ("N3 categorical code", sys.getsizeof(codes), cat_vocab)]:
    print(fmt([label, cb, ob, cb + ob, round((cb + ob) / len(region_col), 1)]))
print(f"N1 -> N3 gain {round((container + objects) / (sys.getsizeof(codes) + cat_vocab), 1)}x; "
      f"ratio between two measurements of the same column "
      f"{round((container + objects) / (container + shared), 1)}x")
```

```
column has 3199 values, 3199 distinct objects, 5 distinct values
region column type            list bytes  object bytes         total     bytes/row
N1 object list                     26040        147797        173837          54.3
N1 if counted as shared            26040           229         26269           8.2
N2 fixed-width text                19251             0         19251           6.0
N3 categorical code                 3279           349          3628           1.1
N1 -> N3 gain 47.9x; ratio between two measurements of the same column 6.6x
```

The gain lies between the two ends of the table: the same five-valued column takes 173,837 bytes as an object list
and 3,628 bytes as a categorical code, which is **47.9 times**. The reason for the drop from 54.3 bytes per row to
1.1 bytes is a single observation: the column has 3,199 values but only five distinct ones. The categorical dtype
turns that gap into structure; the column becomes an array of one-byte codes, and the five strings sit once in a
vocabulary.

The second number sits in the middle of the table and concerns the measurement itself. The first row and the
second row measure **the same column**, and there are 6.6 times between them. The distinction is this: in a column
read from a file, each of the 3,199 values is a separate object, because the parser produces a new string for every
row. Had the same column been built from a fixed list, five objects would have been shared and the object share
would have been 229 bytes instead of 147,797 bytes. The measuring function counts the pointer and does not follow
the object it points to; which number is correct depends on where the column came from, and both are defensible.
**If a memory report does not state which rule was used, that report is a choice.**

Fixed-width text stands in the middle and carries its own decision: every row occupies as much space as the
longest region name. Six bytes were chosen for the longest name; if a seven-letter region name enters the column,
either the column is rebuilt or the name is truncated. The categorical dtype moves this decision into the
vocabulary and removes the width problem.

## The Number That Width Choice Moves

```python
# width.py — the reading column is held in four representations; for each one the
# allocated bytes, the total and the number of rows whose tariff tier changed are printed side by side.
THRESHOLD = 0.05   # readings are reported with one decimal, a difference below this comes from rounding
reading_col = [float(s[4]) for s in rows]
f64 = array("d", reading_col)                            # 8-byte float
u16 = array("H", [round(o * 10) for o in reading_col])   # 2-byte deci-m3
u8 = array("B", [round(o) for o in reading_col])         # 1-byte whole m3
as_objects = sys.getsizeof(reading_col) + sum(sys.getsizeof(o) for o in reading_col)
exact = round(sum(reading_col), 1)

G = (22, 13, 12, 12, 17)
fmt2 = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0])
                         for j, v in enumerate(h))
print(fmt2(["reading column type", "total bytes", "bytes/row", "total m3", "tier changed"]))
for label, size, series in [("object list", as_objects, reading_col),
                            ("8-byte float", sys.getsizeof(f64), list(f64)),
                            ("2-byte deci-m3", sys.getsizeof(u16), [v / 10 for v in u16]),
                            ("1-byte whole m3", sys.getsizeof(u8), [float(v) for v in u8])]:
    changed = sum(1 for a, b in zip(reading_col, series) if tier(a) != tier(b))
    print(fmt2([label, size, round(size / len(reading_col), 1), round(sum(series), 1), changed]))
drift = abs(sum(u8) - exact)
print(f"the 1-byte choice shrinks memory {round(as_objects / sys.getsizeof(u8), 1)}x; "
      f"total drifts {drift:.1f} m3, {round(drift / THRESHOLD)}x the threshold")

# The same transformation finds 1329 distinct values in the meter_id column: the code grows
# to two bytes and the vocabulary becomes larger than the column itself.
meter_col = [s[1] for s in rows]
m_vocab = sorted(set(meter_col))
m_codes = array("H", [m_vocab.index(a) for a in meter_col])
m_categorical = sys.getsizeof(m_codes) + sys.getsizeof(m_vocab) + sum(
    sys.getsizeof(a) for a in m_vocab)
m_fixed = sys.getsizeof(bytearray(len(meter_col) * 6))
r_fixed, r_categorical = sys.getsizeof(fixed), sys.getsizeof(codes) + cat_vocab
print(f"region  : {len(vocab):5d} distinct values, fixed-width {r_fixed} bytes, "
      f"categorical {r_categorical} bytes -> {round(r_fixed / r_categorical, 1)}x gain")
print(f"meter_id: {len(m_vocab):5d} distinct values, fixed-width {m_fixed} bytes, "
      f"categorical {m_categorical} bytes -> {round(m_categorical / m_fixed, 1)}x loss")

# The whole dataset in two layouts: row records and narrow types chosen per column.
row_layout = sys.getsizeof(rows) + sum(
    sys.getsizeof(s) + sum(sys.getsizeof(a) for a in s) for s in rows)
p_vocab, t_vocab = sorted({s[2] for s in rows}), sorted({s[5] for s in rows})
columns = (array("I", [int(s[0]) for s in rows]), m_codes,
           array("B", [p_vocab.index(s[2]) for s in rows]), codes, u16,
           array("B", [t_vocab.index(s[5]) for s in rows]))
column_layout = sum(sys.getsizeof(v) for v in columns) + sum(
    sys.getsizeof(sz) + sum(sys.getsizeof(a) for a in sz)
    for sz in (m_vocab, p_vocab, vocab, t_vocab))
print(f"row layout {row_layout} bytes, column layout {column_layout} bytes -> "
      f"{round(row_layout / column_layout, 1)}x; {round(column_layout / len(rows))} bytes "
      f"per row instead of {round(row_layout / len(rows))} bytes")
```

```
reading column type     total bytes   bytes/row    total m3     tier changed
object list                  102816        32.1     66667.9                0
8-byte float                  25672         8.0     66667.9                0
2-byte deci-m3                 6478         2.0     66667.9                0
1-byte whole m3                3279         1.0     66661.0              108
the 1-byte choice shrinks memory 31.4x; total drifts 6.9 m3, 138x the threshold
region  :     5 distinct values, fixed-width 19251 bytes, categorical 3628 bytes -> 5.3x gain
meter_id:  1329 distinct values, fixed-width 19251 bytes, categorical 79653 bytes -> 4.1x loss
row layout 1298875 bytes, column layout 109714 bytes -> 11.8x; 34 bytes per row instead of 406 bytes
```

The first three rows are free of charge. Going from the object list to an array of eight-byte numbers shrinks
memory 4 times, and going to the two-byte deci-m³ representation shrinks it 15.9 times, and the total does not
change: readings are reported with one decimal, one decimal becomes an integer when multiplied by ten, and two bytes
comfortably carry the value 370. Up to this point the gain of the abstraction really is free.

The fourth row shows the cost. The one-byte whole m³ representation shrinks memory 31.4 times relative to the
object list and moves the total by only 6.9 m³, and that difference is about one ten-thousandth of the total;
looking at the report, it seems harmless. But the last column gives another number: **108 rows change tariff
tier**. Rounding cancels out in the total, but at the boundary it does not; a meter that reads 10.4 m³ drops into
the first tier when rounded to an integer, and that subscriber's bill changes. Comparing the 6.9 m³ drift in the
total with the threshold leads to the same place: the difference is 138 times the threshold, so it cannot be
counted as coming from rounding. **This is the hidden default:** narrowing the width looks like a memory decision,
but it is a measurement decision that determines which row falls into which tier.

The last two rows draw the limit of the categorical dtype. The same transformation gains 5.3 times on the region
column relative to fixed-width text and loses 4.1 times on the meter number column. The reason is a single number:
1,329 distinct values do not fit in one byte, the code grows to two bytes and the vocabulary becomes larger than
the column itself. "Switch the text column to the categorical dtype" is not a rule; it is a decision that depends
on the number of distinct values, and it cannot be given without writing that number down.

The final row gives the sum of the decisions. The same 3,199 records take 1,298,875 bytes as row records and
109,714 bytes with narrow types chosen per column: 34 bytes per row instead of 406 bytes, **11.8 times**. This
number is the composition of six separate type decisions, and none of them comes by itself; which width and which
representation was chosen for each column was written down one by one. A call that reads the dataset gives all of
these decisions by default and does not report the result as a single number.

## Summary

- The region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a
  categorical code; the gain is 47.9 times, a drop from 54.3 bytes per row to 1.1 bytes.
- Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the
  pointer; a column read from a file holds 3,199 separate objects, a column built from a fixed list holds five.
- In the reading column, going from eight bytes down to two bytes does not change the total; going down to one
  byte shrinks memory 31.4 times relative to the object list but moves the total by 6.9 m³, and that difference
  is 138 times the threshold.
- The same rounding changes the tariff tier of 108 rows; the cost that does not show in the total appears at the
  boundary.
- The categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with
  1,329 distinct values; the gain depends not on the number of rows but on the number of distinct values.

## Next Step

This lesson narrowed the memory cost of a column and found a difference of up to 31 times between four
representations. But narrowing has a floor: 3,199 rows fit in memory with every type, and 320 million rows fit
with none. At that point the question is no longer choosing a type but giving up on holding the whole dataset in
memory at once. The next lesson reads the dataset in chunks and measures the balance that chunk size strikes
between the memory peak and the number of passes; it counts which operations can be computed chunk by chunk and
which cannot, and shows the wrong average that chunking silently produces.