A Tour of the Standard Library

Every package measured in the previous lesson was produced for the measurement: their bodies were one line, their only job was counting themselves. Yet there is a large set of modules and packages present in every run, requiring no installation; it comes bundled with the language itself and already sits on the search path. This lesson looks at that set’s scope.

The question is not “what does each module do” — a reference list is not this lesson’s job. The question is: what is the difference when the same job is written with a standard-library module versus by hand? Three axes are measured: how many objects get created, how many get held, and how many lines get written. A fourth thing is also counted: in how many cases the hand-written version silently diverges.

The Library’s Scope

The standard library is part of the language and is imported without bringing anything in from outside. The areas it covers gather into a few groups.

Text and data formats: re gives pattern matching, textwrap text wrapping, unicodedata character properties; json, csv, and configparser read and write common data formats. Date and calendar: datetime carries date and time arithmetic, calendar calendar queries, zoneinfo time zones.

File system and resources: pathlib gives path objects, shutil bulk file operations, tempfile temporary space, glob pattern-based file selection. Data structures: collections gives specialized containers, heapq priority access, bisect position-finding in a sorted list, array a uniformly typed number sequence.

Functional tools: itertools gives iterator combination and slicing, functools wrapping and caching, operator operators’ function counterparts. Numbers: math, statistics, decimal, fractions, and random cover separate numeric domains. Types and contracts: typing, dataclasses, enum, and abc.

System and execution: sys, os, subprocess, argparse, and logging. Concurrency: threading, multiprocessing, asyncio, and concurrent.futures. Networking and transport: socket, http, urllib. Compression and digests: zipfile, tarfile, gzip, hashlib, hmac, secrets. Testing: unittest and doctest.

This is not memorized as a list; the useful reading is asking, when starting a job, which group that job falls into. Once the group is found, the module is found.

The Cost of Importing the Library

The standard library’s modules also fall under the rule measured in this topic’s first two lessons; they have no separate mechanism. When a library module is requested for the first time, its body runs once, the module object gets created and written to the cache; every later request for the same module runs 0 bodies and creates no new object. The is comparison measured in the first lesson showed this directly.

In the same way, the second lesson’s eager-import measurement applies to the library’s package parts: when a package’s submodule is requested, every level in the chain’s body runs. This means using the library has a cost — but it is a cost paid once, and it is not the same kind as the cost hand-written code makes you pay. Hand-written code runs on every call; import runs once, then does no more work than a dictionary lookup.

This is why “avoiding unnecessary imports” and “avoiding unnecessary object creation” are two separate jobs. The first is a one-time thing and is measured in module count; the second repeats on every call and is measured in item count. The measurement’s first table is of the second kind.

Three Jobs, Two Notations

The measurement is run on three jobs, and each job is written twice. The first is taking the first three items from two sources: in the hand-written version, two lists are built and concatenated, and the first three items are sliced off at the end; in the library version, itertools chains the two streams and the first three items are taken.

The second is splitting a file name into its root and extension: in the hand-written version, the name is split at the last dot; in the library version, pathlib builds a path object and reads the two pieces from it. The third is adding days to a date: in the hand-written version, month lengths and the leap-year rule are written out; in the library version, datetime adds two objects together.

The three jobs are deliberately chosen to be of different kinds. In the first, the two notations give the same result, and the distinction is purely in cost; in the second and third, the result itself diverges on some inputs. This heads off a one-dimensional reading like “the library writes shorter” from the start: the gain shows up in the object in one job, in the line count in another, in correctness in a third.

For an honest line count, all six notations are kept as source text; the line count is counted from that text, and the functions are produced from that same text. This way, the line counted and the line that runs are the same.

The measurement’s assumptions:

• MP20 — The Item class is taken from the shared setup; its counting scheme is unchanged, and the counter is reset before every measurement.
• MP21 — In the stream measurement, each of the two sources carries 400 values; the total is 800.
• MP22 — “Held in the intermediate container” is the number of items sitting in an intermediate list the moment the function returns its result; because the lazy notation builds no such list, 0 is written for it.
• MP23 — Line count is the number of lines in the function’s source text with blank ends stripped; the same text produces the function that runs.
• MP24 — Six names and six dates are chosen by hand, to cover edge cases, not at random.
• MP25 — The oracle is the standard library’s result; the hand-written version diverges from or agrees with it.
• MP26 — The hand-written leap-year rule is the divisible-by-four rule; the century correction is deliberately not written.
• MP27 — The dates in the day-adding measurement cover year, month, and century boundaries; results are compared as a year, month, day triple.
• MP28 — The measurement touches no file; path objects are built purely as names and never asked of the file system.

Measurement

"""Standard library: how many objects and lines when the same job is written by hand."""

from datetime import date, timedelta
from itertools import chain, islice
from pathlib import PurePosixPath

COUNTER = {"created": 0}


class Item:
    """Counts every creation of itself; the shared setup's counting scheme is unchanged."""

    def __init__(self, value):
        COUNTER["created"] += 1
        self.value = value


SOURCE = {
    "manual_stream": """
def manual_stream(a, b):
    combined = [Item(d) for d in a] + [Item(d) for d in b]
    return combined[:3], len(combined)
""",
    "library_stream": """
def library_stream(a, b):
    stream = chain((Item(d) for d in a), (Item(d) for d in b))
    return list(islice(stream, 3)), 0
""",
    "manual_split_name": """
def manual_split_name(name):
    base = name.rsplit("/", 1)[-1]
    if "." not in base:
        return base, ""
    root, dot, tail = base.rpartition(".")
    return root, dot + tail
""",
    "library_split_name": """
def library_split_name(name):
    path = PurePosixPath(name)
    return path.stem, path.suffix
""",
    "manual_add_days": """
def manual_add_days(y, m, d, days):
    lengths = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
    for _ in range(days):
        days_in_month = lengths[m - 1]
        if m == 2 and y % 4 == 0:
            days_in_month = 29
        d += 1
        if d > days_in_month:
            d, m = 1, m + 1
            if m > 12:
                m, y = 1, y + 1
    return y, m, d
""",
    "library_add_days": """
def library_add_days(y, m, d, days):
    result = date(y, m, d) + timedelta(days=days)
    return result.year, result.month, result.day
""",
}

NAMESPACE = {"Item": Item, "chain": chain, "islice": islice, "date": date,
            "PurePosixPath": PurePosixPath, "timedelta": timedelta}
for source in SOURCE.values():
    exec(source, NAMESPACE)


def line_count(name):
    return len(SOURCE[name].strip().splitlines())


def measure_stream(function_name, n=400):
    COUNTER["created"] = 0
    result, intermediate = NAMESPACE[function_name](range(n), range(n))
    return COUNTER["created"], intermediate, len(result)


NAMES = ["data.csv", "archive.tar.gz", ".hidden", "name.", "folder/name.txt", "name"]
DATES = [(2024, 2, 28, 1), (2023, 2, 28, 1), (2100, 2, 28, 1),
            (2000, 2, 28, 1), (2024, 12, 31, 1), (2024, 1, 31, 45)]

print(f"{'first three items from two sources':<26s} {'lines':>5s} {'created':>10s}"
      f" {'held in intermediate':>17s} {'returned':>10s}")
for name, label in (("manual_stream", "manual"), ("library_stream", "library")):
    created, intermediate, returned = measure_stream(name)
    print(f"  {label:<24s} {line_count(name):5d} {created:10d} {intermediate:17d}"
          f" {returned:10d}")

print()
print(f"name splitting: manual {line_count('manual_split_name')} lines, "
      f"library {line_count('library_split_name')} lines")
diverging = [(a, NAMESPACE["manual_split_name"](a), NAMESPACE["library_split_name"](a))
           for a in NAMES
           if NAMESPACE["manual_split_name"](a) != NAMESPACE["library_split_name"](a)]
print(f"names tried {len(NAMES)}, diverging {len(diverging)}")
for a, m, l in diverging:
    print(f"  {a:16s} manual {m}  library {l}")

print()
print(f"adding days: manual {line_count('manual_add_days')} lines, "
      f"library {line_count('library_add_days')} lines")
deviating = [(t, NAMESPACE["manual_add_days"](*t), NAMESPACE["library_add_days"](*t))
         for t in DATES
         if NAMESPACE["manual_add_days"](*t) != NAMESPACE["library_add_days"](*t)]
print(f"dates tried {len(DATES)}, deviating {len(deviating)}")
for t, m, l in deviating:
    print(f"  {t}  manual {m}  library {l}")

first three items from two sources lines    created held in intermediate   returned
  manual                       3        800               800          3
  library                      3          3                 0          3

name splitting: manual 6 lines, library 3 lines
names tried 6, diverging 2
  .hidden          manual ('', '.hidden')  library ('.hidden', '')
  name.            manual ('name', '.')  library ('name.', '')

adding days: manual 12 lines, library 3 lines
dates tried 6, deviating 1
  (2100, 2, 28, 1)  manual (2100, 2, 29)  library (2100, 3, 1)

Same Lines, Different Objects

The first table is this lesson’s most contrarian result: the two notations’ line count is equal, both 3. The library does not shorten the writing here. Where they part is in the object counted: the hand-written version creates 800 items, the library version 3. Returned is 3 in both.

The difference comes from at which point the work is done. In the hand-written version, two lists are built with a comprehension; the moment a comprehension is built, it creates every item, then the two get concatenated, and only at the very end are the first three taken. In the other notation, itertools chains the two streams without consuming them, and when the first three items are requested, only three items get created; the remaining 797 never get produced at all.

The intermediate-container column shows the holding separately: in the hand-written version, 800 items are still sitting in a list the moment the result returns; in the library version, 0. This is the standard library’s face of the course’s second claim — the lazy notation here does not merely delay production, it never produces what is never requested at all, because the stream is not consumed to the end.

This distinction’s boundary must not be misread. The library’s chained stream does not avoid production; it only ties production to consumption. Had the same stream been consumed to the end, it too would have created 800 items, and the production difference between the two notations would have zeroed out; the only difference left standing would be the item count held in the intermediate list. The source of the gain is not laziness itself, it is how little was requested: three items were requested, three items were produced.

The rule that follows is: what the standard library earns you is not always lines. Some modules shorten the writing, some leave the writing the same and lower the objects created instead. The two are separate gains and are measured separately. The question to ask when choosing a module should not be “how many lines does it shorten” but “which axis does it change.”

Line Difference and Silent Divergence

In the second and third measurements, the table flips: the line difference is large. Name splitting is 6 lines by hand, 3 with the library; adding days is 12 by hand, 3 with the library. A quarter-ratio difference — but the real cost is not in the lines.

In name splitting, 2 of the 6 names tried diverge. For the leading-dot name .hidden, the hand-written version leaves the root empty and counts the extension as .hidden; the library, by contrast, takes the whole name as the root and the extension as empty. For the trailing-dot name name., the hand-written version makes the extension ., and the library again takes the whole name as the root. For the multi-extension archive.tar.gz and the directory-carrying folder/name.txt, the two notations agree — the divergence is not in every case, only at the edges.

In adding days, 1 of the 6 dates tried deviates, and the deviation sits at a century boundary. The hand-written leap-year rule looks only at divisibility by four; since 2100 divides by four, the hand-written version counts February as 29 days and gives (2100, 2, 29). The library gives (2100, 3, 1), because it also applies the century rule. In the remaining five cases — a year boundary, a leap year, and a forty-five-day jump — the two notations give the same result.

The lesson shared by both measurements is this: hand-written code does not give a wrong result loudly. No exception is raised, no warning appears; the result is merely, silently, different. Because agreement holds in most of the tried cases, if the test set does not include the edges, the divergence never shows at all. What the standard library mainly earns you is that those edge cases are already written.

Why the Two Divergences Are Not the Same Kind

Even though the numbers sit in the same column, the two divergences are not the same thing, and confusing them is this measurement’s easiest misreading.

The deviation in adding days is a mistake. The calendar rule is not the language’s preference, it is a rule defined outside it; a notation that does not write the century correction fails to follow that rule, and the date it gives is not correct under any reading. Here the library is the oracle, because it applies the whole rule.

The divergence in name splitting, by contrast, is a difference in contract. Whether a leading-dot name has an extension is not an externally fixed fact the way the calendar is; the library chose one rule, the hand-written version chose another. Treating hidden file names as extensionless is a common convention and the library applies it; but an application may need the opposite. The 2 diverging cases are not the hand-written version’s defect, they are the point where two contracts part ways.

The criterion that follows from this: the decision to write a job by hand is justified when it can show that the library’s contract does not fit the job. A hand-written solution written without knowing what the library does fails this criterion; a solution written knowing the difference meets it. The two situations’ source text looks the same; the only thing that tells them apart is whether the difference was measured.

Summary

• The standard library is part of the language; it covers text, data format, calendar, file system, data structure, functional tool, number, system, concurrency, network, and testing groups.
• In the stream job, the two notations’ line count is equal (3 and 3), but the hand-written version creates 800 items, the lazy version 3; held in the intermediate container is 800 against 0.
• The library’s gain is not always lines: some modules shorten the writing, some lower the objects created instead; the two gains are measured separately.
• In name splitting, the hand-written version costs 6 lines and diverges from the library on 2 of 6 names tried; the divergence sits only at the edge names.
• In adding days, the hand-written version costs 12 lines and deviates on 1 of 6 dates tried; the deviation comes from not writing the century rule.
• A hand-written mistake raises no exception; the result comes out silently different, which is why a test set with no edge cases never shows the divergence at all.

Next Step

In this lesson, everything was imported: the library modules and the measurement functions alike were defined inside a body and called from outside. Yet the same file can run in two separate forms — one as an imported module, the other as a directly run script. The same body runs in both forms, but the module’s own name changes, and a block that looks at that name runs in only one case. The next lesson measures this duality: when the same file is run in two forms, how many module objects get created, and what happens to the old objects when a module gets reloaded?