Package Structure

The previous lesson searched for modules in a flat list: every source sat directly inside a directory on the search path, and its name was a single piece. In that layout, two files carrying the same name were a clash, and order alone decided the winner. When code grows, this layout is not enough; files doing related work want to sit close to each other, carrying the same short name in different contexts.

A package is the layout where directories join the namespace: a directory and the __init__.py file inside it together give a module object, and what is inside that directory is attached to it with a dot. This makes a name two-part — it gets a path and a short name. The lesson’s question is: while the same short name exists at two separate levels, which object does relative and absolute notation resolve to, and how many bodies does a single request run?

How a Directory Becomes a Module

A package is a directory with __init__.py inside it. That file is the package’s body, and it runs once when the package is first requested; being empty is also an option, in which case the body binds no name, but the module object still gets created. The files and subdirectories inside a package are its submodules and subpackages.

A dotted name is resolved left to right, and every level is a separate module object. A request for pkg.sub.tool needs not one object but three: pkg, pkg.sub, and pkg.sub.tool. All three are written to the cache separately, and all three bodies run in order. This is a direct consequence of the previous lesson’s rule — every name gets its own record.

Every resolved level is also bound as an attribute one level up. Once pkg.sub has been imported, the name sub exists in the pkg module object’s dictionary, and that name is bound to the very same pkg.sub object in the cache. This is how dotted access works: the dot is not a search-path lookup, it is an attribute read.

Two Notations, Two Starting Points

The same submodule can be reached by two routes. Absolute notation starts the name from the root: the name’s first piece is searched for on the search path, and the chain is resolved from there. Relative import, by contrast, starts the name from the requesting module’s package; a leading dot means “the package I am in,” two dots mean “the package one level up.”

The two produce two separate addresses carrying the same letters. If a package’s root has a module named tool, and its subpackage also has another module named tool, a relative request written from inside the subpackage gives the neighboring one, an absolute request gives the one at the root. The names are the same; the objects they resolve to are not.

Relative import has one condition: the requesting module has to belong to a package. For a module that is not inside a package, there is no such thing as “the package I am in,” and the request cannot be resolved. The measurement tests this case too.

Whose Bill Is the Package’s Body

__init__.py is a package’s door, and it runs whenever any module belonging to the package is requested. This means whatever is written there gets billed to every user of the package. There are two common notations, and the difference between them is measurable.

In the plain notation, __init__.py only binds the package’s own names, it does not import the submodules; whichever submodule the user wants, they write it separately. In the eager notation, __init__.py imports the submodules itself and carries their names up to package level; the user reaches all of them from a single name. The second earns shorter notation, but it makes a user requesting a single leaf run all the siblings’ bodies. The measurement’s last section compares the same single-leaf request across the two notations.

How Many Bodies a Request Runs

The previous lesson measured that import is an act of creation. In packages, this count grows with the chain’s length, and one side of the chain also comes from the imported module’s own requests. If a submodule imports other modules inside its body, those run within the same request too. The measurement writes down the bodies a single request runs, in run order.

The measurement’s assumptions:

• MP11 — The measurement leaves no file in the repository; the package tree is produced in a temporary directory, deleted when the job ends, and the directory’s path is not printed.
• MP12 — The bodies that ran are counted by every body appending its own name to a shared list; the counting side is the running module itself, and the list preserves run order.
• MP13 — The root and the subpackage carry two modules with the same short name; the two differ only in the value of a module-level name.
• MP14 — The relative and absolute requests are made from the body of a third, measured module; the two results are read from that module’s dictionary.
• MP15 — Which name gets bound is measured in a separate namespace dictionary; names other than the two defined ones are filtered out.
• MP16 — A relative request is tried in a module that does not belong to a package; the oracle is the exception’s class name, not its message text.
• MP17 — When the measurement ends, the added path entry and all cache records are rolled back.
• MP18 — Plain and eager notation carry the same tree shape: one package and three leaves. The only difference is a single line in the eager notation’s __init__.py file.
• MP19 — In comparing the two notations, a single leaf is requested, and the count is reset before each measurement; the two measurements do not carry each other’s count.

Measurement

"""Package structure: which object does the same short name resolve to in two notations."""

import importlib
import sys
import tempfile
from pathlib import Path


def body_marker(name, extra=""):
    return f"import log\nlog.RUNS.append('{name}')\n{extra}"


SOURCES = {
    "log.py": "RUNS = []\n",
    "pkg/__init__.py": body_marker("pkg"),
    "pkg/tool.py": body_marker("pkg.tool", "NAME = 'pkg.tool'\n"),
    "pkg/sub/__init__.py": body_marker("pkg.sub"),
    "pkg/sub/tool.py": body_marker("pkg.sub.tool", "NAME = 'pkg.sub.tool'\n"),
    "pkg/sub/user.py": body_marker("pkg.sub.user",
                                   "from . import tool as relative\n"
                                   "from pkg import tool as absolute\n"),
    "alone.py": "from . import tool\n",
    "plain/__init__.py": body_marker("plain"),
    "eager/__init__.py": body_marker("eager", "from . import x, y, z\n"),
}
for root_name in ("plain", "eager"):
    for leaf in ("x", "y", "z"):
        SOURCES[f"{root_name}/{leaf}.py"] = body_marker(f"{root_name}.{leaf}")

TOPS = ("log", "alone", "pkg", "plain", "eager")


def setup(root):
    for name, text in SOURCES.items():
        path = root / name
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(text, encoding="utf-8")
    sys.path.insert(0, str(root))
    importlib.invalidate_caches()


def teardown(root):
    sys.path.remove(str(root))
    for name in list(sys.modules):
        if name.split(".")[0] in TOPS:
            sys.modules.pop(name)


def measure(root):
    setup(root)
    user = importlib.import_module("pkg.sub.user")
    log = sys.modules["log"]
    ran = list(log.RUNS)
    registered = sorted(a for a in sys.modules if a.split(".")[0] == "pkg")
    relative, absolute = user.relative, user.absolute
    attribute_bound = (sys.modules["pkg"].sub is sys.modules["pkg.sub"]
                 and sys.modules["pkg.sub"].tool
                 is sys.modules["pkg.sub.tool"])
    importlib.import_module("pkg.sub.tool")
    second_request = len(log.RUNS) - len(ran)

    ns_dotted, ns_from = {}, {}
    exec("import pkg.tool", ns_dotted)
    exec("from pkg import tool", ns_from)
    bound_dotted = sorted(a for a in ns_dotted if not a.startswith("__"))
    bound_from = sorted(a for a in ns_from if not a.startswith("__"))
    same_object = ns_dotted["pkg"].tool is ns_from["tool"]

    try:
        importlib.import_module("alone")
        error = "none"
    except Exception as e:
        error = type(e).__name__

    log.RUNS.clear()
    importlib.import_module("plain.x")
    plain_result = list(log.RUNS)
    log.RUNS.clear()
    importlib.import_module("eager.x")
    eager_result = list(log.RUNS)

    teardown(root)
    return {"ran": ran, "registered": registered, "second": second_request,
            "same": relative is absolute,
            "relative_name": relative.NAME, "absolute_name": absolute.NAME,
            "attribute_bound": attribute_bound, "bound_dotted": bound_dotted,
            "bound_from": bound_from, "same_object": same_object, "error": error,
            "plain": plain_result, "eager": eager_result}


with tempfile.TemporaryDirectory() as d:
    s = measure(Path(d))

print(f"single request ran {len(s['ran'])} bodies: {' -> '.join(s['ran'])}")
print(f"package modules written to cache {len(s['registered'])}: {s['registered']}")
print(f"bodies run when the same submodule is requested a second time {s['second']}")
print()
print(f"relative notation -> {s['relative_name']}")
print(f"absolute notation -> {s['absolute_name']}")
print(f"same object       -> {s['same']}")
print()
print(f"name the dotted notation binds {s['bound_dotted']}, "
      f"name the from notation binds {s['bound_from']}")
print(f"the module both reach is the same object -> {s['same_object']}")
print(f"the subpackage is bound as an attribute on the parent package -> {s['attribute_bound']}")
print(f"relative import in a package-less module -> {s['error']}")
print()
print(f"{'single-leaf request':<22s} {'bodies run':>12s}  run order")
print(f"{'plain package':<22s} {len(s['plain']):12d}  {' -> '.join(s['plain'])}")
print(f"{'eager importer':<22s} {len(s['eager']):12d}  "
      f"{' -> '.join(s['eager'])}")

single request ran 5 bodies: pkg -> pkg.sub -> pkg.sub.user -> pkg.sub.tool -> pkg.tool
package modules written to cache 5: ['pkg', 'pkg.sub', 'pkg.sub.tool', 'pkg.sub.user', 'pkg.tool']
bodies run when the same submodule is requested a second time 0

relative notation -> pkg.sub.tool
absolute notation -> pkg.tool
same object       -> False

name the dotted notation binds ['pkg'], name the from notation binds ['tool']
the module both reach is the same object -> True
the subpackage is bound as an attribute on the parent package -> True
relative import in a package-less module -> ImportError

single-leaf request      bodies run  run order
plain package                     2  plain -> plain.x
eager importer                    4  eager -> eager.x -> eager.y -> eager.z

Five Bodies, Five Objects

The first line gives the cost of a single request, in order. pkg.sub.user was requested; bodies run came to 5. The order explains itself: first pkg, then pkg.sub — the dotted name resolves left to right, and every level’s body runs in its own turn. Third in line, the requested module’s own body starts, and that body makes two more requests: the relative request runs pkg.sub.tool, the absolute one runs pkg.tool.

Module objects created: 5, kept: 5 — all stay registered in the cache. The third line pays this off: when the same submodule is requested a second time, bodies run: 0. The second request creates no new object, runs no body, only hands back the cached record.

The measure that follows is: a package request’s cost does not end at the requested module. Every level in the chain, and every request made in those levels’ bodies, enters the same bill. Whatever is inside a package’s __init__.py file gets paid by anyone requesting any module belonging to that package.

Same Name, Two Objects

The middle section is the lesson’s central measure. Inside pkg/sub/user.py, the same short name is written twice; the relative notation gives pkg.sub.tool, the absolute notation gives pkg.tool, and the is comparison is False. Two names, two separate objects. Created: 2, shared: 0.

The distinction sits entirely in the starting point. Relative notation starts the name from the requesting module’s package; because user sits inside pkg.sub, it finds its neighbor. Absolute notation starts the name from the search path and finds the package at the root. The short name written in the source files is the same in both; only the notation itself makes the difference.

There is a reverse side to this too, and it follows from the rule the previous lesson measured: a cache record is kept by name, not by file. If the same source is requested under two separate names, two separate records open, and two separate module objects get created — the exact situation measured in the previous lesson when a cache record was removed. Because those two objects’ dictionaries are independent, classes defined inside them are separate objects too; an instance produced from one does not give the expected answer when tested against the other’s class. Being reachable both by package name and directly is therefore not a convenience for the same tree, it is a namespace split in two.

This has a consequence on the reading side: seeing only the short name in a file does not say which module is meant. The same line’s meaning depends on which package the file sits in. When the file is moved to a different package, a relative request silently resolves to a different object, or does not resolve at all; an absolute request is unaffected by the move. The choice between the two notations is not a matter of style, it is a decision about which context the name binds to.

Which Name Gets Bound

The last section gives the two notations’ name-binding difference. The dotted notation only binds the name pkg on the requesting side — not the submodule’s name. The submodule is reached from that name with a dot, because every level of the chain is bound as an attribute one level up. The from notation, by contrast, only binds the name tool; the package’s name never appears on the requesting side.

Even though the two notations bind different names, they reach the same object: the is comparison is True. New objects created: 0; both share the single record in the cache. The next line shows the package side of this — the name sub in the parent package’s dictionary and the pkg.sub record in the cache are the same object, and the same holds one level below too.

The next line is relative notation’s boundary. A relative request in a module that does not belong to a package results in ImportError. The reason is clear: a relative name starts from the requesting module’s package; for a module with no package, there is no place to start from. This is a situation that can come up depending on how a file is run, and it is measured in this topic’s last lesson.

The Cost of Eager Importing

The last table compares the same request in the two package notations, and the difference is born from a single line. In the plain package, requesting a single leaf runs 2 bodies: the package itself and the requested leaf. In eager notation, the same request runs 4 bodies — the package’s body, and all three leaves it requests from within itself. The requested leaf is already among those three; looking at the order, eager.x ran during the package’s own body, and the outside request found it already sitting in the cache once its turn came.

The numbers read as a ratio: the user requested one leaf, paid for three. This ratio grows as the leaf count grows; in a ten-leaf package, a single leaf request runs all eleven. Module objects created: 2 in plain notation, 4 in eager; kept is the same, because all of them stay in the cache.

This is not a defect, it is a trade-off, and both sides of it are measurable. Eager notation gives the user a short name and presents the package as a single surface; its cost is that it makes anyone requesting any part of the package pay for every part’s body. Plain notation leaves the cost to the requester; its cost is that it requires the user to know what is in which submodule. What decides the choice is whether the package’s parts are used together or separately.

Summary

• A package is a directory that gives a module object together with __init__.py; a dotted name resolves left to right, every level is a separate object, and every level is bound as an attribute one level up.
• A single submodule request ran 5 bodies and wrote 5 module objects to the cache; the same module’s second request ran 0 bodies.
• The same short name gave pkg.sub.tool in relative notation, pkg.tool in absolute notation, and is came out False; the only thing making the difference is where the name starts from.
• Dotted notation binds only the root package’s name, from notation binds only the taken name; both create 0 new objects and share the same module object.
• When __init__.py eagerly imports its submodules, a single-leaf request runs 4 bodies instead of 2; the user requests one leaf, pays for three.
• A relative import in a module that does not belong to a package gives ImportError, because there is no package for the name to start from.

Next Step

Every package measured up to here was produced for the measurement: their bodies were one line, their job was counting themselves. Yet there is a large set of packages present in every run, requiring no installation, that comes bundled with the language itself. The next lesson looks at that set’s scope and measures it with a single question: writing the same job with a standard-library module versus writing it by hand — how many objects and how many lines differ, and under which conditions does the hand-written version silently diverge?