The Module System

Everything up to this point sat inside a single file. Closures, generators, wrappers, and decorators all stood in the same namespace; a name was visible from where it was defined, and where it was not, scope was the reason. When code spreads across more than one file, the shape of this problem changes: where do names get written now, and by what operation does one file get another’s name?

The answer comes from this course’s measure. Import is not a name-copying operation, it is an act of creation: the first time a file is requested, a module object is created from it, its body runs once, and every module-level name is written into that object’s dictionary. A second request does not create a new object. This lesson’s question is: when the same module is requested more than once, how many objects get created, how many times does the body run, and which record blocks the second run?

What Import Creates

An import request does three things. First, it checks whether the name is registered in the module cache; if it is, the object there is handed back and the job is done. If not, the source is searched for on the search path, a module object is created from the source found, that object is written to the cache first, and then its body is run. In the last step, the name in the request is bound to the module object on the requesting side.

The order detail matters: the object is written to the cache before the body runs. This is why, even if a module’s body indirectly imports another module that requests it back, no infinite loop is born — the second request finds, in the cache, a module whose body has not finished but whose object already exists.

A direct consequence follows: import is not cheap, it is expensive once. Whatever is in the body — class definitions, module-level computations, tables — all of it runs once on the first request, and the results are held in the module object. Subsequent requests do no more work than a dictionary lookup.

Two Notations, the Same Object

Import has two notations, and both create the same module object; where they differ is which name gets bound to what on the requesting side. The first notation binds the module object itself to a name; the second reads a single value from the module’s dictionary and binds that value to a local name. The second does not create a new object — it gives a second name to an object already created.

The observable consequence of the distinction is this: the two names share the object, they do not share the binding. If the name in the module’s dictionary is later bound to a different object, the name taken outside stays on the old object — because that name was written not into the module’s dictionary but into its own. By contrast, if the shared object is mutated in place, both sides see the same change. The measurement’s last section tests these two cases separately.

The Search Path Is a List

For a name not found in the cache, next comes the search path. The search path is a list of directory names, and it is ordered: the first matching source wins, the rest are not looked at. If two sources in two different directories carry the same name, which one gets imported is decided not by the files’ content but by their order in the list.

The practical consequence of the order shows up when a file carrying the same name as a standard-library module sits in a directory near the front of the search path: the request finds that file, and the standard library’s module is never searched for. There is no error message, because the rule worked correctly — the first match won. The sign of a name clash is that the expected name cannot be found in the module; the reason is that the name came from a different source.

These two mechanisms — cache and search path — work together, and which one has the final say first is a measurable question. If we change the search path’s order after a module has been imported, does the next request follow the new order, or does it get the old object from the cache? The measurement answers this.

Where a Module-Level Name Is Written

Scope rules are not this course’s subject; the four levels — local, enclosing, global, built-in — were built in the Python Fundamentals course’s scope lesson and are not repeated here. What is added here is this: of those four levels, the global one is not an abstract space, it is the dictionary of a concrete object. Every name bound at module level is written into that module object’s __dict__ dictionary; the vars built-in gives that same dictionary. Assigning an attribute to a module from outside is also writing into that same dictionary.

This explains why the phrase “global variable” is misleading. Nothing is global; what exists belongs to a module. Two modules’ variables carrying the same name are independent of each other, because they sit in two separate dictionaries.

The measurement’s assumptions:

• MP1 — The measurement leaves no file in the repository. Module sources are produced in a temporary directory, and the directory is deleted when the job ends; the directory’s path is not printed.
• MP2 — An import from the standard library is a real import; the oracle is an is comparison and no identity number is printed.
• MP3 — How many times the body ran is counted by a record the body itself appends to a shared list. The counting side is the measured module itself.
• MP4 — The number of module objects created is found by collecting the objects seen into a list and telling them apart from each other with is.
• MP5 — In the search-path measurement, two separate directories carry two sources with the same name; the two differ only in the value of a module-level name.
• MP6 — After the order is changed, the cache is checked twice: once full, once cleared.
• MP7 — When the measurement ends, the added path entries and cache records are rolled back; the interpreter’s starting state does not carry over into the measurement.
• MP8 — Where a name is written is read with vars; that the result of vars and __dict__ is the same object is also tested with is.
• MP9 — In comparing the two notations, the second notation is run in a separate namespace dictionary; the measurement does not touch the main run’s own names.
• MP10 — Testing the sharing takes two steps: first the object is mutated in place, then the name in the module is bound to a different object. The two steps are read separately.

Measurement

"""Module system: how many objects does import create, how many times does the body run."""

import importlib
import sys
import tempfile
from pathlib import Path

LOG_SOURCE = "RUNS = []\n"
MEASUREMENT_SOURCE = ("import log\n"
                "log.RUNS.append('run')\n"
                "LABEL = 'measurement'\n")
CHOICE_SOURCE = "SOURCE = '{name}'\n"
NAMES = ("log", "measurement", "choice")


def distinct_objects(objects):
    """Counts how many separate objects were seen, without printing identity."""
    distinct = []
    for n in objects:
        if not any(n is f for f in distinct):
            distinct.append(n)
    return len(distinct)


def clear_cache():
    for name in NAMES:
        sys.modules.pop(name, None)


def import_twice():
    import json as first
    import json as second
    return first is second, "json" in sys.modules


def how_many_body_runs(root):
    (root / "log.py").write_text(LOG_SOURCE, encoding="utf-8")
    (root / "measurement.py").write_text(MEASUREMENT_SOURCE, encoding="utf-8")
    sys.path.insert(0, str(root))
    importlib.invalidate_caches()
    seen = [importlib.import_module("measurement")]
    seen.append(importlib.import_module("measurement"))
    log = sys.modules["log"]
    cached_count = len(log.RUNS)
    sys.modules.pop("measurement")
    seen.append(importlib.import_module("measurement"))
    result = (len(seen), distinct_objects(seen), cached_count,
             len(log.RUNS), seen[-1].log is log)
    sys.path.remove(str(root))
    clear_cache()
    return result


def search_path(root):
    for name in ("one", "two"):
        (root / name).mkdir()
        (root / name / "choice.py").write_text(CHOICE_SOURCE.format(name=name),
                                           encoding="utf-8")
    one, two = str(root / "one"), str(root / "two")
    sys.path[:0] = [one, two]
    importlib.invalidate_caches()
    before = importlib.import_module("choice").SOURCE
    sys.path.remove(one)
    sys.path.insert(1, one)
    cached = importlib.import_module("choice").SOURCE
    sys.modules.pop("choice")
    uncached = importlib.import_module("choice").SOURCE
    sys.path.remove(one)
    sys.path.remove(two)
    clear_cache()
    return before, cached, uncached


def module_level_name(root):
    (root / "choice.py").write_text(CHOICE_SOURCE.format(name="one"),
                                  encoding="utf-8")
    sys.path.insert(0, str(root))
    importlib.invalidate_caches()
    m = importlib.import_module("choice")
    same_dict = vars(m) is m.__dict__
    present = "SOURCE" in vars(m)
    m.ADDED = "added"
    from_outside = vars(m)["ADDED"] == "added"
    sys.path.remove(str(root))
    clear_cache()
    return same_dict, present, from_outside


def two_notations(root):
    (root / "choice.py").write_text("SOURCE = ['one']\n", encoding="utf-8")
    sys.path.insert(0, str(root))
    importlib.invalidate_caches()
    m = importlib.import_module("choice")
    namespace = {}
    exec("from choice import SOURCE", namespace)
    shares = namespace["SOURCE"] is m.SOURCE
    m.SOURCE.append("changed")
    saw_mutation = "changed" in namespace["SOURCE"]
    m.SOURCE = ["new"]
    saw_rebinding = namespace["SOURCE"] is m.SOURCE
    sys.path.remove(str(root))
    clear_cache()
    return shares, saw_mutation, saw_rebinding


same, registered = import_twice()
print(f"module imported twice is the same object -> {same}, "
      f"registered in the cache -> {registered}")

with tempfile.TemporaryDirectory() as d:
    requests, objects, cached, after_clear, shared = how_many_body_runs(Path(d))
print()
print(f"import requests {requests} | module objects created {objects} | "
      f"body runs while cached {cached}, after cache cleared {after_clear}")
print(f"log module the two modules see is the same object -> {shared}")

with tempfile.TemporaryDirectory() as d:
    before, cached, uncached = search_path(Path(d))
print()
print(f"search path order one-two -> {before}")
print(f"order two-one, cache full -> {cached}")
print(f"order two-one, cache cleared -> {uncached}")

with tempfile.TemporaryDirectory() as d:
    same_dict, present, from_outside = module_level_name(Path(d))
print()
print(f"vars(m) is m.__dict__ -> {same_dict} | module-level name in the dict "
      f"-> {present} | name written from outside in the same dict -> {from_outside}")

with tempfile.TemporaryDirectory() as d:
    shares, mutation, rebinding = two_notations(Path(d))
print()
print(f"imported name is the same object as the module's -> {shares} | "
      f"saw the object's mutation -> {mutation} | "
      f"saw the rebinding -> {rebinding}")

module imported twice is the same object -> True, registered in the cache -> True

import requests 3 | module objects created 2 | body runs while cached 1, after cache cleared 2
log module the two modules see is the same object -> True

search path order one-two -> one
order two-one, cache full -> one
order two-one, cache cleared -> two

vars(m) is m.__dict__ -> True | module-level name in the dict -> True | name written from outside in the same dict -> True

imported name is the same object as the module's -> True | saw the object's mutation -> True | saw the rebinding -> False

Three Requests, Two Objects, Two Runs

The first line is the shared measurement’s fifth reading: when the same module is requested twice, two names bind to the same object, and that object is registered in the cache. Module objects created: 1, kept: 1, shared by the two names: 1.

The second section shows this bond’s cost. Three import requests were made; module objects created came to 2. Between the first two requests, because the cache was full, the body run count stayed at 1 — the second request did not create a new object, did not rerun the body, only handed back the cached object. Before the third request, the record was removed from the cache; that request found the source again, created a new module object, and ran the body a second time. Body run count went up to 2.

Three separate things are counted here. Module objects created: 2. Kept — that is, registered in the cache at the measurement’s end: 1. And the shared one: the log module. The last line states this: the log imported inside the measurement module’s body and the log the main run gets from the cache are the same object. The measurement module was created twice, but the log both see is singular, because its record was never removed.

The rule that follows is: the count of module objects depends not on the number of files but on how many times the cache record dropped. Two module objects can be created from the same file, and the two objects’ dictionaries are independent of each other — a class defined in one is a separate object from the class defined in the other. Because the two objects were produced from the same source, they carry the same names; the objects those names are bound to, though, are not shared. An identity test gives an unexpected result in this case: two classes coming from the same file, despite having the same name, do not stand in for each other.

Order or Cache

The third section puts the two mechanisms face to face. While directory one sits at the front of the search path, import gives one, as expected. Then the order was changed: two was moved to the front. Because the cache was full, the result did not change — it was still one. Once the cache record was removed, the same request gave two.

This is direct proof that the cache comes before the search path. The search path only decides while the cache has no record; if there is a record, the path is never even read. Changing the path while the program runs has no effect on any module already imported up to that point.

The fourth section settles where the name is written. vars and __dict__ give the same dictionary, the module-level bound name sits there, and an attribute assigned to the module from outside is also written to the same dictionary. This shows that a module’s namespace and a module’s attributes are not two separate things: writing SOURCE = "one" inside the file and writing m.SOURCE = "one" from outside bind the same key in the same dictionary.

The Object Is Shared, the Binding Is Not

The last section gives the two notations’ difference in three values. The imported name and the module’s name are bound to the same object: objects shared: 1, new objects created: 0. The second notation created nothing; it gave a second name to an object already created.

The second value shows the sharing is live. When the module’s object was mutated in place, the imported name saw the change — since they look at the same object, it could not have been otherwise. The third value draws the boundary: when the module’s name was rebound to a different object, the imported name did not see it. That name now sits on the old object and has no link to the module’s dictionary.

Reduced to a single sentence: import shares the object, it copies the name. If a module’s name that will change later needs to be tracked, the module itself is imported and access is made through the module every time; for an object that will not change, the second notation earns a saved lookup each time. The difference between the two is not a matter of style, it is a measured behavioral difference.

Summary

• Import is not a name-copying operation, it is an act of creation: the source is read once, a module object is created from it, and the body runs once.
• The order is cache, search path, body; the object is written to the cache before the body runs, and a module imported twice is the same object.
• Three requests, with the cache cleared once, create 2 module objects and run the body 2 times; without the disruption, both would have stayed at 1.
• The search path’s order only decides while there is no record in the cache: change the order and leave the cache full, the result does not change; clear the record, it does.
• Every module-level name is written into the module object’s __dict__; vars gives the same dictionary, and an attribute assigned from outside goes there too.
• Taking a single name from a module creates 0 new objects and shares the object: the object’s in-place mutation is visible, the module’s name being rebound is not.

Next Step

In the measurement, modules stood flat: all of them sat directly inside one directory on the search path, and their names were single pieces. When code grows, names get split by dots and directories get nested. At that point, the same short name can be found in more than one place — one in a parent directory, another in a subdirectory. The next lesson measures this: when the same short name is called with two requests, one from a neighboring location and one from the root, which object does it resolve to, and how many bodies of the nested directories run?