Submodules

Up to this point, history was inside a single repository. Bisect, blame, and rewriting were measured; object count stood behind these measurements — the variable that determined cost, not the thing being measured itself.

In this topic, the background moves to the foreground. When a repository grows and pieces too large to fit in a single history start being carried, the question asked changes: what happens when the object count itself becomes the work? The first case is one repository containing another repository’s history inside it. This lesson’s question is not how a submodule is set up — setup is two commands. The question is: how many separate histories does a single identity written to the main repository bind a commit in that repository to, and how does clone cost respond to that?

What the Main Repository Records

A submodule is another repository that stands in a repository’s working tree and has its own history. The main repository does not copy its files into its own object store; all it records is which commit the submodule stands at. This record is called the pinned version and it stands in the main repository’s tree as a single directory entry.

Two files are the whole job. .gitmodules declares the submodule’s name and where it is fetched from; the tree entry holds the pinned version’s identity. The first is a tracked text file, the second is a field rewritten at every commit.

# taught commands and file content — example dump, not executed

git submodule add <submodule-address> components/metrics-core
git submodule update --init --recursive
git clone --recurse-submodules <main-repository-address>

# .gitmodules
[submodule "components/metrics-core"]
	path = components/metrics-core
	url = <submodule-address>

The --recurse-submodules option does not finish cloning with the main repository: it clones each submodule separately as well. A clone without the option leaves submodule directories empty — the main repository’s history arrives complete, but the working tree does not work. update --init does the same job afterward.

The Pinned Version’s Two Faces

The pinned version is cheap inside the main repository and expensive outside it, and the lesson’s entire tension is here.

The cheap face: advancing the submodule is a single commit in the main repository. The identity in the tree entry changes, and a new tree plus a blob carrying the message are written — two objects. However many commits have piled up in the submodule during that interval, the object count written to the main repository is independent of it.

The expensive face: those two objects bind a commit in the submodule’s history to the main repository’s history. Reproducing a commit in the main repository now requires reaching not one history, but the main repository plus every submodule’s history at once. In a repository with three submodules, building a single commit’s tree requires reading four identities from four separate histories.

This second face is something measurable, and the measurement separates two questions: how many objects are written in the main repository, and how much history that write binds.

It Tracks a Commit, Not a Branch

The pinned version is a commit identity, not a branch name. This has a direct consequence: once a submodule directory is cloned, it does not stand on any branch — it is in a detached-HEAD state. Detached HEAD was established in the Branching and Collaboration course in the context of a commit falling outside history; here it is not an accident, it is the submodule’s normal state. The main repository cannot pin a branch, because a branch moves, and a moving reference cannot be a pinned version.

# taught commands — example dump, not executed

git -C components/metrics-core checkout main
git -C components/metrics-core pull
git add components/metrics-core
git commit -m "Advance metrics-core pinned version"

git submodule update --remote components/metrics-core
git submodule status

Advancing is four steps, and all four are done by hand: a branch is switched to inside the submodule, new commits are fetched, and in the main repository the submodule directory is staged and committed. If the third step is skipped, the submodule directory shows a new commit but the main repository still records the old one; status output reports this, history does not. The --remote option automates the first two steps according to the branch written in .gitmodules, but the third step is still done by whoever writes it: there is no advance that is not written to the main repository.

This is where a submodule differs from a dependency manager. A dependency manager declares a range and resolves it itself; a submodule declares a single identity and there is no such thing as resolution. In exchange, a submodule brings the full history: every commit up to the pinned version is there, readable, and searchable.

The Measurement’s Assumptions

• LR1 — The main repository is the shared setup’s 200-commit scale; each of the three submodules is an independent history produced from the same setup with a separate seed. Submodule histories share no object with the main repository.
• LR2 — The main-repository commits that advance the pinned version are picked from a separate generator; each advance consumes between 1 and 20 submodule commits and does not go past the submodule’s end.
• LR3 — The commit that advances the pinned version writes 2 objects to the main repository: the new tree and the blob carrying the message. This number is independent of how many commits were passed in the submodule.
• LR4 — “Covered commits” is the number of submodule-history commits consumed up to the main repository’s tip; it is not visible in the main repository’s history, but is required to reproduce the state at the main repository’s tip.
• LR5 — “Ids needed” is the number of identities that must be read to build a single main-repository commit’s tree: the main repository’s own identity plus every submodule’s pinned version.
• LR6 — A recursive clone fetches every submodule’s full history. Cost is counted in the touched object unit; network, wall-clock time, and disk are outside the measurement.

Measurement

"""Submodules: the history the pinned version binds, and the clone cost.

Part 1 - each submodule's advance count and clone objects.
Part 2 - as submodule count grows: bound history and recursive clone.
"""
SEED = 20260814
FILES = ("metrics.py", "report.py", "identity.py", "config.py", "document.md")
MAIN_SCALE = 200
SUBMODULES = (("metrics-core", 300, SEED + 1),
              ("report-format", 150, SEED + 2),
              ("identity-library", 80, SEED + 3))


def rng(seed):
    d = seed % 2147483646 + 1

    def draw(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return draw


def history(n, seed=SEED):
    """n-commit linear history; each commit touches one file."""
    draw, commits = rng(seed), []
    for i in range(n):
        file = FILES[draw(5)]
        binary = draw(11) == 0
        commits.append({"no": i + 1, "file": file, "binary": binary,
                         "objects": 2 + (40 if binary else 0)})
    return commits


def clone_cost(t, include_binary=True):
    return sum(x["objects"] for x in t if include_binary or not x["binary"])


def advances(main, sub, seed):
    """Which main-repository commits advance the pinned version."""
    draw, record, position = rng(seed), [], 0
    for x in main:
        advance = draw(12) == 0
        step = 1 + draw(20)
        if advance and position < len(sub):
            position = min(position + step, len(sub))
            record.append({"main_no": x["no"], "pinned": position})
    return record


main = history(MAIN_SCALE)
print(f"main repository {len(main)} commits, clone objects {clone_cost(main)}, "
      f"without binaries {clone_cost(main, False)}")
print()
print("submodule            commits  advances  covered commits  objects in main"
      "  clone objects")
records = []
for name, n, seed in SUBMODULES:
    sub = history(n, seed)
    a = advances(main, sub, seed + 100)
    records.append((name, sub, a))
    print(f"  {name:18s} {n:6d} {len(a):10d} {a[-1]['pinned']:16d} "
          f"{len(a) * 2:17d} {clone_cost(sub):13d}")
print()
print("submodule count  bound history  ids needed  main clone objects"
      "  recursive clone objects   ratio")
for k in range(len(SUBMODULES) + 1):
    recursive = clone_cost(main) + sum(clone_cost(a) for _, a, _ in records[:k])
    print(f"{k:16d} {k + 1:14d} {k + 1:11d} {clone_cost(main):18d} "
          f"{recursive:25d} {recursive / clone_cost(main):5.2f}")
print()
total_advances = sum(len(a) for _, _, a in records)
total_covered = sum(a[-1]["pinned"] for _, _, a in records)
print(f"main repository commits that advance a pinned version {total_advances}, "
      f"objects written to main {total_advances * 2}, "
      f"submodule commits bound {total_covered}")
print(f"main repository history shows {len(main)} commits; "
      f"commits covered {len(main) + total_covered}")

main repository 200 commits, clone objects 920, without binaries 374

submodule            commits  advances  covered commits  objects in main  clone objects
  metrics-core          300         18              161                36          2200
  report-format         150         14              137                28           980
  identity-library       80          6               80                12           440

submodule count  bound history  ids needed  main clone objects  recursive clone objects   ratio
               0              1           1                920                       920  1.00
               1              2           2                920                      3120  3.39
               2              3           3                920                      4100  4.46
               3              4           4                920                      4540  4.93

main repository commits that advance a pinned version 38, objects written to main 76, submodule commits bound 378
main repository history shows 200 commits; commits covered 578

The Bound History

The last two lines give the lesson’s core. In the main repository, 38 commits advance a pinned version, and these commits write 76 objects total to the main repository — a small share of a two-hundred-commit history’s 920 objects. The same 38 commits bind 378 commits from the submodule histories to the main repository.

The ratio reads directly: every 2 objects written to the main repository bring in an average of ten submodule commits. This is the asymmetry the lesson measures. Someone looking at the main repository’s history sees 200 commits; the number of commits needed to reproduce the state at the main repository’s tip is 578. The visible history and the covered history are not the same thing, and the gap is not written anywhere in the main repository.

This asymmetry has a diagnostic consequence. A search run in the main repository scans only the main repository’s 200 commits; the submodules’ 378 covered commits are outside that search. If a defect went in through a submodule, a search in the main repository cannot find it — the narrowest range it can find is the pinning commit that brought the defect in. The search’s second round has to be rebuilt in the submodule’s own history. Even if the range narrowing in the main repository is two commits, the number of submodule commits packed between those two commits can climb as high as twenty.

The second table’s first two columns say the same thing in terms of identity. With no submodules, building a commit’s tree requires 1 identity and looks at 1 history. With three submodules, this number climbs to 4. All four have to be found: if one of the submodules is unreachable, the commit in the main repository stands complete but cannot be reproduced. The main repository’s history is not corrupted; the history that is missing is a different one.

Adding Up the Clone Cost

The last three columns of the lower table give the clone cost, and the unit is touched object.

The main repository alone is 920 objects. The submodules bring 2200, 980, and 440 objects respectively. Once all three are added, the recursive clone climbs to 4540 objects — 4.93 times the main-repository clone. The main repository’s history did not grow by even one commit in the process.

The largest contribution comes from the first submodule, and the reason is written in the table: its 300-commit history is larger than the main repository’s. The submodule’s length has nothing to do with the main repository’s length; the main repository does not look at the submodule’s size when it writes the identity that pins it. Nothing stands in the way of a setup where the main repository is small and the clone cost is large.

The ratio column on the right does not climb linearly: 3.39, then 4.46, then 4.93. The increase shrinks at each step because the submodules being added are progressively shorter. If the order were reversed, the numbers would look different, but the last row would stay the same — the total is independent of order. The ratio itself does not say anything by itself; what says something is that the sum of the submodule histories is more than three times the main repository’s history. This ratio does not show up in any measurement that looks at the main repository alone.

Recursion is not limited to one layer either. If a submodule has its own submodule, the --recursive option fetches that too, and the number of bound histories and the number of identities needed climb to five. The measurement does not build this, but the scheme’s rule does not change: every layer writes two objects to the main repository and binds the whole of its own history behind it.

Two practical conclusions follow from this. First: a non-recursive clone fetches the main repository’s history complete and fetches none of the submodules’ objects; this is the right choice for someone who wants to read the history. Second: someone who needs the working tree has to do a recursive clone and pays the ratio read from the table. Two separate costs exist in the same repository, and which one is paid is determined by the option used.

The Broken Link and Recovery

A pinned version is an identity, and an identity is valid as long as the commit it points to is reachable. If history is rewritten in the submodule’s upstream repository — the operation measured in this topic’s first lesson — old identities no longer point to any commit. How many of the main repository’s 38 advance commits break depends on which point the rewrite starts from.

This is not a data loss, but it makes past points in the main repository’s history unreproducible, and the record of that loss is not kept in the main repository. Recovery has three paths. If the submodule’s old commits still stand somewhere — in a copy, in a backup branch, or if the reflog window has not closed — those commits can be brought back from a separate reference and identities resolved again. If they still stand. The second path is writing a fix-up commit in the main repository that moves pinned versions to the new identities; this does not recover the history, it only repairs the tip. The third and cheapest path is never incurring the loss at all: before history is rewritten in a repository connected as a submodule, the main repositories that pin it need to be known. The link’s direction is one-way — a submodule does not know who pins it.

Summary

• A submodule writes an identity to the main repository, not a history; the record consists of the .gitmodules file and the pinned version in the tree entry.
• Advancing the pinned version writes 2 objects to the main repository, and this number is independent of how many commits were passed in the submodule; in the measurement, 76 objects bind 378 submodule commits.
• The main repository’s history shows 200 commits; the number of commits needed to reproduce the state at its tip is 578, and this gap is not written in the main repository.
• With three submodules, building a single commit’s tree requires 4 identities from 4 separate histories; if one is unreachable, the commit does not corrupt but cannot be reproduced.
• A recursive clone climbs from 920 objects to 4540 — a 4.93 ratio; a non-recursive clone fetches the main repository’s history complete and takes 0 objects from a submodule.

Next Step

In submodules, the thing that grew cost was commit count: a three-hundred-commit history brought in five times more objects than an eighty-commit one. This is a case where object count follows commit count, and it matches intuition. The next lesson looks at the case where intuition breaks: in the same history, when fewer than a tenth of the commits produce more than two-thirds of the total objects, the question of what weighs the history down has to be asked again — and the answer is not commit count. In that case, splitting the repository is no remedy, because the weight does not come from how commits are distributed but from the type of object they carry; what needs to come out of history is not a repository, but a class of file.