Large File Storage

In the previous lesson, object count tracked commit count: a three-hundred-commit submodule brought in roughly five times more objects than an eighty-commit one. The ratio was nearly one-to-one because every commit left a record of the same weight.

This lesson looks at the case where that intuition breaks. In the same history, a small minority of commits can produce the majority of objects, and then the question “how big is the history?” gets the wrong answer if it is answered with commit count. Our question is: is the thing that weighs down a history how many commits were written, or what type of object those commits produce? Storage cost itself is not discussed here — that axis was established in the Relational Database Management and Cloud Storage and Data courses. The only thing measured here is object share in history.

Why a Binary Asset Is a Separate Class

Version control can store text files as a diff: the lines that change between two versions are much smaller than the whole file. This is what keeps a hundred versions of a file from being a hundred copies.

This mechanism does not work for binary assets. An image, an audio recording, or a compiled output is a compressed byte sequence; when a single field in it changes, the entire byte sequence changes. There is no line structure, so there is no line diff either. The result: every version of a binary asset is a new, whole object in history. The shared setup models this directly — a commit carrying a binary asset writes 42 objects against a text commit’s 2.

This accumulation has an irreversible side. Once a binary asset is committed, it enters history, and deleting it in a later commit does not remove it from history; it only removes it from the working tree. Everyone who clones keeps getting that object, because a clone fetches not the working tree but history.

The Pointer Scheme

Large file storage is a filter scheme that moves a binary body outside of history. A file pattern in the repository attributes is bound to a filter name; the filter works in both directions. The clean filter engages while a file is being committed, writes the body to a separate store, and puts a small pointer file in history in the body’s place. The smudge filter engages while a file is being written to the working tree, reads the pointer, and fetches the body from the separate store.

# attributes file and pointer content — example dump, not executed

# .gitattributes
*.bin  filter=large-file -text
*.png  filter=large-file -text

# pointer file that enters history (not the body)
version https://<filter-spec>/v1
oid     sha256:<body's digest>
size    41943040

What stands in history is this three-line text. The -text attribute marks the file as binary and stops line-ending conversion from being applied to it — that distinction is measured in this topic’s fourth lesson. The filter’s name is not a product name, it is a name defined in configuration; the same pattern could be bound to another filter.

The scheme’s one real gain is this: the clone now fetches not every version, but only the pointers. The body is requested for, and only for, the version that actually needs to be written to the working tree.

The Measurement’s Assumptions

• LR7 — The measurement uses the shared setup’s four scales as they are: 50, 200, 1000, 4000 commits. Commits carrying a binary asset come from the setup’s own distribution; they are not selected.
• LR8 — A text commit writes 2 objects (tree and blob). A commit carrying a binary asset adds 40 objects to these; this is that version of the body’s counterpart in history.
• LR9 — In the pointer scheme, the binary body moves outside of history and a 1 object pointer blob is left in its place; a binary commit writes 3 objects.
• LR10 — “Separate store” is outside the measurement. The only thing counted is the object in history; how much room the bodies take up outside is not this measure’s subject.
• LR11 — A single binary version’s body is required to be able to work after a clone; the measurement adds this on top of the clone as 40 objects.
• LR12 — The “without binaries” column gives the state where commits carrying a binary asset were never written; this is not a cleanup result, it is the comparison baseline.
• LR13 — Cost is counted in the touched object unit. Bytes, disk, and network duration are outside the measurement.

Measurement

"""Large file storage: is what weighs down history commit count or object type.

Part 1 - binary-commit share and object share at four scales.
Part 2 - the same history with the pointer scheme: where does clone cost land.
"""
SEED = 20260814
FILES = ("metrics.py", "report.py", "identity.py", "config.py", "document.md")
SCALES = (50, 200, 1000, 4000)
POINTER = 1        # pointer blob: the single object that replaces the binary body
BINARY_BODY = 40   # a binary version's objects in the shared setup


def rng(seed):
    d = seed % 2147483646 + 1

    def draw(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return draw


def history(n, seed=SEED):
    """n-commit linear history; each commit touches one file."""
    draw, commits = rng(seed), []
    for i in range(n):
        file = FILES[draw(5)]
        binary = draw(11) == 0
        commits.append({"no": i + 1, "file": file, "binary": binary,
                         "objects": 2 + (40 if binary else 0)})
    return commits


def clone_cost(t, include_binary=True):
    return sum(x["objects"] for x in t if include_binary or not x["binary"])


def pointer_cost(t):
    """The binary body moves outside; a single pointer blob is left in history."""
    return sum(2 + (POINTER if x["binary"] else 0) for x in t)


def working_copy(t):
    """The single binary version needed to work after cloning."""
    return BINARY_BODY if any(x["binary"] for x in t) else 0


print("commits  binary commits  binary share  clone objects  no binaries  binary objects  object share")
for n in SCALES:
    t = history(n)
    binary = sum(1 for x in t if x["binary"])
    full, without = clone_cost(t), clone_cost(t, False)
    print(f"{n:6d} {binary:14d} {binary / n:12.4f} {full:14d} {without:11d} "
          f"{full - without:14d} {(full - without) / full:12.4f}")
print()
print("commits  clone in history  pointer scheme  + one version  gained objects   ratio")
for n in SCALES:
    t = history(n)
    full, pointer = clone_cost(t), pointer_cost(t)
    works = pointer + working_copy(t)
    print(f"{n:6d} {full:16d} {pointer:14d} {works:13d} {full - works:16d} "
          f"{full / works:5.2f}")
print()
n = 1000
t = history(n)
binary = sum(1 for x in t if x["binary"])
full, without = clone_cost(t), clone_cost(t, False)
print(f"history {n} commits, carrying a binary asset {binary}, total objects {full}, "
      f"binary share {(full - without) / full:.4f}")
print(f"if {n - binary} text commits were deleted, objects gained {without}; "
      f"if {binary} binary commits were deleted, objects gained {full - without}")

commits  binary commits  binary share  clone objects  no binaries  binary objects  object share
    50              7       0.1400            380          86            294       0.7737
   200             13       0.0650            920         374            546       0.5935
  1000             91       0.0910           5640        1818           3822       0.6777
  4000            365       0.0912          22600        7270          15330       0.6783

commits  clone in history  pointer scheme  + one version  gained objects   ratio
    50              380            107           147              233  2.59
   200              920            413           453              467  2.03
  1000             5640           2091          2131             3509  2.65
  4000            22600           8365          8405            14195  2.69

history 1000 commits, carrying a binary asset 91, total objects 5640, binary share 0.6777
if 909 text commits were deleted, objects gained 1818; if 91 binary commits were deleted, objects gained 3822

Object Count Does Not Track Commit Count

The third row pays off the topic’s fourth reading. In a thousand-commit history, the number of commits carrying a binary asset is 91 — 0.0910 of commits, fewer than a tenth. The same 91 commits produce 0.6777 of the total objects. The remaining 909 commits — close to nine-tenths of history — write only a third of the objects.

The last line sharpens this with a thought experiment. If all nine hundred nine text commits were deleted, 1818 objects would drop from history. If the ninety-one binary commits were deleted, 3822 objects would drop — more than twice as many, with fewer than a tenth of the commits. What weighs down history is not commit count, it is object type. This is the exact opposite of the situation in this topic’s first lesson: there, what determined cost was history’s length; here, it is its composition.

The top table also carries a warning. Object share comes out to 0.7737, 0.5935, 0.6777, and 0.6783 at the four scales; there is no orderly trend. The reason is on the left side of the table, not the right: in a fifty-commit history, the number of binary commits is 7, and the smallest measurable difference at this scale is 1/50 = 0.0200. A single commit’s position moving swings the share by two points. At a thousand and four thousand commits, the share settles between 0.6777 and 0.6783; this is the real value, and the spread at small scale does not lend itself to drawing a conclusion from it.

What the Pointer Scheme Pays

The lower table’s unit is again touched object. In a thousand-commit history, the clone drops from 5640 objects to 2091; once the single binary version needed to work is added, 2131. The gained objects are 3509, the ratio 2.65. At four thousand commits, the gain climbs to 14195 objects, and the ratio stays at 2.69 — the scheme’s gain grows with scale, but its ratio stays fixed, because the gain is directly tied to binary commit share.

The two-hundred-commit row comes out below the ratio: 2.03. The reason is written in the top table — at that scale, binary commit share is at its lowest value, 0.0650. The ratio is directly a function of this share and is not an independent metric. The same scheme gains nothing in a repository carrying no binary assets at all; every number that looks like a gain is, in fact, measuring that repository’s binary share.

Where the gain does not come from needs attention. The pointer scheme removes no commit; history is still a thousand commits. What it removes is every binary version’s body, and these bodies do not vanish, they only move outside of history, into a separate store. The scheme’s cost is there too: the cloning side now has to reach two sources at once, and if it cannot reach the second source, its history stands complete but it cannot build the working tree. This is another form of the dependency measured in submodules — there, what was missing was a history; here, a body store.

There is an axis the measure does not see, and it needs to be named. The object unit does not ask how many bytes a body is; in the setup, a binary version is 40 objects, and this number is independent of the body’s size. In a real repository, a large body and a small body produce the same object count but do not cost the same. This lesson measures object share; the byte axis is outside the measure, and no disk conclusion can be drawn from the table.

A second limit: the scheme delivers its full gain only when set up from the start. If the filter is defined today, today’s commits onward have their bodies moved out; the 3822 objects already standing in history stay in place, and every clone keeps fetching them.

The Binary Cannot Be Merged

Alongside object count, there is a second consequence, and it is paid not in history but in work. When two branches write separate versions of the same binary asset, a merge conflict is born and the conflict cannot be resolved: a merge works by blending two texts’ lines, and a binary body has no lines to blend. The only decision left is choosing one of the two versions.

This is why binary patterns are also given the -merge attribute; the attribute stops the tool from trying to blend and asks for a direct choice instead. The -diff attribute is the display side of the same reasoning: comparing two binary bodies line by line has no meaning, the output is unreadable. How much unnecessary work is born when these two attributes are not set is measured in this topic’s fourth lesson.

Narrowing the Clone Itself

The pointer scheme moves the object outside of history. Two more schemes enter the same cost from a different angle, both leaving the object in place and changing when it gets fetched.

# taught commands — example dump, not executed

git clone --filter=blob:none <repository-address>
git clone --depth 1 <repository-address>
git sparse-checkout set components/metrics-core

A partial clone does not fetch blobs at clone time; commit and tree objects arrive, and blobs are only requested once a file is actually read. A shallow clone cuts history at a given depth and never fetches older commits at all. Sparse checkout looks not at the object store but at the working tree: history arrives complete, and only the selected paths are written to the working tree.

What all three share is that they are a deferral: the object is still in history and is requested when needed. The pointer scheme is different from this — it removes the body from history. This lesson measures the second; the first three could be measured with the same unit but need a different setup, because their gain is tied not to the repository but to the access pattern. A shallow clone has a separate limit of its own too: bisect and blame run over the truncated history never see the cut range at all.

Retroactive Cleanup and Recovery

Removing binary bodies from an existing history has only one way: rewriting history. This is the operation measured in this topic’s first lesson, and its rules apply here too.

The operation changes every binary commit’s identity and the identity of every commit after it; in a thousand-commit history, if the first binary commit is at an early point, almost every identity changes. It cannot be undone: old identities no longer point to anything; every tag, every pinned version, and every external reference relying on those identities breaks. Every copy that has cloned the repository has to be re-cloned; a single branch pushed from the old copy brings the deleted bodies back, and the operation has to be redone from scratch. The command’s full form is not given here, because it gives no warning of loss when run with the wrong range.

Recovery paths are set up beforehand, not afterward. First: before the operation is run, a complete copy of the repository is kept in a separate place; as long as this copy is not synchronized with the new history, it is the sole source of the old identities. Second: the transformation is tried in this separate copy first, and the result is counted — how many commits’ identities changed, how many objects dropped. Third: after a locally mistaken transformation, the reflog returns old tips before the garbage-collection window closes. Once the window closes, this path closes too. Fourth, and cheapest: write the pattern into the attributes file on the repository’s first day. The gain is read in the table; its cost is only that it was not paid from the start.

Summary

• A binary asset does not lend itself to a line diff; every version accumulates in history as a new, whole object, and deleting it after it is committed does not remove it from history.
• In a thousand-commit history, 91 commits carry a binary asset — 0.0910 of commits — and produce 0.6777 of total objects; the clone is 5640 objects, 1818 without binaries.
• If 909 text commits were deleted, 1818 objects would drop; if 91 binary commits were deleted, 3822 would: what weighs down history is not commit count, it is object type.
• The pointer scheme drops the clone from 5640 to 2131 objects at a thousand commits (a 2.65 ratio) but removes no commit; bodies move to a separate store and a second access condition is born.
• Retroactive cleanup rewrites history, cannot be undone, and requires every copy to be re-cloned; the safe path is trying it in a separate copy and writing the pattern on the repository’s first day.

Next Step

The pointer scheme dropped clone cost to a third, but the table still says clone: the cost is paid again every time. When it becomes necessary to work on two branches of the same repository together, the usual solution is a second clone, and that clone copies every object all over again. The next lesson questions this assumption: can a second working area be set up that shares the same repository’s object store, and if so, how many objects go uncopied compared to a clone per branch?