--- title: 'Large File Storage' source: 'https://academia.sh/en/courses/advanced-git/large-file-storage' course: 'Advanced Git' language: en updated: '2026-08-17T18:10:42+00:00' license: 'CC BY-SA 4.0' --- # Large File Storage What weighs down history is not commit count but object type: fewer than a tenth of a thousand commits — 91 binary commits — produce 0.6777 of the total objects; the clone is 5640 objects and drops to 1818 without binaries. In the previous lesson, object count tracked commit count: a three-hundred-commit submodule brought in roughly five times more objects than an eighty-commit one. The ratio was nearly one-to-one because every commit left a record of the same weight. This lesson looks at the case where that intuition breaks. In the same history, a small minority of commits can produce the majority of objects, and then the question "how big is the history?" gets the wrong answer if it is answered with commit count. Our question is: is the thing that weighs down a history how many commits were written, or **what type of object** those commits produce? Storage cost itself is not discussed here — that axis was established in the Relational Database Management and Cloud Storage and Data courses. The only thing measured here is **object share in history**. ## Why a Binary Asset Is a Separate Class Version control can store text files as a **diff**: the lines that change between two versions are much smaller than the whole file. This is what keeps a hundred versions of a file from being a hundred copies. This mechanism does not work for binary assets. An image, an audio recording, or a compiled output is a compressed byte sequence; when a single field in it changes, the entire byte sequence changes. There is no line structure, so there is no line diff either. The result: **every version of a binary asset is a new, whole object in history.** The shared setup models this directly — a commit carrying a binary asset writes **42** objects against a text commit's **2**. This accumulation has an irreversible side. Once a binary asset is committed, it enters history, and deleting it in a later commit does not remove it from history; it only removes it from the working tree. Everyone who clones keeps getting that object, because a clone fetches not the working tree but **history**. ## The Pointer Scheme **Large file storage** is a filter scheme that moves a binary body outside of history. A file pattern in the repository attributes is bound to a filter name; the filter works in both directions. The **clean filter** engages while a file is being committed, writes the body to a separate store, and puts a small **pointer file** in history in the body's place. The **smudge filter** engages while a file is being written to the working tree, reads the pointer, and fetches the body from the separate store. ```text # attributes file and pointer content — example dump, not executed # .gitattributes *.bin filter=large-file -text *.png filter=large-file -text # pointer file that enters history (not the body) version https:///v1 oid sha256: size 41943040 ``` What stands in history is this three-line text. The `-text` attribute marks the file as binary and stops line-ending conversion from being applied to it — that distinction is measured in this topic's fourth lesson. The filter's name is not a product name, it is a name defined in configuration; the same pattern could be bound to another filter. The scheme's one real gain is this: **the clone now fetches not every version, but only the pointers.** The body is requested for, and only for, the version that actually needs to be written to the working tree. ## The Measurement's Assumptions - **LR7** — The measurement uses the shared setup's four scales as they are: **50, 200, 1000, 4000** commits. Commits carrying a binary asset come from the setup's own distribution; they are not selected. - **LR8** — A text commit writes **2** objects (tree and blob). A commit carrying a binary asset adds **40** objects to these; this is that version of the body's counterpart in history. - **LR9** — In the pointer scheme, the binary body moves outside of history and a **1** object pointer blob is left in its place; a binary commit writes **3** objects. - **LR10** — "Separate store" is outside the measurement. The only thing counted is **the object in history**; how much room the bodies take up outside is not this measure's subject. - **LR11** — A single binary version's body is required to be able to work after a clone; the measurement adds this on top of the clone as **40** objects. - **LR12** — The "without binaries" column gives the state where commits carrying a binary asset were **never written**; this is not a cleanup result, it is the comparison baseline. - **LR13** — Cost is counted in the **touched object** unit. Bytes, disk, and network duration are outside the measurement. ## Measurement ```python """Large file storage: is what weighs down history commit count or object type. Part 1 - binary-commit share and object share at four scales. Part 2 - the same history with the pointer scheme: where does clone cost land. """ SEED = 20260814 FILES = ("metrics.py", "report.py", "identity.py", "config.py", "document.md") SCALES = (50, 200, 1000, 4000) POINTER = 1 # pointer blob: the single object that replaces the binary body BINARY_BODY = 40 # a binary version's objects in the shared setup def rng(seed): d = seed % 2147483646 + 1 def draw(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return draw def history(n, seed=SEED): """n-commit linear history; each commit touches one file.""" draw, commits = rng(seed), [] for i in range(n): file = FILES[draw(5)] binary = draw(11) == 0 commits.append({"no": i + 1, "file": file, "binary": binary, "objects": 2 + (40 if binary else 0)}) return commits def clone_cost(t, include_binary=True): return sum(x["objects"] for x in t if include_binary or not x["binary"]) def pointer_cost(t): """The binary body moves outside; a single pointer blob is left in history.""" return sum(2 + (POINTER if x["binary"] else 0) for x in t) def working_copy(t): """The single binary version needed to work after cloning.""" return BINARY_BODY if any(x["binary"] for x in t) else 0 print("commits binary commits binary share clone objects no binaries binary objects object share") for n in SCALES: t = history(n) binary = sum(1 for x in t if x["binary"]) full, without = clone_cost(t), clone_cost(t, False) print(f"{n:6d} {binary:14d} {binary / n:12.4f} {full:14d} {without:11d} " f"{full - without:14d} {(full - without) / full:12.4f}") print() print("commits clone in history pointer scheme + one version gained objects ratio") for n in SCALES: t = history(n) full, pointer = clone_cost(t), pointer_cost(t) works = pointer + working_copy(t) print(f"{n:6d} {full:16d} {pointer:14d} {works:13d} {full - works:16d} " f"{full / works:5.2f}") print() n = 1000 t = history(n) binary = sum(1 for x in t if x["binary"]) full, without = clone_cost(t), clone_cost(t, False) print(f"history {n} commits, carrying a binary asset {binary}, total objects {full}, " f"binary share {(full - without) / full:.4f}") print(f"if {n - binary} text commits were deleted, objects gained {without}; " f"if {binary} binary commits were deleted, objects gained {full - without}") ``` ``` commits binary commits binary share clone objects no binaries binary objects object share 50 7 0.1400 380 86 294 0.7737 200 13 0.0650 920 374 546 0.5935 1000 91 0.0910 5640 1818 3822 0.6777 4000 365 0.0912 22600 7270 15330 0.6783 commits clone in history pointer scheme + one version gained objects ratio 50 380 107 147 233 2.59 200 920 413 453 467 2.03 1000 5640 2091 2131 3509 2.65 4000 22600 8365 8405 14195 2.69 history 1000 commits, carrying a binary asset 91, total objects 5640, binary share 0.6777 if 909 text commits were deleted, objects gained 1818; if 91 binary commits were deleted, objects gained 3822 ``` ## Object Count Does Not Track Commit Count The third row pays off the topic's fourth reading. In a thousand-commit history, the number of commits carrying a binary asset is **91** — **0.0910** of commits, fewer than a tenth. The same 91 commits produce **0.6777** of the total objects. The remaining **909** commits — close to nine-tenths of history — write only a third of the objects. The last line sharpens this with a thought experiment. If all nine hundred nine text commits were deleted, **1818** objects would drop from history. If the ninety-one binary commits were deleted, **3822** objects would drop — more than twice as many, with fewer than a tenth of the commits. **What weighs down history is not commit count, it is object type.** This is the exact opposite of the situation in this topic's first lesson: there, what determined cost was history's length; here, it is its composition. The top table also carries a warning. Object share comes out to **0.7737**, **0.5935**, **0.6777**, and **0.6783** at the four scales; there is no orderly trend. The reason is on the left side of the table, not the right: in a fifty-commit history, the number of binary commits is **7**, and the smallest measurable difference at this scale is **1/50 = 0.0200**. A single commit's position moving swings the share by two points. At a thousand and four thousand commits, the share settles between **0.6777** and **0.6783**; this is the real value, and the spread at small scale does not lend itself to drawing a conclusion from it. ## What the Pointer Scheme Pays The lower table's unit is again **touched object**. In a thousand-commit history, the clone drops from **5640** objects to **2091**; once the single binary version needed to work is added, **2131**. The gained objects are **3509**, the ratio **2.65**. At four thousand commits, the gain climbs to **14195** objects, and the ratio stays at **2.69** — the scheme's gain grows with scale, but its **ratio** stays fixed, because the gain is directly tied to binary commit share. The two-hundred-commit row comes out below the ratio: **2.03**. The reason is written in the top table — at that scale, binary commit share is at its lowest value, **0.0650**. The ratio is directly a function of this share and is not an independent metric. The same scheme gains **nothing** in a repository carrying no binary assets at all; every number that looks like a gain is, in fact, measuring that repository's binary share. Where the gain does not come from needs attention. The pointer scheme removes no commit; history is still a thousand commits. What it removes is **every binary version's body**, and these bodies do not vanish, they only move outside of history, into a separate store. The scheme's cost is there too: the cloning side now has to reach two sources at once, and if it cannot reach the second source, its history stands complete but it cannot build the working tree. This is another form of the dependency measured in submodules — there, what was missing was a history; here, a body store. There is an axis the measure does not see, and it needs to be named. The object unit does not ask how many **bytes** a body is; in the setup, a binary version is **40** objects, and this number is independent of the body's size. In a real repository, a large body and a small body produce the same object count but do not cost the same. This lesson measures object share; the byte axis is outside the measure, and no disk conclusion can be drawn from the table. A second limit: the scheme delivers its full gain only when set up **from the start**. If the filter is defined today, today's commits onward have their bodies moved out; the **3822** objects already standing in history stay in place, and every clone keeps fetching them. ## The Binary Cannot Be Merged Alongside object count, there is a second consequence, and it is paid not in history but in work. When two branches write separate versions of the same binary asset, a merge conflict is born and the conflict **cannot be resolved**: a merge works by blending two texts' lines, and a binary body has no lines to blend. The only decision left is choosing one of the two versions. This is why binary patterns are also given the `-merge` attribute; the attribute stops the tool from trying to blend and asks for a direct choice instead. The `-diff` attribute is the display side of the same reasoning: comparing two binary bodies line by line has no meaning, the output is unreadable. How much unnecessary work is born when these two attributes are not set is measured in this topic's fourth lesson. ## Narrowing the Clone Itself The pointer scheme moves the object outside of history. Two more schemes enter the same cost from a different angle, both leaving the object in place and changing **when it gets fetched**. ```text # taught commands — example dump, not executed git clone --filter=blob:none git clone --depth 1 git sparse-checkout set components/metrics-core ``` A **partial clone** does not fetch blobs at clone time; commit and tree objects arrive, and blobs are only requested once a file is actually read. A **shallow clone** cuts history at a given depth and never fetches older commits at all. **Sparse checkout** looks not at the object store but at the working tree: history arrives complete, and only the selected paths are written to the working tree. What all three share is that they are a deferral: the object is still in history and is requested when needed. The pointer scheme is different from this — it **removes** the body from history. This lesson measures the second; the first three could be measured with the same unit but need a different setup, because their gain is tied not to the repository but to the access pattern. A shallow clone has a separate limit of its own too: bisect and blame run over the truncated history never see the cut range at all. ## Retroactive Cleanup and Recovery Removing binary bodies from an existing history has only one way: **rewriting history.** This is the operation measured in this topic's first lesson, and its rules apply here too. The operation changes every binary commit's identity and the identity of every commit after it; in a thousand-commit history, if the first binary commit is at an early point, **almost every identity** changes. **It cannot be undone:** old identities no longer point to anything; every tag, every pinned version, and every external reference relying on those identities breaks. Every copy that has cloned the repository has to be **re-cloned**; a single branch pushed from the old copy brings the deleted bodies back, and the operation has to be redone from scratch. The command's full form is not given here, because it gives no warning of loss when run with the wrong range. Recovery paths are set up beforehand, not afterward. **First:** before the operation is run, a complete copy of the repository is kept in a separate place; as long as this copy is not synchronized with the new history, it is the sole source of the old identities. **Second:** the transformation is tried in this separate copy first, and the result is counted — how many commits' identities changed, how many objects dropped. **Third:** after a locally mistaken transformation, the reflog returns old tips before the garbage-collection window closes. Once the window closes, this path closes too. **Fourth, and cheapest:** write the pattern into the attributes file on the repository's first day. The gain is read in the table; its cost is only that it was not paid from the start. ## Summary - A binary asset does not lend itself to a line diff; every version accumulates in history as a new, whole object, and deleting it after it is committed does not remove it from history. - In a thousand-commit history, **91** commits carry a binary asset — **0.0910** of commits — and produce **0.6777** of total objects; the clone is **5640** objects, **1818** without binaries. - If **909** text commits were deleted, **1818** objects would drop; if **91** binary commits were deleted, **3822** would: what weighs down history is not commit count, it is object type. - The pointer scheme drops the clone from **5640** to **2131** objects at a thousand commits (a **2.65** ratio) but removes no commit; bodies move to a separate store and a second access condition is born. - Retroactive cleanup rewrites history, **cannot be undone**, and requires every copy to be re-cloned; the safe path is trying it in a separate copy and writing the pattern on the repository's first day. ## Next Step The pointer scheme dropped clone cost to a third, but the table still says **clone**: the cost is paid again every time. When it becomes necessary to work on two branches of the same repository together, the usual solution is a second clone, and that clone copies every object all over again. The next lesson questions this assumption: can a second working area be set up that shares the same repository's object store, and if so, how many objects go uncopied compared to a clone per branch?