Bulk History Transformation

The previous lesson kept rewriting on top of a single branch. When the plan stayed close to the tip, the cost was small: a three-commit plan touched six objects. Even when started from the middle of history, only about half of the commits changed identity, and the rest stayed as they were. That half was the deepest point the measurement reached.

Some jobs do not leave even that much untouched. An asset committed in a repository’s early days that should never have been there, an author field misspelled across every commit, a directory that needs to move under the root — all of these are forced to start from the first commit of history. These are called bulk history transformations, and although they come from the same family as the previous lesson’s tool, they operate at a different scale. This lesson’s question is this: what does the cost become when the transformation starts from the first commit, and what is left in the repository’s other copies once the operation is done?

Transformation as a Rule

In interactive mode, the plan was a list: you saw the lines and wrote an action for each one. In bulk transformation there is no plan; there is a rule. The rule is applied to every commit in order, the commit’s tree is run through the rule, and a new commit is produced from the result. The difference between ten commits and a thousand is only duration; the rule written is the same rule.

Four typical requirements fall into this class. Removing an asset: if a binary file or a leaked secret that should never have been committed sits in history, deleting it from the last commit is not enough; it stays readable from the old commits. Metadata correction: if the author or committer field is wrong throughout history. Tree relocation: turning a subdirectory into the repository’s root, or the reverse. Retroactive normalization: applying a line-ending or encoding fix not only to new commits but to old ones as well.

The tool has a subcommand that does this job, and there are also filtering tools that run outside the repository. The command is not given in full form in this lesson. The reason is not pedagogy but responsibility: a copy-and-run bulk-transformation line changes the repository’s history irreversibly and invalidates every copy of it. The dump below carries a placeholder instead of a filter and cannot be run.

# example dump — not executed, the command is not given in full form

$ git filter-branch <filter> -- --all
Rewrite 9c4d7e1 (1/1000) (0 seconds passed, remaining 0 predicted)
Rewrite 4a5b6c7 (2/1000) (0 seconds passed, remaining 0 predicted)
...
Ref 'refs/heads/main' was rewritten

The dump’s counter shows this lesson’s measure directly: the counter runs from one to a thousand, because a thousand of the thousand commits are being regenerated.

The line at the end of the dump says a second thing. Interactive mode operates from a single branch’s tip; bulk transformation can be applied to all references at once, and usually is. The reason is directly tied to the measurement: a branch or a tag left out of the transformation keeps pointing at the old chain, that chain stays reachable, and the asset meant to be removed does not leave the repository. Forgetting a branch gives the same result as not doing the transformation at all. This is why the scope decision comes before the rule itself: which references will enter the transformation, which will be deleted before the transformation runs.

Where the Transformation Starts

The rule is applied to every commit, but identity change does not start at every commit. Commits produced up to the first one where the rule’s content actually changes something come out identical to the old ones and get the same identity. The change starts at that first commit and, by the previous lesson’s rule, runs all the way to the tip.

This reduces the job of removing a file from history to a single question: when was that file first touched. The answer is usually unpleasant. If a repository has only a few files and every commit touches one of them, every file’s first appearance sits near the very beginning of history. The measurement asks this separately for five files and one binary asset.

The measurement’s assumptions:

• RH9 — The setup is the previous lesson’s setup and is unchanged: four scales (50, 200, 1000, 4000), each commit touches one of five files, one in eleven carries a binary asset.
• RH10 — Bulk transformation is a rule applied to every commit. The measurement does not count what the rule does, only how many commits it regenerates; this is why an author correction and a file removal sit in the same column.
• RH11 — Identity change starts at the first commit whose content the rule actually changes, and runs to the tip of history.
• RH12 — Removing a file starts at the first commit that touches that file; for a binary asset, the start is the first commit that carries the binary.
• RH13 — This lesson’s unit of cost is the touched object. The number of commits whose identity changes is the width of the set; the step unit is not used in this lesson.
• RH14 — The shared prefix is the commits the new history keeps in common with the old one: the range up to one before the starting position.
• RH15 — Objects to fetch are the objects the other side does not already hold; every object outside the shared prefix is retransferred. This is not a network measurement, it is an object count.
• RH16 — The set’s resolution at the four scales is 1/50, 1/200, 1/1000, 1/4000 respectively.

Measurement

"""Bulk history transformation: the cost of rewriting from the first commit.

Part 1 - transformation from the first commit at four scales.
Part 2 - which asset is removed determines where the transformation must start.
Part 3 - the prefix the old and new history share.
"""
SEED = 20260814
FILES = ("metrics.py", "report.py", "identity.py", "config.py", "document.md")
BINARY_ASSET = "presentation.bin"
SCALES = (50, 200, 1000, 4000)


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def history(n, seed=SEED):
    """n-commit linear history; each commit touches one file."""
    draw, commits = rng(seed), []
    for i in range(n):
        file = FILES[draw(5)]
        binary = draw(11) == 0
        commits.append({"no": i + 1, "file": file, "binary": binary,
                         "objects": 2 + (40 if binary else 0)})
    return commits


def rewrite(t, position):
    after = [x for x in t if x["no"] >= position]
    return len(after), sum(x["objects"] for x in after)


def first_touch(t, name):
    """The first commit that touches the asset to be removed: the transformation starts there."""
    for x in t:
        if (x["binary"] if name == BINARY_ASSET else x["file"] == name):
            return x["no"]
    return 0


print(f"{'commits':>7s} {'identity changed':>17s} {'touched objects':>16s}"
      f" {'ratio':>6s} {'shared prefix':>14s}")
for n in SCALES:
    t = history(n)
    changed, touched = rewrite(t, 1)
    print(f"{n:7d} {changed:17d} {touched:16d} {changed / n:6.3f}"
          f" {len(t) - changed:14d}")

print()
T = history(1000)
print(f"{'removed asset':<17s} {'first touch':>12s} {'identity changed':>17s}"
      f" {'touched objects':>16s} {'ratio':>6s}")
for name in FILES + (BINARY_ASSET,):
    k = first_touch(T, name)
    changed, touched = rewrite(T, k)
    print(f"{name:<17s} {k:12d} {changed:17d} {touched:16d} {changed / len(T):6.3f}")

print()
print(f"{'starting position':>18s} {'shared prefix':>14s} {'shared objects':>15s}"
      f" {'objects to fetch':>17s}")
for position in (1, 2, 9, 501):
    shared = sum(x["objects"] for x in T if x["no"] < position)
    print(f"{position:18d} {position - 1:14d} {shared:15d}"
          f" {sum(x['objects'] for x in T) - shared:17d}")

commits  identity changed  touched objects  ratio  shared prefix
     50                50              380  1.000              0
    200               200              920  1.000              0
   1000              1000             5640  1.000              0
   4000              4000            22600  1.000              0

removed asset      first touch  identity changed  touched objects  ratio
metrics.py                   1              1000             5640  1.000
report.py                    8               993             5586  0.993
identity.py                  4               997             5634  0.997
config.py                    3               998             5636  0.998
document.md                  2               999             5638  0.999
presentation.bin             6               995             5630  0.995

 starting position  shared prefix  shared objects  objects to fetch
                 1              0               0              5640
                 2              1               2              5638
                 9              8              56              5584
               501            500            2640              3000

The Ratio Locks to One

The top table does the same operation at four scales, and the ratio column gives a single value: 1.000. In a thousand-commit history, 1000 identities change and 5640 objects are touched. The previous lesson’s rewrite from the middle of the same history meant 501 identities and 3002 objects; starting from the first commit roughly doubles that number. But the real difference is in the column on the right: shared prefix 0. In rewriting from the middle, the old and new history shared the first 500 commits; here, not a single commit stays in common.

The ratio locking in place does not mean scale is unimportant — it says the opposite. Because the ratio is constant, the absolute cost grows linearly with history: touched objects are 22600 in the four-thousand-commit history, sixty times the fifty-commit figure. The measurement does not measure duration, but its practical implication is clear: the transformation is not a momentary task, it has to be planned.

The middle table shows that the starting point is not something that can be chosen. The five files’ and the binary asset’s first touches sit at commits 1, 8, 4, 3, 2, and 6. The latest to appear is the eighth commit, and removing even that one still rewrites 993 commits — 0.993 of history. This is not a coincidence; it is a direct consequence of the count: in a repository shared among a handful of files, every file appears within the first few commits. In bulk transformation there is no such thing as “part of history”; the choice is between all of history and nearly all of it.

The bottom table shows where the cost falls. When the starting position is 1, none of the objects on the other side is any use: shared objects are 0, objects to fetch are 5640 — the entire repository. If the position were 501, shared objects would be 2640 and the amount to transfer would drop to 3000. The difference is not only quantitative but qualitative: with a common ancestor in place the other side can fetch; without one, the only thing it can do is clone. Fetching adds on top of what is already held; once there is no base left to add onto, fetching stops being a fetch in anything but name.

What the Transformation Does Not Change

The numbers give only the cost; they do not say what the transformation does to history. On that side, the situation is surprisingly calm. If the rule touches only metadata — such as the author field — every commit’s actual change stays as it is, order is preserved, branch structure is preserved. The previous course’s measured questions get the same answers; what changes is only the identities those answers are attached to. In most cases, a bulk transformation changes history’s addresses, not its answers.

In one case this is not true: if the rule removes an asset, the change carried by the commits that touched that asset is lost along with it. That file’s line history can no longer be asked; the commits that touched it no longer appear to have touched it. This is the information that is lost, and it is usually exactly what the transformation is for. The distinction has to be made: a metadata correction does not delete information, removing an asset does. The two pay the same cost and are equally irreversible, but they are not the same operation as far as history’s readability goes.

Irreversibility and the Safe Path

Bulk transformation is an irreversible operation, and for two separate reasons.

Locally, the old chain stays unreachable. The reflog shows it for a while, then garbage collection cleans the objects up; the recovery window is measured in this topic’s fourth lesson. This in-between period has a consequence that is easy to miss: the repository that ran the transformation is not clean the moment it finishes. The removed asset’s objects are unreachable but stay in place; anyone who knows their identity can still read them. What is clean is a fresh clone taken from that repository, because cloning only transfers reachable objects. This is why “work on a separate clone” is the correct path for outcome, not just for safety.

In the copies, there is no such thing as recovery, because there is no loss to recover from: every copy still holds the old chain intact. That is the problem. If a copy keeping the old chain is merged with the new history, the removed asset comes back and the transformation is wasted. This is why every copy has to be re-cloned, and this rule is not a suggestion, it is part of the operation’s definition. Until your copies are replaced, the bulk transformation is not complete.

The transformation’s effect reaches beyond the repository too. If old identities are written into an issue tracker, a release note, or a build record, those identities no longer point at anything. Tags bound to old commits need to be handled separately.

The safe path has four parts. First is a separate clone: the transformation is tried not on the actual repository but on a fresh copy of it, and the actual repository waits untouched until verification finishes. Second is verification: on the new history, it is measured whether the removed asset is truly gone and whether the commit count matches expectations. Verification looks for three answers — the commit count after the transformation, whether the removed asset’s name appears in any commit’s tree, and whether a reference was left out of the transformation. All three are read operations with no destructive side; the measurement already does this counting, and the same counting can be done in a real repository. Third is coordination: when the copies will be changed is announced in advance, because a copy that misses the change can bring the old chain back. Fourth is the leak rule: a secret removed from history does not become invalid by being removed. Once a key has been written, it has to be rotated; the transformation makes it harder to read, not never having happened.

The coordination item turns into a sequence in practice. The transformation is bound to a moment: everyone writes their in-hand work to the other side up to that moment, from that moment on no one derives new work from the old chain, the transformation is applied, the copies are refreshed, and only after that does work continue. Whichever step in the sequence gets skipped, the result is the same: the moment a single commit derived from the old chain enters the new history, the transformation is broken and has to be redone. This turns bulk transformation from a technical operation into a scheduling task, and it is also the largest unmeasured line item in its cost.

Summary

• Bulk transformation is not a plan but a rule: it is applied to every commit and regenerates every commit. The tool’s subcommand and outside filtering tools do this job; the command is not given in full form here.
• Identity change starts at the first commit whose content the rule changes. Removing a file requires going back to that file’s first appearance.
• Transformation from the first commit touches 380 / 920 / 5640 / 22600 objects at four scales and the ratio is 1.000 at every scale; in a thousand-commit history, 1000 identities change.
• The five files’ and the binary asset’s first touches lie between 1 and 8; removing even the latest-appearing asset rewrites 0.993 of history.
• Once the shared prefix drops to 0, there is no common ancestor left: the other side cannot fetch and has to re-clone the entire 5640-object set.
• The operation cannot be undone. The safe path is a separate clone, verification, coordination, and rotating the leaked key.

Next Step

In this lesson the cost was counted once and stayed within a single repository: a thousand identities, five thousand six hundred forty objects, zero common ancestor. What the number leaves out is this — writing the rewritten history to the other side means overwriting the chain on the other side, and the person who does this does not pay the share of the cost that falls to the other side. The next lesson measures that distinction: in a thousand-commit history, while fixing the last commit and fixing the first commit differ by a thousandfold in identity terms, whether there is any difference at all in terms of the repository’s copies.