---
title: 'Bulk History Transformation'
source: 'https://academia.sh/en/courses/advanced-git/bulk-history-transformation'
course: 'Advanced Git'
language: en
updated: '2026-08-17T18:10:43+00:00'
license: 'CC BY-SA 4.0'
---

# Bulk History Transformation

A repository-wide transformation starts from the first commit: in a thousand-commit history, 1000 identities change, 5640 objects are touched, and the new history does not share a single commit with the old one — so every copy has to be re-cloned and the operation cannot be undone.

The previous lesson kept rewriting on top of a single branch. When the plan stayed close
to the tip, the cost was small: a three-commit plan touched six objects. Even when
started from the middle of history, only about half of the commits changed identity, and
the rest stayed as they were. That half was the deepest point the measurement reached.

Some jobs do not leave even that much untouched. An asset committed in a repository's
early days that should never have been there, an author field misspelled across every
commit, a directory that needs to move under the root — all of these are forced to start
from the **first** commit of history. These are called **bulk history transformations**,
and although they come from the same family as the previous lesson's tool, they operate
at a different scale. This lesson's question is this: what does the cost become when the
transformation starts from the first commit, and what is left in the repository's other
copies once the operation is done?

## Transformation as a Rule

In interactive mode, the plan was a list: you saw the lines and wrote an action for each
one. In bulk transformation there is no plan; there is a **rule**. The rule is applied to
every commit in order, the commit's tree is run through the rule, and a new commit is
produced from the result. The difference between ten commits and a thousand is only
duration; the rule written is the same rule.

Four typical requirements fall into this class. **Removing an asset:** if a binary file
or a leaked secret that should never have been committed sits in history, deleting it
from the last commit is not enough; it stays readable from the old commits. **Metadata
correction:** if the author or committer field is wrong throughout history. **Tree
relocation:** turning a subdirectory into the repository's root, or the reverse.
**Retroactive normalization:** applying a line-ending or encoding fix not only to new
commits but to old ones as well.

The tool has a subcommand that does this job, and there are also filtering tools that
run outside the repository. The command is **not given in full form** in this lesson.
The reason is not pedagogy but responsibility: a copy-and-run bulk-transformation line
changes the repository's history irreversibly and invalidates every copy of it. The dump
below carries a placeholder instead of a filter and cannot be run.

```text
# example dump — not executed, the command is not given in full form

$ git filter-branch <filter> -- --all
Rewrite 9c4d7e1 (1/1000) (0 seconds passed, remaining 0 predicted)
Rewrite 4a5b6c7 (2/1000) (0 seconds passed, remaining 0 predicted)
...
Ref 'refs/heads/main' was rewritten
```

The dump's counter shows this lesson's measure directly: the counter runs from one to a
thousand, because a thousand of the thousand commits are being regenerated.

The line at the end of the dump says a second thing. Interactive mode operates from a
single branch's tip; bulk transformation can be applied to **all references at once**,
and usually is. The reason is directly tied to the measurement: a branch or a tag left
out of the transformation keeps pointing at the old chain, that chain stays reachable,
and the asset meant to be removed does not leave the repository. Forgetting a branch
gives the same result as not doing the transformation at all. This is why the scope
decision comes before the rule itself: which references will enter the transformation,
which will be deleted before the transformation runs.

## Where the Transformation Starts

The rule is applied to every commit, but **identity change** does not start at every
commit. Commits produced up to the first one where the rule's content actually changes
something come out identical to the old ones and get the same identity. The change
starts at that first commit and, by the previous lesson's rule, runs all the way to the
tip.

This reduces the job of removing a file from history to a single question: when was that
file **first** touched. The answer is usually unpleasant. If a repository has only a few
files and every commit touches one of them, every file's first appearance sits near the
very beginning of history. The measurement asks this separately for five files and one
binary asset.

The measurement's assumptions:

- **RH9** — The setup is the previous lesson's setup and is unchanged: four scales
  (**50, 200, 1000, 4000**), each commit touches one of five files, one in eleven
  carries a binary asset.
- **RH10** — Bulk transformation is a rule applied to every commit. The measurement does
  not count **what** the rule does, only how many commits it regenerates; this is why an
  author correction and a file removal sit in the same column.
- **RH11** — Identity change starts at the first commit whose content the rule actually
  changes, and runs to the tip of history.
- **RH12** — Removing a file starts at the first commit that touches that file; for a
  binary asset, the start is the first commit that carries the binary.
- **RH13** — This lesson's unit of cost is the **touched object**. The number of commits
  whose identity changes is the width of the set; the step unit is not used in this
  lesson.
- **RH14** — The shared prefix is the commits the new history keeps in common with the
  old one: the range up to one before the starting position.
- **RH15** — Objects to fetch are the objects the other side does not already hold;
  every object outside the shared prefix is retransferred. This is not a network
  measurement, it is an object count.
- **RH16** — The set's resolution at the four scales is **1/50, 1/200, 1/1000, 1/4000**
  respectively.

## Measurement

```python
"""Bulk history transformation: the cost of rewriting from the first commit.

Part 1 - transformation from the first commit at four scales.
Part 2 - which asset is removed determines where the transformation must start.
Part 3 - the prefix the old and new history share.
"""
SEED = 20260814
FILES = ("metrics.py", "report.py", "identity.py", "config.py", "document.md")
BINARY_ASSET = "presentation.bin"
SCALES = (50, 200, 1000, 4000)


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def history(n, seed=SEED):
    """n-commit linear history; each commit touches one file."""
    draw, commits = rng(seed), []
    for i in range(n):
        file = FILES[draw(5)]
        binary = draw(11) == 0
        commits.append({"no": i + 1, "file": file, "binary": binary,
                         "objects": 2 + (40 if binary else 0)})
    return commits


def rewrite(t, position):
    after = [x for x in t if x["no"] >= position]
    return len(after), sum(x["objects"] for x in after)


def first_touch(t, name):
    """The first commit that touches the asset to be removed: the transformation starts there."""
    for x in t:
        if (x["binary"] if name == BINARY_ASSET else x["file"] == name):
            return x["no"]
    return 0


print(f"{'commits':>7s} {'identity changed':>17s} {'touched objects':>16s}"
      f" {'ratio':>6s} {'shared prefix':>14s}")
for n in SCALES:
    t = history(n)
    changed, touched = rewrite(t, 1)
    print(f"{n:7d} {changed:17d} {touched:16d} {changed / n:6.3f}"
          f" {len(t) - changed:14d}")

print()
T = history(1000)
print(f"{'removed asset':<17s} {'first touch':>12s} {'identity changed':>17s}"
      f" {'touched objects':>16s} {'ratio':>6s}")
for name in FILES + (BINARY_ASSET,):
    k = first_touch(T, name)
    changed, touched = rewrite(T, k)
    print(f"{name:<17s} {k:12d} {changed:17d} {touched:16d} {changed / len(T):6.3f}")

print()
print(f"{'starting position':>18s} {'shared prefix':>14s} {'shared objects':>15s}"
      f" {'objects to fetch':>17s}")
for position in (1, 2, 9, 501):
    shared = sum(x["objects"] for x in T if x["no"] < position)
    print(f"{position:18d} {position - 1:14d} {shared:15d}"
          f" {sum(x['objects'] for x in T) - shared:17d}")
```

```
commits  identity changed  touched objects  ratio  shared prefix
     50                50              380  1.000              0
    200               200              920  1.000              0
   1000              1000             5640  1.000              0
   4000              4000            22600  1.000              0

removed asset      first touch  identity changed  touched objects  ratio
metrics.py                   1              1000             5640  1.000
report.py                    8               993             5586  0.993
identity.py                  4               997             5634  0.997
config.py                    3               998             5636  0.998
document.md                  2               999             5638  0.999
presentation.bin             6               995             5630  0.995

 starting position  shared prefix  shared objects  objects to fetch
                 1              0               0              5640
                 2              1               2              5638
                 9              8              56              5584
               501            500            2640              3000
```

## The Ratio Locks to One

The top table does the same operation at four scales, and the ratio column gives a
single value: **1.000**. In a thousand-commit history, **1000** identities change and
**5640** objects are touched. The previous lesson's rewrite from the middle of the same
history meant 501 identities and 3002 objects; starting from the first commit roughly
doubles that number. But the real difference is in the column on the right: shared
prefix **0**. In rewriting from the middle, the old and new history shared the first 500
commits; here, not a single commit stays in common.

The ratio locking in place does not mean scale is unimportant — it says the opposite.
Because the ratio is constant, the absolute cost grows linearly with history: touched
objects are **22600** in the four-thousand-commit history, sixty times the fifty-commit
figure. The measurement does not measure duration, but its practical implication is
clear: the transformation is not a momentary task, it has to be planned.

The middle table shows that the starting point is not something that can be chosen. The
five files' and the binary asset's first touches sit at commits **1, 8, 4, 3, 2**, and
**6**. The latest to appear is the eighth commit, and removing even that one still
rewrites **993** commits — **0.993** of history. This is not a coincidence; it is a
direct consequence of the count: in a repository shared among a handful of files, every
file appears within the first few commits. **In bulk transformation there is no such
thing as "part of history";** the choice is between all of history and nearly all of
it.

The bottom table shows where the cost falls. When the starting position is **1**, none
of the objects on the other side is any use: shared objects are **0**, objects to fetch
are **5640** — the entire repository. If the position were **501**, shared objects would
be **2640** and the amount to transfer would drop to **3000**. The difference is not only
quantitative but qualitative: with a common ancestor in place the other side can
**fetch**; without one, the only thing it can do is **clone**. Fetching adds on top of
what is already held; once there is no base left to add onto, fetching stops being a
fetch in anything but name.

## What the Transformation Does Not Change

The numbers give only the cost; they do not say what the transformation does to history.
On that side, the situation is surprisingly calm. If the rule touches only metadata —
such as the author field — every commit's actual change stays as it is, order is
preserved, branch structure is preserved. The previous course's measured questions get
the same answers; what changes is only the identities those answers are attached to. **In
most cases, a bulk transformation changes history's addresses, not its answers.**

In one case this is not true: if the rule removes an asset, the change carried by the
commits that touched that asset is lost along with it. That file's line history can no
longer be asked; the commits that touched it no longer appear to have touched it. This
is the information that is lost, and it is usually exactly what the transformation is
for. The distinction has to be made: a metadata correction does not delete information,
removing an asset does. The two pay the same cost and are equally irreversible, but they
are not the same operation as far as history's readability goes.

## Irreversibility and the Safe Path

Bulk transformation is an irreversible operation, and for two separate reasons.

**Locally**, the old chain stays unreachable. The reflog shows it for a while, then
garbage collection cleans the objects up; the recovery window is measured in this
topic's fourth lesson. This in-between period has a consequence that is easy to miss:
**the repository that ran the transformation is not clean the moment it finishes.** The
removed asset's objects are unreachable but stay in place; anyone who knows their
identity can still read them. What is clean is a fresh clone taken from that
repository, because cloning only transfers reachable objects. This is why "work on a
separate clone" is the correct path for outcome, not just for safety.

**In the copies**, there is no such thing as recovery, because there is no loss to
recover from: every copy still holds the old chain intact. That is the problem. If a
copy keeping the old chain is merged with the new history, the removed asset comes back
and the transformation is wasted. This is why **every copy has to be re-cloned**, and
this rule is not a suggestion, it is part of the operation's definition. Until your
copies are replaced, the bulk transformation is not complete.

The transformation's effect reaches beyond the repository too. If old identities are
written into an issue tracker, a release note, or a build record, those identities no
longer point at anything. Tags bound to old commits need to be handled separately.

The safe path has four parts. **First is a separate clone:** the transformation is
tried not on the actual repository but on a fresh copy of it, and the actual repository
waits untouched until verification finishes. **Second is verification:** on the new
history, it is measured whether the removed asset is truly gone and whether the commit
count matches expectations. Verification looks for three answers — the commit count
after the transformation, whether the removed asset's name appears in any commit's
tree, and whether a reference was left out of the transformation. All three are read
operations with no destructive side; the measurement already does this counting, and
the same counting can be done in a real repository. **Third is coordination:** when the
copies will be changed is announced in advance, because a copy that misses the change
can bring the old chain back. **Fourth is the leak rule:** a secret removed from
history does not become invalid by being removed. Once a key has been written, it has
to be rotated; the transformation makes it harder to read, not never having happened.

The coordination item turns into a sequence in practice. The transformation is bound to a
moment: everyone writes their in-hand work to the other side up to that moment, from
that moment on no one derives new work from the old chain, the transformation is applied,
the copies are refreshed, and only after that does work continue. Whichever step in the
sequence gets skipped, the result is the same: the moment a single commit derived from
the old chain enters the new history, the transformation is broken and has to be redone.
This turns bulk transformation from a technical operation into a **scheduling** task, and
it is also the largest unmeasured line item in its cost.

## Summary

- Bulk transformation is not a plan but a **rule**: it is applied to every commit and
  regenerates every commit. The tool's subcommand and outside filtering tools do this
  job; the command is not given in full form here.
- Identity change starts at the first commit whose content the rule changes. Removing a
  file requires going back to that file's **first appearance**.
- Transformation from the first commit touches **380 / 920 / 5640 / 22600** objects at
  four scales and the ratio is **1.000** at every scale; in a thousand-commit history,
  **1000** identities change.
- The five files' and the binary asset's first touches lie between **1** and **8**;
  removing even the latest-appearing asset rewrites **0.993** of history.
- Once the shared prefix drops to **0**, there is no common ancestor left: the other
  side cannot fetch and has to re-clone the entire **5640**-object set.
- The operation cannot be undone. The safe path is a separate clone, verification,
  coordination, and rotating the leaked key.

## Next Step

In this lesson the cost was counted once and stayed within a single repository: a
thousand identities, five thousand six hundred forty objects, zero common ancestor. What
the number leaves out is this — writing the rewritten history to the other side means
**overwriting** the chain on the other side, and the person who does this does not pay
the share of the cost that falls to the other side. The next lesson measures that
distinction: in a thousand-commit history, while fixing the last commit and fixing the
first commit differ by a thousandfold in identity terms, whether there is any difference
at all in terms of the repository's copies.