Secret Scanning in Repository History

When an access key is typed by hand into a configuration line, the build passes, the tests pass, and the previous three checks all come back clean: the value is a valid string, it produces no error at a running endpoint, it matches no dependency record. The check that will see it looks at the string constants inside the files themselves.

Secret scanning is the check that runs the string constants in a repository’s content through a pattern set and pulls out the ones that could be secret. The Secure Coding course measured this same repository history once already, but what it measured was the secret’s exposure: how many commits a value stayed open across, what deleting it fails to close. Here what is measured is the check itself — how many findings each pattern produces, where the noise comes from, which secret class matches no pattern at all, and how many minutes all of this takes from the pipeline.

• AC19. The repository history is 1,800 commits and 940 string constants shared across three repositories; it is fictional, and no real secret format is ever written. A string constant is an immutable text value written into a file; whether it is actually secret is a fact of the model, the scanner only looks at shape.
• AC20. The pattern set has four members. Prefix format and block header catch only classes with a fixed shape. Keyword assignment looks not at the value but at the name the value is assigned to. Entropy threshold takes every string that stays above the threshold and is the only pattern that can be tuned.
• AC21. Entropy is a model measure, from 0 to 1, of how disordered a string’s character distribution is; random values sit high, dictionary words sit low.
• AC22. Removing a string from the working tree does not remove it from history, and it is not rotation; this was measured in the Secure Coding course and is taken here as input.
• AC23. The scanner is installed at commit 900 and has never run before that; builds are taken every ten commits, 90 builds in total. Scanning costs 0.02 model minutes per item, reviewing a false positive costs 4. The durations are data, not a measurement.

Which Pattern Produces the Noise

// secrets.mjs — string constants across the repository history are run through a pattern
// set; what is measured is matches, real secrets, false positives, and items no pattern
// catches. THIS IS A MODEL.
export const SEED = 20260329, COMMITS = 1800, STRING_COUNT = 940, ITEM_MIN = 0.02;
const rng = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = rng(SEED);

// [name, share, secret probability, entropy floor, entropy ceiling, name-assignment probability, format]
// secretProb = the true probability that this class is actually secret; format = the shape
// the prefix/block pattern recognizes.
export const CLASSES = [
  ["prefixed access key", 0.09, 0.93, 0.70, 0.96, 0.55, "prefix"],
  ["random session secret", 0.11, 0.82, 0.66, 0.94, 0.72, ""],
  ["private key block", 0.04, 0.96, 0.55, 0.80, 0.10, "block"],
  ["connection string", 0.10, 0.68, 0.34, 0.72, 0.86, ""],
  ["word passphrase", 0.08, 0.62, 0.16, 0.40, 0.34, ""],
  ["hash value", 0.17, 0.00, 0.74, 0.95, 0.12, ""],
  ["test placeholder", 0.23, 0.03, 0.28, 0.82, 0.78, ""],
  ["unique identifier", 0.18, 0.00, 0.58, 0.86, 0.18, ""],
].map(([name, share, secretProb, e0, e1, nameProb, format]) =>
  ({ name, share, secretProb, e0, e1, nameProb, format }));
const pickClass = (u) => { let t = 0;
  for (const c of CLASSES) { t += c.share; if (u < t) return c; } return CLASSES.at(-1); };

// Every string enters at one commit; part of it is removed from the working tree at a
// later commit. Removal does not invalidate the secret; it stays where it is in history.
export const STRINGS = Array.from({ length: STRING_COUNT }, (_, i) => {
  const c = pickClass(r());
  const entryCommit = Math.floor(r() * COMMITS), ent = c.e0 + r() * (c.e1 - c.e0);
  const removedCommit = r() < 0.30
    ? entryCommit + 1 + Math.floor(r() * (COMMITS - entryCommit)) : null;
  return { i, cls: c, secret: r() < c.secretProb, ent, keyAssigned: r() < c.nameProb,
    entryCommit, removedCommit, revisions: 1 + Math.floor(r() * 9) };
});

export const patterns = (threshold) => [
  { name: "prefix format", hit: (d) => d.cls.format === "prefix" },
  { name: "block header", hit: (d) => d.cls.format === "block" },
  { name: "keyword assignment", hit: (d) => d.keyAssigned },
  { name: "entropy threshold", hit: (d) => d.ent >= threshold },
];
export const matches = (d, threshold) => patterns(threshold).some((p) => p.hit(d));

const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  const secrets = STRINGS.filter((d) => d.secret);
  console.log(`model (seed ${SEED}): ${COMMITS} commits, ${STRING_COUNT} string constants, ` +
    `${secrets.length} of them really secret; ${STRINGS.filter((d) => d.removedCommit).length} ` +
    `strings removed from the working tree`);
  const y1 = gen([16, 10, 9, 9, 12, 12]);
  console.log("\nentropy threshold varies, the other three patterns are fixed");
  console.log(y1(["threshold", "matches", "real", "false+", "false+ %", "unseen"]));
  for (const t of [0.60, 0.80, 1.01]) {
    const e = STRINGS.filter((d) => matches(d, t)), g = e.filter((d) => d.secret).length;
    console.log(y1([t > 1 ? "off" : t.toFixed(2), e.length, g, e.length - g,
      (100 * (e.length - g) / e.length).toFixed(1), secrets.length - g]));
  }

  const THRESHOLD = 0.80, ps = patterns(THRESHOLD);
  const y2 = gen([24, 10, 9, 9, 12]);
  console.log(`\ncontribution per pattern at threshold ${THRESHOLD.toFixed(2)}`);
  console.log(y2(["pattern", "matches", "real", "false+", "sole source"]));
  for (const p of ps) {
    const l = STRINGS.filter((d) => p.hit(d));
    const sole = l.filter((d) => ps.filter((q) => q.hit(d)).length === 1 && d.secret).length;
    console.log(y2([p.name, l.length, l.filter((d) => d.secret).length,
      l.filter((d) => !d.secret).length, sole]));
  }

  const y3 = gen([24, 7, 12, 13, 13]);
  console.log(`\nunseen secrets by class at threshold ${THRESHOLD.toFixed(2)}`);
  console.log(y3(["string class", "strings", "real", "matched", "unseen"]));
  for (const c of CLASSES) {
    const l = STRINGS.filter((d) => d.cls === c), sl = l.filter((d) => d.secret);
    const e = sl.filter((d) => matches(d, THRESHOLD)).length;
    console.log(y3([c.name, l.length, sl.length, e, sl.length - e]));
  }
}

model (seed 20260329): 1800 commits, 940 string constants, 310 of them really secret; 269 strings removed from the working tree

entropy threshold varies, the other three patterns are fixed
threshold          matches     real   false+    false+ %      unseen
0.60                   855      274      581        68.0          36
0.80                   677      260      417        61.6          50
off                    519      245      274        52.8          65

contribution per pattern at threshold 0.80
pattern                    matches     real   false+ sole source
prefix format                   93       89        4          12
block header                    39       37        2          32
keyword assignment             442      172      270          86
entropy threshold              281      105      176          15

unseen secrets by class at threshold 0.80
string class            strings        real      matched       unseen
prefixed access key          93          89           89            0
random session secret       102          83           71           12
private key block            39          37           37            0
connection string            89          56           44           12
word passphrase              66          39           13           26
hash value                  179           0            0            0
test placeholder            214           6            6            0
unique identifier           158           0            0            0

The first table turns the one tunable knob all the way. At an entropy threshold of 0.60, matches are 855 and the false-positive rate is 68.0 percent; turning the pattern off entirely brings matches down to 519, but the rate only drops to 52.8 percent. Two-thirds of the noise is somewhere the threshold never touches. The cost of turning it off is in the same row: real findings drop from 274 to 245, unseen secrets rise from 36 to 65.

The second table names the source. The keyword-assignment pattern alone produces 442 matches, 270 of them false positives — two-thirds of all the noise at threshold 0.80. The reason is what the pattern does: it looks not at the value but at the name it is assigned to; any line whose name resembles a password field matches, even when the value inside it is a test placeholder. The prefix-format pattern gives 4 false positives out of 93 matches, block header gives 2 out of 39. A pattern set’s false-positive rate is not an average value; almost all of it comes from a single pattern, and the rate does not drop until that pattern is removed.

The last column makes the decision to remove it harder. The same pattern is the sole source of 86 real secrets; block header is the sole source for 32 items, entropy threshold for 15, prefix format for 12. The noisiest pattern is also the pattern with the most unique contribution, so the decision is not to remove it but to put its findings into a separate queue: when 270 false positives land in the same list, 86 unique real findings get lost inside it.

The third table gives the blind spot that is independent of the threshold. 26 of the word-passphrase class’s 39 real secrets match no pattern at all: their entropy is low, they carry no prefix, and two-thirds of them are not assigned to a password-like name. The connection-string and random-session-secret classes each miss twelve items too; in total, 50 secrets go unseen despite having been scanned. The opposite extreme is in the same table: the hash-value and unique-identifier classes hold 337 strings between them and not a single secret — both sit high on the entropy measure, so as the threshold drops, these are what fill up the report.

The Cost of History and the Incremental Scan’s Blind Spot

// history.mjs — the same pattern set runs under three placements: working tree every
// build, incremental diff every build, and a one-time full history. Durations are MODEL minutes.
import { STRINGS, COMMITS, ITEM_MIN, matches } from "./secrets.mjs";

const THRESHOLD = 0.80, BUILDS = 90, START = 900, REVIEW_MIN = 4;
const commitAt = (y) => START + Math.round((y + 1) * (COMMITS - START) / BUILDS);
const live = (d, c) => d.entryCommit <= c && (d.removedCommit === null || d.removedCommit > c);
const SECRETS = STRINGS.filter((d) => d.secret);
const BUILD_COMMITS = Array.from({ length: BUILDS }, (_, y) => commitAt(y));
// A string that lives only in history: not in the working tree at any build moment.
const historyOnly = (d) => !BUILD_COMMITS.some((c) => live(d, c));

// A placement says which item set is scanned at each build. The working-tree placement
// sees whatever files exist at that moment, the incremental placement only sees what
// entered since the previous build; full history reads every past revision once at setup.
const PRIOR = STRINGS.filter((d) => d.entryCommit <= START);
function measure(name, set, oneOff = 0) {
  const seen = new Set(), blockedBuilds = new Set(), looked = new Set();
  let scanned = oneOff;
  if (oneOff) for (const d of PRIOR) {
    looked.add(d); if (matches(d, THRESHOLD)) seen.add(d);
  }
  for (let y = 0; y < BUILDS; y++) {
    const list = set(commitAt(y), y ? commitAt(y - 1) : START);
    scanned += list.length;
    for (const d of list) {
      looked.add(d);
      if (seen.has(d) || !matches(d, THRESHOLD)) continue;
      seen.add(d); blockedBuilds.add(y);
    }
  }
  const real = [...seen].filter((d) => d.secret);
  const falsePos = seen.size - real.length;
  return { name, scanned, matches: seen.size, real: real.length, falsePos,
    notLooked: SECRETS.filter((d) => !looked.has(d)).length,
    missed: SECRETS.filter((d) => looked.has(d) && !seen.has(d)).length,
    blocked: blockedBuilds.size, min: Math.round(scanned * ITEM_MIN),
    reviewMin: falsePos * REVIEW_MIN,
    historyOnly: real.filter(historyOnly).length };
}

const REVISIONS = PRIOR.reduce((a, d) => a + d.revisions, 0);
const PLACEMENTS = [
  measure("working tree every build", (c) => STRINGS.filter((d) => live(d, c))),
  measure("incremental diff/build",
    (c, o) => STRINGS.filter((d) => d.entryCommit > o && d.entryCommit <= c)),
  measure("full history + incremental",
    (c, o) => STRINGS.filter((d) => d.entryCommit > o && d.entryCommit <= c), REVISIONS),
];

const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const y = gen([26, 9, 8, 9, 11, 13, 8, 11, 9]);
console.log(`${BUILDS} builds, scanner installed at commit ${START}, ${SECRETS.length} real ` +
  `secrets; full history reads ${REVISIONS} string revisions, ${ITEM_MIN} model minutes per item`);
console.log(y(["placement", "matches", "real", "false+", "not looked", "missed",
  "blocked", "scan min", "rev min"]));
for (const o of PLACEMENTS) {
  console.log(y([o.name, o.matches, o.real, o.falsePos, o.notLooked, o.missed, o.blocked,
    o.min, o.reviewMin]));
}

const full = PLACEMENTS.at(-1);
console.log(`\nfull history costs ${Math.round(REVISIONS * ITEM_MIN)} minutes once and finds ` +
  `${full.real} secrets; ${full.historyOnly} of them are not in the working tree at any ` +
  `build — removed, but still valid since it was never rotated`);
console.log(`pattern blindness is independent of placement: in the full history, ${full.missed} ` +
  `secrets are looked at but match no pattern. What is expensive is not the scan but the ` +
  `review: ${full.min} scan minutes against ${full.reviewMin} review minutes, because ` +
  `${full.matches} matches include ${full.falsePos} false positives`);

90 builds, scanner installed at commit 900, 310 real secrets; full history reads 2378 string revisions, 0.02 model minutes per item
placement                   matches    real   false+ not looked       missed blocked   scan min  rev min
working tree every build        640     246      394         18           46      89       1057     1576
incremental diff/build          341     127      214        164           19      89          9      856
full history + incremental      677     260      417          0           50      89         57     1668

full history costs 48 minutes once and finds 260 secrets; 14 of them are not in the working tree at any build — removed, but still valid since it was never rotated
pattern blindness is independent of placement: in the full history, 50 secrets are looked at but match no pattern. What is expensive is not the scan but the review: 57 scan minutes against 1668 review minutes, because 677 matches include 417 false positives

All three rows carry the same four patterns; what separates them is what they look at. The most common setup is the second row: at every build, scan only the diff that build brings in. It costs 9 minutes and finds 127 secrets. The same row records that 164 secrets are never looked at — items that entered the repository before the scanner was installed. An incremental check never visits them and reports clean on every run, because the set it looks at is always new. An incremental scan’s coverage list is the commit it was installed at; unless that date is written in the report, zero findings cannot be told apart from never having looked.

The first row closes this gap expensively. Scanning the entire working tree at every build costs 1,057 minutes — a hundred and seventeen times the incremental placement. In return, it sees secrets that entered before installation and are still there: the never-looked-at count drops from 164 to 18. The remaining 18 items are not in the working tree at any build. The blocked column does not separate the three placements: all three stop 89 of the 90 builds at least once, because what blocks is not the real secret but every new match that lands in the queue.

The third row flips the ranking. Scanning the entire history once at installation reads 2,378 string revisions and takes 48 minutes; a twenty-second of the 1,057 minutes the working-tree placement spends. In return, the never-looked-at count drops to zero, real findings rise to 260, and 14 of them are not in the working tree at any build — removed, but still valid because removal is not rotation. These 14 items appear in this placement alone. Scanning history is not the expensive part; the expensive part is scanning it again at every build.

The last two columns write down where the cost sits. In the full-history placement, 57 minutes of scanning produces 1,668 minutes of review: 97 percent of the cost is opening and closing 417 false positives. The measure of the time a secret scan takes from the pipeline is not run time, it is the false-positive rate, and where that rate comes from was written in the previous section.

Summary

• The entropy threshold, the only tunable pattern, brings the false-positive rate down from 68.0 percent to only 52.8 percent, and real findings drop from 274 to 245 along the way.
• The noise has a single source: keyword assignment marks 270 of its 442 matches wrong, because it looks at the name the value is assigned to, not the value. The same pattern is the sole source of 86 real secrets, so the decision is not to remove it but to route its findings into a separate queue.
• 50 of the 310 secrets match no pattern despite having been scanned; 26 of them sit in a single class — low-entropy, prefix-less word passphrases not assigned to a password-like name.
• The three placements carry three separate blind spots: incremental diff runs in 9 minutes but never sees the 164 secrets that predate installation; working tree at every build costs 1,057 minutes and misses 18; a one-time full history reads the entire past in 48 minutes and surfaces the 14 secrets that exist only there.
• 97 percent of the cost is review, not scanning: 57 minutes of run time buys 1,668 minutes spent on 417 false positives.

Next Step

All four checks so far stopped at the same place: the files in the repository. But what ships to production is not a file, it is an image — compiled output, the required libraries, and the operating layer underneath all merge into a single build. This build’s vulnerabilities come from two separate places, and their fix paths look nothing alike: part sits in the layers the team added, part is inherited from a base layer nobody has touched. For the second kind there is no line of code to write; what closes it is an upgrade. The next lesson separates these two sources in an image’s layer list and counts how many vulnerabilities upgrading the base layer closes.