Feature Flags

In the previous lesson’s two-environment scheme, rollback dropped to a single switch: traffic flips to the old environment, and the previous behavior returns within minutes. What that switch rolled back was the entire code. A release carries a version; every change inside that version ships together and comes back together. But what usually needs rolling back is a single behavior.

A feature flag separates this unit: the code deploys to production, the behavior turns on with a separate switch. This lesson’s measure is not whether a flag exists, but which number this separation changes. The same change sequence is run under both schemes, the rollback unit and round are counted, the code paths a flag opens are counted, and at the end of a period, the flag debt left behind is written down.

• RS25. The flag value lives outside the pipeline, in a config record the running process reads; changing it does not require a new release. Propagating the setting to every node takes one round.
• RS26. Flags are independent of one another: any combination is a valid code path.
• RS27. Drift in production is seen from the verification service’s mismatch report; the report is produced once every two deployment rounds. Detection time is the same under both schemes.
• RS28. In the without-flags scheme, the rollback unit is a version: every change inside that version goes back together.
• RS29. A flag’s read point is where the condition is written in the code; a flag can be read at more than one point, and cleanup cost is proportional to that number.
• RS30. The fiction is the regional measurement network’s billing service: an invoice amount is produced from a meter reading. The service, the ledger, and the figures below are all fictional.

The Rollback Unit

A period of six changes gets released. The third — the late fee — is defective, and the defect is seen only after the following changes have also reached production. In the without-flags scheme, the changes split into three releases of two. In the with-flags scheme, every change releases separately, and the defective one ships dark behind a flag before it is turned on.

// scheme.mjs — the same change sequence is released under two schemes; the release scheme is a MODEL

const CHANGES = ['rate table', 'tiered rate', 'late fee',
  'invoice summary field', 'work order link', 'batch job report'];
const DEFECTIVE = 2;      // late fee drifts once the tier is on
const INVOICE_ROUND = 40; // invoices computed in one deployment round
const CONFIG_ROUND = 4;   // invoices computed in one config-propagation round
const NOTICE = 2;         // the mismatch report surfacing the defect: two deployment rounds

function withoutFlags() {
  const release = [[0, 1], [2, 3], [4, 5]];   // three releases, each release a version
  let version = release.length, round = 0;
  while (release.slice(0, version).flat().includes(DEFECTIVE)) { version -= 1; round += 1; }
  const remaining = release.slice(0, version).flat();
  const pulled = release.flat().filter((i) => !remaining.includes(i));
  return { name: 'without flags', unit: 'code', round, pulled,
    affected: (NOTICE + round) * INVOICE_ROUND };
}

function withFlags() {
  // every change released separately; the defective one shipped dark behind a flag, then turned on
  const flag = { 'late fee': 'on' };
  let round = 0;
  flag['late fee'] = 'off'; round += 1;
  const pulled = Object.entries(flag).filter(([, d]) => d === 'off')
    .map(([name]) => CHANGES.indexOf(name));
  return { name: 'with flags', unit: 'behavior', round, pulled,
    affected: NOTICE * INVOICE_ROUND + round * CONFIG_ROUND };
}

console.log('scheme         rollback unit  round  pulled  of these sound  affected invoices');
for (const d of [withoutFlags(), withFlags()]) {
  const sound = d.pulled.filter((i) => i !== DEFECTIVE);
  console.log(`${d.name.padEnd(15)} ${d.unit.padEnd(14)} ${String(d.round).padEnd(6)} ` +
    `${String(d.pulled.length).padEnd(7)} ${String(sound.length).padEnd(17)} ${d.affected}`);
  console.log(`  pulled from production: ${d.pulled.map((i) => CHANGES[i]).join(', ')}`);
}

scheme         rollback unit  round  pulled  of these sound  affected invoices
without flags   code           2      4       3                 160
  pulled from production: late fee, invoice summary field, work order link, batch job report
with flags      behavior       1      1       0                 84
  pulled from production: late fee

In the without-flags scheme, two rounds of rewinding were needed, because the defective change sat in the middle version and one more version had shipped after it. Four changes were pulled from production; three of them were sound. The owners of those three changes had nothing to do with the defect, and their work got pulled from production for a reason that was not their fault. In the with-flags scheme, the only thing rolled back is a behavior: not one sound change left production.

The difference in affected-invoice count comes from how long the rollback round takes. Detection takes the same time under both schemes — eighty invoices. The split happens after that: rewinding a version takes two deployment rounds, turning off a flag takes one config-propagation round. 160 against 84. All of this difference comes from the rollback path getting shorter, not from the defect being seen any earlier.

The Code Paths a Flag Opens

A flag is a condition, and every condition splits the code in two. Three flags open eight code paths. The computation below actually runs: the invoice amount is computed separately for every combination of the three flags, and compared against a separately written expected behavior for that combination.

// flag.mjs — the code paths opened by three flags are counted; the billing computation is a MODEL
// Regional measurement network fiction: an invoice amount is produced from a meter reading.

const BASE = 12.4;
const TIER = [[10, 12.4], [25, 18.6], [Infinity, 27.9]];
const READING = [
  { no: 'S-0431', m3: 8, delayMo: 0 }, { no: 'S-0432', m3: 8, delayMo: 2 },
  { no: 'S-0433', m3: 17, delayMo: 0 }, { no: 'S-0434', m3: 17, delayMo: 1 },
  { no: 'S-0435', m3: 34, delayMo: 3 }, { no: 'S-0436', m3: 52, delayMo: 0 },
];
const FLAG = ['tier', 'delay', 'summary'];
const cents = (x) => Math.round(x * 100) / 100;

function tiered(m3) {
  let remaining = m3, floor = 0, amount = 0;
  for (const [ceiling, unit] of TIER) {
    const slice = Math.min(remaining, ceiling - floor);
    amount += slice * unit; remaining -= slice; floor = ceiling;
    if (remaining <= 0) break;
  }
  return amount;
}

// Code in production: every flag is a branch point.
function production(o, b) {
  let amount = b.tier ? tiered(o.m3) : o.m3 * BASE;
  const base = o.m3 * BASE;
  if (b.delay && o.delayMo > 0) amount += base * 0.015 * o.delayMo;
  const f = { amount: cents(amount) };
  if (b.summary) f.summary = { m3: o.m3, delayMo: o.delayMo };
  return f;
}

// Every combination's expected behavior is written separately: the fee runs on its own amount.
function expected(o, b) {
  let amount = b.tier ? tiered(o.m3) : o.m3 * BASE;
  if (b.delay && o.delayMo > 0) amount += amount * 0.015 * o.delayMo;
  return cents(amount);
}

const paths = [];
for (let m = 0; m < 2 ** FLAG.length; m += 1) {
  const b = Object.fromEntries(FLAG.map((name, i) => [name, Boolean(m & (1 << i))]));
  const diverging = READING.filter((o) => production(o, b).amount !== expected(o, b));
  paths.push({ mask: FLAG.map((name) => (b[name] ? '1' : '0')).join(''), b, diverging });
}

// Tested set: production's current state, and every flag on its own.
const TESTED = new Set(['000', '100', '010', '001']);
console.log('path  tier  delay  summary  diverging  tested');
for (const y of paths) {
  console.log(`${y.mask}  ${FLAG.map((a) => (y.b[a] ? 'on    ' : 'off   ')).join(' ')}` +
    `  ${String(y.diverging.length).padEnd(11)} ${TESTED.has(y.mask) ? 'yes' : 'no'}`);
}
const defective = paths.filter((y) => y.diverging.length > 0);
console.log('');
console.log(`total code paths                : ${paths.length}`);
console.log(`tested paths                    : ${TESTED.size} (${(TESTED.size / paths.length * 100).toFixed(0)}% share)`);
console.log(`paths producing diverging behavior: ${defective.length} — ${defective.map((y) => y.mask).join(', ')}`);
console.log(`of these, tested                : ${defective.filter((y) => TESTED.has(y.mask)).length}`);
console.log('');
console.log('per-invoice difference on the first path where the defect appears');
for (const o of defective[0].diverging) {
  const y = defective[0];
  console.log(`  ${o.no} ${String(o.m3).padStart(3)} m3  ${o.delayMo} mo delay` +
    `  production=${production(o, y.b).amount}  expected=${expected(o, y.b)}`);
}

path  tier  delay  summary  diverging  tested
000  off    off    off     0           yes
100  on     off    off     0           yes
010  off    on     off     0           yes
110  on     on     off     2           no
001  off    off    on      0           yes
101  on     off    on      0           no
011  off    on     on      0           no
111  on     on     on      2           no

total code paths                : 8
tested paths                    : 4 (50% share)
paths producing diverging behavior: 2 — 110, 111
of these, tested                : 0

per-invoice difference on the first path where the defect appears
  S-0434  17 m3  1 mo delay  production=257.36  expected=258.01
  S-0435  34 m3  3 mo delay  production=673.07  expected=683.53

The defect does not sit in a single flag; it sits at the intersection of two flags: the late fee runs on the base amount instead of the tiered amount. While the tier is off, the two values are equal, so the computation comes out right; once the tier turns on, it drifts. Because the test set tries each flag on its own, it never sees four of the eight paths, and both of the two defective paths sit inside that four.

The size of the drift is also worth noting: between 0.65 and 10.46 per invoice. This is a drift that produces no error message, runs from a quarter of a percent to one and a half percent as a share, and shows up only for subscribers who happen to have a delay. That is why the question of how many of the eight paths get tested matters: adding a flag does not add to testing obligation, it multiplies it.

The Gate’s Three Numbers

The flag condition is a gate, and like the other gates it is evaluated on three numbers. First, what it catches. The number of defects the flag condition catches is zero: the condition lets both branches through, and tests nothing. The only thing it catches is exposure — 84 invoices instead of 160, a difference of 76 invoices. All of that difference comes from the rollback path, and it is what only this gate delivers: none of the pipeline’s other gates can change the rollback unit.

Second, what it misses and what it wrongly holds. The flag gate produces the defect it misses by itself: the four untested paths are paths that did not exist before the three flags were added. Wrongly holding is measured by sound behavior sitting off — in the ledger below, at the end of the twelve-week period, there is a change with nothing to do with the defect, complete and deployed, that is still sitting off.

Third, the time it adds. This gate adds no minutes to lead time: the code still ships in a single release, the pipeline’s stages do not change. What it adds is a deferred round — writing the read points, and later deleting them. Because deferral looks free, it goes unpaid and accumulates as debt. In this lesson, the gate’s third number is not minutes; it is the count of uncleared read points.

Flag Debt

A flag’s cost is not paid the moment it is added; it is paid for as long as it is not removed. A twelve-week period’s ledger is counted: which flag was added when, whether it was cleared, and at how many points it gets read.

// debt.mjs — one period's flag ledger is counted; the ledger is a MODEL
// [name, week added, week cleared (null: still standing), read points, today's value]
const PERIOD = 12;
const LEDGER = [
  ['tiered-rate', 1, 6, 3, 'on'], ['late-fee', 2, null, 2, 'off'],
  ['invoice-summary', 2, 7, 1, 'on'], ['work-order-link', 3, null, 4, 'on'],
  ['batch-job-report', 4, null, 2, 'on'], ['verifier-second-pass', 5, 9, 2, 'on'],
  ['reading-retry', 7, null, 5, 'on'], ['contract-warning', 8, null, 3, 'off'],
  ['summary-new-field', 10, null, 2, 'on'],
];

const cleared = LEDGER.filter(([, , t]) => t !== null);
const remaining = LEDGER.filter(([, , t]) => t === null);
const points = remaining.reduce((t, [, , , n]) => t + n, 0);
const oldest = remaining.reduce((e, k) => (k[1] < e[1] ? k : e));
console.log(`period                           : ${PERIOD} weeks`);
console.log(`added / cleared / remaining      : ${LEDGER.length} / ${cleared.length} / ${remaining.length}`);
console.log(`average lifetime of cleared flags: ` +
  `${(cleared.reduce((t, [, e, k]) => t + (k - e), 0) / cleared.length).toFixed(1)} weeks`);
console.log(`paths opened by remaining flags  : ${2 ** remaining.length}`);
console.log(`path running today               : 1 (remaining flags' values are fixed)`);
console.log(`dead code paths                  : ${2 ** remaining.length - 1}`);
console.log(`if every flag is tested alone    : ${remaining.length + 1} paths` +
  ` (${((remaining.length + 1) / 2 ** remaining.length * 100).toFixed(0)}%)`);
console.log(`read points left to clean        : ${points}`);
console.log(`off flags waiting to be turned on: ` +
  remaining.filter(([, , , , d]) => d === 'off').map(([a, e]) => `${a} (${PERIOD - e} weeks)`).join(', '));
console.log(`oldest uncleared flag            : ${oldest[0]} — ${PERIOD - oldest[1]} weeks`);

period                           : 12 weeks
added / cleared / remaining      : 9 / 3 / 6
average lifetime of cleared flags: 4.7 weeks
paths opened by remaining flags  : 64
path running today               : 1 (remaining flags' values are fixed)
dead code paths                  : 63
if every flag is tested alone    : 7 paths (11%)
read points left to clean        : 18
off flags waiting to be turned on: late-fee (10 weeks), contract-warning (4 weeks)
oldest uncleared flag            : late-fee — 10 weeks

Six flags remained standing and opened sixty-four code paths. Only one of these paths runs today, because all six flags’ values are fixed. The remaining sixty-three are dead code paths: none of them execute, all of them sit in the code, and each is assumed to be a behavior that can be tested. If the test set widens with the same pattern — every flag on its own — it covers seven of the sixty-four paths; the share drops from 50% to 11%. As the flag count grows linearly, coverage thins out exponentially.

Eighteen read points is the number of conditions that need deleting to clear these flags. The longer this work is deferred, the less readable the code becomes — but more importantly, which branch is live stops being readable from the code itself. The three cleared flags’ average lifetime was 4.7 weeks; one of the remaining ones has been sitting for ten weeks. Two flags are waiting, off: one is defective, one is complete but never turned on. The second is the change this gate wrongly holds — a behavior that has sat in production for four weeks and reached not one subscriber.

The gate that needs removing becomes visible right here. tiered-rate was cleared in week six: the flag was turned on, the behavior became permanent, and the condition was deleted. Removing a gate belongs on the plan exactly as much as building one does; when it does not, the ledger’s every number grows.

Summary

• The flag reduced the rollback unit from the entire code to a single behavior: in the without-flags scheme, four changes were pulled from production and three were sound; in the with-flags scheme, the only thing pulled was the defective behavior.
• The rollback round dropped from two to one, affected invoices from 160 to 84; all of that difference came from the rollback path getting shorter, not from the defect being seen any earlier.
• Three flags opened eight code paths; a set testing each flag on its own covered four of them, and both of the two paths producing diverging behavior sat outside that coverage.
• A flag condition catches no defects — what it catches is exposure; the paths it misses, it produces by itself.
• In twelve weeks, nine flags were added, three were cleared; the remaining six flags opened sixty-four paths, sixty-three of which sat as dead code, and eighteen read points waited to be cleared.

Next Step

Turning off a flag works when what needs rolling back is a behavior. If the defect does not sit behind a flag — configuration changed, data was transformed, or the change shipped without a flag at all — the way back is not a single switch, it is an ordered list of steps. How many steps that list has, how many rounds each step takes, and after which step the way back closes are each measured separately. The next lesson treats rollback not as a button but as a plan, and counts how many steps a plan’s not having been rehearsed costs on its first run.