The Difference Between DevSecOps and DevOps

The previous curriculum spent ten courses counting the paths that defense closed, and at its close asked which item a scarce budget should go to. All of those measurements shared one silent assumption: the software was already written. The path set was extracted from a running system; the check was placed on a running system. This course removes that assumption and takes up the same measurement network’s software while it is being built.

The unit of measure changes. Instead of a path, a stage is counted: in which stage a vulnerability or a wrong decision is born, in which stage it can be caught, and how many stages apart the two are. This first lesson measures the crudest form of that difference — the difference between security being built into the flow and standing at the end of the flow as a separate gate. Both setups are run on the same item set; three columns come out: how many items are caught, how many stages late they are caught, and what is paid in return.

• DA1. The network and its software are fictional: the regional measurement network’s subscriber portal, meter collection service, billing, and field application are developed by five teams. The change flow and the item set are modeled with node; what is measured is this model.
• DA2. The stage set has six members: requirement, design, code, build, deployment, runtime. An item is born in exactly one stage and cannot be caught before the stage it is born in.
• DA3. The visibility vector is a modeling decision: it says whether a check placed at that stage can see the item or not. In reality this mapping is worked out by trial; the point of the lesson is not the mapping but the count run on top of it.
• DA4. A check either sees an item or it does not. Partial visibility and the probability of missing it are not modeled; both would lower the number of items caught without changing the direction of the result.
• DA5. Gate review happens at the batch level. If a batch contains an item the gate can see, the whole batch is sent back; the other changes in the batch wait along with it.
• DA6. The alert counts and minute values are given model values. The false-positive count is not given directly; it is found by subtracting the items actually caught from the alert count.

The Item Set and the Change Flow

The model keeps two things separate: the changes that pass through the flow, and the items that a portion of those changes carry. An item is defined by the stage it is born in and a six-digit visibility vector.

// flow.mjs — the change flow and item set of the fictional measurement network's software (model)
export const STAGE = ['requirement', 'design', 'code', 'build', 'deployment', 'runtime'];
export const TEAM = [['subscriber-portal', 10], ['meter-collection', 8], ['billing', 6],
  ['field-app', 5], ['shared-library', 6]];
export const DAY = 60; // 12-week measurement window, business days
// item: name | stage born in | visibility vector (1/0 per stage: can a check placed
// there see this item?). The vectors are a modeling decision, not a measurement.
export const ITEM = `
  no-data-classification        0 100000
  undefined-retention-period    0 100001
  unclear-personal-data-scope   0 100010
  unauthenticated-internal-call 1 011110
  weak-session-model            1 010011
  trust-boundary-free-flow      1 010000
  missing-role-in-authz-model   1 011001
  missing-authorization-check   2 001101
  embedded-secret-key           2 001110
  unvalidated-input             2 001001
  personal-data-in-logs         2 001001
  incomplete-error-path         2 001000
  weak-randomness                2 001000
  unbounded-query-size          2 000001
  outdated-dependency           3 000110
  unsigned-build-output         3 000110
  unnecessary-build-tool        3 000100
  default-password              4 000011
  overly-broad-network-rule     4 000011
  unencrypted-backup            4 000010
  open-admin-endpoint           4 000011
  race-under-scale              5 000001
  third-party-disclosed-flaw    5 000001`
  .trim().split('\n').map((r) => {
    const [name, d, g] = r.trim().split(/\s+/);
    return { name, born: +d, g };
  });
export const DAILY = TEAM.reduce((s, [, n]) => s + n, 0) / 5;
export const TOTAL_CHANGES = DAILY * DAY;
// The change that carries an item is placed at even intervals in the flow (model).
export const CARRIER = new Map(ITEM.map((k, i) =>
  [Math.floor((i * TOTAL_CHANGES) / ITEM.length), k]));
// First visible stage: the first stage at or after birth where the visibility vector is 1.
export const firstSeen = (k, stages) =>
  stages.find((i) => i >= k.born && k.g[i] === '1') ?? null;

The vectors carry the lesson’s real information. no-data-classification can be seen only in the requirement stage; no later check can find it, because there is no faulty line to find, only a missing decision. unbounded-query-size is born in the code stage but becomes visible only at runtime: there is nothing wrong in the code, only the absence of a limit that shows up once the data grows.

The Separate Gate’s Queue

The first setup puts security at a single point: a review gate standing before deployment. The gate’s daily capacity is limited, while the arrival rate is the sum of the teams.

// run1.mjs — single gate: changes queue at one point before deployment
import { DAY, DAILY, TOTAL_CHANGES, CARRIER, ITEM } from './flow.mjs';

const CAPACITY = 5; // reviews the gate can finish per day (model)

function run(b) {
  const batch = [];
  for (let i = 0; i < TOTAL_CHANGES; i += b) {
    const last = Math.min(i + b, TOTAL_CHANGES) - 1, contents = [];
    for (let c = i; c <= last; c++) if (CARRIER.has(c)) contents.push(CARRIER.get(c));
    batch.push({ first: i, n: last - i + 1, ready: Math.floor(last / DAILY), contents, done: null });
  }
  for (let d = 0; d < DAY * 20; d++) {
    let slot = CAPACITY;
    for (const p of batch) if (p.done === null && p.ready <= d && slot-- > 0) p.done = d;
  }
  // A change's wait: from the day it arrives to the day its batch finishes.
  let wait = 0;
  for (const p of batch)
    for (let c = p.first; c < p.first + p.n; c++) wait += p.done - Math.floor(c / DAILY);
  const open = batch.filter((p) => p.ready < DAY && p.done >= DAY);
  return { daily: batch.length / DAY, pending: open.reduce((s, p) => s + p.n, 0),
    avg: wait / TOTAL_CHANGES,
    sentBack: batch.filter((p) => p.contents.some((k) => k.g[4] === '1')).reduce((s, p) => s + p.n, 0) };
}

const seen = ITEM.filter((k) => k.g[4] === '1').length;
console.log(`${DAILY} changes/day, ${DAY} days, ${TOTAL_CHANGES} changes; ` +
  `gate ${CAPACITY} reviews/day; items the gate can see ${seen}/${ITEM.length}`);
console.log('\nbatch  reviews/day   changes pending at day 60     average wait days     changes sent back');
for (const b of [1, 2, 3, 5, 7]) {
  const r = run(b);
  console.log(String(b).padEnd(7) + r.daily.toFixed(1).padEnd(14) +
    String(r.pending).padEnd(30) + r.avg.toFixed(1).padEnd(22) + r.sentBack);
}

7 changes/day, 60 days, 420 changes; gate 5 reviews/day; items the gate can see 10/23

batch  reviews/day   changes pending at day 60     average wait days     changes sent back
1      7.0           120                           12.0                  10
2      3.5           0                             0.1                   20
3      2.3           0                             0.1                   30
5      1.4           0                             0.3                   50
7      1.0           0                             0.0                   70

The first row gives the gate’s capacity problem: seven changes arrive per day, the gate can finish five. Because the remaining two pile up, 120 changes are still in the queue on day sixty, with an average wait of 12 days. This is not a slowdown, it is a saturated queue: as long as the gap does not close, the wait time keeps growing as time passes.

The teams’ response to this is batching — piling up changes and sending them into a single review. The table shows that batching works: once the batch size rises to two, the queue closes and the average wait drops to a tenth of a day. The last column writes down the cost. While the number of items the gate actually sees stays fixed at ten in every row, the number of changes sent back because of those ten items rises from 10 to 70. At a batch size of seven, six extra legitimate changes are held up for the sake of a single item.

The gate’s second limit stands in the very first row: of the twenty-three items, only ten are in a form the gate can look at. The remaining thirteen items pass through the gate without any trouble, because the gate does not look at a requirements decision or a design gap — the only thing in its hands is the build itself, about to be deployed.

The Same Item Set, a Distributed Check

The second setup takes the same item set and spreads the check across six stages. The comparison is fair for exactly this reason: the only thing that changes is where the check stands.

This is where the naming of the two setups earns its meaning. Development and operations merging into a single flow (DevOps) is not a tool choice, it is a feedback rule: the shorter the loop in which a decision sees its outcome, the more cheaply it gets fixed. The gate setup applies this rule to delivery speed but leaves the security item outside the rule; security’s feedback loop, however fast it runs, passes through a single point at the very end of the flow. Building security into the flow (DevSecOps) consists of applying the same rule to the security item as well. What sets it apart is not adding a new kind of check — five of the six checks in the table could just as well be set up in the gate setup. What sets it apart is placing the check into the loop in which the item is born.

// run2.mjs — same item set, two flows: check at a single gate and spread across stages
import { STAGE, ITEM, firstSeen } from './flow.mjs';

// Per-stage output of a check built into the flow (model): unit run, alerts produced,
// developer minutes per alert.
const CHECK = [['requirement review', 18, 5, 25], ['design session', 12, 9, 30],
  ['static code analysis', 420, 96, 6], ['dependency scan', 420, 54, 8],
  ['configuration audit', 60, 22, 12], ['runtime observation', 60, 14, 20]];

const flows = [['single gate', [4, 5]], ['integrated', [0, 1, 2, 3, 4, 5]]];
console.log('flow         caught     never seen      total delay stages    average delay');
const result = new Map();
for (const [name, stages] of flows) {
  const y = ITEM.map((k) => [k, firstSeen(k, stages)]);
  const found = y.filter(([, i]) => i !== null);
  const late = found.reduce((s, [k, i]) => s + i - k.born, 0);
  result.set(name, y);
  console.log(name.padEnd(13) + String(found.length).padEnd(11) +
    String(ITEM.length - found.length).padEnd(16) + String(late).padEnd(22) +
    (late / found.length).toFixed(2));
}

console.log('\nstage           born   caught in gate flow     caught in integrated flow');
for (const [i, a] of STAGE.entries())
  console.log(a.padEnd(16) + String(ITEM.filter((k) => k.born === i).length).padEnd(7) +
    String(result.get('single gate').filter(([, j]) => j === i).length).padEnd(24) +
    result.get('integrated').filter(([, j]) => j === i).length);

console.log('\ncheck                   unit   alert  actual  false positive  developer minutes');
let alertTotal = 0, minuteTotal = 0;
for (const [i, [name, unit, alerts, minutes]] of CHECK.entries()) {
  const actual = result.get('integrated').filter(([, j]) => j === i).length;
  alertTotal += alerts; minuteTotal += alerts * minutes;
  console.log(name.padEnd(24) + String(unit).padEnd(7) + String(alerts).padEnd(7) +
    String(actual).padEnd(8) + String(alerts - actual).padEnd(16) + alerts * minutes);
}
console.log(`\ntotal ${alertTotal} alerts, ${minuteTotal} minutes (${(minuteTotal / 60).toFixed(1)} hours), ` +
  `per item ${(minuteTotal / 60 / 23).toFixed(1)} hours`);
const gateHours = 300 * 3;
console.log(`gate: 300 reviews x 3 hours = ${gateHours} hours, 10 items, ` +
  `per item ${(gateHours / 10).toFixed(0)} hours`);

flow         caught     never seen      total delay stages    average delay
single gate  18         5               35                    1.94
integrated   23         0               3                     0.13

stage           born   caught in gate flow     caught in integrated flow
requirement     3      0                       3
design          4      0                       4
code            7      0                       6
build           3      0                       3
deployment      4      10                      4
runtime         2      8                       3

check                   unit   alert  actual  false positive  developer minutes
requirement review      18     5      3       2               125
design session          12     9      4       5               270
static code analysis    420    96     6       90              576
dependency scan         420    54     3       51              432
configuration audit     60     22     4       18              264
runtime observation     60     14     3       11              280

total 200 alerts, 1947 minutes (32.5 hours), per item 1.4 hours
gate: 300 reviews x 3 hours = 900 hours, 10 items, per item 90 hours

The first table gives two separate numbers, and they must not be mixed up. In the gate flow, 18 items are caught, but only ten of these are caught at the gate; the remaining eight surface later through a runtime event. Surfacing is not the same as catching, but at least the item becomes known. 5 items, on the other hand, are items that neither setup sees: the classification gap in the requirement, the trust-boundary-free flow in the design, the incomplete error path and the weak randomness in the code, the unnecessary tool in the build. These are not in any record, do not enter any ranking, do not take up any budget.

The delay column reduces the difference to a single number. Items caught in the gate flow are found a total of 35 stages late, 1.94 stages per item. In the integrated flow, total delay is 3 stages, and all of it comes from a single item: unbounded-query-size, born in the code stage, becomes visible only at runtime.

The second table shows this one item directly. Seven items are born in the code stage; the integrated flow catches six of them in the code stage. The seventh belongs to a class no early check can look at, and it stays there no matter how far left you push the check. This course’s second lesson counts exactly that class.

The Cost of Spreading It Out

The third table is the bill for the integrated flow. Six checks produce a total of 200 alerts; 23 of these are real items, 177 are false positives. The ratio varies from check to check: the requirement review gets three of its five alerts right, static code analysis gets six of its ninety-six. As frequency rises, the hit rate falls, because code analysis runs on every one of the 420 changes while the requirement review is convened only eighteen times.

The total load is 1,947 developer-minutes, that is, 32.5 hours. The time the gate flow spends in the same window is 300 reviews times three hours, that is, 900 hours. The cost paid per item is 1.4 hours in the integrated flow, 90 hours in the gate flow — a sixty-four-fold difference. It should be noted that the two numbers do not measure the same work: the gate looks at every change, the integrated check looks only at the ones that produce an alert. That is also where the difference comes from.

The second item of cost does not show up in the table. 177 false positives means interrupting a developer’s work 177 times; ninety of these come from a single check. A check’s real cost is not the time it runs but how many of the alerts it produces have to be read by a human. When the false-positive rate rises, alerts stop getting read, and the real items that check catches become invisible too — this produces the same outcome as never putting the check in place at all, except the cost keeps being paid.

Summary

• In the fictional measurement network’s software, 420 changes flow over 60 business days and carry 23 items; items are born in all six stages, most of all in the code stage (7 items).
• A single gate standing before deployment can see only 10 of the 23 items. 7 changes arrive per day, 5 reviews get finished; by day 60, 120 changes are waiting in the queue and the average wait is 12 days.
• Batching closes the queue, but its cost is legitimate changes sent back: at batch size 1 it is 10, at batch size 7 it is 70 changes held up because of the same 10 items.
• When the check is spread across six stages, all 23 items are caught; items never seen drop from 5 to 0, total delay drops from 35 stages to 3 stages, and delay per item drops from 1.94 to 0.13 stages.
• The cost of spreading it out is that 177 of 200 alerts are false positives; 576 of the 1,947 developer-minutes go to the 96 alerts produced by a single check.
• Of the 7 items born in the code stage, 6 are caught in the code stage; one becomes visible only at runtime, and an early check does not lower that item’s delay.

Next Step

This lesson moved the check and counted that the number of items caught went up while the delay went down. The one thing not measured is what that delay actually costs. A delay of thirty-five stages is a number, but the difference between fixing an item at the design stage and fixing it at runtime is not a stage count, it is the work spent: changing a design decision is a meeting; taking the same decision back out of a deployed system is a data migration, a release plan, and an outage window. The next lesson puts the per-stage fix cost into the model, runs the same item set through two flows with late and early checks, and calculates the overall multiplier. The same lesson also asks a second question: for which items that multiplier does not hold — that is, how large is the class of items for which catching them early gains nothing at all.