What Is DevOps?

The previous lesson measured handover as a contract: twenty of the twenty-eight responsibility items were handed over, eight stayed in development for lack of information, tooling, or procedure. The contract wrote down who would do what. What it did not write down was who would see what. Two roles can be jointly responsible for the same service and still see the same occurrence at a different step, on a different surface, sometimes not at all.

Shared responsibility holds the development and operations roles jointly accountable for a service’s outcome. A handover contract divides items; shared responsibility divides the outcome, and the second requires, as a precondition, that both roles can see the same occurrence. This lesson measures that precondition: the same set of occurrences is dropped into two roles’ observation scope, how many occurrences stay visible to only one side is counted, and the bill for combining scope is worked out.

Observation Scope Is a Data Structure

The example is this curriculum’s fictional regional measurement network: software that collects readings from a municipality’s water meters, verifies the readings, turns them into invoices, and opens work orders for field crews. The network and the teams are both fictional. CF1: there are three environments (development, test, production), two teams (development and operations), and one nightly batch job. CF2: there are eight observation surfaces; three sit in the development team’s scope, four in the operations team’s scope, and invoice-statement sits in no one’s scope. CF3: twenty-two occurrences are born within one window; each occurrence has a class (error, delay, drift), a birth step, and the surfaces it appears on together with its delay on each surface. CF4: every occurrence has an owner who can close it — development, operations, or shared; a shared occurrence cannot be closed until it is visible in both scopes, and closing takes 4, 2, and 5 steps respectively. CF5: an occurrence visible in a scope writes 2 steps of triage for that role; an occurrence that has not reached its owner within 30 steps is surfaced from above, and because the context has to be rebuilt secondhand this adds 2 extra steps. CF6: the development team works toward a release during its idle steps, and a release needs 8 steps of work; every occurrence attaches to the last release before it, and a release with an attached occurrence that never reaches its owner counts as failed. The window is 140 steps.

The occurrences, the surfaces, and the two roles’ rotation are a process-internal model; there is no real alert stream or incident log.

// observation/scope.mjs — the fictional regional measurement network's occurrence and observation-scope
// model (model): there is no real alert stream or incident log, steps are a data structure. No randomness.

export const WINDOW = 140;                   // measured step window
export const TRIAGE = 2;                     // cost to one role of triaging an occurrence (steps)
export const RELEASE_WORK = 8;               // work steps a release needs
export const CLOSURE = { dev: 4, ops: 2, shared: 5 };
export const CEILING = 30;                   // an occurrence that never reaches its owner surfaces at this delay
export const SECOND_HAND = 2;                // extra context cost of an occurrence surfaced this way

export const SURFACE = {                     // surface -> whose scope it sits in, in the separate arrangement
  "app-record": "dev", "error-tracking": "dev", "release-log": "dev",
  "work-schedule": "ops", "machine-metrics": "ops", "alert-stream": "ops",
  "support-record": "ops", "invoice-statement": "-",
};
export const ROLE = ["dev", "ops"];

const table = (s, f) => s.trim().split("\n").map((r) => f(r.trim().split(/\s+/)));

// occurrence: birth step, class, role that can close it, name, surfaces it appears on (surface:delay)
export const OCCURRENCE = table(`
    2  error  dev    reading-parser            app-record:0 support-record:12
    6  delay  ops    batch-overflow            work-schedule:1 machine-metrics:2
   11  drift  dev    untested-setting          release-log:0 alert-stream:9
   17  error  ops    queue-jam                 machine-metrics:0 alert-stream:1
   21  delay  dev    query-slowdown            app-record:3 machine-metrics:1
   26  error  dev    duplicate-invoice         support-record:16
   31  drift  ops    disk-full                 machine-metrics:0
   36  error  dev    blank-address-work-order  error-tracking:2 support-record:9
   42  delay  ops    network-latency           machine-metrics:1 alert-stream:2
   46  drift  dev    library-drift             release-log:1
   51  error  shared nightly-job-incomplete    work-schedule:1 app-record:4
   57  error  dev    negative-consumption      invoice-statement:0 support-record:14
   61  delay  ops    backup-collision          work-schedule:2 machine-metrics:3
   66  drift  shared timezone-difference       app-record:6 work-schedule:3
   71  error  dev    rounding-difference       invoice-statement:0
   77  delay  dev    cache-warmup              error-tracking:2 machine-metrics:1
   81  error  ops    connection-pool           alert-stream:1 app-record:0
   86  drift  dev    missing-env-var           app-record:0 alert-stream:5
   91  error  dev    meter-id-conflict         error-tracking:2 support-record:10
   96  delay  ops    job-rerun                 work-schedule:1
  102  drift  dev    format-break              invoice-statement:0 support-record:15
  107  error  shared duplicate-work-order      app-record:3 support-record:8`,
([born, cls, owner, name, ...y]) => ({ born: +born, cls, owner, name,
  surface: y.map((p) => ({ name: p.split(":")[0], delay: +p.split(":")[1] })) }));

// Arrangement: on top of the separate arrangement, which role additionally sees which surface (contrast run).
export const ARRANGEMENT = {
  separate: { dev: [], ops: [] },
  selected: { dev: ["support-record"], ops: ["release-log", "invoice-statement"] },
  combined: { dev: Object.keys(SURFACE), ops: Object.keys(SURFACE) },
};

export const scope = (d) => Object.fromEntries(ROLE.map((r) =>
  [r, new Set([...Object.keys(SURFACE).filter((y) => SURFACE[y] === r), ...ARRANGEMENT[d][r]])]));

export function run(d) {
  const S = scope(d);
  const G = OCCURRENCE.map((o) => {              // in which step each occurrence is seen by each role
    const g = Object.fromEntries(ROLE.map((r) => {
      const v = o.surface.filter((y) => S[r].has(y.name)).map((y) => y.delay);
      return [r, v.length ? Math.min(...v) : null];
    }));
    const closeStep = o.owner === "shared"        // a shared occurrence must be seen in both scopes
      ? (g.dev === null || g.ops === null ? null : Math.max(g.dev, g.ops))
      : g[o.owner];
    return { o, g, closeStep, ownerRole: o.owner === "shared" ? "dev" : o.owner };
  });

  const queue = Object.fromEntries(ROLE.map((r) => [r, []]));
  const remaining = Object.fromEntries(ROLE.map((r) => [r, 0]));
  const load = Object.fromEntries(ROLE.map((r) => [r, 0]));
  const closure = new Map(), releases = [];
  let accumulated = 0, start = 0;

  for (let t = 0; t <= WINDOW; t++) {
    for (const x of G) {
      for (const r of ROLE)                       // an occurrence seen in scope opens triage work
        if (x.g[r] !== null && t === x.o.born + x.g[r]) queue[r].push([TRIAGE, null]);
      if (x.closeStep !== null && t === x.o.born + x.closeStep)
        queue[x.ownerRole].push([CLOSURE[x.o.owner], x]);
      else if (x.closeStep === null && t === x.o.born + CEILING)   // surfaced from above
        queue[x.ownerRole].push([CLOSURE[x.o.owner] + SECOND_HAND, x]);
    }
    for (const r of ROLE) {
      if (remaining[r] === 0 && queue[r].length) [remaining[r]] = queue[r][0];
      if (remaining[r] > 0) {                      // role is busy with an occurrence this step
        remaining[r]--; load[r]++;
        if (remaining[r] === 0) { const [, x] = queue[r].shift(); if (x) closure.set(x.o.name, t + 1); }
        continue;
      }
      if (r !== "dev") continue;                   // idle step: the dev team works toward a release
      if (accumulated === 0) start = t;
      if (++accumulated === RELEASE_WORK) { releases.push({ t: t + 1, start, occurrence: [] }); accumulated = 0; }
    }
  }
  for (const x of G) {                             // an occurrence attaches to the release before it
    const y = [...releases].reverse().find((v) => v.t <= x.o.born);
    if (y) y.occurrence.push(x);
  }
  return { G, load, closure, releases };
}

export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-");

export function measure(d) {
  const R = run(d), count = (f) => R.G.filter(f).length;
  const reach = R.G.map((x) => (x.closeStep === null ? CEILING : x.closeStep));   // ceiling for those never reached
  const duration = R.G.map((x) => (R.closure.get(x.o.name) ?? WINDOW) - x.o.born);
  const broken = R.releases.filter((y) => y.occurrence.some((x) => x.closeStep === null)).length;
  return { R,
    devOnly: count((x) => x.g.dev !== null && x.g.ops === null),
    opsOnly: count((x) => x.g.ops !== null && x.g.dev === null),
    both: count((x) => x.g.dev !== null && x.g.ops !== null),
    neither: count((x) => x.g.dev === null && x.g.ops === null),
    unreached: count((x) => x.closeStep === null), reachStep: avg(reach),
    noiseDev: count((x) => x.g.dev !== null && x.o.owner === "ops"),
    noiseOps: count((x) => x.g.ops !== null && x.o.owner === "dev"),
    loadDev: R.load.dev, loadOps: R.load.ops, recovery: avg(duration),
    releaseCount: R.releases.length, leadTime: avg(R.releases.map((y) => y.t - y.start)),
    changeFailureRate: R.releases.length ? (100 * broken / R.releases.length).toFixed(0) + "%" : "-" };
}

The four delivery metrics are read from the same run: deployment frequency is the number of releases in the window, lead time is the steps from when work starts on a release to when it goes out, change failure rate is the share of failed releases, time to restore is the steps from an occurrence’s birth to its closure.

// observation/measure.mjs — how an occurrence falls into the two roles' observation scope, and what combining the scopes costs.
import { OCCURRENCE, SURFACE, ROLE, measure, scope, CEILING } from "./scope.mjs";

const print = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const D = ["separate", "selected", "combined"];
const R = Object.fromEntries(D.map((d) => [d, measure(d)]));

console.log(`${OCCURRENCE.length} occurrences, ${Object.keys(SURFACE).length} observation surfaces; ` +
  ["error", "delay", "drift"].map((s) =>
    `${s} ${OCCURRENCE.filter((o) => o.cls === s).length}`).join(", "));
for (const r of ROLE) console.log(`  ${r}: ${[...scope("separate")[r]].join(", ")}`);
console.log(`  in no one's scope: ` +
  Object.keys(SURFACE).filter((y) => SURFACE[y] === "-").join(", "));

const A = [-33, 9, 9, 11];
console.log("\n1. same 22 occurrences, three observation arrangements");
print(A, "measure", "separate", "selected", "combined");
for (const [ad, k] of [["dev-only scope", "devOnly"],
  ["ops-only scope", "opsOnly"], ["both scopes", "both"],
  ["no scope", "neither"], ["never reaches owner", "unreached"],
  [`owner reach (avg. step)`, "reachStep"], ["noise to dev (occurrences)", "noiseDev"],
  ["noise to ops (occurrences)", "noiseOps"], ["dev load (steps)", "loadDev"],
  ["ops load (steps)", "loadOps"]]) print(A, ad, ...D.map((d) => R[d][k]));

console.log("\n2. four delivery metrics, same window");
print(A, "metric", "separate", "selected", "combined");
for (const [ad, k] of [["deployment frequency (releases)", "releaseCount"],
  ["lead time (avg. step)", "leadTime"], ["change failure rate", "changeFailureRate"],
  ["time to restore (avg. step)", "recovery"]]) print(A, ad, ...D.map((d) => R[d][k]));

const B = [-11, 12, 10, 10, 22];
console.log("\n3. in the separate arrangement, which scope an occurrence class falls into");
print(B, "class", "occurrence", "dev-only", "ops-only", "never reaches owner");
for (const s of ["error", "delay", "drift"]) {
  const g = R.separate.R.G.filter((x) => x.o.cls === s);
  print(B, s, g.length, g.filter((x) => x.g.dev !== null && x.g.ops === null).length,
    g.filter((x) => x.g.ops !== null && x.g.dev === null).length,
    g.filter((x) => x.closeStep === null).length);
}

console.log("\n4. occurrences that never reach their owner in the separate arrangement (ceiling " + CEILING + " steps)");
const C = [-26, -9, 10, 22];
print(C, "occurrence", "owner", "closure", "scope it appears in");
for (const x of R.separate.R.G.filter((x) => x.closeStep === null))
  print(C, x.o.name, x.o.owner, R.separate.R.closure.get(x.o.name) - x.o.born,
    ROLE.filter((r) => x.g[r] !== null).join("+") || "neither");

22 occurrences, 8 observation surfaces; error 10, delay 6, drift 6
  dev: app-record, error-tracking, release-log
  ops: work-schedule, machine-metrics, alert-stream, support-record
  in no one's scope: invoice-statement

1. same 22 occurrences, three observation arrangements
measure                           separate selected   combined
dev-only scope                           1        0          0
ops-only scope                           9        7          0
both scopes                             11       15         22
no scope                                 1        0          0
never reaches owner                      4        1          0
owner reach (avg. step)                7.0      5.0        1.6
noise to dev (occurrences)               1        1          7
noise to ops (occurrences)              10       12         12
dev load (steps)                        95       95        107
ops load (steps)                        54       58         58

2. four delivery metrics, same window
metric                            separate selected   combined
deployment frequency (releases)          5        5          4
lead time (avg. step)                 25.8     27.0       34.8
change failure rate                    40%      20%         0%
time to restore (avg. step)           13.5     11.9       10.2

3. in the separate arrangement, which scope an occurrence class falls into
class        occurrence  dev-only  ops-only   never reaches owner
error                10         0         3                     3
delay                 6         0         4                     0
drift                 6         1         2                     1

4. occurrences that never reach their owner in the separate arrangement (ceiling 30 steps)
occurrence                owner       closure   scope it appears in
duplicate-invoice         dev              42                   ops
negative-consumption      dev              42                   ops
rounding-difference       dev              40               neither
format-break              dev              36                   ops

The numbers are of the measurement kind; their inputs are the assumptions above.

The Same Occurrence, Two Separate Scopes

The first table’s separate column is today’s split. Eleven of the twenty-two occurrences appear in both scopes, nine only in operations’ scope, one only in development’s scope, and one appears in neither scope — rounding-difference, which falls on the invoice-statement surface, is on no one’s rotation. The third table breaks this down by class: four of the six delays sit only in operations’ scope, because delay falls on machine metrics and the work schedule; one of the six drifts sits only in development’s scope, because only the development team reads the release log.

The real number is in the bottom row: four occurrences never reach their owner. The fourth table lists them by name, and the owner of all four is the development team. Three of them appear in operations’ scope — they fall on the support record, so the operations team sees them but cannot close them; one appears nowhere. Being surfaced from above takes 30 steps, and their closure falls between 36 and 42 steps. Averaged together with the occurrences that do reach their owner within the same window, the average time for a signal to reach its owner is 7.0 steps.

The ten occurrences in operations’ scope are occurrences operations cannot close; in other words, in the separate arrangement the operations team is already looking at a pile of occurrences it cannot close. A split observation scope does not destroy the signal — it puts it on the wrong side.

The Bill for Combining Scope

The combined column gives both roles all eight surfaces. Visibility becomes complete: all twenty-two occurrences appear in both scopes, zero never reach their owner, and the average reach time drops from 7.0 to 1.6 steps. The cost is in the same column: the noise landing on the development team — occurrences it cannot close — rises from 1 to 7, and its triage load rises from 95 to 107 steps.

Those twelve steps show up in the second table as a delivery metric. Deployment frequency falls from 5 to 4 releases, and lead time rises from 25.8 to 34.8 steps. In exchange, the change failure rate falls from 40% to 0%, and time to restore falls from 13.5 to 10.2 steps. Two of the four metrics improve, and two get worse; a sentence that looks at only one metric cannot carry this decision.

The selected column is a third option: only three of the eight surfaces are shared — the development team sees the support record, and the operations team sees the release log and the invoice statement. Occurrences that never reach their owner fall from 4 to 1, average reach time falls from 7.0 to 5.0 steps, the change failure rate falls from 40% to 20%, and time to restore falls from 13.5 to 11.9 steps. Deployment frequency stays at 5 releases, and lead time rises by only 1.2 steps. The development team’s load does not change at all: 95 steps stays 95 steps. The triage work brought by the three surfaces exactly covers the secondhand context-building cost of the three occurrences that are no longer surfaced from above.

Where the Difference Hides

In this lesson, the difference hides in the split of the observation scope, and its number is this: four of the eight surfaces sit in a single role, one sits in no role; ten of the twenty-two occurrences appear in a single scope, one in neither. The declared part of the difference is the surface split — which team looks at what is known. The undeclared part is which occurrence that split leaves on which side; that is only visible in the run.

How many steps it takes the signal to reach whom also reads from here: in the separate arrangement it reaches its owner in an average of 7.0 steps, in 5.0 steps once three surfaces are shared, and in 1.6 steps once the scopes are combined. This is the technical side of shared responsibility — who the surfaces are opened to; the organizational side is whose flow the triage load of an opened surface is deducted from.

Summary

• Twenty-two occurrences are split across the separate observation scopes: 11 appear in both scopes, 9 only in operations’ scope, 1 only in development’s scope, 1 in neither; 4 occurrences never reach their owner, and the owner of all four is the development team.
• The signal reaching its owner takes an average of 7.0 steps in the separate arrangement, 5.0 steps once three surfaces are shared, and 1.6 steps once the scopes are combined.
• The cost of combining falls on the development team’s flow: noise rises from 1 to 7 occurrences and load rises from 95 to 107 steps; deployment frequency falls from 5 to 4 releases and lead time stretches from 25.8 to 34.8 steps.
• In exchange, the change failure rate falls from 40% to 0% and time to restore falls from 13.5 to 10.2 steps: two of the four metrics improve, and two get worse.
• Selectively sharing three surfaces brings most of the improvement (1 unreached, 5.0-step reach, 20% change failure rate) and leaves the development team’s load fixed at 95 steps.

Next Step

When observation scope is split, the signal arrives late or never — that is now measured. But the split is not only in observation: the work itself is split too, and a change gets thrown from team to team. The next lesson runs the same set of changes through two arrangements — separate teams and shared responsibility — and counts the wait per boundary, the changes that bounce back because context could not cross it, and the number of errors that surface in production.