Graceful Degradation

The previous lesson built the gate and openly rejected the rejected request. One check on the rejection was left undone: when a request drops, everything gathered for it up to that point drops with it. A tracking query’s response is not a single piece — the shipment’s state, route, estimated delivery time, and fee are separate fields from separate dependencies. When one dependency goes down, dropping the whole request also throws away the other three fields already available.

This lesson takes up the last option: deciding which part of the response is core, and still answering when non-core fields do not arrive. Its name is graceful degradation, not to be confused with graceful shutdown: graceful shutdown is a process finishing the work it holds while shutting down and taking no new work, established in the Server-Side Fundamentals and Caching, Queues and Asynchronous Processing courses. Graceful degradation, by contrast, is a still-standing system shrinking the content of its response.

Fields and the Core Set

AY14 — core set, fallback store, and two failure scenarios. The core set is the state and route fields; estimate and fee are optional. Rationale: the tracking page is meaningless without showing where the shipment is and what state it is in, while estimated delivery and fee are supporting information. The fallback store keeps the last known value for 5000 shipments, and a stale value is served for at most 60 rounds. AY14a: the billing service stops, duration 15 minutes, once a month. AY14b: the projection read model stops, duration 15 minutes, once a month. Both durations sit a little above K01’s ten-minute recovery-time assumption, and neither is added to K01’s table.

The fields’ byte sizes split K01’s 480-byte tracking response: state 180, route 120, estimate 90, fee 90. Three policies are compared. All or nothing drops the request if any field is missing. Field skip requires the core but drops a missing optional field from the response. Stale fallback additionally serves the last known value — itself a form of degradation, its width set by the staleness bound.

// degradation/response.mjs — MODEL of the tracking response: fields are gathered from separate
// dependencies. A round is an abstract step (one round = one second) and each round represents
// one sample request from the flow. No real service, network, or store is set up.
export const K01 = { readPeak: 416.67, responseBytes: 480, failureShare: 28.2 };

// AY14: core set {state, route}; total bytes give K01's 480-byte tracking response.
export const FIELD = [
  { name: "state", source: "projection", core: true, bytes: 180 },
  { name: "route", source: "projection", core: true, bytes: 120 },
  { name: "estimate", source: "estimate", core: false, bytes: 90 },
  { name: "fee", source: "billing", core: false, bytes: 90 },
];
export const MARKER_BYTES = 6;           // freshness marker per field
export const FALLBACK_RECORDS = 5000;    // shipment count kept in the fallback store
export const STALENESS_BOUND = 60;       // AY14: a stale value is served for at most 60 rounds
const TRANSIENT = 40;                    // transient timeout interval on a healthy dependency

export function run({ policy, mode, warmup = 120, failure = 900 }) {
  const fallback = new Map();            // field -> round of the last fresh value
  const s = {
    full: 0, reduced: 0, dropped: 0, freshField: 0, staleField: 0, missingField: 0,
    falseDegradation: 0, totalBytes: 0, lastResponseRound: 0,
  };

  for (let t = 1; t <= warmup + failure; t += 1) {
    const failing = t > warmup;
    const outcome = [];
    FIELD.forEach((a, i) => {
      const health = failing ? mode[a.source] : "healthy";
      const transient = health === "healthy" && (t + i * 10) % TRANSIENT === 0;
      if (health === "healthy" && transient === false) {
        fallback.set(a.name, t); outcome.push({ a, status: "fresh" }); return;
      }
      const age = fallback.has(a.name) ? t - fallback.get(a.name) : Infinity;
      if (policy === "fallback" && age <= STALENESS_BOUND) {
        outcome.push({ a, status: "stale" });
        if (transient && failing) s.falseDegradation += 1;   // fresh value existed, transient timeout cut it
        return;
      }
      outcome.push({ a, status: "missing" });
    });
    if (failing === false) continue;      // warmup rounds are not counted

    const missingCore = outcome.some((o) => o.a.core && o.status === "missing");
    const anyMissing = outcome.some((o) => o.status !== "fresh");
    if (policy === "all" ? anyMissing : missingCore) { s.dropped += 1; continue; }

    let bytes = 0;
    for (const o of outcome) {
      if (o.status === "fresh") s.freshField += 1;
      else if (o.status === "stale") s.staleField += 1;
      else { s.missingField += 1; continue; }
      bytes += o.a.bytes + MARKER_BYTES;
    }
    s.totalBytes += bytes;
    s.lastResponseRound = t - warmup;
    if (anyMissing) s.reduced += 1; else s.full += 1;
  }
  s.responses = s.full + s.reduced;
  s.avgBytes = s.responses === 0 ? 0 : s.totalBytes / s.responses;
  s.round = failure;
  return s;
}

Warmup rounds represent the failure starting after the system has run healthy for a while. The fallback store can only serve a value that was filled in during that time; a field never seen before has no fallback.

// degradation/run.mjs — comparing three policies over three days, and isolation's limit
import { K01, FIELD, MARKER_BYTES, FALLBACK_RECORDS, STALENESS_BOUND, run } from "./response.mjs";

const FAILURE = 900;                 // AY14: 15 minutes = 900 rounds
const POLICY = [["all or nothing", "all"], ["field skip", "skip"], ["+ stale fallback", "fallback"]];
const HEALTHY = { projection: "healthy", estimate: "healthy", billing: "healthy" };
const DAY = [
  ["failure-free", HEALTHY],
  ["AY14a: billing stops (optional field)", { ...HEALTHY, billing: "down" }],
  ["AY14b: projection stops (core field)", { ...HEALTHY, projection: "down" }],
];

console.log(`${FAILURE} rounds (after 120 rounds of warmup); staleness bound ${STALENESS_BOUND} rounds`);
console.log(`fields: ` + FIELD.map((a) => `${a.name}(${a.source}, ${a.core ? "core" : "optional"}, ${a.bytes} B)`).join(", "));
console.log(`total field bytes ${FIELD.reduce((x, a) => x + a.bytes, 0)} = K01 tracking response ${K01.responseBytes} B\n`);

for (const [day, mode] of DAY) {
  console.log(`-- ${day} --`);
  console.log(`${"policy".padEnd(20)}${"responses".padStart(10)}${"full".padStart(5)}${"reduced".padStart(11)}` +
    `${"dropped".padStart(8)}${"fresh/stale/missing field".padStart(26)}${"avg. bytes".padStart(11)}${"last response round".padStart(20)}`);
  for (const [label, p] of POLICY) {
    const r = run({ policy: p, mode, failure: FAILURE });
    console.log(`${label.padEnd(20)}${String(r.responses).padStart(10)}${String(r.full).padStart(5)}` +
      `${String(r.reduced).padStart(11)}${String(r.dropped).padStart(8)}` +
      `${`${r.freshField}/${r.staleField}/${r.missingField}`.padStart(26)}${r.avgBytes.toFixed(1).padStart(11)}` +
      `${String(r.lastResponseRound).padStart(20)}`);
  }
  console.log();
}

const a = run({ policy: "all", mode: HEALTHY, failure: FAILURE });
const y = run({ policy: "fallback", mode: HEALTHY, failure: FAILURE });
console.log(`-- failure-free day's cost --`);
console.log(`freshness marker ${MARKER_BYTES} B x ${FIELD.length} fields = ${MARKER_BYTES * FIELD.length} B/response` +
  ` = ${((MARKER_BYTES * FIELD.length) / K01.responseBytes * 100).toFixed(1)}% of K01's response,` +
  ` ${((K01.readPeak * MARKER_BYTES * FIELD.length) / 1024).toFixed(2)} KiB/s at peak read`);
console.log(`fallback store ${FALLBACK_RECORDS} shipments x ${K01.responseBytes} B = ` +
  `${((FALLBACK_RECORDS * K01.responseBytes) / 1024 / 1024).toFixed(2)} MiB; every fresh field is one store write ->` +
  ` ${(K01.readPeak * FIELD.length).toFixed(2)} writes/s`);
console.log(`the strict policy drops ${a.dropped}/${FAILURE} requests on a failure-free day` +
  ` -> ${((K01.readPeak * a.dropped) / FAILURE).toFixed(2)} req/s at peak read`);
console.log(`with stale fallback, the same day drops ${y.dropped}, serves ${y.staleField} fields stale` +
  ` = ${((y.staleField / (FAILURE * FIELD.length)) * 100).toFixed(2)}% of fields`);

console.log(`\n-- impact on the downtime budget (AY14: 15 min, once a month) --`);
console.log(`${"scenario".padEnd(9)}${"policy".padEnd(21)}${"answered".padStart(11)}` +
  `${"outage(min)".padStart(12)}${"of failure share(28.2)".padStart(23)}`);
for (const [label, mode] of [["AY14a", DAY[1][1]], ["AY14b", DAY[2][1]]]) {
  for (const [pad, p] of POLICY) {
    const r = run({ policy: p, mode, failure: FAILURE });
    const min = 15 * (r.dropped / FAILURE);
    console.log(`${label.padEnd(9)}${pad.padEnd(21)}${String(r.responses).padStart(11)}` +
      `${min.toFixed(2).padStart(12)}${`${((min / K01.failureShare) * 100).toFixed(1)}%`.padStart(23)}`);
  }
}

console.log(`\n-- isolation's limit: what each source going down leaves --`);
console.log(`${"failed source".padEnd(16)}${"affected field".padStart(15)}${"core?".padStart(12)}` +
  `${"responses".padStart(10)}${"avg. bytes".padStart(11)}${"last response round".padStart(20)}`);
for (const source of ["billing", "estimate", "projection"]) {
  const affected = FIELD.filter((x) => x.source === source);
  const r = run({ policy: "fallback", mode: { ...HEALTHY, [source]: "down" }, failure: FAILURE });
  console.log(`${source.padEnd(16)}${affected.map((x) => x.name).join("+").padStart(15)}` +
    `${(affected.some((x) => x.core) ? "yes" : "no").padStart(12)}` +
    `${String(r.responses).padStart(10)}${r.avgBytes.toFixed(1).padStart(11)}${String(r.lastResponseRound).padStart(20)}`);
}

900 rounds (after 120 rounds of warmup); staleness bound 60 rounds
fields: state(projection, core, 180 B), route(projection, core, 120 B), estimate(estimate, optional, 90 B), fee(billing, optional, 90 B)
total field bytes 480 = K01 tracking response 480 B

-- failure-free --
policy               responses full    reduced dropped fresh/stale/missing field avg. bytes last response round
all or nothing             810  810          0      90                  3240/0/0      504.0                 899
field skip                 856  810         46      44                 3378/0/46      498.8                 900
+ stale fallback           900  810         90       0                 3510/90/0      504.0                 900

-- AY14a: billing stops (optional field) --
policy               responses full    reduced dropped fresh/stale/missing field avg. bytes last response round
all or nothing               0    0          0     900                     0/0/0        0.0                   0
field skip                 856    0        856      44                2545/0/879      405.4                 900
+ stale fallback           900    0        900       0              2633/127/840      414.4                 900

-- AY14b: projection stops (core field) --
policy               responses full    reduced dropped fresh/stale/missing field avg. bytes last response round
all or nothing               0    0          0     900                     0/0/0        0.0                   0
field skip                   0    0          0     900                     0/0/0        0.0                   0
+ stale fallback            59    0         59     841                 115/121/0      504.0                  59

-- failure-free day's cost --
freshness marker 6 B x 4 fields = 24 B/response = 5.0% of K01's response, 9.77 KiB/s at peak read
fallback store 5000 shipments x 480 B = 2.29 MiB; every fresh field is one store write -> 1666.68 writes/s
the strict policy drops 90/900 requests on a failure-free day -> 41.67 req/s at peak read
with stale fallback, the same day drops 0, serves 90 fields stale = 2.50% of fields

-- impact on the downtime budget (AY14: 15 min, once a month) --
scenario policy                  answered outage(min) of failure share(28.2)
AY14a    all or nothing                 0       15.00                  53.2%
AY14a    field skip                   856        0.73                   2.6%
AY14a    + stale fallback             900        0.00                   0.0%
AY14b    all or nothing                 0       15.00                  53.2%
AY14b    field skip                     0       15.00                  53.2%
AY14b    + stale fallback              59       14.02                  49.7%

-- isolation's limit: what each source going down leaves --
failed source    affected field       core? responses avg. bytes last response round
billing                     fee          no       900      414.4                 900
estimate               estimate          no       900      414.4                 900
projection          state+route         yes        59      504.0                  59

All these numbers belong to the computed value class: they are counted over a deterministic sequence of rounds.

Failure-Free Day’s Cost

Three costs are paid, all three even without a failure.

The field it carries. Every field carries a freshness marker: whether the value is fresh or stale, and if stale, how old. Six bytes, four fields, 24 bytes per response — 5.0 percent of K01’s 480-byte tracking response, 9.77 KiB/s at the peak read rate. Without this marker, degradation is silent, and silent degradation is its most dangerous form: the client mistakes a stale fee for a current one.

The fallback store. The last known value for five thousand shipments takes 2.29 MiB, but the real cost is not memory, it is writes: every fresh field updates the store, 1666.68 writes per second at peak read — a load paid continuously even with no failure, and the pattern’s most expensive item.

Fields served stale. With stale fallback on, even on a failure-free day, 90 of 3600 fields are served stale — 2.50 percent. These are transient timeouts: the fresh value would have arrived the next round, but the fallback engaged immediately. The fallback path’s own freshness is itself a cost.

Against this, the first table also shows the strict policy’s own cost: on a failure-free day it drops 90 of 900 requests, 41.67 requests per second at peak read, with no dependency down at all. The strict policy is not a baseline; it is a defect in its own right.

The Failing Day’s Gain

The AY14a table shows what the pattern buys. When the billing service stops for fifteen minutes, the strict policy drops 900 of 900 requests: the entire fifteen minutes is an outage, eating 53.2 percent of K01’s monthly 28.2-minute failure share — a single failure spending more than half the month’s allowance.

Field skip answers 856 requests in the same failure, and the outage drops to 0.73 minutes — 2.6 percent of the share. Adding stale fallback answers 900 of 900, and the outage zeroes out. The response’s average size drops from 504 to 414.4 bytes; the user sees where the shipment is and its state, but not the fee.

A design decision becomes visible here. The fee field is served stale for the first sixty rounds, then dropped once the staleness bound is exceeded: the table shows 127 stale and 840 missing. A stale fee is close to true for a while; after fifteen minutes it is outright wrong. The bound is the name for that “while.”

Isolation’s Limit

The AY14b table writes this topic’s closing sentence. When the projection read model — the sole source of both core fields — stops, all three policies collapse. The strict policy and field skip drop 900 of 900 requests: fifteen minutes of outage, 53.2 percent of the share. Stale fallback answers only 59 requests and falls silent at round 59; outage 14.02 minutes, 49.7 percent.

The gain is one minute, exactly the staleness bound. Degradation does not carry the core, it only postpones. The last table sums this up in three rows: an optional source going down leaves the response count at 900; a core source going down drops it to 59, with the last response given at round 59.

This says something about the topic as a whole. Every pattern here contains the failure: it takes the broken dependency out of the loop, separates resource pools, bounds retries, budgets time, reports overload to the source, rejects requests exceeding capacity, trims the response’s perimeter. None of them removes the failure. The broken part stays broken; the only difference is that the system around it does not break with it.

When a core part breaks, containment has nothing left to give. What is needed then is a copy to stand in for the broken part — and no such copy was ever designed in this topic. Where the copy stands, when it takes over, how many requests drop during the takeover, and how much a wrongly made takeover decision costs are all open questions.

Summary

• Graceful degradation splits the response’s fields into core and optional sets and answers if the core is present; distinct from graceful shutdown — one is a shutting-down process finishing its work, the other a still-standing system shrinking its response.
• The failure-free day’s cost has three items: a 24-byte freshness marker per response (5.0 percent of K01’s response, 9.77 KiB/s at peak read), a 2.29 MiB fallback store with 1666.68 writes per second, and 2.50 percent of fields served stale even without a failure.
• Under AY14a (optional dependency down), the strict policy drops every one of 900 requests and, at 15.00 minutes, eats 53.2 percent of K01’s 28.2-minute failure share; field skip brings this to 0.73 minutes, stale fallback to 0.00. The response’s average size drops from 504 to 414.4 bytes.
• Stale fallback does not carry the core, it postpones it: under AY14b (core dependency down), only 59 requests are answered, the outage falls from 15.00 to 14.02 minutes — a gain exactly the staleness bound.
• Every pattern in this topic contains the failure, none removes it; when a core part breaks, containment has nothing left to give.

Next Step

The gap this topic leaves fits in one sentence: there is nothing to stand in for the broken part. The isolation patterns sealed the failure inside a box and kept the system outside it standing; the part inside is still broken, and no mechanism to fix it was designed. AY14b’s 14.02 minutes is exactly the measure of that gap — containment cannot reach past it. The next topic looks inside the box and asks the first question: if a copy stands ready to replace the broken part, how is its takeover decided. The decision itself is a source of failure, because a slowed-down part and a stopped one look the same from the outside, and a wrongly made takeover decision can cost more than never taking over at all. That topic takes up the failover window, what triggers the decision, and how many minutes a takeover writes to the downtime budget.