External Configuration Store

The previous three lessons’ decisions sit as numbers embedded in code: the claim check’s lifetime is a constant, the key duration is a constant, the transfer pool’s size is a constant, the idempotency ledger’s window is a constant. If a failure calls for shortening the key duration or closing the proof path entirely, the only way is a redeploy, and a redeploy draws from the 15.0-minute planned share within K01’s monthly 43.2-minute outage budget.

The external configuration store pattern keeps runtime settings in a single place outside the application unit, with the replicas reading from there. The Application Scaffolding course established defining configuration by schema, validating it at startup, and separating out secrets; those rules are not repeated here. This lesson’s question is runtime: how long a change takes to reach the replicas, how much caching cuts the store’s load, what happens when the store goes down, and how many replicas a wrongly set value spreads to.

Propagation and the Mixed-Value Window

The rig is an in-process model: there is no real store, network, or replica process; time advances in modeled seconds, and no number depends on this machine’s speed. Replica count and per-replica request rate are taken from the Stateless Services lesson in the Application Layer and Service Interaction course: 3 replicas, 171.30 requests/s per replica.

DD16 — refresh interval 30 seconds. Rationale: the longest acceptable wait for a setting to take effect is kept under a minute. Its sensitivity is given at 5 and 300 seconds. DD17 — a wrong value’s detection time is 120 seconds. Rationale: a wrong value shows up as a spike in the error rate, and confirming it takes time; how the spike is detected is not this course’s subject — it belongs to the Performance Anti-Patterns and Monitoring course.

// config/store.mjs — in-process model of the external configuration store and the replica caches.
// There is no real store, network, or replica process; time advances in modeled seconds, counters are explicit.

// rollout: "all-at-once" = the value is valid for every replica; "staged" = replica 0 first, then the rest once confirmed
export function target(replicaIndex, t, { rollout, wrongAt, detection }) {
  const inWindow = t >= wrongAt && t < wrongAt + detection;
  if (rollout === "staged") return replicaIndex === 0 && inWindow ? "wrong" : "correct";
  return inWindow ? "wrong" : "correct";
}

export function run(a) {
  const { duration, replicas, interval, stagger, replicaRate, downAt, upAt, restartAt, diskCache } = a;
  const value = Array.from({ length: replicas }, () => "correct");
  const up = Array.from({ length: replicas }, () => true);
  const s = { faultyResponses: 0, droppedRequests: 0, storeReads: 0, failedReads: 0,
    firstSeen: -1, lastSeen: -1, mixed: 0, propagation: -1 };

  for (let t = 0; t < duration; t += 1) {
    if (t === restartAt)                          // a replica restarts after failover
      up[replicas - 1] = diskCache || !(t >= downAt && t < upAt);
    if (t >= upAt && !up[replicas - 1]) up[replicas - 1] = true;
    for (let k = 0; k < replicas; k += 1) {
      if (t >= k * stagger && (t - k * stagger) % interval === 0) {
        s.storeReads += 1;
        if (t >= downAt && t < upAt) s.failedReads += 1;   // last known good value is kept
        else value[k] = target(k, t, a);
      }
      if (!up[k]) { s.droppedRequests += replicaRate; continue; }
      if (value[k] === "wrong") s.faultyResponses += replicaRate;
    }
    const n = value.filter((d) => d === "wrong").length;
    const expected = value.map((_, k) => target(k, t, a)).filter((d) => d === "wrong").length;
    if (n > 0 && s.firstSeen < 0) s.firstSeen = t;
    if (n > 0) s.lastSeen = t;
    if (n > 0 && n < replicas) s.mixed += 1;
    if (s.propagation < 0 && expected > 0 && n === expected) s.propagation = t - a.wrongAt;
  }
  return s;
}

// config/measure.mjs — blast radius of a wrong value, and behavior when the store goes down
import { run } from "./store.mjs";

const REPLICAS = 3, REPLICA_RATE = 171.30;         // M19/K03: 3 replicas, 171.30 requests/s per replica
const EDGE_PEAK = 513.89, BUDGET = 28.2 * 60;       // K01: peak requests/s at the edge, monthly failure share
const INTERVAL = 30, DETECTION = 120, DURATION = 600;  // DD16 refresh interval, DD17 detection time
const BASE = { duration: DURATION, replicas: REPLICAS, interval: INTERVAL, replicaRate: REPLICA_RATE, wrongAt: 65,
  detection: DETECTION, downAt: DURATION, upAt: DURATION, restartAt: -1, diskCache: true };

console.log(`${REPLICAS} replicas, ${REPLICA_RATE} requests/s per replica (M19/K03), edge peak ${EDGE_PEAK}`);
console.log(`DD16 refresh interval ${INTERVAL} s, DD17 detection time ${DETECTION} s; ` +
  `wrong value written at second 65`);
console.log(`\n${"rollout".padEnd(13)}${"stagger".padStart(9)}${"propagation".padStart(13)}` +
  `${"first seen".padStart(12)}${"last seen".padStart(11)}${"mixed s".padStart(9)}` +
  `${"faulty replies".padStart(16)}${"affected replicas".padStart(20)}`);
for (const rollout of ["all-at-once", "staged"])
  for (const stagger of [0, 10]) {
    const r = run({ ...BASE, rollout, stagger });
    console.log(`${rollout.padEnd(13)}${String(stagger).padStart(9)}${String(r.propagation).padStart(13)}` +
      `${String(r.firstSeen).padStart(12)}${String(r.lastSeen).padStart(11)}` +
      `${String(r.mixed).padStart(9)}${r.faultyResponses.toFixed(0).padStart(16)}` +
      `${String(rollout === "all-at-once" ? REPLICAS : 1).padStart(20)}`);
  }

// Store outage: goes down at second 200, 180 s down; a replica restarts after failover at second 260.
console.log(`\nstore outage: down for 180 s; a replica restarts after failover at second 260`);
console.log(`${"behavior".padEnd(24)}${"failed reads".padStart(17)}${"dropped requests".padStart(19)}` +
  `${"equiv. outage s".padStart(19)}${"of failure share".padStart(19)}`);
for (const [label, diskCache] of [["last known good value", true], ["store required", false]]) {
  const r = run({ ...BASE, rollout: "all-at-once", stagger: 10, wrongAt: DURATION,
    downAt: 200, upAt: 380, restartAt: 260, diskCache });
  const outage = r.droppedRequests / EDGE_PEAK;   // equivalent outage seconds at the edge peak rate
  console.log(`${label.padEnd(24)}${String(r.failedReads).padStart(17)}` +
    `${r.droppedRequests.toFixed(0).padStart(19)}${outage.toFixed(2).padStart(19)}` +
    `${`${((100 * outage) / BUDGET).toFixed(2)}%`.padStart(19)}`);
}

console.log(`\n${"DD16".padStart(8)}${"store reads/s".padStart(16)}${"worst-case propagation".padStart(25)}` +
  `${"faulty reply ceiling".padStart(25)}`);
for (const a of [5, 30, 300])
  console.log(`${`${a} s`.padStart(8)}${(REPLICAS / a).toFixed(4).padStart(16)}` +
    `${`${a} s`.padStart(25)}${(EDGE_PEAK * (DETECTION + a)).toFixed(0).padStart(25)}`);
console.log(`if every request read the store, store reads would be ${EDGE_PEAK}/s; at DD16 = ${INTERVAL} s that is ` +
  `${(EDGE_PEAK / (REPLICAS / INTERVAL)).toFixed(0)}x fewer`);

3 replicas, 171.3 requests/s per replica (M19/K03), edge peak 513.89
DD16 refresh interval 30 s, DD17 detection time 120 s; wrong value written at second 65

rollout        stagger  propagation  first seen  last seen  mixed s  faulty replies   affected replicas
all-at-once          0           25          90        209        0           61668                   3
all-at-once         10           25          70        209       40           61668                   3
staged               0           25          90        209      120           20556                   1
staged              10           25          90        209      120           20556                   1

store outage: down for 180 s; a replica restarts after failover at second 260
behavior                     failed reads   dropped requests    equiv. outage s   of failure share
last known good value                  18                  0               0.00              0.00%
store required                         18              20556              40.00              2.36%

    DD16   store reads/s   worst-case propagation     faulty reply ceiling
     5 s          0.6000                      5 s                    64236
    30 s          0.1000                     30 s                    77084
   300 s          0.0100                    300 s                   215834
if every request read the store, store reads would be 513.89/s; at DD16 = 30 s that is 5139x fewer

These numbers belong to the measurement class, but the time axis is modeled seconds; the run is arithmetic and does not depend on the environment.

Propagation Opens a Window

The change is written to the store at second 65, and the replicas finish converging 25 seconds later. Propagation time can run shorter than the refresh interval — it depends on the distance between the write instant and the next refresh — but its upper bound is the interval. The bottom table’s last column confirms this: at DD16 = 5, 30, 300 seconds, worst-case propagation is 5, 30, 300 seconds respectively.

The second row shows what staggering does and does not do. When the replicas’ refresh is spread out with a 10-second stagger, the first replica to see the value sees it at second 70 instead of 90, and a 40-second mixed-value window opens: some replicas answer with the new value, some with the old. Total faulty responses stay at 61,668 in both rows; staggering does not speed up propagation, it only spreads it out over time.

The mixed-value window is not a defect. One client request can be served with the new value and the next with the old; a configuration change is not atomic, and values must make sense side by side. Writing two dependent settings one at a time produces an invalid combination inside the mixed window; that is why dependent settings are written as a single record.

Blast Radius and the Store Going Down

The third and fourth rows measure a wrong value’s scope. When the value is written valid for every replica, it spreads to all three at once and produces 61,668 faulty responses within the DD17 = 120-second detection window. When the same value is first marked on a single replica and handed to the others only after confirmation, the blast radius stays at 1 replica and faulty responses come to 20,556 — exactly a third. The staged rollout’s mixed-value window is 120 seconds, and that is not a defect but the pattern itself: the time the two values coexist is time set aside for detection.

The bottom table has the store going down: it stays down for 180 seconds, and during that time a replica restarts after failover. Two behaviors split apart. Choosing the last known good value gives 18 failed reads, the replicas keep running on their cached value, the restarting replica loads the value from its disk copy, and dropped requests are 0. Choosing the store required gives the same 18 failed reads, but the restarting replica cannot come up: it drops 20,556 requests over 120 seconds. That is a 40.00-second equivalent outage at the edge peak rate, spending 2.36 percent of K01’s monthly 28.2-minute failure share in a single event.

The conclusion is a design rule: the external configuration store must be read in a way that does not bring the application down when it goes down itself. Otherwise a single helper component caps the service availability of every service depending on it. The startup validation rule does not conflict with this; validation runs against the last known good value — the value is missing, not the store.

The Numbers for Two Days

The failure-free day’s cost has three items. The first is store reads, and caching keeps it small: at DD16 = 30 seconds, the three replicas together make 0.1000 reads a second. If every request read the store, that would be 513.89 reads/s — 5139 times more. The second is delay: a setting change does not take effect instantly; in the worst case it waits up to DD16. The third is the mixed-value window: replicas run on different values for a while, which is why dependent settings must be kept in a single record.

DD16 ties these three items together and pulls in both directions. Shortening the interval to 5 seconds raises store reads to 0.6000/s (sixfold) but lowers the faulty-reply ceiling from 77,084 to 64,236. Stretching the interval to 300 seconds drops reads to 0.0100/s and raises the ceiling to 215,834. The refresh interval is not a performance parameter, it is a rollback time: it sets how long a wrong value stays in effect.

The failing day’s gain has two items. When the store goes down, running on the last known good value drops dropped requests from 20,556 to 0 and the equivalent outage from 40.00 seconds to zero, preserving 2.36 percent of the monthly failure share. And a wrong value’s blast radius under staged rollout falls from three replicas to one, faulty responses from 61,668 to 20,556. Neither improves the failure-free day; the first adds a read path, the second a rollout procedure.

Summary

• A change propagated to three replicas in 25 seconds; the upper bound is the refresh interval, and at DD16 = 5/30/300 seconds it is 5, 30, 300 seconds respectively.
• Staggering the refresh does not speed up propagation, it opens a 40-second mixed-value window; total faulty responses stay at 61,668 in both cases. That is why dependent settings are written as a single record.
• Cached reads keep store load at 0.1000 reads/s; if every request read the store it would be 513.89, 5139 times more.
• Staged rollout drops the affected replicas from 3 to 1, faulty responses from 61,668 to 20,556; in exchange it makes the 120-second mixed-value window permanent.
• When the store goes down, running on the last known good value drops dropped requests from 20,556 to 0: a 40.00-second equivalent outage, 2.36 percent of K01’s monthly 28.2-minute failure share.
• DD16 is a rollback time, not a performance parameter: at 5 seconds the faulty-reply ceiling is 64,236, at 300 seconds it is 215,834.

Next Step

The five patterns in this topic ran inside the same system. The idempotency ledger, the compensation chain, the claim check, the valet key, and the external configuration store — all parts of the new arrangement, all sitting on K01’s numbers. But the shipment tracking and billing service was not born from scratch: the old system is still standing, still answering the same tracking numbers, still producing the same invoice lines, next to the new arrangement. Nothing these five lessons designed said how that system would be taken over — which route moves when, where the unmoved routes go, how state held in both systems stays consistent, and how long the migration takes were never asked. The next lesson takes on migrating the old system in stages.