Throttling and Load Shedding

The previous lesson carried the backpressure signal to the service boundary and showed it turning into a rejection there. The rejection itself was not designed: the rule “reject if the buffer is full” did not choose which request would go. At the edge there are 513.89 requests per second, and three flows pass through the same gate — the tracking query, the carrier’s status event, the end-of-day billing request. When capacity falls short, are all three throttled the same.

Two decisions are tangled in that question; this lesson separates them.

Two Separate Criteria

Throttling cuts off what exceeds a known rate. Its criterion is the flow itself: a ceiling derived from the usual rate is set per flow, and a request exceeding it is rejected. The decision never looks at the system’s current state — even with capacity wide open, a request over the ceiling does not get through.

Load shedding rejects according to instantaneous capacity. Its criterion is the system’s current fullness: while capacity suffices, nothing is rejected; as saturation nears, requests are rejected by a shedding-class order — the same request accepted when idle, rejected when full.

Admission control, setting the threshold on concurrency, and load shedding preventing collapse were established in the Caching, Queues and Asynchronous Processing course and are not retold here. The question here is what the two do together.

Two naming collisions need separating up front. The Traffic Layer course’s gateway offloading shares the words but is separate: it moves work like authentication or TLS termination out of the application to the gateway — the work still happens, only its location changes. The load shedding here never accepts the request at all. Similarly, throttling in the Frontend Quality course means limiting a connection’s bandwidth; here it is the same idea applied to request rate.

Gate

The model is a single admission gate at the edge layer; three flows share the same capacity, and work is counted in units.

AY12 — capacity and unit costs. The edge layer processes 700 units per round. A tracking query and a status event cost one unit each; an end-of-day billing request costs 60. Rationale: the batch request is rare but runs a read at K01’s 833.33 records/s, two orders of magnitude above one read query. Usual demand at these values is 603.89 units per round — 86.3 percent of capacity.

AY13 — two failure scenarios. AY13a: the end-of-day billing job moves into the peak hour; its rate climbs from 1.5 per round to 4.5, duration 30 minutes, twice a month. Rationale: K01’s four-hour window is fixed; a job running long or starting late enters the peak hour. AY13b: an edge node goes down; capacity drops by a quarter (700 → 525), duration 8 minutes, twice a month. Rationale: losing a node and its replacement arriving stays under K01’s ten-minute recovery assumption. Their sensitivities are below, not added to K01’s table.

The shedding-class order follows what each flow costs to lose: losing a status event permanently shortens the tracking chain (class 1), rejecting a tracking query makes the user wait but it can be retried (class 2), the end-of-day job can be deferred (class 3). Shedding starts with the lowest-priority class.

// gate/edge.mjs — MODEL of the admission gate at the edge layer. A round is an abstract step
// (one round = one second; the cycle coefficient is an assumption). No real node or server is set up.
export const K01 = { tracking: 416.67, event: 97.22, edgePeak: 513.89 };

// AY12: unit costs and edge capacity. One unit = one tracking query's work.
export const FLOW = [
  { name: "event", unit: 1, rate: K01.event, tier: 1 },     // carrier status write
  { name: "tracking", unit: 1, rate: K01.tracking, tier: 2 },   // read returned to the user
  { name: "batch", unit: 60, rate: 1.5, tier: 3 },          // end-of-day billing request
];
export const CAPACITY = 700;                  // AY12: units processed per round
export const CEILING_RATIO = 1.2;             // throttling ceiling = usual rate x 1.2
export const OSCILLATION = [0.85, 0.85, 0.85, 1.45];   // arrival oscillation of the tracking flow

export function run({ roundCount, throttle, shed, capacity = CAPACITY,
                       batchRate = 1.5, oscillate = true }) {
  const rates = FLOW.map((a) => (a.name === "batch" ? batchRate : a.rate));
  const ceiling = FLOW.map((a) => a.rate * CEILING_RATIO);   // derived from the known usual rate
  const carry = FLOW.map(() => 0);
  const queue = [];                                     // { i, unit, round }
  const s = {
    arrived: FLOW.map(() => 0), accepted: FLOW.map(() => 0),
    throttled: FLOW.map(() => 0), shedRejected: FLOW.map(() => 0),
    oldestAge: 0,
  };
  const LIMIT = 2 * capacity;                           // one round's work may wait

  for (let t = 1; t <= roundCount; t += 1) {
    const arrivals = [];
    const remainingCeiling = [...ceiling];              // ceiling does not carry over rounds
    FLOW.forEach((a, i) => {                            // arrival: fractional rate with carry
      const multiplier = a.name === "tracking" && oscillate ? OSCILLATION[(t - 1) % OSCILLATION.length] : 1;
      carry[i] += rates[i] * multiplier;
      while (carry[i] >= 1) {
        carry[i] -= 1; s.arrived[i] += 1;
        if (throttle && remainingCeiling[i] < 1) { s.throttled[i] += 1; continue; }
        remainingCeiling[i] -= 1; arrivals.push(i);
      }
    });

    let pending = queue.reduce((x, o) => x + o.unit, 0);
    arrivals.sort((x, y) => FLOW[x].tier - FLOW[y].tier);   // higher tier accepted first
    for (const i of arrivals) {
      if (shed && pending + FLOW[i].unit > LIMIT) { s.shedRejected[i] += 1; continue; }
      queue.push({ i, unit: FLOW[i].unit, round: t });
      pending += FLOW[i].unit; s.accepted[i] += 1;
    }

    let budget = capacity;                              // the round's processing capacity
    while (queue.length > 0 && queue[0].unit <= budget) {
      const o = queue.shift();
      budget -= o.unit;
      s.oldestAge = Math.max(s.oldestAge, t - o.round);
    }
  }
  s.backlog = queue.reduce((x, o) => x + o.unit, 0);
  s.pendingAge = queue.length === 0 ? 0 : roundCount - queue[0].round;
  return s;
}

// gate/run.mjs — throttling and load shedding measured separately and together over three days
import { K01, FLOW, CAPACITY, CEILING_RATIO, run } from "./edge.mjs";

const ROUNDS = 600;                    // comparison window; totals scale linearly with duration
const SETTING = [
  ["neither", { throttle: false, shed: false }],
  ["throttling only", { throttle: true, shed: false }],
  ["shedding only", { throttle: false, shed: true }],
  ["both", { throttle: true, shed: true }],
];
const DAY = [
  ["failure-free", {}],
  ["AY13a: batch job moves into peak hour (1.5 -> 4.5 req/round)", { batchRate: 4.5 }],
  ["AY13b: an edge node goes down (capacity 700 -> 525)", { capacity: 525 }],
];

const usual = FLOW.reduce((x, a) => x + a.rate * a.unit, 0);
console.log(`capacity ${CAPACITY} units/round; usual demand ${usual.toFixed(2)} units/round` +
  ` (utilization ${((usual / CAPACITY) * 100).toFixed(1)}%)`);
console.log(`flows: ${FLOW.map((a) => `${a.name}(tier ${a.tier}, ${a.unit} unit, ${a.rate}/round)`).join(", ")}`);
console.log(`throttling ceilings (usual rate x ${CEILING_RATIO}): ` +
  FLOW.map((a) => `${a.name} ${(a.rate * CEILING_RATIO).toFixed(2)}`).join(", "));

for (const [day, extra] of DAY) {
  console.log(`\n-- ${day} --`);
  console.log(`${"setting".padEnd(18)}${"accepted event/tracking/batch".padStart(30)}` +
    `${"throttled".padStart(15)}${"shed".padStart(16)}${"oldest age".padStart(13)}` +
    `${"backlog".padStart(8)}${"pending age".padStart(13)}`);
  for (const [label, y] of SETTING) {
    const r = run({ roundCount: ROUNDS, ...y, ...extra });
    console.log(`${label.padEnd(18)}${r.accepted.join("/").padStart(30)}` +
      `${r.throttled.join("/").padStart(15)}${r.shedRejected.join("/").padStart(16)}` +
      `${String(r.oldestAge).padStart(13)}${String(r.backlog).padStart(8)}` +
      `${String(r.pendingAge).padStart(13)}`);
  }
}

console.log(`\n-- the peak AY13b keeps --`);
const shed = run({ roundCount: ROUNDS, throttle: false, shed: true, capacity: 525 });
const kept = (shed.accepted[0] + shed.accepted[1]) / ROUNDS;
console.log(`with shedding on, accepted event+tracking = ${kept.toFixed(2)} req/round;` +
  ` K01 peak edge ${K01.edgePeak} req/s -> ratio ${(kept / K01.edgePeak).toFixed(4)}`);
console.log(`batch flow's unit load ${(FLOW[2].rate * FLOW[2].unit).toFixed(2)}/round, node loss's gap` +
  ` ${(FLOW.reduce((x, a) => x + a.rate * a.unit, 0) - 525).toFixed(2)}/round`);
const [MIN, TIMES, FAILURE_SHARE] = [8, 2, 28.2];      // AY13b duration and K01's monthly failure share (min)
console.log(`gateless, AY13b is an outage: ${MIN} min x ${TIMES} = ${MIN * TIMES} min/mo =` +
  ` ${(((MIN * TIMES) / FAILURE_SHARE) * 100).toFixed(1)}% of K01's failure share (${FAILURE_SHARE} min)`);
console.log(`gated, the same duration is not an outage but a partial rejection:` +
  ` ${(shed.shedRejected.reduce((x, y) => x + y, 0) / ROUNDS).toFixed(2)} req/round rejected`);

console.log(`\n-- failure-free day's cost --`);
const th = run({ roundCount: ROUNDS, throttle: true, shed: false });
const thFlat = run({ roundCount: ROUNDS, throttle: true, shed: false, oscillate: false });
const sh = run({ roundCount: ROUNDS, throttle: false, shed: true });
const neither = run({ roundCount: ROUNDS, throttle: false, shed: false });
console.log(`throttling: ${th.throttled[1]} valid tracking queries rejected = ` +
  `${((th.throttled[1] / th.arrived[1]) * 100).toFixed(2)}% of arrivals, ${((K01.tracking * th.throttled[1]) / th.arrived[1]).toFixed(2)} req/s at peak read`);
console.log(`same setting with oscillation off: ${thFlat.throttled[1]} rejections -> what exceeds the ceiling is not average rate but the peak`);
console.log(`shedding: ${sh.shedRejected.reduce((x, y) => x + y, 0)} rejections, oldest age ${sh.oldestAge} round(s)` +
  ` (gateless: ${neither.oldestAge} round(s))`);
console.log(`every request passes two checks: flow ceiling and utilization -> 2 checks per request`);

console.log(`\n-- AY13's sensitivity --`);
console.log(`${"scenario".padEnd(26)}${"both: event/tracking/batch".padStart(31)}${"total rejected".padStart(15)}`);
for (const [label, extra] of [["batch 3.0 req/round", { batchRate: 3 }], ["batch 6.0 req/round", { batchRate: 6 }],
                        ["capacity 595 (15% loss)", { capacity: 595 }], ["capacity 455 (35% loss)", { capacity: 455 }]]) {
  const r = run({ roundCount: ROUNDS, throttle: true, shed: true, ...extra });
  const rejected = r.throttled.reduce((x, y) => x + y, 0) + r.shedRejected.reduce((x, y) => x + y, 0);
  console.log(`${label.padEnd(26)}${r.accepted.join("/").padStart(31)}${String(rejected).padStart(15)}`);
}

capacity 700 units/round; usual demand 603.89 units/round (utilization 86.3%)
flows: event(tier 1, 1 unit, 97.22/round), tracking(tier 2, 1 unit, 416.67/round), batch(tier 3, 60 unit, 1.5/round)
throttling ceilings (usual rate x 1.2): event 116.66, tracking 500.00, batch 1.80

-- failure-free --
setting            accepted event/tracking/batch      throttled            shed   oldest age backlog  pending age
neither                         58331/250002/900          0/0/0           0/0/0            1     122            0
throttling only                 58331/234372/600    0/15630/300           0/0/0            0       0            0
shedding only                   58331/250002/900          0/0/0           0/0/0            1     122            0
both                            58331/234372/600    0/15630/300           0/0/0            0       0            0

-- AY13a: batch job moves into peak hour (1.5 -> 4.5 req/round) --
setting            accepted event/tracking/batch      throttled            shed   oldest age backlog  pending age
neither                        58331/250002/2700          0/0/0           0/0/0           67   52266           66
throttling only                 58331/234372/600   0/15630/2100           0/0/0            0       0            0
shedding only                  58331/250002/1872          0/0/0         0/0/828            1     680            0
both                            58331/234372/600   0/15630/2100           0/0/0            0       0            0

-- AY13b: an edge node goes down (capacity 700 -> 525) --
setting            accepted event/tracking/batch      throttled            shed   oldest age backlog  pending age
neither                         58331/250002/900          0/0/0           0/0/0           81   48983           80
throttling only                 58331/234372/600    0/15630/300           0/0/0           29   15624           28
shedding only                   58331/229912/454          0/0/0     0/20090/446            1     525            0
both                            58331/229852/455    0/15630/300      0/4520/145            1     525            0

-- the peak AY13b keeps --
with shedding on, accepted event+tracking = 480.40 req/round; K01 peak edge 513.89 req/s -> ratio 0.9348
batch flow's unit load 90.00/round, node loss's gap 78.89/round
gateless, AY13b is an outage: 8 min x 2 = 16 min/mo = 56.7% of K01's failure share (28.2 min)
gated, the same duration is not an outage but a partial rejection: 34.23 req/round rejected

-- failure-free day's cost --
throttling: 15630 valid tracking queries rejected = 6.25% of arrivals, 26.05 req/s at peak read
same setting with oscillation off: 0 rejections -> what exceeds the ceiling is not average rate but the peak
shedding: 0 rejections, oldest age 1 round(s) (gateless: 1 round(s))
every request passes two checks: flow ceiling and utilization -> 2 checks per request

-- AY13's sensitivity --
scenario                       both: event/tracking/batch total rejected
batch 3.0 req/round                      58331/234372/600          16830
batch 6.0 req/round                      58331/234372/600          18630
capacity 595 (15% loss)                  58331/234372/600          15930
capacity 455 (35% loss)                    58331/214820/5          36077

All these numbers belong to the computed value class: they are counted over a deterministic arrival sequence.

Failure-Free Day’s Cost

Throttling costs on a failure-free day too. The tracking flow’s ceiling is 500.00 requests per round; when arrival oscillation peaks at 604.17, the difference gets rejected. Over six hundred rounds, 15,630 valid tracking queries are lost — 6.25 percent of arrivals, 26.05 requests per second at K01’s peak read rate. With oscillation off, the same setting rejects nothing — so what exceeds the ceiling is not average rate but the peak. Capacity was idle at the time too (oldest age and backlog both zero): throttling rejects without letting available capacity be used.

Load shedding costs nothing on a failure-free day. Fullness never nears its limit, so nothing is rejected, and oldest age matches the gateless arrangement: 1 round. Its cost is not rejection but reacting late.

The shared cost is small: every request passes two checks, flow ceiling and fullness.

Two Failures, Two Mechanisms

The second and third tables show the two mechanisms catching different failure modes.

Under AY13a (the batch job moves into peak hour), throttling wins. In the gateless arrangement, delay climbs to 67 rounds and backlog to 52,266 units — a collapse. Holding the batch flow’s ceiling at 1.80 per round, throttling cuts 2100 extra requests at the gate; delay zero, backlog zero. Shedding also prevents the collapse but differently: letting idle capacity be used, it accepts 1872 batch requests — 3.1 times what throttling accepts — at the cost of a one-round delay and a 680-unit backlog.

Under AY13b (an edge node goes down), throttling is useless. No flow exceeds its ceiling — the flows run at usual rates. Throttling rejects nothing, and with capacity down to 525, delay climbs to 29 rounds and backlog to 15,624 units. Shedding, in the same scenario, keeps delay at 1 round and backlog at 525 units.

The difference comes from the criterion: throttling’s ceiling is derived from a known rate and does not change with capacity, so a capacity loss violates no ceiling and throttling cannot see it.

The last row runs both together and gives the best result in both scenarios: neither makes the other unnecessary.

The Peak Protected

The original question was which request gets protected. Under AY13b with shedding on, satisfied status events plus tracking queries total 480.40 requests per round — 0.9348 of K01’s 513.89 requests/s peak. The status-event flow is fully protected (58,331 requests); what gets cut is 20,090 tracking queries and 446 batch requests.

The arithmetic is in the table: the node loss’s gap is 78.89 units per round, the batch flow’s load 90.00. Cutting the batch flow entirely would close the gap — but shedding reacts only at saturation, so the queue hits its limit at arrival peaks and tracking queries get cut in those rounds too. The shedding-class order decides what gets cut first, not that nothing gets cut.

Without the gate, this would be an outage: delay climbs to 81 rounds and no request is satisfied on time. At two occurrences a month, that is 16 minutes — 56.7 percent of K01’s 28.2-minute failure share. With the gate, the same duration is a partial rejection, not an outage — zero minutes written to the budget, at the cost of 34.23 rejected requests per round.

The last table is sensitivity. Whether the batch rate is 3.0 or 6.0, or capacity loss stays at 15 percent, the satisfied-request count does not change: throttling pins the ceiling, so the anomaly’s size at the source never reaches the outcome. At 35 percent loss, though, tracking falls to 214,820 and batch from 600 to 5 requests — the shedding-class order exhausts the lowest-priority class first, then enters the protected one.

Summary

• Throttling cuts what exceeds a known rate at the source (criterion: flow); load shedding rejects by shedding-class order according to instantaneous capacity (criterion: fullness); the Traffic Layer course’s gateway offloading moves work to the gateway, while this load shedding never accepts the request at all.
• The failure-free day’s cost belongs only to throttling: 15,630 valid tracking queries rejected (6.25 percent of arrivals, 26.05 requests/s at peak read), zeroing out with oscillation off. Shedding rejects nothing on a failure-free day.
• Under AY13a (batch job moves into peak hour), the gateless arrangement collapses with a 67-round delay; throttling zeroes it, and shedding also prevents the collapse while letting idle capacity be used, passing 3.1 times more batch requests.
• Under AY13b (an edge node goes down), throttling rejects nothing and delay climbs to 29 rounds — its criterion is the flow, so it cannot see the capacity loss. Shedding keeps delay at 1 round.
• Shedding protects 0.9348 of K01’s 513.89 requests/s peak under AY13b and keeps the status-event flow fully up; in the gateless arrangement, the same 16 minutes would have eaten 56.7 percent of K01’s 28.2-minute failure share.

Next Step

The gate kept its promise to the accepted request and openly rejected what it did not accept. One gap remained. The 20,090 tracking queries cut under AY13b went entirely to waste — yet a tracking query’s response is not a single piece: the shipment’s status, estimated delivery time, and fee are separate fields from separate dependencies. When one dependency goes down, dropping the whole request throws away the fields that were already available too. The next lesson takes this up: deciding which part of the response is core, still answering when non-core fields do not arrive, and counting what that costs on a failure-free day.