Rate Limiting and Throttling

The subject of the previous five patterns was always the caller: what happens when this system calls another service was what got regulated. The panel’s optional fields were filled with fallbacks, call sites that throw were counted, and two endpoints outside the pattern’s coverage were found. All of them looked downstream.

The upstream side was never regulated. The loan service does not only call, it also gets called, with no limit on how many calls it will accept. When an overnight batch job falls into a badly written loop, the timeout, retry, and breaker in front of that service do nothing useful — all of them regulate what happens to a request already accepted. The limit is placed on the accepting side and its name is rate limiting; the form that delays instead of rejecting is called throttling.

The limiter’s effect and a comparison of algorithms — fixed window, sliding window, token bucket — were measured in Resilience and Reliability and in a case study on rate-limiter design. Here the algorithm is held fixed and two other questions are asked: which layer of the code the limiter sits in, and whom the limit applies to.

Three Layers, One Limiter

The limiter is a single function; it can be placed in three spots. At the gateway, it sits in one place in front of every service. In a service’s middle layer, it protects only that service, but every one of its routes. At the call site, it regulates only the call it wraps — this third case is throttling on the way out, and its coverage is counted the same way as the wrapped-call-site measure from the previous lesson. The first two are built here.

// limit/layer.mjs — the limiter and the two layers that can host it: gateway and middle layer.
import http from "node:http";

// Fixed window. RS12: the window was chosen longer than the run; algorithm choice is not measured here.
export function limiter(key, limit = 20, window = 60_000) {
  const bucket = new Map();
  let start = Date.now();
  const s = { passed: 0, rejected: 0, member: new Map(), seen: 0 };
  return { s, bucket, allow(i) {
    if (Date.now() - start >= window) { bucket.clear(); start = Date.now(); }
    s.seen += 1;
    const n = (bucket.get(key(i)) ?? 0) + 1;
    bucket.set(key(i), n);
    const t = s.member.get(i.member) ?? { passed: 0, rejected: 0 };
    if (n <= limit) { t.passed += 1; s.passed += 1; } else { t.rejected += 1; s.rejected += 1; }
    s.member.set(i.member, t);
    return n <= limit;
  } };
}

const parseRequest = (req) => ({ path: req.url, client: req.headers["x-client"], member: req.headers["x-member"] });
const respond = (res, code, g) => { res.writeHead(code, { "content-type": "application/json" }); res.end(JSON.stringify(g)); };

// Service: covers every one of its routes if given a middle layer, none if not.
export function service(routes, middleLayer) {
  return http.createServer((req, res) => {
    const i = parseRequest(req);
    if (routes.includes(i.path) === false) return respond(res, 404, { path: i.path });
    if (middleLayer && middleLayer.allow(i) === false) return respond(res, 429, { limit: 1 });
    respond(res, 200, { path: i.path });
  });
}

// Gateway: the limit applies only if the written prefix matches. Other prefixes pass through.
export function gateway(limit, prefix, target) {
  return http.createServer(async (req, res) => {
    const i = parseRequest(req);
    if (i.path.startsWith(prefix) && limit.allow(i) === false) return respond(res, 429, { limit: 1 });
    const y = await fetch(`http://127.0.0.1:${target(i.path)}${i.path}`,
      { headers: { "x-client": i.client ?? "", "x-member": i.member ?? "" } });
    respond(res, y.status, await y.json());
  });
}

export async function listen(s) { await new Promise((r) => s.listen(0, "127.0.0.1", r)); return s.address().port; }
export async function sendRequest(port, path, client, member) {
  const y = await fetch(`http://127.0.0.1:${port}${path}`, { headers: { "x-client": client, "x-member": member } });
  await y.json();
  return y.status;
}

The measurement has two parts. First, coverage: four paths, two access routes, and two placements are set up; for every request, which limiter sees it is counted. Then, the key: the placement with wider coverage is held fixed, and four separate key choices run on the same request stream.

// limit/measurement.mjs — first coverage (which path is counted at which layer), then key selection.
import { limiter, service, gateway, listen, sendRequest } from "./layer.mjs";

const LOAN = ["/loan/borrow", "/loan/list", "/loan-bulk/list"];
const CATALOG = ["/catalog/search"];
const PREFIX = "/loan/";                     // the single prefix written into the gateway

// Two separate setups: in A the limit is only at the gateway, in B only in the service's middle layer.
const gwA = limiter(() => "global", 1e9);
const midB = limiter(() => "global", 1e9);
const setup = async (mid) => {
  const o = await listen(service(LOAN, mid)), c = await listen(service(CATALOG, null));
  return { o, c, target: (u) => (u.startsWith("/catalog") ? c : o) };
};
const A = await setup(null), B = await setup(midB);
const gwPortA = await listen(gateway(gwA, PREFIX, A.target));
const gwPortB = await listen(gateway(limiter(() => "global", 1e9), "/none/", B.target));

console.log("-- coverage: which placement counts a request --");
console.log(`${"path".padEnd(20)}${"access".padEnd(11)}${"gateway".padStart(10)}${"middle layer".padStart(14)}`);
const coverage = { gateway: 0, middle: 0, none: 0, total: 0 };
for (const path of [...LOAN, ...CATALOG]) {
  for (const access of ["gateway", "direct"]) {
    const g0 = gwA.s.seen, a0 = midB.s.seen;
    await sendRequest(access === "direct" ? (path.startsWith("/catalog") ? A.c : A.o) : gwPortA, path, "measure", "u0");
    await sendRequest(access === "direct" ? (path.startsWith("/catalog") ? B.c : B.o) : gwPortB, path, "measure", "u0");
    const g = gwA.s.seen > g0, a = midB.s.seen > a0;
    coverage.total += 1; if (g) coverage.gateway += 1; if (a) coverage.middle += 1; if (!g && !a) coverage.none += 1;
    console.log(`${path.padEnd(20)}${access.padEnd(11)}${(g ? "counted" : "skipped").padStart(10)}${(a ? "counted" : "skipped").padStart(14)}`);
  }
}
console.log(`of ${coverage.total} path-access pairs, the gateway sees ${coverage.gateway}, the middle layer sees ${coverage.middle}; ` +
  `${coverage.none} pairs are outside every placement\n`);

// RS13: the overnight batch job runs uninterrupted first, the mobile client cycles through its members, web arrives last.
const STREAM = [["batch", "u5", 40], ["mobile", "u1", 12], ["mobile", "u2", 8], ["mobile", "u3", 4], ["web", "u4", 6]];
const PLAN = [];
for (const [client, member, n] of STREAM)
  for (let i = 0; i < n; i += 1) PLAN.push({ client, member, path: client === "mobile" ? LOAN[i % 3] : LOAN[1] });

const KEY = [["fixed", () => "global"], ["client", (i) => i.client],
  ["member", (i) => i.member], ["client+path", (i) => `${i.client} ${i.path}`]];
const MEMBER = ["u1", "u2", "u3", "u4", "u5"], LIMIT = 20, ACTIVE_MEMBERS = 5000;
console.log(`-- key selection: ${PLAN.length} requests, limit per bucket ${LIMIT}, placement middle layer --`);
console.log(`${"key".padEnd(12)}${"buckets".padStart(8)}${"cap".padStart(7)}${"passed".padStart(8)}${"rejected".padStart(9)}` +
  MEMBER.map((u) => `${u} p/r`.padStart(11)).join(""));
for (const [name, f] of KEY) {
  const lim = limiter(f, LIMIT);
  const port = await listen(service(LOAN, lim));
  for (const p of PLAN) await sendRequest(port, p.path, p.client, p.member);
  console.log(`${name.padEnd(12)}${String(lim.bucket.size).padStart(8)}${String(lim.bucket.size * LIMIT).padStart(7)}` +
    `${String(lim.s.passed).padStart(8)}${String(lim.s.rejected).padStart(9)}` +
    MEMBER.map((u) => `${lim.s.member.get(u).passed}/${lim.s.member.get(u).rejected}`.padStart(11)).join(""));
}
console.log(`\nthe one written parameter is ${LIMIT}; the real cap is multiplied by the bucket count. If the key is member, the ` +
  `bucket count is the member count: RS14's ${ACTIVE_MEMBERS} active members mean the same configuration is ${ACTIVE_MEMBERS * LIMIT} requests/window.`);
process.exit(0);

-- coverage: which placement counts a request --
path                access        gateway  middle layer
/loan/borrow        gateway       counted       counted
/loan/borrow        direct        skipped       counted
/loan/list          gateway       counted       counted
/loan/list          direct        skipped       counted
/loan-bulk/list     gateway       skipped       counted
/loan-bulk/list     direct        skipped       counted
/catalog/search     gateway       skipped       skipped
/catalog/search     direct        skipped       skipped
of 8 path-access pairs, the gateway sees 2, the middle layer sees 6; 2 pairs are outside every placement

-- key selection: 70 requests, limit per bucket 20, placement middle layer --
key          buckets    cap  passed rejected     u1 p/r     u2 p/r     u3 p/r     u4 p/r     u5 p/r
fixed              1     20      20       50       0/12        0/8        0/4        0/6      20/20
client             3     60      46       24       12/0        8/0        0/4        6/0      20/20
member             5    100      50       20       12/0        8/0        4/0        6/0      20/20
client+path        5    100      50       20       12/0        8/0        4/0        6/0      20/20

the one written parameter is 20; the real cap is multiplied by the bucket count. If the key is member, the bucket count is the member count: RS14's 5000 active members mean the same configuration is 100000 requests/window.

The Path Outside the Limit

The gateway placement sees two of the eight path-access pairs. This has nothing to do with the location itself; it comes down to a single line at the gateway: the /loan/ prefix. The loan service’s third path is named /loan-bulk/list, which does not match the prefix. The path was added later, the prefix list was never updated, and the limiter never saw it. Nothing in the code raises an error; the rule stays silently incomplete.

The second, heavier gap is in the access route. The gateway can only see a request that passes through it. Every request reaching the service’s own listener directly — another internal service’s call, a maintenance script, a health check — never touches the limiter. All four direct rows show skipped in the gateway column. A gateway is not a boundary, it is a gate; walk around the gate and there is no boundary.

The middle layer sees six pairs, because it wraps the entire handler table: whatever the path’s name, wherever the request comes from, every request reaching that service gets counted. Its cost is narrow coverage — only that one process. Catalog is a separate process with no middle layer of its own; /catalog/search never enters any limiter’s view under either access route. Even with both layers built together, these two pairs would stay open, because the gap is not in the layer, it is in the process the limiter was never built into.

The throttling form does not change this table. Even if the limiter queues and delays the request instead of returning 429, the requests it can queue are still only the ones it sees. The choice between rejecting and delaying is about the request’s fate; the coverage question is independent of it, and comes first.

Whom the Key Limits

The second table runs the same placement, the same limit, and the same seventy-request stream through four separate keys. The stream is RS13: the overnight batch job starts with forty requests, the mobile client cycles through its three members in turn, and the web client arrives last.

A fixed key — the key function returns the same value for every request — pools every client into one bucket. The batch job exhausts it with its first twenty requests, and the remaining fifty are rejected: all four other members end up with zero passed requests. Not one of the mobile client’s twelve requests or the web client’s six reaches the system. This is worse than no limiter at all: without it, the batch job’s load would only have slowed the service down; with it, the batch job shuts everyone else out entirely. The wrong key turns a protection tool into a tool of contagion.

The client key splits the bucket into three and confines the batch job to its own quota; passed requests rise from twenty to forty-six. But behind the mobile client sit three separate members sharing one bucket. The first two exhaust the mobile quota, and all four of the third member’s requests get rejected — a member who sent four requests and had none pass, for behavior that belonged to someone else. The client key feeds users behind a shared client on each other’s quota.

The member key fixes this: all three mobile members pass every one of their requests, and only the batch job gets limited. In the measured stream, client+path gives the same result, because the mobile client’s twenty-four requests spread across three paths and no bucket reaches the limit.

The Real Cap

The last two columns show the sneakiest side of the configuration surface. The one number written in the configuration is 20. The upper bound the system actually accepts is 20 × bucket count: 20 under the fixed key, 60 under the client key, 100 under the member key.

The bucket count is not a configuration value; it comes from the data. Under the member key, the bucket count equals the active-member count. RS14 assumes five thousand active members here; the same value of 20 then means one hundred thousand requests per window. Someone reading the configuration assumes the service is limited to twenty requests. Key choice is more decisive than the limit value itself, and since the two sit in separate places, their product is written nowhere.

This also explains why the limiter is not enough alone. A per-member limit throttles a single member; it does not protect the service’s total capacity. The limit that protects total capacity carries a fixed key, with the cost already seen in the first table. Both keys need to be built in together — two bucket sets, two limit values, two reasons for rejection.

Summary

• The limiter’s effect and algorithm choice were measured elsewhere; what is measured here is placement and key. The same limiter function was built into the gateway and the middle layer.
• The gateway sees 2 of 8 path-access pairs, the middle layer sees 6. Two causes: the /loan/ prefix does not catch the later-added /loan-bulk/list path, and no direct request passes through the gateway.
• 2 pairs sit outside every placement: catalog is a separate process the limiter was never built into. The gap is not in the choice of layer, it is in the missing process.
• A fixed key pools every client into one bucket: the batch job exhausts it with twenty requests, and all thirty of the other members’ requests are rejected — worse than no limiter at all.
• The client key feeds members behind a shared client on each other’s quota: a member who sent four requests was rejected on all four. Under the member key, that member passed all four.
• The one written parameter is 20, the real cap is 20 × bucket count: 20 fixed, 60 client, 100 member, 100,000 at RS14’s five thousand members. Bucket count comes from data, not config.

Next Step

Six patterns were built in this topic, each written against a specific way things break: no response arrives, the response is delayed, a dependency stays down, resources run out, a field is missing, too many calls come in. Every pattern has a place in the code, a coverage, and call sites outside that coverage.

Which failure mode each pattern corresponds to, though, was never spelled out. When a dependency refuses a connection, answers slowly, returns an empty body, or returns a wrong value that looks valid, application code sees each differently — one as an error object, one as a timeout, one with no symptom at all. The next lesson produces each of these appearances one by one, writes down how each looks at the call site, matches which pattern catches which, and counts what is left over: the failure modes no pattern meets.