--- title: 'Noisy Neighbor' source: 'https://academia.sh/en/courses/performance-and-monitoring/noisy-neighbor' course: 'Performance Anti-Patterns and Monitoring' language: en updated: '2026-08-23T07:01:28+00:00' license: 'CC BY-SA 4.0' --- # Noisy Neighbor Recognizing insufficient isolation on a shared resource from its symptom: separating the same latency growth's two causes via request share versus work share, the noise ratio climbing from 1.01 to 1.43, the work share jumping from 0.1702 to 0.2415 while the request share never moves, and fair share allocation's cost in the largest tenant's own delay rising from 0.47 to 2.96 rounds. The seven patterns so far each measured a single owner's work. One assumption was never questioned: that everyone using a shared resource is on the same side. The shipment tracking and pricing service does not serve a single customer; sellers all pass through the same edge, the same store, and the same worker pool. The symptom is this: the average wait small sellers see has quadrupled, but the total request rate at the edge never changed. K01's peak read rate stays at 416.67 requests/s; capacity is the same too. This is called a **noisy neighbor**: on a shared resource, one **tenant's** behavior changes the performance the others see. ## Same Symptom, Two Causes **First cause: the small sellers' own rate grew.** No one is bothering anyone; demand genuinely grew and the queue lengthened accordingly. This is not noise, it is a capacity question. **Second cause: a neighbor grew its share.** The small sellers' rate never changed; the largest seller tightened its periodic scan and began consuming a large portion of the shared pool's capacity. Both produce the same symptom. The measurement that separates them is counting **request share** and **work share** separately, per tenant. The work a request generates varies by hundreds of times from one stream to another — the sixth lesson measured this: a periodic scan does work proportional to the seller's own record count. A measurement that counts share by request can never see the noise. ## The Shared Pool The run below is a **model**: a round is an abstract step (one round is one second; the cycle coefficient is an assumption), and no real cluster or tenant is set up. Two regimes are compared — a shared queue that processes in arrival order, and work-conserving round-robin allocation. **KK10 — seller volume.** Two hundred sellers' shipment volume is skewed by rank, and its total is K01's assumption of 400,000 shipments per day. Rationale: a single large seller's volume can carry the whole queue by itself. **KK11 — unit and capacity.** A tracking query is 1 unit, a periodic scan is units equal to one percent of the seller's shipment count; the pool processes 540 units per round. **KK12 — two scenarios.** A: sellers past the fifth rank have a request rate 1.20 times higher. B: the largest seller tightens its scan fivefold. Both grow the usual demand by a similar amount. These assumptions are not added to K01's table. ```js // tenant/sharing.mjs — MODEL of a shared work pool. A round is an abstract step // (one round = one second; the cycle coefficient is an assumption). No real // cluster or tenant is set up. export const CAPACITY = 540; // units/round export const TENANTS = 200; export const INTERVAL = 60; // periodic-scan interval (rounds) export const K01 = { peakReads: 416.67, dailyShipments: 400_000 }; // KK10: seller volume is skewed (share ~ 1/rank), its total is K01's daily shipments. export function sellers(n = TENANTS) { const h = Array.from({ length: n }, (_, i) => 1 / (i + 1)); const t = h.reduce((a, b) => a + b, 0); return h.map((x, i) => ({ name: `S${i + 1}`, share: x / t, shipments: Math.round((x / t) * K01.dailyShipments) })); } // A periodic scan's unit cost is proportional to the seller's own record count (the sixth lesson's measure). export const scan = (s) => Math.max(1, Math.round(s.shipments / 100)); export function run({ round = 600, fairShare = false, bigFrequent = false, smallGrew = false, chunk = 0 }) { const S = sellers(); const backlog = S.map(() => 0); const queue = S.map(() => []), head = S.map(() => 0); // per-tenant queue and head pointer const fifo = []; let fifoHead = 0; // shared-regime overall arrival order const s = S.map(() => ({ requests: 0, units: 0, done: 0, delay: 0, maxDelay: 0 })); const enqueue = (i, units, t) => { // chunk > 0: a large request is split into chunks const n = chunk > 0 ? Math.ceil(units / chunk) : 1; for (let c = 0; c < n; c += 1) { const o = { tenant: i, remaining: Math.ceil(units / n), arrival: t }; if (fairShare) queue[i].push(o); else fifo.push(o); s[i].requests += 1; s[i].units += o.remaining; } }; const finish = (o, t) => { const d = t - o.arrival; s[o.tenant].done += 1; s[o.tenant].delay += d; s[o.tenant].maxDelay = Math.max(s[o.tenant].maxDelay, d); }; for (let t = 1; t <= round; t += 1) { S.forEach((x, i) => { const factor = i >= 5 && smallGrew ? 1.20 : 1; // A: the small sellers' own rate is growing backlog[i] += K01.peakReads * x.share * factor; while (backlog[i] >= 1) { backlog[i] -= 1; enqueue(i, 1, t); } const freq = i === 0 && bigFrequent ? 5 : 1; // B: the largest neighbor is tightening its scan if ((t + i) % Math.round(INTERVAL / freq) === 0) enqueue(i, scan(x), t); }); let budget = CAPACITY; if (fairShare) { // fair share allocation: work-conserving round-robin for (let pass = 0; pass < 12 && budget > 0; pass += 1) { const active = []; for (let i = 0; i < TENANTS; i += 1) if (head[i] < queue[i].length) active.push(i); if (active.length === 0) break; const allotment = Math.max(1, Math.floor(budget / active.length)); const start = t % active.length; // starting point rotates, order favors no one for (let n = 0; n < active.length && budget > 0; n += 1) { const i = active[(start + n) % active.length]; let left = allotment; while (left > 0 && budget > 0 && head[i] < queue[i].length) { const o = queue[i][head[i]]; const taken = Math.min(o.remaining, left, budget); o.remaining -= taken; left -= taken; budget -= taken; if (o.remaining === 0) { finish(o, t); head[i] += 1; } } } } } else { // shared queue: in arrival order while (budget > 0 && fifoHead < fifo.length) { const o = fifo[fifoHead]; const taken = Math.min(o.remaining, budget); o.remaining -= taken; budget -= taken; if (o.remaining === 0) { finish(o, t); fifoHead += 1; } } } } return { S, s }; } ``` ```js // tenant/measure.mjs — splitting request share from work share, and fair share allocation's cost import { CAPACITY, TENANTS, INTERVAL, K01, sellers, scan, run } from "./sharing.mjs"; const WIDTHS = [10, 9, 11, 11, 10, 12, 12]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" ")); const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" ")); const PICK = [0, 9, 49, 199]; // S1, S10, S50, S200 const avg = (x) => (x.delay / x.done).toFixed(2); const smallAvg = (r) => { // S6-S200 together const k = r.s.slice(5); return (k.reduce((a, x) => a + x.delay, 0) / k.reduce((a, x) => a + x.done, 0)).toFixed(2); }; const S = sellers(); const baseline = S.reduce((a, x) => a + K01.peakReads * x.share + scan(x) / INTERVAL, 0); console.log(`capacity ${CAPACITY} units/round, ${TENANTS} sellers, baseline demand ${baseline.toFixed(2)} units/round` + ` (utilization ${(baseline / CAPACITY).toFixed(4)}); the largest seller's daily shipments ${S[0].shipments},` + ` the smallest's ${S[TENANTS - 1].shipments}; periodic scan ${scan(S[0])} and ${scan(S[TENANTS - 1])} units\n`); const DAYS = [["fault-free", {}], ["A: small sellers sped up", { smallGrew: true }], ["B: the largest neighbor tightened its scan", { bigFrequent: true }]]; const SHARED = [], ALLOCATED = []; for (const [name, extra] of DAYS) { const r = run(extra); SHARED.push(r); ALLOCATED.push(run({ ...extra, fairShare: true })); const tReq = r.s.reduce((a, x) => a + x.requests, 0), tUnit = r.s.reduce((a, x) => a + x.units, 0); console.log(`-- ${name} --`); write(["seller", "requests", "req share", "work share", "noise", "avg delay", "max delay"]); line(); for (const i of PICK) { const x = r.s[i]; write([r.S[i].name, x.requests, (x.requests / tReq).toFixed(4), (x.units / tUnit).toFixed(4), (x.units / tUnit / (x.requests / tReq)).toFixed(2), avg(x), x.maxDelay]); } const k = r.s.slice(5); console.log(`S6-S200: avg delay ${smallAvg(r)} rounds, work share ${(k.reduce((a, x) => a + x.units, 0) / tUnit).toFixed(4)},` + ` req share ${(k.reduce((a, x) => a + x.requests, 0) / tReq).toFixed(4)}\n`); } console.log("-- with fair share allocation on: work-conserving round-robin allocation --"); write(["day", "S1 delay", "S10 delay", "S50 delay", "S200 delay", "S6-S200 avg", "S1 max delay"]); line(); DAYS.forEach(([name], i) => { const r = ALLOCATED[i]; write([name.split(":")[0], ...PICK.map((i) => avg(r.s[i])), smallAvg(r), r.s[0].maxDelay]); }); console.log(`\nthe cost: on day B, S1's own average delay goes from ${avg(SHARED[2].s[0])} -> ${avg(ALLOCATED[2].s[0])} rounds,` + ` max from ${SHARED[2].s[0].maxDelay} -> ${ALLOCATED[2].s[0].maxDelay} rounds`); console.log(`even on the fault-free day S1 pays: ${avg(SHARED[0].s[0])} -> ${avg(ALLOCATED[0].s[0])} rounds`); console.log(`accounting: ${TENANTS} counters, 1 touch per request; S1's indivisible request is ${scan(S[0])} units,` + ` its per-round share ${Math.floor(CAPACITY / TENANTS)} units`); console.log("\nsame day B, with the large request chunked (no fair share allocation):"); for (const chunk of [681, 50]) { const r = run({ bigFrequent: true, chunk }); console.log(` largest chunk ${String(chunk).padStart(3)} units (${(chunk / CAPACITY).toFixed(2)} of the round's capacity)` + ` -> S6-S200 avg delay ${smallAvg(r)} rounds`); } ``` ``` capacity 540 units/round, 200 sellers, baseline demand 483.34 units/round (utilization 0.8951); the largest seller's daily shipments 68050, the smallest's 340; periodic scan 681 and 3 units -- fault-free -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42541 0.1689 0.1702 1.01 0.10 2 S10 4263 0.0169 0.0170 1.01 0.20 2 S50 860 0.0034 0.0034 1.00 0.25 2 S200 222 0.0009 0.0008 0.95 0.33 2 S6-S200: avg delay 0.24 rounds, work share 0.6114, req share 0.6143 -- A: small sellers sped up -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42541 0.1506 0.1540 1.02 0.48 2 S10 5113 0.0181 0.0180 1.00 0.72 3 S50 1030 0.0036 0.0036 0.99 0.90 3 S200 265 0.0009 0.0009 0.95 1.12 3 S6-S200: avg delay 0.85 rounds, work share 0.6485, req share 0.6560 -- B: the largest neighbor tightened its scan -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42581 0.1690 0.2415 1.43 0.47 3 S10 4263 0.0169 0.0156 0.92 0.85 3 S50 860 0.0034 0.0031 0.91 1.02 3 S200 222 0.0009 0.0008 0.87 1.23 3 S6-S200: avg delay 0.98 rounds, work share 0.5589, req share 0.6142 -- with fair share allocation on: work-conserving round-robin allocation -- day S1 delay S10 delay S50 delay S200 delay S6-S200 avg S1 max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ fault-free 0.48 0.00 0.00 0.00 0.00 7 A 3.27 0.00 0.00 0.00 0.00 13 B 2.96 0.00 0.00 0.00 0.00 9 the cost: on day B, S1's own average delay goes from 0.47 -> 2.96 rounds, max from 3 -> 9 rounds even on the fault-free day S1 pays: 0.10 -> 0.48 rounds accounting: 200 counters, 1 touch per request; S1's indivisible request is 681 units, its per-round share 2 units same day B, with the large request chunked (no fair share allocation): largest chunk 681 units (1.26 of the round's capacity) -> S6-S200 avg delay 0.98 rounds largest chunk 50 units (0.09 of the round's capacity) -> S6-S200 avg delay 1.00 rounds ``` Delays and shares are in the **measurement** class: they are counted over a deterministic arrival sequence. Capacity, unit costs, and the seller distribution are **assumption**; the rates coming from K01 are **computed**. ## Request Share Does Not Reveal the Noise The two scenarios produce the same symptom. Small sellers' average wait is 0.24 rounds on the fault-free day; 0.85 on day A, 0.98 on day B. Someone looking only at this number cannot distinguish the two days. Looking at the shares sharpens the distinction. **On day A** the small sellers' request share climbs from 0.6143 to 0.6560, work share from 0.6114 to 0.6485; the two move together. The largest seller's noise ratio — work share over request share — stays at 1.01. What grows the load is the waiters themselves. **On day B** the small sellers' request share does not change at all: 0.6143 versus 0.6142. Their work share, however, drops from 0.6114 to **0.5589**. The share they lose passes to the largest seller: its work share climbs from 0.1702 to **0.2415**, its noise ratio from 1.01 to **1.43**. Its request share, meanwhile, sits between 0.1689 and 0.1690 — that is, fixed. This is the measurement that makes the distinction. An accounting kept by request count shows no stir in any tenant on day B: the largest seller sends only a handful of extra requests, but each carries 681 units. A small seller's same request is 3 units — a **227-fold** difference. A system that measures share by request cannot see the noisy neighbor, because the quantity it measures is not the quantity the resource is exhausted by. ## Fair Share Allocation and Its Cost When round-robin allocation is turned on, small sellers' wait drops to **0.00 rounds** on all three days. Order is now given by tenant rather than arrival moment, and the remaining budget is not wasted — the regime is work-conserving. The bulkhead pattern from the Resilience and Reliability course is a harder solution to the same problem: the pool is partitioned up front, neighborliness disappears, and its cost was measured there as idle capacity. Fair share allocation here does not leave the resource idle; its cost is elsewhere. The cost falls on the largest tenant. On day B, S1's own average delay rises from 0.47 to **2.96 rounds**, its max delay from 3 to 9 rounds. The reason is in the last line: its one indivisible request is 681 units, but its per-round share is 2 units. It pays even on the fault-free day — 0.10 to 0.48 rounds — while nothing has broken. The second cost is accounting: two hundred counters are kept, and every request adds a record touch. This overhead grows with the tenant count. A third measurement rules out one wrong fix. Chunking the large request by itself changes nothing: when the largest chunk shrinks from 681 to 50 units, the small sellers' wait rises from 0.98 to 1.00 rounds — the chunks still enter the same queue back to back. What fixes it is not the chunk's smallness but giving out turns **round-robin** across tenants. One warning: fair share allocation removes the symptom without distinguishing the cause. On day A too, small sellers' wait drops to zero and S1 pays the cost again (3.27 rounds), even though S1 did not grow that day's load. A share rule imposed without measurement punishes the wrong tenant. ## The Condition Under Which Sharing Is Correct Sharing is not wrong at every scale, and its limit is given by two numbers. **As long as the noise ratio stays close to one.** On the fault-free day, four sellers' ratios fall between 0.95 and 1.01; the pool runs at 0.8951 utilization and average wait is 0.24 rounds. Under this condition, sharing is not just harmless but profitable: no tenant has idle capacity set aside for it. **As long as no single request exceeds the per-round capacity.** S1's scan is 681 units, the round's capacity is 540 — a ratio of 1.26. If an indivisible request takes longer than one round, every tenant behind it waits at least one round, and no rate rule can prevent that. ## Summary - The symptom is small tenants' wait increasing; it has two causes — their own rate grew, or a neighbor grew its share. Both push the wait from 0.24 to 0.85 and 0.98 rounds. - The measurement that separates them is splitting request share from work share. On day A the two grow together; on day B request share goes from 0.6143 to 0.6142 — essentially unchanged — while work share drops to 0.5589. - The noise ratio for the largest seller climbs from 1.01 to 1.43 and its work share jumps from 0.1702 to 0.2415; its request share stays fixed. An accounting kept by request cannot see the noise. - Fair share allocation drops small tenants' wait to 0.00 rounds on all three days; in exchange, S1's delay rises from 0.47 to 2.96 rounds, and from 0.10 to 0.48 rounds even on the fault-free day, and 200 counters are kept. - Chunking the large request alone is not enough (0.98 → 1.00 rounds); what fixes it is giving out turns round-robin across tenants. - Sharing is correct when the noise ratio is close to one and no single request exceeds the per-round capacity; S1's 681-unit scan exceeds the 540-unit round by 1.26 times. ## Next Step Every measurement in this lesson held one number fixed: that the pool processes 540 units per round. Where that number comes from was never asked. The work a process can carry is bounded not by its processor's speed but by how many requests it can advance at the same time, and that bound usually fills up long before the processor does. The next lesson picks up the symptom from there: throughput stops growing even as workers are added, and processor utilization is low. This too has two causes — the downstream dependency is saturated, or the workers are waiting on blocking calls. The measure that separates the two is not utilization but how much of each worker-round goes to waiting; and that share comes from the product of the blocking call's duration and the concurrency limit.