Namespaces and Quotas

The nightly job became a cluster object and can open twelve slices at once. Every number measured so far was on a single application’s own account; that the cluster also belongs to others was never taken into account. Two other applications run on the same six nodes, and the objects of all three sit in the same naming space, the same resource pool.

The namespace is the object that divides this space. The word is familiar: in the runtime measurements, namespace was the visibility mapping that determines which processes, which network, and which filesystem the kernel shows a process; here the same word describes a partition that divides the object names and resource accounting inside the cluster — one narrows what a process sees, the other separates a tenant’s naming and resource space. The two are different objects at different layers.

A namespace alone separates only names; it does not separate resources. The object that separates resources is the quota. This lesson names the devolved decision as follows: whether a request is accepted. Once the decision moves from a human to the cluster, the cluster needs to know three items per request — which tenant, how much it wants, how much it actually uses. The third item is in no declaration; it only comes from measurement, and this lesson’s entire drift comes from there.

Three Tenants, One Cluster

The three measured cases use the same request sequence: no quota, quota, and quota with a default applied to undeclared pods. The sequence’s order does not change, because the order itself is one of the things measured.

• CO49 — Cluster capacity, tenants, request sequence, and quota accounting are a model built in node; no real cluster runs.
• CO50 — The cluster is 96 processor units (6 nodes × 16). A single dimension is measured; memory is not counted separately because it works the same arithmetic.
• CO51 — Three tenants are fictional: measurement network, field service, reporting. Sixty requests arrive; requests 11–30 are the measurement network’s nightly burst, the rest round-robin.
• CO52 — Request sizes come from the generator by tenant: measurement network 1–6, field service 1–4, reporting 1–2 units.
• CO53 — Roughly a third of requests declare no resource request. Actual usage is 0.4–1.1 times the declared value.
• CO54 — Quota accounting looks at the declared request, not actual usage. Seed 20260819.

// measure/quota.mjs — three tenants sharing a cluster: no quota, with quota, and with a default request.
// MODEL: cluster capacity, tenants, request sequence, and quota accounting are a model built in
// node; no real cluster runs. The numbers below are the model's numbers.
const CAPACITY = 96, SEED = 20260819;                 // 6 nodes x 16 processor units
const TENANTS = ["measurement-network", "field-service", "reporting"];
let s = SEED % 2147483647;
const random = () => (s = (s * 48271) % 2147483647) / 2147483647;

const REQUESTS = [];                                       // order: round-robin -> burst -> round-robin
for (let i = 0; i < 60; i += 1) {
  const k = i >= 10 && i < 30 ? TENANTS[0] : TENANTS[i % 3];
  const requested = 1 + Math.floor(random() * (k === "reporting" ? 2 : k === "field-service" ? 4 : 6));
  REQUESTS.push({ k, requested, declared: random() >= 0.3, actual: Math.max(1, Math.round(requested * (0.4 + random() * 0.7))) });
}
const count = (f) => TENANTS.map((k) => REQUESTS.filter((i) => i.k === k && f(i)).length);

// --- 1. no quota: first come, first served ---
let remaining = CAPACITY;
const accepted = new Set();
for (const [n, i] of REQUESTS.entries()) if (i.requested <= remaining) { remaining -= i.requested; accepted.add(n); }
console.log(`${"tenant".padEnd(20)}${"incoming".padStart(11)}${"accepted".padStart(10)}${"rejected".padStart(10)}` +
  `${"accept rate".padStart(13)}${"consumed".padStart(11)}${"cluster share".padStart(15)}`);
for (const k of TENANTS) {
  const g = REQUESTS.filter((i) => i.k === k), a = g.filter((i) => accepted.has(REQUESTS.indexOf(i)));
  const t = a.reduce((x, i) => x + i.requested, 0);
  console.log(`${k.padEnd(20)}${String(g.length).padStart(11)}${String(a.length).padStart(10)}${String(g.length - a.length).padStart(10)}` +
    `${`%${Math.round(a.length / g.length * 100)}`.padStart(13)}${String(t).padStart(11)}${`%${Math.round(t / CAPACITY * 100)}`.padStart(15)}`);
}
console.log(`no quota: ${accepted.size} requests accepted, ${REQUESTS.length - accepted.size} rejected, idle capacity ${remaining}`);

// --- 2. quota: two splits ---
function quota(limits, defaultValue = null) {
  const usage = Object.fromEntries(TENANTS.map((k) => [k, 0]));
  const acceptedK = Object.fromEntries(TENANTS.map((k) => [k, 0]));
  const quotaRejected = Object.fromEntries(TENANTS.map((k) => [k, 0]));
  let free = CAPACITY, actual = 0, charged = 0, pressure = 0;
  for (const i of REQUESTS) {
    const amount = i.declared ? i.requested : (defaultValue === null ? i.requested : defaultValue);
    if (usage[i.k] + amount > limits[i.k] || amount > free) { quotaRejected[i.k] += 1; continue; }
    usage[i.k] += amount; free -= amount; acceptedK[i.k] += 1;
    charged += amount; actual += i.actual; if (i.actual > amount) pressure += 1;
  }
  return { usage, acceptedK, quotaRejected, free, actual, charged, pressure };
}
const SPLITS = { equal: { "measurement-network": 32, "field-service": 32, reporting: 32 },
  weighted: { "measurement-network": 48, "field-service": 32, reporting: 16 } };
console.log(`\n${"split".padEnd(10)}${"tenant".padEnd(20)}${"quota".padStart(6)}${"accepted".padStart(10)}${"quota rejections".padStart(18)}` +
  `${"used".padStart(6)}${"quota headroom".padStart(16)}`);
for (const [label, limits] of Object.entries(SPLITS)) {
  const r = quota(limits);
  for (const k of TENANTS)
    console.log(`${label.padEnd(10)}${k.padEnd(20)}${String(limits[k]).padStart(6)}${String(r.acceptedK[k]).padStart(10)}` +
      `${String(r.quotaRejected[k]).padStart(18)}${String(r.usage[k]).padStart(6)}${String(limits[k] - r.usage[k]).padStart(16)}`);
  const totalRejected = TENANTS.reduce((a, k) => a + r.quotaRejected[k], 0);
  console.log(`  -> ${label}: accepted ${REQUESTS.length - totalRejected}, rejected ${totalRejected}, idle cluster capacity ${r.free} ` +
    `(quota total ${Object.values(limits).reduce((a, b) => a + b, 0)})`);
}

// --- 3. default request value ---
console.log(`\n${"default request".padEnd(17)}${"accepted".padStart(9)}${"charged to quota".padStart(18)}${"actual usage".padStart(15)}` +
  `${"overallocation".padStart(16)}${"node pressure".padStart(15)}${"idle capacity".padStart(15)}`);
for (const v of [0, 1, 2, 4, null]) {
  const r = quota(SPLITS.weighted, v);
  const acceptedT = TENANTS.reduce((a, k) => a + r.acceptedK[k], 0);
  console.log(`${(v === null ? "declaration required" : String(v)).padEnd(17)}${String(acceptedT).padStart(9)}${String(r.charged).padStart(18)}` +
    `${String(r.actual).padStart(15)}${String(r.charged - r.actual).padStart(16)}${String(r.pressure).padStart(15)}${String(r.free).padStart(15)}`);
}
console.log(`${REQUESTS.length} requests, ${count((i) => !i.declared).reduce((a, b) => a + b, 0)} do not declare a request`);
console.log(`information items: 3 per request (tenant, request, actual usage) = ${REQUESTS.length * 3}, seed ${SEED}`);

tenant                 incoming  accepted  rejected  accept rate   consumed  cluster share
measurement-network          34        26         8          %76         77            %80
field-service                13         5         8          %38         13            %14
reporting                    13         4         9          %31          6             %6
no quota: 35 requests accepted, 25 rejected, idle capacity 0

split     tenant               quota  accepted  quota rejections  used  quota headroom
equal     measurement-network     32        13                21    32               0
equal     field-service           32        12                 1    30               2
equal     reporting               32        13                 0    21              11
  -> equal: accepted 38, rejected 22, idle cluster capacity 13 (quota total 96)
weighted  measurement-network     48        17                17    48               0
weighted  field-service           32        12                 1    30               2
weighted  reporting               16        10                 3    16               0
  -> weighted: accepted 39, rejected 21, idle cluster capacity 2 (quota total 96)

default request   accepted  charged to quota   actual usage  overallocation  node pressure  idle capacity
0                       53                79             92             -13             22             17
1                       47                88             77              11             11              8
2                       41                94             67              27              2              2
4                       32                96             52              44              0              0
declaration required       39                94             64              30              0              2
60 requests, 22 do not declare a request
information items: 3 per request (tenant, request, actual usage) = 180, seed 20260819

Without a Quota: the Starved Tenant

In a quota-free cluster the rule is one line: accept if there is room. The result is in the first table.

The measurement network sent 34 requests, 26 were accepted — an accept rate of 76% — and it held 80% of the cluster. Field service got 5 of 13 requests through (38%), reporting 4 of 13 (31%). Reporting’s cluster share is 6%. The cluster did not sit idle; capacity was fully exhausted and 25 requests were rejected.

The cause of starvation is not request size, it is order. The measurement network’s nightly burst flows uninterrupted from request 11 to request 30, and it consumes most of the capacity in that stretch; the small reporting requests that come after are left bidding for scraps. Reporting’s requests are the smallest (1–2 units) and still get the lowest accept rate. So under quota-free sharing, share is distributed not by need or courtesy but by timing. One tenant raising its own pod count is a direct effect on another’s requests, and this effect is declared nowhere.

What happens to the 25 rejected requests also has to be stated separately. The request is not rejected, it waits: the pod object is declared, it just has no room on any node. Reporting’s user does not see this as an error, they see it as slowness, and they look for the cause in their own namespace. This is the most expensive part of quota-free sharing — one tenant’s problem is born from another tenant’s behavior, and it sits somewhere neither side can see.

A namespace does not change this table. Even if the three tenants’ objects sit in separate namespaces, they share the same 96 units; a namespace only lets the name nightly-job be used three times and lets authorization be granted per partition. A second object is needed for resource separation.

Once a Quota Is Set: Two Costs

The second table shows together what the quota buys and what it makes you pay in exchange.

Under the equal split (32/32/32), the measurement network hits its quota: it gets 21 quota rejections and stays at 32 units. In exchange reporting gets all 13 of its requests through, without a single rejection. Starvation is over. The cost is in the last column: reporting never uses 11 units of its quota, and 13 units sit idle across the cluster. While the measurement network’s 21 requests are rejected, 13 units sit empty in the cluster — this is exactly the quota’s waste. Total acceptance rises from 35 under no quota to 38, so the quota did not lower throughput; but idle capacity rose from zero to 13.

Under the weighted split (48/32/16), waste falls: idle capacity is 2 units, total acceptance 39. The measurement network’s rejections fall from 21 to 17. The cost has shifted — reporting now takes 3 rejections. Weighting requires knowing demand; this split is only correct because who would want how much was measured in advance. If demand changes — say reporting grows during monthly close — the same split produces waste in the opposite direction.

What a quota counts is not processor alone. The same accounting can be kept separately for memory, for the total size of persistent volume claims, and for object counts — how many service objects, how many secret objects, how many persistent volume claims a namespace can hold at most. Count items protect something different from the processor item: not capacity, but the control plane’s own load. A tenant declaring thousands of secret objects fills no node, but it fills the state store and the reconciliation loop. The measured table tracks a single item; the others work by the same arithmetic.

Reading the two rows together shows the quota’s real nature: a quota is not a capacity allocation, it is an upper bound. Unused quota does not pass to anyone; the idle 13 units are not opened up to the measurement network, they just sit there. The devolved decision is partly taken back here: the cluster makes the “accept this request” decision, but the “who gets how much” decision still belongs to the human who wrote the quota, and that human does not know next week’s demand.

The Default Request Value

The third table asks where the number quota accounting rests on comes from. 22 of 60 requests declare no resource request. A quota has to see a number, so a default is applied to undeclared pods.

There are two separate numbers here, and they must not be confused. The resource request is the accounting and placement number: the quota sums it, and the scheduler looks for that much room on a node. The limit is the ceiling enforced at run time; the pod cannot exceed it. A quota can count both separately, and a separate default can be defined for each. The sweep below changes the request default, because that is the number that decides acceptance; the limit default’s effect shows up somewhere else — if the limit is set low, the pod clears the quota, finds room on a node, and then hits its own ceiling, and work slows down or terminates. When the two are not considered together, the quota table looks clean while unexplained terminations pile up on the application side.

At a default of 0, these pods eat nothing from the quota: acceptance jumps to 53, the total charged to the quota stays at 79, and the cluster thinks it has 17 units idle. Actual usage, though, is 92 — 13 units more than declared, and the 22 pods are using most of what they never declared. The cluster is comfortable in its accounting ledger while its nodes are under pressure. This is the quota’s most dangerous setting: the rule looks like it is being enforced, and it is not.

As the default grows, the table turns the other way. At 2 units node pressure falls to 2, at 4 to 0; acceptance drops from 53 to 32, and overallocation rises to 44 units — 44 of the 96 units charged to the quota are never used. Safety was bought, and in exchange nearly half the cluster was spent on paper.

The last row gives the third path: making declaration mandatory. Acceptance 39, overallocation 30, node pressure 0, idle capacity 2. No single value of a fixed default catches all of this at once, because the default is a single number and the real need of 22 pods is not a single number. The cost of mandating declaration, though, is paid not on the cluster’s side but on the side of whoever writes the declaration.

This topic’s devolution balance sheet is this: the acceptance decision rests on three items per request — which tenant, how much it wants, how much it actually uses; 180 items for sixty requests. The first two items come from declaration and are cheap; the third comes only from measurement, and the quota never sees it at all. Because the system does not know actual usage, it either overallocates and wastes capacity, or underallocates and puts nodes under pressure.

Summary

• A namespace and a quota do separate jobs: a namespace divides names and authorization, not resources; the object that divides resources is the quota.
• In a quota-free cluster, share is distributed by order, not by need: the measurement network takes 80% of the cluster with a 76% accept rate, while reporting stays at 6% with a 31% accept rate, and 25 requests are rejected.
• A quota ends starvation but makes you pay two costs: under the equal split, the measurement network gets 21 quota rejections while 11 units of reporting’s quota and 13 units of the cluster sit idle.
• The weighted split brings waste down from 13 to 2 units, shifting rejection onto reporting in exchange; its correctness depends on demand being known in advance.
• Unused quota passes to nobody: a quota is not a capacity allocation, it is an upper bound.
• At a default request of 0, the cluster thinks it has 17 units idle while actual usage is 92 and 22 pods create node pressure; at a default of 4, pressure drops to zero but 44 units are overallocated. Mandating declaration brings pressure down to 0 while keeping waste at 30.

Next Step

Throughout this topic, objects were declared one by one: the smallest deployment unit, the replica set and version object, the service object and the ingress rule, the configuration and secret object, the persistent volume claim, the stateful set, the job and the cron job, the namespace and the quota. All of them share a common gap, and this lesson exposed it with the last table: declarations have a request, quotas have an upper bound, but nowhere does it say “this pod will open on which node.” The quota accepting a request does not finish the job; the accepted pod has to land on one of six nodes, and that node genuinely has to have room. The storage measurement already showed half of this problem: a volume in single-writer mode eliminated twenty-six of thirty-six options. So what makes the choice among the remaining options, what information does it look at when deciding, what happens if the information it needs to look at has gone stale, and where does a pod wait when no option fits? The next topic writes this decision as a constraint solver and measures the placement decision, the count of staying unsolvable, and the scaling decision’s own delay.