Service Mesh

The previous lesson set two rules on traffic at the edge and the origin. Both sat in the request’s path and applied to traffic coming from outside.

Services also send requests to each other. In the fictional measurement network, the verifier connects to billing, billing to work orders, work orders to the notification gateway, and none of that traffic passes through the edge proxy. These calls have rules too: how many seconds the timeout is, how many times to retry, which service can call which, whether the call travels encrypted.

These rules can go in two places. In application code: each service configures its own client. In a sidecar proxy: a proxy is placed next to each service instance, all outbound and inbound traffic passes through that proxy, and the rules are written into the proxy. The name of the second arrangement is service mesh.

This lesson compares the two arrangements on the same rule change and asks a single question: is what the sidecar gains bigger than what it costs?

• WS31. Every service has two instances, and one sidecar goes to each instance.
• WS32. A “work item” is a fictional unit: a change, a review, or a release. In the code arrangement, a rule change costs three work items per service; in the mesh arrangement, two work items total.
• WS33. The mesh arrangement’s fixed annual cost is forty work items (the control plane), and every sidecar instance adds two work items a year (version upgrade, identity rotation). Six traffic rules change per year.
• WS34. A sidecar instance consumes sixty-four megabytes of memory and a tenth of a core; the latency a hop adds is fictionally ten times forty microseconds, that is, 0.4 milliseconds.
• WS35. The real measurement block does not print duration: the expected hop and connection counts are written as derived from the input, and the measured extra latency is written as yes or no based only on whether it falls between zero and fifty milliseconds.
• WS36. The new-surface-item count is this: one management endpoint and one identity per sidecar instance, plus the control plane and the rule store.

The Hop Itself

The first thing a sidecar adds to every request is a hop. The block below does not model this, it actually runs it: an app server and a sidecar proxy placed in front of it are set up, and both paths are tested with the same number of requests.

// sidecar/hop.mjs — real measurement: the hop a sidecar proxy adds to a request.
// Servers are set up in this file and closed when the measurement ends. Raw duration is not
// printed: expected values are derived from the input, the measured extra latency only goes through a band check.
import { createServer, request } from "node:http";

const REQUESTS = 200, WARMUP = 50, BAND = [0, 50];         // band: milliseconds
const APP = 8951, SIDECAR = 8950;
const app = createServer((i, y) => { y.sendDate = false; y.end("measurement"); }).listen(APP, "127.0.0.1");
const sidecar = createServer((inReq, inRes) => {           // one job: carry the request one hop onward
  inRes.sendDate = false;
  const outReq = request({ port: APP, path: inReq.url, method: inReq.method, headers: inReq.headers },
    (outRes) => { inRes.writeHead(outRes.statusCode, outRes.headers); outRes.pipe(inRes); });
  outReq.on("error", () => inRes.writeHead(502).end("cut off"));
  inReq.pipe(outReq);
}).listen(SIDECAR, "127.0.0.1");

const call = (port) => new Promise((resolve) => {
  request({ port, path: "/measurement", agent: false }, (y) => { y.resume(); y.on("end", resolve); }).end();
});
async function measure(port, n) {                          // ms per request
  const b = process.hrtime.bigint();
  for (let i = 0; i < n; i += 1) await call(port);
  return Number(process.hrtime.bigint() - b) / 1e6 / n;
}

await measure(APP, WARMUP); await measure(SIDECAR, WARMUP);   // warmup: both paths run once
const extra = (await measure(SIDECAR, REQUESTS)) - (await measure(APP, REQUESTS));
app.close(); sidecar.close();

console.log(`requests / path                    ${REQUESTS}`);
console.log(`expected hops                     direct 1, sidecar 2   (diff 1)`);
console.log(`expected connections / request     direct 1, sidecar 2   (diff 1)`);
console.log(`extra latency ${BAND[0]}-${BAND[1]} ms band          ${extra >= BAND[0] && extra <= BAND[1] ? "yes" : "no"}`);

requests / path                    200
expected hops                     direct 1, sidecar 2   (diff 1)
expected connections / request     direct 1, sidecar 2   (diff 1)
extra latency 0-50 ms band          yes

The measured duration does not go into the output, because it varies machine to machine and cannot be compared. Two numbers derived from the input do not change: on the sidecar path, every request spends two hops and two connections, on the direct path, one. This is for one direction of a call; in a call between two services, since a sidecar stands at both ends, the extra-hop count climbs to two.

What a Rule Change Touches

The second block keeps the ledger for the two arrangements.

// sidecar/cost.mjs — model: applying the same traffic rules in two places. "Work item" is a
// fictional unit (a change, a review, a release); every number is derived from the constants
// below. There is no product, there is an arrangement.
const INSTANCES = 2;        // instance count per service
const CODE_ITEM = 3;        // per service in the code arrangement: change + review + release
const MESH_ITEM = 2;        // per rule in the mesh arrangement, total: config + rollout check
const CONTROL_PLANE = 40;   // control plane's fixed annual work items
const SIDECAR_UPKEEP = 2;   // per sidecar instance, annual: version upgrade + identity rotation
const RULE = 6;             // traffic rules changed per year
const HOP_MS = 0.4;         // fictional: latency a sidecar hop adds
const MEMORY = 64, CORES = 0.1;   // per sidecar instance

const SERVICE = [3, 5, 8, 20, 40];
const codeWork = (n) => RULE * n * CODE_ITEM;
const meshWork = (n) => RULE * MESH_ITEM + CONTROL_PLANE + n * INSTANCES * SIDECAR_UPKEEP;

const E = [9, 11, 17, 14, 15, 14, 12, 15, 10];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["service", "instances", "code touched", "code work/yr", "mesh touched",
  "mesh rule/yr", "mesh fixed", "mesh total/yr", "winner"]));
for (const n of SERVICE)
  console.log(write([n, n * INSTANCES, n, codeWork(n), 0, RULE * MESH_ITEM,
    CONTROL_PLANE + n * INSTANCES * SIDECAR_UPKEEP, meshWork(n), codeWork(n) < meshWork(n) ? "code" : "mesh"]));
const breakeven = [...Array(60).keys()].map((i) => i + 1).find((n) => codeWork(n) >= meshWork(n));
console.log(`break-even service count: ${breakeven}  (below this count, the code arrangement is cheaper)`);

// Per-request cost and the new surface the sidecar introduces.
const P = [9, 20, 12, 19, 16, 12, 20];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + write2(["service", "internal calls/req", "extra hops", "extra latency(ms)",
  "sidecar MB", "cores", "new surface items"]));
for (const n of SERVICE) {
  const calls = Math.min(n - 1, 4), hops = 2 * (calls + 1), instances = n * INSTANCES;
  console.log(write2([n, calls, hops, (hops * HOP_MS).toFixed(1), instances * MEMORY,
    (instances * CORES).toFixed(1), instances * 2 + 2]));
}

service    instances     code touched  code work/yr   mesh touched  mesh rule/yr  mesh fixed  mesh total/yr    winner
3                  6                3            54              0            12          52             64      code
5                 10                5            90              0            12          60             72      mesh
8                 16                8           144              0            12          72             84      mesh
20                40               20           360              0            12         120            132      mesh
40                80               40           720              0            12         200            212      mesh
break-even service count: 4  (below this count, the code arrangement is cheaper)

service    internal calls/req  extra hops  extra latency(ms)      sidecar MB       cores   new surface items
3                           2           6                2.4             384         0.6                  14
5                           4          10                4.0             640         1.0                  22
8                           4          10                4.0            1024         1.6                  34
20                          4          10                4.0            2560         4.0                  82
40                          4          10                4.0            5120         8.0                 162

The Number of Services Touched

The first table’s real column is the third one. A traffic rule — say, the timeout drops from five seconds to two — touches each service separately in the code arrangement. Three touches in a three-service system, forty in a forty-service one. Every touch is a change, a review, and a release; in a forty-service system, a single rule change costs one hundred twenty work items, and because the releases are independent of each other, there is no moment when the rule is in effect everywhere at once.

In the mesh arrangement, the number of services touched is zero. The rule is written once, distributed to the sidecars, and the application code does not change. Two work items per rule change, independent of the service count. In a forty-service system, the annual seven hundred twenty work items drop to two hundred twelve.

Looking at this one column alone, the service mesh wins at every scale. The other columns break that.

Three Services: Cost Outweighs Gain

The first row is where the measurement breaks. In a three-service system, the code arrangement costs fifty-four work items a year, the mesh arrangement sixty-four. The mesh is more expensive, and the reason is the fixed cost: the control plane’s forty work items are bigger than three services’ total rule-change cost.

The break-even service count is four. Below that, the work gained from rule changes is smaller than the control plane’s upkeep.

The second table reads the same row from the other direction. In a three-service system, the sidecars consume three hundred eighty-four megabytes of memory and six-tenths of a core, add six hops and two-point-four milliseconds to every request, and introduce fourteen new surface items. None of these three costs show up in the work-item ledger.

This is the case where the service mesh gains nothing, and it is not rare: three-service systems are the majority. Where the decision is made with the sentence “a service mesh is good practice,” the price paid is not the return on six gained work items — it is minus ten work items plus fourteen surface items.

What the Central Rule Wrongly Closes

The zero in the mesh touched column depends on one condition: the rule fitting every service the same way. That condition does not hold for most rules.

The rule that drops the timeout to two seconds is correct for thirty-eight services; it is not correct for the service that calls the nightly batch job, because that call is already slow. The rule written as a single line at the center cuts off all of that service’s calls at two seconds. This problem never arises in the code arrangement — the rule is already written separately in each service, and that service keeps its own value. In the central arrangement, the fix is an exception, and an exception costs as many work items as the rule itself: by the model’s constants, every exception costs two work items a year, just like a rule does.

This is the service mesh’s counterpart to the course’s “wrong closure” column. The step that drops the number of services touched by a rule change to zero does not remove the question of how well the rule fits which service; it moves that question from the codebase to a configuration file, where its visibility drops. A service owning its own exception in its own code gets seen in a review; an exception in a central rule set depends on the attention of whoever reads the set.

In numbers, the result is this: as the exception count rises, the mesh side’s annual work items grow the way the rule count does, and the break-even service count shifts up from four. In a system with ten exceptions, the mesh side costs twenty more work items a year, and the break-even point climbs to six services.

What Gets Paid Per Request

The middle of the second table does not saturate as scale grows. The internal-call count settles at four — chain length does not grow without limit with the service count — and extra hops settle at ten, extra latency at four milliseconds. This is the service mesh’s good news: the per-request cost does not grow with scale.

Four milliseconds looks small. The measurement network’s nightly batch job makes hundreds of thousands of internal calls on every run, and there four milliseconds turns into hours. On a dashboard request that passes through the edge cache, it never shows up at all. What the extra latency means is decided by the call count, not by the number itself.

Resource consumption, though, grows linearly with scale. In a forty-service system, the sidecars hold five gigabytes of memory and eight cores. That is the resources of eight app instances; the service mesh sets aside 15% to 20% of the system’s hardware just to run, and this line never comes up in any rule discussion.

The Sidecar’s Own Surface

The last column is the course’s unit of measure. Every sidecar instance introduces two items.

The first is the management endpoint. It shows the sidecar’s configuration, dumps its metrics, reloads itself. This endpoint listens next to the app, in the same network namespace; the “management endpoint” item the first lesson counted now exists not per machine but per instance. Eighty management endpoints in a forty-service system.

The second is identity. Sidecars speak to each other encrypted, and each one has a certificate. This is the extreme case of the third lesson’s “terminating at the app” row: the certificate now sits in eighty places, not six. The fourth lesson’s automation becomes mandatory here — renewing eighty certificates by hand gets forgotten on eighty items, not nine.

The third is a single item and the heaviest: the control plane. The component that distributes rules to every sidecar is a single place with write authority to every service. Together with the rule store, that is two items, and neither can be closed — they are the service mesh’s reason for existing.

This count, climbing from fourteen items to one hundred sixty-two in a forty-service system, has to be placed next to the seventeen items the first lesson counted on a single machine. The service mesh collected the traffic rules in one place and spread the management endpoints everywhere.

Summary

• On a rule change, the code arrangement touches every service (three for three services, forty for forty), the mesh arrangement touches none; in a forty-service system, the annual work-item count drops from seven hundred twenty to two hundred twelve.
• The real measurement confirmed that the sidecar path adds one hop and one connection to every request; the measured extra latency stayed inside the zero-to-fifty-millisecond band, and raw duration was not printed.
• In a three-service system, the mesh arrangement is more expensive than the code arrangement: sixty-four work items against fifty-four. The break-even service count is four, and the control plane’s fixed forty work items decide that threshold.
• The per-request cost does not saturate with scale — ten hops, four milliseconds — but resource consumption grows linearly: five gigabytes of memory and eight cores at forty services.
• The sidecar adds two surface items to every instance (management endpoint and identity); fourteen items at three services, one hundred sixty-two at forty. The control plane and the rule store are items that cannot be closed.

Next Step

This topic built the request’s path from start to finish. The path passes through three roles, the four pieces of information lost passing through the proxy chain get relayed, encryption terminates at one point and its certificate refreshes itself, the response gets stored in two layers, and inter-service traffic gets bound to a single rule set. All of these measurements took one thing as given.

The client connected somewhere. Where it connected to had a name, and that name resolved to an address. The second lesson used a name, measurement.example.invalid; the third lesson assumed the certificate was issued for that name; the fifth lesson never asked how a client landing on an edge node got there. The name itself, which address that name points to, how long the old one stays valid after it changes, and who that name is registered to were not measured once in this topic. The next topic takes on the name itself: the default state of the record set, the consequences of treating the lifetime as a decision, and who owns the record that carries the name.