Service Mesh

Where the previous lesson left off, there were three aggregation layers, and all three had a bare outbound call: no timeout, no retry, no counter. In a real system, these three jobs would have to be rewritten in every service’s code. In the library loan system, the loan service talks to the fee service; the trip goes over a network, the transport is secure, and the other side occasionally returns a temporary error.

This lesson’s question is: if retry, timeout, secure transport, and metric collection are pulled out of the application code and handed to a separate process running alongside the main one, how many lines get deleted from the application code, what is left behind, and what is the cost. The sidecar and the ambassador were defined in The Traffic Layer course and measured there at the deployment-unit level; here the measure is on the code side. When a process like this runs next to every service, and all of them are given policy from a single place, the layout that results is called a service mesh, and the place the policy is given from is called the control plane.

Assumptions: SB13 the fee service talks over secure transport, and the certificate is generated at measurement time. SB14 the fee service rejects the first two read requests with a temporary error; the payment endpoint processes the record, but its first response is treated as lost. SB15 the measurement does not measure duration; it counts lines, processes, hops, and collections.

The Fee Service

The fee service’s work endpoint runs over secure transport. A plain management endpoint sits alongside it; that endpoint is the measurement rig — it reads and resets the counters.

// mesh/fee.mjs — fee service: work endpoint 8751 over secure transport, management endpoint 8757 plain
import { createServer as tls } from "node:https";
import { createServer } from "node:http";
import { readFileSync } from "node:fs";

const counter = { read: 0, payment: 0, collected: 0 };
const reset = () => Object.keys(counter).forEach((k) => (counter[k] = 0));

tls({ key: readFileSync("mesh/key.pem"), cert: readFileSync("mesh/cert.pem") },
  (q, y) => {
    if (q.url === "/penalty/pay") {
      counter.payment += 1;
      counter.collected += 1;                       // the record is processed; the first response is treated as lost in transit
      const code = counter.payment === 1 ? 503 : 200;
      y.writeHead(code, { "Content-Type": "application/json" });
      return y.end(JSON.stringify({ collected: counter.collected, amount: 12.5 }));
    }
    counter.read += 1;
    const code = counter.read <= 2 ? 503 : 200;  // the first two read requests are temporarily rejected
    y.writeHead(code, { "Content-Type": "application/json" });
    y.end(JSON.stringify(code === 200 ? { balance: 12.5 } : { error: "temporary" }));
  }).listen(8751, "127.0.0.1", () => console.log("fee 127.0.0.1:8751"));

createServer((q, y) => {
  if (q.url === "/reset") reset();
  y.writeHead(200, { "Content-Type": "application/json" });
  y.end(JSON.stringify(counter));
}).listen(8757, "127.0.0.1");

Network Concerns in the Application Code

In the first implementation, the loan service does everything itself. The lines carrying a network concern are marked; the measurement will count that mark.

// mesh/loan-thick.mjs — every network concern lives in the application code, 8752
import { createServer } from "node:http";
import { request } from "node:https";                                   // [tls]
import { readFileSync } from "node:fs";                                 // [tls]

const CA = readFileSync("mesh/cert.pem");                                    // [tls]
const TIMEOUT = 400;                                                    // [timeout]
const RETRY = 3;                                                        // [retry]
const metric = {};                                                      // [metric]

const call = (path, method) => new Promise((resolve, reject) => {       // [tls]
  const q = request({ host: "127.0.0.1", port: 8751, path,              // [tls]
    method, ca: CA }, (y) => {                                          // [tls]
    let g = "";
    y.on("data", (p) => (g += p));
    y.on("end", () => resolve({ code: y.statusCode, body: g }));
  });
  q.setTimeout(TIMEOUT, () => q.destroy(new Error("timeout")));         // [timeout]
  q.on("error", reject);                                                // [timeout]
  q.end();
});

const safe = (method) => method === "GET";        // only an idempotent call is retried [retry]
const attempt = async (path, method, action) => {
  const o = (metric[action] ??= { requests: 0, tries: 0, errors: 0 });   // [metric]
  o.requests += 1;                                                      // [metric]
  let last;
  for (let i = 0; i < (safe(method) ? RETRY : 1); i += 1) {             // [retry]
    o.tries += 1;                                                       // [metric]
    last = await call(path, method);
    if (last.code < 500) break;                                         // [retry]
  }
  if (last.code >= 500) o.errors += 1;                                  // [metric]
  return last;
};

const json = (y, code, g) => {
  y.writeHead(code, { "Content-Type": "application/json" });
  y.end(JSON.stringify(g));
};

createServer(async (q, y) => {
  const [root, member] = q.url.split("/").filter(Boolean);
  if (root === "metric") return json(y, 200, metric);                    // [metric]
  const job = root === "pay" ? { path: "/penalty/pay", method: "POST" }
    : { path: `/penalty/${member}`, method: "GET" };
  const s = await attempt(job.path, job.method, root);
  json(y, s.code, { action: root, code: s.code, body: s.body });
}).listen(8752, "127.0.0.1", () => console.log("loan-thick 127.0.0.1:8752"));

The Same Job, a Thin Application, and a Sidecar

In the second implementation, the business logic stays line-for-line the same; the only thing that changes is the body of the attempt function. The application now makes a plain call to a local address.

// mesh/loan-thin.mjs — network concerns live outside the application, 8753: plain call to a local address
import { createServer } from "node:http";

const attempt = (path, method) => fetch(`http://127.0.0.1:8754${path}`, { method })
  .then(async (y) => ({ code: y.status, body: await y.text() }));

const json = (y, code, g) => {
  y.writeHead(code, { "Content-Type": "application/json" });
  y.end(JSON.stringify(g));
};

createServer(async (q, y) => {
  const [root, member] = q.url.split("/").filter(Boolean);
  const job = root === "pay" ? { path: "/penalty/pay", method: "POST" }
    : { path: `/penalty/${member}`, method: "GET" };
  const s = await attempt(job.path, job.method, root);
  json(y, s.code, { action: root, code: s.code, body: s.body });
}).listen(8753, "127.0.0.1", () => console.log("loan-thin 127.0.0.1:8753"));

Every deleted line moves to the sidecar. What the sidecar sees is a path, a method, and a status code; it does not know what job is being done.

// mesh/sidecar.mjs — sidecar 8754: timeout, retry, secure transport, metrics
import { createServer } from "node:http";
import { request } from "node:https";
import { readFileSync } from "node:fs";

const CA = readFileSync("mesh/cert.pem");
const TIMEOUT = 400;
const RETRY = 3;
const RETRIED_CODE = [502, 503, 504];      // the policy is a status code; it does not know what the job is
const metric = {};

const call = (path, method) => new Promise((resolve, reject) => {
  const q = request({ host: "127.0.0.1", port: 8751, path, method, ca: CA }, (y) => {
    let g = "";
    y.on("data", (p) => (g += p));
    y.on("end", () => resolve({ code: y.statusCode, body: g }));
  });
  q.setTimeout(TIMEOUT, () => q.destroy(new Error("timeout")));
  q.on("error", reject);
  q.end();
});

createServer(async (q, y) => {
  if (q.url === "/metric" || q.url === "/reset") {
    const g = JSON.stringify(metric);
    if (q.url === "/reset") Object.keys(metric).forEach((a) => delete metric[a]);
    y.writeHead(200, { "Content-Type": "application/json" });
    return y.end(g);
  }
  const o = (metric[q.url] ??= { requests: 0, tries: 0, errors: 0 });
  o.requests += 1;
  let last;
  for (let i = 0; i < RETRY; i += 1) {
    o.tries += 1;
    last = await call(q.url, q.method);
    if (!RETRIED_CODE.includes(last.code)) break;
  }
  if (last.code >= 500) o.errors += 1;
  y.writeHead(last.code, { "Content-Type": "application/json" });
  y.end(last.body);
}).listen(8754, "127.0.0.1", () => console.log("sidecar 127.0.0.1:8754"));

Measurement

The measurement runs the same two jobs in both layouts: a balance query and a penalty payment. It then reads the fee service’s counters, the metrics on both sides, and the code’s lines.

// mesh/measure.mjs — two layouts run the same job: outcome, retries, collected, deleted lines
import { readFileSync } from "node:fs";

const get = async (u) => JSON.parse(await (await fetch(u)).text());
const code = async (u) => (await fetch(u)).status;
const meaningful = (d) => readFileSync(d, "utf8").split("\n").map((s) => s.trim())
  .filter((s) => s && !s.startsWith("//") && /[A-Za-z]/.test(s));
const netConcern = (d) => readFileSync(d, "utf8").split("\n")
  .filter((s) => /\/\/ \[/.test(s)).length;                 // line marked as a network concern

const LAYOUT = [["thick application", 8752, null], ["thin + sidecar", 8753, 8754]];
const outcome = [];
for (const [name, app, tool] of LAYOUT) {
  await get("http://127.0.0.1:8757/reset");            // fee counters
  if (tool) await get(`http://127.0.0.1:${tool}/reset`);
  const balance = await code(`http://127.0.0.1:${app}/balance/U-41`);
  const pay = await code(`http://127.0.0.1:${app}/pay/U-41`);
  const counter = await get("http://127.0.0.1:8757/count");
  const metric = await get(`http://127.0.0.1:${tool ?? app}/metric`);
  outcome.push({ name, balance, pay, counter, metric, processes: tool ? 3 : 2, hops: tool ? 2 : 1 });
}

const B = ["balance code", "payment code", "reached fee", "collected", "processes", "local hops"];
console.log(`${"layout".padEnd(18)}${B.map((b) => b.padStart(14)).join("")}`);
for (const s of outcome) {
  console.log(`${s.name.padEnd(18)}${[s.balance, s.pay, s.counter.read + s.counter.payment,
    s.counter.collected, s.processes, s.hops].map((v) => String(v).padStart(14)).join("")}`);
}
console.log("");
for (const s of outcome) {
  const d = Object.entries(s.metric).map(([a, o]) => `${a} ${o.requests}/${o.tries}/${o.errors}`);
  console.log(`${s.name.padEnd(18)} metric (requests/tries/errors): ${d.join("  ")}`);
}

const SERVICE = 5;                        // catalog, membership, loan, fee, notification
const k = meaningful("mesh/loan-thick.mjs"), i = meaningful("mesh/loan-thin.mjs");
const y = meaningful("mesh/sidecar.mjs");
const deleted = k.filter((s) => !i.includes(s));
console.log(`\nloan-thick = ${k.length} lines, loan-thin = ${i.length} lines, ` +
  `deleted = ${deleted.length}, added = ${i.filter((s) => !k.includes(s)).length}`);
console.log(`lines marked as network concerns: application code ${netConcern("mesh/loan-thick.mjs")} -> 0`);
console.log(`network-concern lines for ${SERVICE} services: application side ${SERVICE * deleted.length}, ` +
  `sidecar side ${y.length} (single file)`);

openssl req -x509 -newkey rsa:2048 -nodes -days 1 -subj "/CN=localhost" \
  -addext "subjectAltName=IP:127.0.0.1" -keyout mesh/key.pem -out mesh/cert.pem 2>/dev/null
node mesh/fee.mjs > /dev/null &
node mesh/loan-thick.mjs > /dev/null &
node mesh/loan-thin.mjs > /dev/null &
node mesh/sidecar.mjs > /dev/null &
for p in 8752 8753 8754 8757; do
  curl -s --retry 30 --retry-connrefused --retry-delay 0 -o /dev/null http://127.0.0.1:$p/metric
done
node mesh/measure.mjs
kill $(jobs -p)

layout              balance code  payment code   reached fee     collected     processes    local hops
thick application            200           503             4             1             2             1
thin + sidecar               200           200             5             2             3             2

thick application  metric (requests/tries/errors): balance 1/3/0  pay 1/1/1
thin + sidecar     metric (requests/tries/errors): /penalty/U-41 1/3/0  /penalty/pay 1/2/0

loan-thick = 38 lines, loan-thin = 13 lines, deleted = 28, added = 3
lines marked as network concerns: application code 18 -> 0
network-concern lines for 5 services: application side 140, sidecar side 33 (single file)

Deleted Lines and the Responsibility Left Behind

The application code drops from 38 meaningful lines to 13; 28 lines are deleted, and 3 come in their place. Eighteen of the deleted lines were marked as network concerns, and none of that mark remains in the application code at all. The measure’s real meaning is in the multiplier: because the same 28 lines would be written in each of five services, the total on the application side is 140 lines; on the sidecar side there is a single 33-line file, and its policy is given from the outside.

The responsibility left behind has three items, and none of them can be handed off to the sidecar. First, which path is reached with which method: the knowledge that the /penalty/pay endpoint is a POST lives in the application. Second, what the returned status code means in business terms — what will the application that sees a 503 show the user. Third, and most important, whether retrying a call is safe. In the thick implementation this knowledge is one line, in the safe function; in the thin implementation, that line was deleted and nothing was put in its place.

The cost shows up in two columns. The process count goes from 2 to 3, and the local hops a request touches go from 1 to 2. Requests reaching the fee service go from 4 to 5. This is the code-side counterpart of The Traffic Layer course’s process-per-deployment-unit measure: the same job, with one extra hop.

The metric rows show the loss. In the thick layout, the counters are kept as balance and pay — that is, by the job’s name; in the thin layout, as /penalty/U-41 and /penalty/pay — that is, by the fee service’s path vocabulary. The sidecar can only guess the difference between “a balance query” and “a penalty payment” by looking at the path. Because the member number is part of the path, every member produces a separate metric key, and aggregating those keys without domain knowledge also turns into a job outside the application.

What the Retry Does Not Know

The most expensive column in the table is the collected column. Both layouts made the same two requests; in the thick layout the fee service processed one collection, in the thin layout it processed two.

The reason is where the policy sits. The fee service’s payment endpoint processes the record and returns its first response as 503 — imitating the case where the response is lost in transit. In the thick application, the safe function had closed the POST call off from retrying: the payment was attempted once, a 503 returned to the application, and collected became 1. The sidecar, though, only looks at the status code; it saw 503, retried the same request, got 200 on the second attempt, and returned 200 to the application. The loan service believes the operation succeeded; the member’s penalty has been collected twice, and there is no error logged on either side.

This is the failure mode that is born: a retry decision made at the network layer replays a call that the business layer had closed off from replay. This failure mode did not exist in the thick layout, because the code making the decision had the knowledge of what the call was.

The fix is to call some of the responsibility back. The sidecar’s policy can be narrowed by method, or the application can put an idempotency key on the request and let the fee service drop a repeated collection. Neither is free: the first puts domain knowledge inside the network policy, the second adds a bookkeeping job to the fee service. What the measurement shows is this: network concerns can be pulled out of the application, but the knowledge of which call can be safely repeated cannot be pulled out — it only changes location.

Summary

• The same job was run in two layouts: application code dropped from 38 lines to 13, 28 lines were deleted and 3 were added; none of the 18 lines marked as network concerns remained in the application.
• In a five-service system, the same concern holds 140 lines on the application side; on the sidecar side there is a single 33-line file, and its policy is given from the outside.
• Responsibility left in the application: choosing the path and method, the status code’s business meaning, and whether repeating a call is safe.
• Cost paid: processes went from 2 to 3, local hops from 1 to 2, requests reaching the fee service from 4 to 5; metric keys fell from the job’s name to the path vocabulary.
• Failure mode born: a status-code-based retry replayed a payment call that was closed off from repetition; collections became 2 instead of 1, and no error was logged on either side.

Next Step

Both implementations shared a silent assumption: the fee service’s address was written into the code as 127.0.0.1:8751. In the thick application this address sits in the application file, in the thin application in the sidecar file; either way, it is a single constant. But a service may not have a single copy — copies can come and go, and a copy that has left can go on looking valid for a while. This topic’s final lesson takes on resolving the address from outside the code: it builds a registry process, gives records a lease, ties health information to routing, and compares this against fixed configuration through two numbers — the file touched in deployment when a copy is added, and the requests sent to a dead address when a copy dies.