Container Networking

Everything measured up to this point was within a single container’s own boundaries: its resources, its environment settings, its writable layer, the directories it mounts. But the measurement network is not a single program. The reading collector feeds data to the verifier, the verifier feeds billing, billing and the work order service look at the same data, and the four run in separate containers. Once the filesystem is isolated, how do these four find each other.

The interesting part of the problem is this: isolation works in two directions here. One container needs to be able to reach another, but it also needs to be unable to reach a third. This lesson builds reachability as a matrix and measures two numbers separately: how many paths network separation closes, and how far the surface opened to the outside drops, from how many endpoints to how many.

A Name Instead of an Address

For one service to reach another, it needs to know its address. An address is a machine name and a port number; it is tied to the machine, it does not travel, and it is part of the environment difference counted in lesson 01 — four of that lesson’s 14 variables (METER_NETWORK_ENDPOINT, READING_QUEUE, DATA_ENDPOINT, FIELD_ENDPOINT) carried exactly this.

This is the difference container networking removes. Every container that joins a network is referred to by a service name, and that name can be resolved from inside another container on the same network. The config no longer holds a machine name, it holds the service’s name; the name is the same on every machine. The difference has been removed from the environment — but not entirely, because whether the name resolves depends on network membership, and membership is defined in the environment.

• RT19 — There are four services, each listening on two ports: an interface port and a metrics port.
• RT20 — The access rule is single: two endpoints reach each other only when they are members of a common network.
• RT21 — Name resolution is tied to the same boundary as access; a service name resolves only on a common network.
• RT22 — In the split layout, the front network holds the collector and the verifier; the back network holds the verifier, billing, and the work order service. The verifier is a member of both.
• RT23 — A published port opens an endpoint on the host machine’s network interface; an unpublished port is reachable only from inside the network.
• RT24 — Network driver classes are modeled as membership arrangements: in the machine-shared layout, all services are in a single set and their ports sit directly on the host machine’s interface; in the networkless layout, every service is alone in its own set.

// runtime/network-matrix.mjs — name resolution, reachability matrix, and outbound surface.
// MODEL: services, networks, and ports are fictional; the access rule is defined in the model as "common network".
const SERVICE = {                                          // RT19: four services, two ports (interface + metrics)
  collector: { ports: [8081, 9101], nets: ["front"] },
  verifier: { ports: [8082, 9102], nets: ["front", "back"] },
  billing: { ports: [8083, 9103], nets: ["back"] },
  workOrder: { ports: [8084, 9104], nets: ["back"] },
};
const NAMES = Object.keys(SERVICE), ALL = NAMES.flatMap((a) => SERVICE[a].ports), DEFAULT = 8080;
const LAYOUT = {                                           // RT22: membership set changes with the layout
  "single network": () => ["shared"],
  "two networks": (a) => SERVICE[a].nets,
  "host-shared": () => ["host"],
  "networkless": (a) => [a],
};
const OUTSIDE = { "single network": 1, "two networks": 1, "host-shared": ALL.length, "networkless": 0 };  // RT23
const shareNet = (a, b, d) => LAYOUT[d](a).some((x) => LAYOUT[d](b).includes(x));                          // RT20

function count(d) {                                        // tally over directed pairs
  let open = 0, closed = 0;
  for (const a of NAMES) for (const b of NAMES) {
    if (a === b) continue;
    if (shareNet(a, b, d)) open += 1; else closed += 1;
  }
  return { open, closed, resolved: open, endpoints: open * 2 };  // RT21: name resolution tied to the same boundary as access
}

console.log(`model: ${NAMES.length} services, ${SERVICE.collector.ports.length} ports per service, ` +
  `${ALL.length} endpoints total; in the two-network layout front = collector + verifier, back = verifier + billing + workOrder`);
console.log(`\n${"source \\ target".padEnd(16)}${NAMES.map((x) => x.padStart(13)).join("")}${"resolved".padStart(12)}`);
for (const a of NAMES) {
  const row = NAMES.map((b) => (a === b ? "-" : shareNet(a, b, "two networks") ? "resolves" : "closed").padStart(13)).join("");
  console.log(`${a.padEnd(16)}${row}` +
    `${String(NAMES.filter((b) => b !== a && shareNet(a, b, "two networks")).length).padStart(12)}`);
}

console.log(`\n${"layout".padEnd(16)}${"open paths".padStart(11)}${"closed paths".padStart(13)}` +
  `${"resolved".padStart(11)}${"in-network".padStart(12)}${"from outside".padStart(14)}`);
const r = {};
for (const d of Object.keys(LAYOUT)) {
  const k = count(d); r[d] = k;
  console.log(`${d.padEnd(16)}${String(k.open).padStart(11)}${String(k.closed).padStart(13)}` +
    `${String(k.resolved).padStart(11)}${String(k.endpoints).padStart(12)}${String(OUTSIDE[d]).padStart(14)}`);
}
const single = r["single network"], split = r["two networks"];
console.log(`network separation closes ${single.open - split.open} paths (${((100 * (single.open - split.open)) / single.open).toFixed(1)}%); ` +
  `closed pairs: ${NAMES.flatMap((a) => NAMES.filter((b) => a !== b && !shareNet(a, b, "two networks")).map((b) => `${a}->${b}`)).join(", ")}`);
console.log(`if all four services listened on the framework default ${DEFAULT}: ${NAMES.length - new Set(NAMES.map(() => DEFAULT)).size} ` +
  `collisions in the host-shared layout, 0 collisions in the separate-namespace layout`);

const PUBLISH = { "all published": ALL, "field interface only": [8084], "none": [] };
console.log(`\n${"publish option".padEnd(22)}${"from outside".padStart(14)}${"in-network".padStart(13)}` +
  `${"open fraction".padStart(16)}`);
for (const [label, ports] of Object.entries(PUBLISH))
  console.log(`${label.padEnd(22)}${String(ports.length).padStart(14)}${String(split.endpoints).padStart(13)}` +
    `${`${ports.length}/${ALL.length}`.padStart(16)}`);
console.log(`the outbound surface drops from ${ALL.length} endpoints to ${PUBLISH["field interface only"].length}; ` +
  `the number of endpoints reachable from within the network stays ${split.endpoints} across all three, none of them asks for identity`);

model: 4 services, 2 ports per service, 8 endpoints total; in the two-network layout front = collector + verifier, back = verifier + billing + workOrder

source \ target     collector     verifier      billing    workOrder    resolved
collector                   -     resolves       closed       closed           1
verifier             resolves            -     resolves     resolves           3
billing                closed     resolves            -     resolves           2
workOrder              closed     resolves     resolves            -           2

layout           open paths closed paths   resolved  in-network  from outside
single network           12            0         12          24             1
two networks              8            4          8          16             1
host-shared              12            0         12          24             8
networkless               0           12          0           0             0
network separation closes 4 paths (33.3%); closed pairs: collector->billing, collector->workOrder, billing->collector, workOrder->collector
if all four services listened on the framework default 8080: 3 collisions in the host-shared layout, 0 collisions in the separate-namespace layout

publish option          from outside   in-network   open fraction
all published                      8           16             8/8
field interface only               1           16             1/8
none                               0           16             0/8
the outbound surface drops from 8 endpoints to 1; the number of endpoints reachable from within the network stays 16 across all three, none of them asks for identity

These numbers are in the measurement class.

Reading the Matrix

The first table gives what each service can see in the split layout. The rightmost column counts how many names each service can resolve, and the four services give four different numbers: collector 1, verifier 3, billing 2, work order 2.

The collector’s row is the narrowest, and that narrowness is the design itself. The collector takes in readings arriving from meters in the field; it is the measurement network’s most outward-facing part, and so it should see the least. In the matrix, the billing and work order columns show closed — from inside the collector, resolving either of those two names is not even possible. Because the name does not resolve, a connection attempt ends before it finds an address; the obstacle, in other words, is not a rejection, it is a state of not knowing.

The verifier’s row is the widest, and that too is by design: it sits at the intersection of both networks, takes readings from the front network, and gives verified data to the back network. It resolves all three names.

This split also says something about how name resolution works. The name is a fixed string inside the application, and it is the same on every machine; what turns it into an address is the network the container is attached to. The same code, the same image, and the same name give two different results under two different network memberships — resolving on one, not resolving on the other. The only thing that changes in the config is the membership line, and every cell of the matrix derives from that line.

The billing and work order rows are identical. Both are on the back network, both see each other and the verifier, neither sees the collector. This symmetry shows that network membership is a set: there is no directional distinction between two endpoints in the same set, each sees the other.

Network Separation Closes Four Paths

The second table’s first two rows give the gain from separation. On a single network there are 12 directed paths between the four services and all are open; all 12 names resolve, and the reachable endpoint count is 24 (12 paths times 2 ports per service). Split into two networks, open paths drop to 8, resolved names to 8, reachable endpoints to 16. Network separation closes 4 of the 12 paths, that is, a third of them.

The four closed paths are two symmetric pairs: collector with billing, collector with work order. These are pairs that were never talking to each other in the first place; what separation does is make the ones that do not talk unable to talk. This separation is not free, and its cost is in writing: a single-network layout writes four membership lines, a two-network layout writes five — the verifier is counted twice. In exchange for one network definition and one extra membership line, four paths close.

The second entry in the cost is on the design side. Making the separation requires knowing in advance which services each network will cover, and that knowledge sits in the run config, not inside the application. When a new service is added, which network it joins is a separate decision; forget it, and the service comes up, runs, and finds nobody, without ever throwing an error.

Three Driver Classes, Three Separate Rows

How a container attaches to the network is a driver choice, and the table’s last two rows show the extremes of that choice. Drivers are referred to here by type name; each corresponds to one membership arrangement.

Separate-namespace driver — the first two rows. Every container has its own network namespace and its own address; service names resolve, and only published endpoints open to the outside: 1.

Host-shared driver — the third row. No separate network namespace is given; containers listen directly in the machine’s address space. Its results show up in three columns at once: open paths climb back to 12, in-network endpoints return to 24, and endpoints reachable from outside jump from 1 to 8. There is no longer a publish decision to make either, because there is no boundary left to publish across — all eight of the eight ports are already on the machine’s interface.

This driver’s second cost is worked out in the last row. Without a separate namespace, port numbers come from a single shared space. If all four services listened on their framework’s default of 8080, the host-shared layout would have 3 collisions: one takes the port, three cannot come up. In the separate-namespace layout the collision count is 0, because every container’s 8080 sits in its own space. This is namespace isolation’s most concrete gain — services do not have to know each other’s port numbers.

Networkless driver — the fourth row. There is no membership at all: 0 open paths, 0 resolved names, 0 endpoints, 0 from outside. This is the one row where every measurement is zero, and it is not useless. For a container like the nightly batch job that only reads and writes to a mounted directory and calls nobody, this is the isolation budget’s cheapest point: the network surface is entirely closed, and closing it costs nothing, because it was never used.

Published Ports and In-Network Access

The third table measures an entirely different boundary. The first two tables counted containers’ access to each other; this table counts access coming from the host machine’s network interface — that is, from entirely outside the container network.

The four services listen on 8 ports in total. If all are published, 8 endpoints open on the host machine’s interface; if only the interface port the field crews use is published, 1 endpoint opens; if none are published, 0. The outbound surface drops from 8 endpoints to 1, and while it drops, the number of in-network reachable endpoints stays constant: 16 in all three options.

This is what the table is really saying. The publish decision and in-network access are independent of each other. A service remains exactly as reachable as before by everyone on its own network whether or not it is published; publishing only adds an endpoint to the host machine’s interface. The metrics ports are a good example of this: none of the four metrics ports is published in any option, yet all of them are reachable from within the network.

In practice the distinction settles here: in-network access is for services to talk to each other, publishing is for a user or tool outside the network to talk to them. Managing the two with a single decision ends with all eight of the eight endpoints left open.

The surface dropping from 8 to 1 is a security gain, and its price is paid on the operations side. An unpublished port cannot be looked at from outside; if understanding the cause of a problem requires asking that port a question, whatever is asking also has to be inside the network.

Where Isolation Is Punctured

There are three holes, and all three show up in the numbers.

First: network membership is not authorization. In the split layout there are 16 endpoints reachable from within the network, and none of them asks for identity. A container on the same network reaches not only its neighbor’s interface port but its metrics port too. Network separation roughly decides who can reach whom; it does not restrict at all what a reaching endpoint may ask for.

Second: the four closed paths are only the direct ones. The verifier is a member of both networks; a request coming from the front network reaches the verifier, and the verifier reaches everything on the back network. So the path closed between the collector and billing can still be covered in two hops. The number network separation closes is 4; the number that can be routed around through the service sitting at the intersection is also 4.

Third: the network namespace is separate, but the host machine’s network stack is shared. Every published port opens a real endpoint on the host machine’s interface, and that endpoint is now under the machine’s rules, not the container network’s. Port 8084 inside the container and the endpoint on the machine’s interface are not the same thing; the second is outside isolation.

Summary

• Container networking removes the address from the environment: the config holds a service name instead of a machine name, and the name is the same on every machine. Four of lesson 01’s 14 variables carried exactly this difference.
• Name resolution is tied to the same boundary as access: in the split layout the collector resolves 1 name, the verifier 3, billing 2, work order 2. The obstacle is not a rejection, it is the name failing to resolve.
• The gain from network separation: where a single network has 12 directed paths, 12 resolved names, and 24 reachable endpoints, two networks bring these down to 8, 8, and 16; 4 paths, a third of the total, close. The cost is one network definition and one extra membership line — five instead of four.
• Three driver classes are three separate rows: the separate-namespace layout has 1 endpoint from outside and 0 port collisions; the host-shared layout has 8 endpoints from outside, 12 open paths, and 3 collisions if all four services listened on 8080; the networkless layout has every measurement at zero.
• Published ports and in-network access are independent: the outbound surface drops from 8 endpoints to 1 while the number of endpoints reachable from within the network stays at 16 across all three options.
• Where isolation is punctured: network membership is not authorization — none of the 16 endpoints asks for identity; the 4 closed direct paths can still be covered in two hops through the verifier, which is a member of both networks; and a published port opens an endpoint outside isolation on the host machine’s interface.

Next Step

The network matrix answered one question and left another wide open. The matrix says which container can reach which endpoint; it does not say who stands behind the request that reaches it. The same gap showed up in the previous lesson: what decided write permission in a mounted directory was not the container’s name but the numeric identity of the user running inside it, and that is why six of the eight nodes were closed. Two measurements are looking at the same place. Which user is the process running inside the container, what does that user correspond to on the host machine, and which files does it access when left at the default? The next lesson measures this identity: how many files stay open with the privileged identity, which operations drop when the identity changes, and how many entries the cost of moving to least privilege requires an ownership fix in. The identity’s capabilities are left to the end of the topic; a separate set of numbers stands there.