End-to-End Encryption Applications

The previous five lessons protected the same thing: the connection. A certificate ties together the two ends of a transport channel, and its job ends when the channel ends. Where the channel ends, though, usually is not where the content is headed. In the fictional measurement network, a meter record leaves the field terminal, passes through the branch gateway, the reverse proxy, the application server, the queue, the processor, the database, and the backup, and reaches the work-order terminal. At every hop the channel is re-established, and between every hop the content is in the clear.

This lesson closes out the topic by carrying the question to the content itself. The unit of measure is no longer the item, it is who can read it: how many intermediary layers see the content, how many items the key staying at the edge pulls off the server, and what capabilities are lost in exchange.

• PK26. A layer “reading” the content means the content exists in the clear at that layer. Transport encryption protects the channel, not the inside of the layer.
• PK27. A trusted item is a separate path that can access the content at a layer: an admin account, a disk, a backup copy, a log entry, debug output. The numbers are fictional.
• PK28. In end-to-end encryption the key exists only at the edges; no layer in between can decrypt the content — it just carries what it carries.
• PK29. The device loss rate, records per device, and record volumes are fictional model values.
• PK30. An escrow key is a separate key that can decrypt the edges’ key, and it is kept somewhere within the organization.

Who Reads the Content

The measurement runs two settings over the same layer list. The only thing that changes is where the key sits; the layers, volumes, and paths stay the same.

// layer.mjs — model of the layers that see content and the capabilities lost. IS A MODEL.
// Fictional regional measurement network: the layer list, volumes, rates, and item counts are made up.
export const LAYER = [
  // name, type, records seen per day (thousands), trusted items required
  ["field terminal", "edge", 4820, 2],
  ["branch gateway", "intermediary", 4820, 3],
  ["reverse proxy", "intermediary", 4820, 4],
  ["application server", "intermediary", 4820, 5],
  ["queue", "intermediary", 3100, 3],
  ["processor", "intermediary", 3100, 4],
  ["database", "intermediary", 1960, 6],
  ["backup and archive", "intermediary", 1960, 4],
  ["log collector", "intermediary", 890, 3],
  ["work-order terminal", "edge", 640, 2],
].map(([name, type, records, items]) => ({ name, type, records, items }));

export const CAPABILITY = [
  // name, present in transport encryption, end-to-end counterpart, extra items for the counterpart
  ["search over content", true, "edge-side index", 4],
  ["server-side summary generation", true, "edge-side computation", 3],
  ["anomaly detection", true, "-", 0],
  ["content inspection", true, "-", 0],
  ["recovery on key loss", true, "recovery key escrow", 5],
  ["access to history from a new device", true, "key transfer", 4],
];

// End-to-end encryption closes the content; metadata fields stay in the clear.
export const METADATA = ["sender endpoint", "recipient endpoint", "timestamp", "record size",
  "send frequency"];

export const DEVICE = 4100;        // meter endpoints plus field terminals combined (fictional)
export const LOSS_RATE = 0.03;   // annual device loss rate (fictional)
export const DEVICE_RECORDS = 1400;  // records stuck on a device, unrecoverable elsewhere (fictional)

export const write = (g) => (h) =>
  h.map((v, i) => (i ? String(v).padStart(g[i]) : String(v).padEnd(g[0]))).join("");

// reader.mjs — who reads the content, items pulled off the server, and capability lost
import { LAYER, CAPABILITY, METADATA, DEVICE, LOSS_RATE, DEVICE_RECORDS, write } from "./layer.mjs";

const sum = (l, f) => l.reduce((t, x) => t + f(x), 0);
const INTERMEDIARY = LAYER.filter((k) => k.type === "intermediary");
const EDGE = LAYER.filter((k) => k.type === "edge");

const fmt1 = write([24, 17, 17, 16, 14]);
console.log(`${LAYER.length} layers: ${EDGE.length} edge, ${INTERMEDIARY.length} intermediary`);
console.log("\n" + fmt1(["layer", "type", "records (k/day)", "in transit", "end-to-end"]));
for (const k of LAYER)
  console.log(fmt1([k.name, k.type, k.records, "reads", k.type === "edge" ? "reads" : "cannot read"]));

const fmt2 = write([30, 14, 16, 18, 14]);
console.log("\n" + fmt2(["metric", "in transit", "end-to-end", "pulled off server", "share"]));
const metric = [
  ["content-visible layers", LAYER.length, EDGE.length],
  ["trusted items required", sum(LAYER, (k) => k.items), sum(EDGE, (k) => k.items)],
  ["records seen in clear (k/day)", sum(LAYER, (k) => k.records), sum(EDGE, (k) => k.records)],
];
for (const [name, a, b] of metric)
  console.log(fmt2([name, a, b, a - b, `${((1 - b / a) * 100).toFixed(0)}%`]));

// Capability lost and the cost of its edge-side counterpart.
const fmt3 = write([40, 34, 15]);
console.log("\n" + fmt3(["capability", "end-to-end counterpart", "extra items"]));
for (const [name, , counterpart, items] of CAPABILITY) console.log(fmt3([name, counterpart, items]));
const noEquivalent = CAPABILITY.filter(([, , k]) => k === "-").length;
console.log(fmt3(["total", `${CAPABILITY.length - noEquivalent} with equivalent, ${noEquivalent} without`,
  sum(CAPABILITY, ([, , , n]) => n)]));

// When the key stays at the edge: devices lost, records unrecoverable.
const lostDevices = Math.round(DEVICE * LOSS_RATE);
console.log(`\ndevice ${DEVICE}, annual loss rate ${LOSS_RATE * 100}% -> ` +
  `${lostDevices} devices (fictional)`);
console.log(`key on server side: unrecoverable records 0`);
console.log(`key only at the edge: unrecoverable records ` +
  `${lostDevices * DEVICE_RECORDS} (${lostDevices} devices x ${DEVICE_RECORDS})`);
const escrow = CAPABILITY.find(([a]) => a === "recovery on key loss")[3];
console.log(`recovery key escrow: unrecoverable records 0, extra items ${escrow}, ` +
  `layers that can read content ${EDGE.length} -> ${EDGE.length + 1}`);

console.log(`\nmetadata fields left in the clear after end-to-end encryption: ${METADATA.length}`);
for (const u of METADATA) console.log(`  ${u}`);

10 layers: 2 edge, 8 intermediary

layer                                type  records (k/day)      in transit    end-to-end
field terminal                       edge             4820           reads         reads
branch gateway               intermediary             4820           reads   cannot read
reverse proxy                intermediary             4820           reads   cannot read
application server           intermediary             4820           reads   cannot read
queue                        intermediary             3100           reads   cannot read
processor                    intermediary             3100           reads   cannot read
database                     intermediary             1960           reads   cannot read
backup and archive           intermediary             1960           reads   cannot read
log collector                intermediary              890           reads   cannot read
work-order terminal                  edge              640           reads         reads

metric                            in transit      end-to-end pulled off server         share
content-visible layers                    10               2                 8           80%
trusted items required                    36               4                32           89%
records seen in clear (k/day)          30930            5460             25470           82%

capability                                          end-to-end counterpart    extra items
search over content                                        edge-side index              4
server-side summary generation                       edge-side computation              3
anomaly detection                                                        -              0
content inspection                                                       -              0
recovery on key loss                                   recovery key escrow              5
access to history from a new device                           key transfer              4
total                                         4 with equivalent, 2 without             16

device 4100, annual loss rate 3% -> 123 devices (fictional)
key on server side: unrecoverable records 0
key only at the edge: unrecoverable records 172200 (123 devices x 1400)
recovery key escrow: unrecoverable records 0, extra items 5, layers that can read content 2 -> 3

metadata fields left in the clear after end-to-end encryption: 5
  sender endpoint
  recipient endpoint
  timestamp
  record size
  send frequency

The first table’s fourth column carries the same value throughout: in transport encryption, all ten of ten layers read the content. This is not a shortcoming of transport encryption — it is its definition. The channel is protected; where the channel ends, the content opens up. A record is encrypted on one hop, re-encrypted on the next, and in between it sits in the clear in some process’s memory.

The summary table sets three measures side by side. Layers that see content drop from ten to two, trusted items required from thirty-six to four, and the volume of records seen in the clear from 30,930 thousand to 5,460 thousand a day. The biggest drop is in the items column, and the reason is clear: a layer seeing content is not a single path, it is every access path at that layer. The database row carries six items — an admin account, a disk, a backup, a log, and two more; end-to-end encryption closes all six of them at once, because there is no key to decrypt with at that layer.

The table’s eight intermediary rows are not equal in weight. The backup-and-archive layer sees only 1,960 thousand records a day, less than most of the others; in exchange, it retains what it sees. A record sits in the queue for seconds, in the processor for milliseconds; in the backup it sits for years. The layer count does not show this difference, because a count measures a moment. The same number has to be read together with its time dimension: eight of ten layers see the content, but the layer that accumulates content is one, and what accumulates there stays retroactively in the clear even if the encryption changes today.

The logic of the measurement shows up in this line. Tightening a control on the server side reduces the item count at that layer; moving the key to the edge takes the layer out of the equation. The first shrinks the item count, the second zeroes it out. This is the reverse of the same shape as the root count in the second lesson: there, a single root gathered items into itself; here, the key scatters items out to the edges.

The Bill for Moving the Key to the Edge

The second table writes what is lost. Four of the six capabilities have an edge-side counterpart, and these counterparts add sixteen items in total. The counterparts do not all do the same job: searching over content requires building an index on the edge side, but that index only covers the records at that edge — the server-side search was over all records. The counterpart is not a replacement, it is a narrowing.

Two capabilities have no counterpart. Anomaly detection and content inspection, by definition, require reading the content; a layer that cannot read cannot do them. These two rows show the decision-like nature of end-to-end encryption: the eight layers it closes shut out not just unwanted access, but wanted access too. In the fictional measurement network, the counterpart to this is that an anomalous record produced by a broken meter gets noticed not at the collector but only at the edge or at the destination.

There is a middle ground the model does not measure. The two settings here put the extremes against each other: the content is either entirely in the clear or entirely closed. In practice the decision is usually made field by field — a record’s subscriber ID might be closed while the meter reading value is left in the clear. In that case the number of closed layers stays the same, but the number of fields each layer sees drops, and in exchange server-side processing keeps working on the fields left open. The unit of measure then is not the layer, it is the layer times the field; this lesson’s table gives the two extreme values of that product.

The third section writes the harshest cost. When the key exists only at the edge, a lost device takes its records down with it: 123 devices a year and 172,200 unrecoverable records. When the key is kept server-side, this number is zero, because there is always a party that can decrypt. The strength of end-to-end encryption and the cause of this loss are the same thing; you cannot have one without the other.

The Layer Escrow Reopens

The known way to zero out the loss is an escrow key: a separate key that can decrypt the edges’ key, kept somewhere within the organization. Unrecoverable records drop to zero and five items get added. But the measurement changes in another column: layers that can read content go from two to three.

This third layer is not like the others. Each intermediary layer only read the volume it itself saw — the queue 3,100 thousand records a day, the log collector 890 thousand. The escrow key, though, makes no distinction; it can decrypt every record the edges encrypt. A decision that closed eight layers reopens a layer wider than all of them combined, for the cost of one item. This is exactly the definition of the course’s measurement axis: the escrow key is a new trust root, and when it breaks, the number of items that drop is larger than the sum of the eight layers it closed.

There is a way to avoid keeping this root in a single place: the key is split into shares, and a certain number of shares have to come together to decrypt it. This raises the number of items required to break it from one to the share threshold, and adds the same number of maintenance jobs to the item column — share storage, share renewal, share-holder handover. The measurement does not change, only another chain enters beneath the root, and that chain’s items get counted too.

An escrow key is not a wrong decision; it is wrong when it is an unmeasured decision. Once measured, the question is not “should there be escrow,” it is “where does this root sit, how many people can reach it, and what drops when it breaks” — the exact question the second lesson asked, over a different item.

The remaining two edges are not outside the count either. After end-to-end encryption, four trusted items remain, and all four sit on the edge devices: the device’s disk and the account that can access that device. When the eight intermediary layers are closed, protection does not disappear — it relocates: the only place content can be read is now the edge device itself. At first glance this is a gain, because four items are fewer than thirty-six; on a second look it is a concentration, because what those four items hold is no longer backed up at any other layer. In the fictional measurement network, a party that can access a field terminal reads the entirety of the 4,820 thousand records that terminal sees per day, and no control anywhere in the chain sees this happen; the certificate is valid, the signature is correct, the handshake is complete.

The final output block writes a limit. End-to-end encryption closes the content, it does not close the relationship: the sender endpoint, recipient endpoint, timestamp, record size, and send frequency stay in the clear at the intermediary layers, because they are needed to route the record. This five-field set of metadata leaves visible when and how often a meter sends data, which terminal a work order was opened for, and which endpoints are talking to each other. A change in a meter’s send frequency can be read as a sign of a malfunction or a tampering attempt, without the record’s content ever being read; the same visibility also shows how much the intermediary layers still know. Content being unreadable does not mean nothing can be known about it.

Summary

• In transport encryption, all 10 of 10 layers read the content; this is not a shortcoming, it is the definition — the channel is protected, and where it ends the content opens up.
• End-to-end encryption drops the layers that see content from 10 to 2, trusted items from 36 to 4 (89%), and records seen in the clear from 30,930 thousand to 5,460 thousand a day. Tightening a control shrinks the item count; moving the key to the edge takes the layer out of the equation.
• In exchange, 6 capabilities are lost: 4 have a 16-item edge-side counterpart with narrower scope; anomaly detection and content inspection have no counterpart, because both require reading the content.
• When the key exists only at the edge, 123 lost devices a year make 172,200 records unrecoverable; when the key is kept server-side, this number is 0. Strength and loss are two faces of the same trait.
• An escrow key zeroes out the loss, adds 5 items, and raises the layers that can read from 2 to 3; this layer sees wider than any of the 8 closed layers ever did. Metadata, though, stays in the clear: even with the content closed, 5 fields remain visible.

Next Step

This topic opened with a question: a signature proves integrity, but what proves who the public key used to verify it belongs to? Six lessons tied that question to a chain — the claims carried by fields, the items carried by the root, revocation’s learning window, the handshake’s identity steps, renewal’s calendar, and the cost of keeping the key at the edge were all counted. Who the key belongs to now rests on a named root.

What is left open is the name itself. An item’s subject reading field-terminal-17 does not say who uses that terminal, what role that person holds, which records they can access, or how that decision gets enforced across thousands of machines. Who is who within the organization, who can access what, and how that gets enforced machine by machine has never been measured. The next topic, Enterprise Identity, takes up these three questions: where identity is centrally held, how many applications go down when that center breaks, and how many rounds it takes a policy to reach how many machines.