Real-Time Data

Up to this point, data has always arrived the moment the client asked for it. The North Slope station, though, produces a measurement without anyone asking: a sensor takes a reading every ten minutes, and the server processes the record. If the user has to refresh the page for a record created while the measurement history screen is open to appear, the screen lags behind reality.

This lesson covers transports that reverse the direction of that flow. Their common point is this: the client always opens the connection — a browser cannot be reached by an incoming connection from outside — but the server decides when to send data. What differs is how long the connection lasts and how many directions it carries.

Three Transport Paths

Path	Direction	Connection	Reconnection	Good fit
Polling	client asks	new each time	not needed	infrequent, latency-tolerant data
Long polling	client asks	open until a response	after every response	events rare, latency matters
Event stream	server pushes	one long HTTP response	built in	one-way notification stream
Socket	two-way	one long connection	set up manually	frequent, mutual communication

Polling means firing a request at fixed intervals. It is the cheapest setup and uses the request layer from the previous two lessons as is. The cost sits in two places: a request is fired even when nothing changed, and worst-case latency equals the interval. A screen polled every thirty seconds sees an event no later than thirty seconds after it happens. Conditional request headers — the validators introduced in the Browser and Web Platform course — can cut down body transfer, but not the number of requests.

Long polling means the server holds the response open until data exists. Request count drops to event count, and latency disappears. In exchange, a new request has to be opened after every event, and the server has to keep open requests pending.

Event stream (Server-Sent Events) means a single HTTP response stays open without closing. The server keeps writing text chunks into that same response body, and the client reads them as they arrive. Because it is an ordinary HTTP response, it is compatible with intermediaries — proxies, compression, authentication headers. But it is one-way only: the client cannot send anything over this channel, and fires a separate request whenever it wants to send something.

Socket (WebSocket) is a two-way channel that starts over HTTP and then switches protocol. Either side can send a message whenever it wants. For work that needs mutual interaction — collaborative editing, a live cursor, a game — it is the only option.

The rule of choice is plain: if data flows in one direction, an event stream is enough and needs fewer moving parts. If the client also needs to send messages frequently over the same channel, a socket is set up.

The Shape of the Event Stream

The stream is a line-based text format. Each line writes field: value, and a blank line completes the event. Four fields are defined: event gives the event’s name, data its body, id its sequence identifier, retry the reconnection interval. A line starting with a colon is a comment and is ignored — the heartbeat that keeps the connection alive is sent this way.

The file below both sets up a server that produces a stream and runs the parser against it. The operating system picks the port; that number varies by the machine running it and does not appear in the output.

// event-stream.mjs — local server producing text/event-stream + stream parser
import http from "node:http";

// --- Server: publishes everything after the last id -------------------------
const MEASUREMENTS = [
  { id: 1, code: "NS-01", value: -4.2 },
  { id: 2, code: "NS-02", value: -6.1 },
  { id: 3, code: "NS-01", value: -4.4 },
  { id: 4, code: "NS-02", value: -5.9 },
];

const server = http.createServer((req, res) => {
  const lastId = Number(req.headers["last-event-id"] ?? 0);
  res.writeHead(200, {
    "content-type": "text/event-stream",
    "cache-control": "no-store",
    connection: "keep-alive",
  });
  res.write("retry: 3000\n\n");            // suggested reconnection interval
  res.write(": stream open\n\n");          // comment line: the heartbeat
  for (const m of MEASUREMENTS.filter((m) => m.id > lastId)) {
    res.write(`id: ${m.id}\n`);
    res.write("event: measurement\n");
    res.write(`data: ${JSON.stringify(m)}\n\n`);
  }
  res.end();
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const BASE = `http://127.0.0.1:${server.address().port}`;

// --- Stream parser: produces events from field lines ------------------------
function parser(apply) {
  let buffer = "";
  let event = { name: "message", data: [], id: null, retry: null };
  const reset = () => { event = { name: "message", data: [], id: null, retry: null }; };

  return function feed(chunk) {
    buffer += chunk;
    let cut;
    while ((cut = buffer.indexOf("\n")) !== -1) {
      const line = buffer.slice(0, cut);
      buffer = buffer.slice(cut + 1);

      if (line === "") {                        // blank line: dispatch the event
        if (event.data.length > 0) apply({ ...event, data: event.data.join("\n") });
        else if (event.retry !== null) apply({ ...event, data: null });
        reset();
        continue;
      }
      if (line.startsWith(":")) continue;       // comment line is ignored

      const colon = line.indexOf(":");
      const field = colon === -1 ? line : line.slice(0, colon);
      let value = colon === -1 ? "" : line.slice(colon + 1);
      if (value.startsWith(" ")) value = value.slice(1);

      if (field === "event") event.name = value;
      else if (field === "data") event.data.push(value);
      else if (field === "id" && !value.includes("\0")) event.id = value;
      else if (field === "retry" && /^\d+$/.test(value)) event.retry = Number(value);
    }
  };
}

// --- Consumer -----------------------------------------------------------
async function consumeStream(lastId, max) {
  const received = [];
  const controller = new AbortController();
  let done = false;
  const feed = parser((event) => {
    if (done) return;                       // enough received: keep parsing the rest
    received.push(event);
    if (received.filter((e) => e.data !== null).length >= max) {
      done = true;
      controller.abort("enough");
    }
  });

  const response = await fetch(`${BASE}/stream`, {
    headers: lastId ? { "last-event-id": String(lastId) } : {},
    signal: controller.signal,
  });
  const decoder = new TextDecoder();
  try {
    for await (const chunk of response.body) feed(decoder.decode(chunk, { stream: true }));
  } catch {
    if (!controller.signal.aborted) throw new Error("stream broke");
  }
  return received;
}

const log = (e) => console.log(
  `  name=${e.name.padEnd(11)} id=${String(e.id).padEnd(4)}` +
  ` retry=${String(e.retry).padEnd(4)} data=${e.data}`);

console.log("first connection (canceled after the first two events):");
const first = await consumeStream(0, 2);
first.forEach(log);

const lastReceivedId = first.filter((e) => e.id !== null).at(-1).id;
console.log(`last received id: ${lastReceivedId}`);

console.log("reconnection (with Last-Event-ID):");
const second = await consumeStream(lastReceivedId, 99);
second.forEach(log);

// --- Applying events to the store: repeated ids are ignored -----------------
const store = new Map();
const seen = new Set();
let skipped = 0;
for (const event of [...first, ...second, ...first]) {  // the first batch repeats on purpose
  if (event.data === null || event.name !== "measurement") continue;
  if (seen.has(event.id)) { skipped++; continue; }
  seen.add(event.id);
  const m = JSON.parse(event.data);
  store.set(m.code, m);
}
console.log("events processed:", seen.size, "| duplicates skipped:", skipped);
console.log("final state     :", JSON.stringify([...store.values()]));

server.close();

first connection (canceled after the first two events):
  name=message     id=null retry=3000 data=null
  name=measurement id=1    retry=null data={"id":1,"code":"NS-01","value":-4.2}
  name=measurement id=2    retry=null data={"id":2,"code":"NS-02","value":-6.1}
last received id: 2
reconnection (with Last-Event-ID):
  name=message     id=null retry=3000 data=null
  name=measurement id=3    retry=null data={"id":3,"code":"NS-01","value":-4.4}
  name=measurement id=4    retry=null data={"id":4,"code":"NS-02","value":-5.9}
events processed: 4 | duplicates skipped: 2
final state     : [{"id":3,"code":"NS-01","value":-4.4},{"id":4,"code":"NS-02","value":-5.9}]

Accepting the Disconnect

A long-lived connection drops. The wireless network changes, an intervening proxy closes an idle connection, a laptop wakes from sleep. This is not an error but the normal condition of stream-based work; the design is not aimed at preventing the drop but at resuming from the right place afterward.

The field that enables resumption is id. The client stores the last id it received, sends it back with the Last-Event-ID header when reconnecting, and the server publishes everything after that id. The output shows this flow: the first connection was cut after two events, the second connection started from three and four. For this mechanism to work, the server has to keep the events it published for a while; if it does not, events from the gap are lost and the client has to pull the full state again after reconnecting.

The retry field suggests a reconnection interval in milliseconds. Code that writes its own parser should honor this value and apply a growing wait instead of a fixed interval; if every client comes back at the same interval when the server crashes, it knocks down the server that is trying to come back up. This calculation itself is the subject of the next lesson.

The last two lines show a third rule: the same event can arrive twice. During reconnection the server may interpret the last id inclusively, or the client may drop before it saves the id it received. Processing code therefore has to be idempotent: applying the same event twice must give the same result as applying it once. In the example this is achieved by keeping seen ids in a set; the first batch’s repeat was ignored and the final state did not change.

The second route to idempotence is writing the event as an announcement, not a command. “Increment the measurement counter by one” breaks when applied twice; “station NS-01’s last measurement is m-114, value −4.4” gives the same result applied twice. Events carried over a stream are designed in the second form whenever possible.

The Two-Way Socket’s Differences

A socket starts with an ordinary HTTP request, and if the server accepts, the connection switches protocol. From that moment HTTP semantics — method, status code, headers — no longer apply; the channel carries text or binary messages. The application has to define its own message format; which field says the message type, where the sequence number sits, is left to an agreement between client and server.

It departs from the event stream on four points. Reconnection is not built in; detecting a drop and rebuilding the connection is the application’s job. A drop is not always noticed; when transmission is cut silently, the connection can keep looking open, so both sides send a poll message at regular intervals and wait for a reply, treating the connection as dead if none arrives. Queuing responsibility sits with the client: messages meant to be sent while the connection is down wait in a queue, drained once the connection is up — or deliberately dropped. Authentication happens at the initial handshake; no header can be sent once the channel is open, so the session’s validity for the channel’s lifetime is a separate problem, covered in the Session Renewal lesson.

Applying the Stream to State

An incoming event has no meaning by itself; its meaning forms from the state it is applied to. Three problems come up in every application.

Meeting the initial state with the stream. When a screen opens, the naive approach fetches the full list first, then listens to the stream. Events produced in the interval between the two steps are missed. The right order is reversed: open the stream first and buffer incoming events, then fetch the full list, apply the buffered events that come after the list’s timestamp, and drain the buffer.

Out-of-order arrival. If events arrive by different routes — one from the stream, one from the response to the user’s own action — an older value can overwrite a newer one. The fix is for every record to carry a version or timestamp, and for the processing code to accept only a newer stamp.

The hidden tab. Keeping the stream open in a background tab consumes a connection on the server and produces work on the client for no benefit. The stream is closed when visibility changes and reopened from the last id once the tab comes back to the front. This is also a natural instance of using the missed-event recovery path.

Summary

• The client always opens the connection; transports differ in how long it lasts and how many directions it carries. Polling is the cheapest setup, and its latency equals the interval.
• An event stream is one-way and an ordinary HTTP response; a socket is two-way and set up by switching protocol, leaving reconnection and liveness checking to the application in exchange.
• The stream format is line-based: event, data, id, and retry fields, events completed by a blank line, a heartbeat sent as a comment line.
• Disconnecting is normal; the last event id is stored and reported at reconnection with Last-Event-ID. If the server does not keep its published events for a while, resumption is not possible.
• The same event can arrive twice; processing code must be idempotent, and events should be designed as final-state announcements rather than increment commands.
• When meeting the initial list with the stream, the stream is opened first and events are buffered; out-of-order events are filtered out with a version stamp.

Next Step

The last three lessons built the ways of getting data; what they share is that data is, for a while, not yet there. The screen has to show something in the meantime: an empty frame, a loading indicator, stale data, or an error. Keeping these states as separate logical flags — loading, has error, has data — quickly produces combinations that contradict each other. The next lesson reduces the states a view can be in to a finite set and tests the transitions with a table.