---
title: 'Real-Time Data'
source: 'https://academia.sh/en/courses/frontend-architecture/real-time-data'
course: 'Application Architecture: Routing, State and Data'
language: en
updated: '2026-08-17T18:11:05+00:00'
license: 'CC BY-SA 4.0'
---

# Real-Time Data

Carrying data the server produces on its own to the client — the choice among polling, event streams, and sockets; parsing the stream format; resuming a dropped connection by id; and applying incoming events to state.

Up to this point, data has always arrived the moment the client asked for it. The North
Slope station, though, produces a measurement without anyone asking: a sensor takes a
reading every ten minutes, and the server processes the record. If the user has to
refresh the page for a record created while the measurement history screen is open to
appear, the screen lags behind reality.

This lesson covers transports that reverse the direction of that flow. Their common
point is this: the client always opens the connection — a browser cannot be reached by an
incoming connection from outside — but the server decides when to send data. What
differs is how long the connection lasts and how many directions it carries.

## Three Transport Paths

| Path | Direction | Connection | Reconnection | Good fit |
|---|---|---|---|---|
| Polling | client asks | new each time | not needed | infrequent, latency-tolerant data |
| Long polling | client asks | open until a response | after every response | events rare, latency matters |
| Event stream | server pushes | one long HTTP response | built in | one-way notification stream |
| Socket | two-way | one long connection | set up manually | frequent, mutual communication |

**Polling** means firing a request at fixed intervals. It is the cheapest setup and uses
the request layer from the previous two lessons as is. The cost sits in two places: a
request is fired even when nothing changed, and worst-case latency equals the interval.
A screen polled every thirty seconds sees an event no later than thirty seconds after it
happens. Conditional request headers — the validators introduced in the Browser and Web
Platform course — can cut down body transfer, but not the number of requests.

**Long polling** means the server holds the response open until data exists. Request
count drops to event count, and latency disappears. In exchange, a new request has to be
opened after every event, and the server has to keep open requests pending.

**Event stream (Server-Sent Events)** means a single HTTP response stays open without
closing. The server keeps writing text chunks into that same response body, and the
client reads them as they arrive. Because it is an ordinary HTTP response, it is
compatible with intermediaries — proxies, compression, authentication headers. But it is
one-way only: the client cannot send anything over this channel, and fires a separate
request whenever it wants to send something.

**Socket (WebSocket)** is a two-way channel that starts over HTTP and then switches
protocol. Either side can send a message whenever it wants. For work that needs mutual
interaction — collaborative editing, a live cursor, a game — it is the only option.

The rule of choice is plain: if data flows in one direction, an event stream is enough
and needs fewer moving parts. If the client also needs to send messages frequently over
the same channel, a socket is set up.

## The Shape of the Event Stream

The stream is a line-based text format. Each line writes `field: value`, and a blank
line completes the event. Four fields are defined: `event` gives the event's name,
`data` its body, `id` its sequence identifier, `retry` the reconnection interval. A line
starting with a colon is a comment and is ignored — the heartbeat that keeps the
connection alive is sent this way.

The file below both sets up a server that produces a stream and runs the parser against
it. The operating system picks the port; that number varies by the machine running it
and does not appear in the output.

```js
// event-stream.mjs — local server producing text/event-stream + stream parser
import http from "node:http";

// --- Server: publishes everything after the last id -------------------------
const MEASUREMENTS = [
  { id: 1, code: "NS-01", value: -4.2 },
  { id: 2, code: "NS-02", value: -6.1 },
  { id: 3, code: "NS-01", value: -4.4 },
  { id: 4, code: "NS-02", value: -5.9 },
];

const server = http.createServer((req, res) => {
  const lastId = Number(req.headers["last-event-id"] ?? 0);
  res.writeHead(200, {
    "content-type": "text/event-stream",
    "cache-control": "no-store",
    connection: "keep-alive",
  });
  res.write("retry: 3000\n\n");            // suggested reconnection interval
  res.write(": stream open\n\n");          // comment line: the heartbeat
  for (const m of MEASUREMENTS.filter((m) => m.id > lastId)) {
    res.write(`id: ${m.id}\n`);
    res.write("event: measurement\n");
    res.write(`data: ${JSON.stringify(m)}\n\n`);
  }
  res.end();
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
const BASE = `http://127.0.0.1:${server.address().port}`;

// --- Stream parser: produces events from field lines ------------------------
function parser(apply) {
  let buffer = "";
  let event = { name: "message", data: [], id: null, retry: null };
  const reset = () => { event = { name: "message", data: [], id: null, retry: null }; };

  return function feed(chunk) {
    buffer += chunk;
    let cut;
    while ((cut = buffer.indexOf("\n")) !== -1) {
      const line = buffer.slice(0, cut);
      buffer = buffer.slice(cut + 1);

      if (line === "") {                        // blank line: dispatch the event
        if (event.data.length > 0) apply({ ...event, data: event.data.join("\n") });
        else if (event.retry !== null) apply({ ...event, data: null });
        reset();
        continue;
      }
      if (line.startsWith(":")) continue;       // comment line is ignored

      const colon = line.indexOf(":");
      const field = colon === -1 ? line : line.slice(0, colon);
      let value = colon === -1 ? "" : line.slice(colon + 1);
      if (value.startsWith(" ")) value = value.slice(1);

      if (field === "event") event.name = value;
      else if (field === "data") event.data.push(value);
      else if (field === "id" && !value.includes("\0")) event.id = value;
      else if (field === "retry" && /^\d+$/.test(value)) event.retry = Number(value);
    }
  };
}

// --- Consumer -----------------------------------------------------------
async function consumeStream(lastId, max) {
  const received = [];
  const controller = new AbortController();
  let done = false;
  const feed = parser((event) => {
    if (done) return;                       // enough received: keep parsing the rest
    received.push(event);
    if (received.filter((e) => e.data !== null).length >= max) {
      done = true;
      controller.abort("enough");
    }
  });

  const response = await fetch(`${BASE}/stream`, {
    headers: lastId ? { "last-event-id": String(lastId) } : {},
    signal: controller.signal,
  });
  const decoder = new TextDecoder();
  try {
    for await (const chunk of response.body) feed(decoder.decode(chunk, { stream: true }));
  } catch {
    if (!controller.signal.aborted) throw new Error("stream broke");
  }
  return received;
}

const log = (e) => console.log(
  `  name=${e.name.padEnd(11)} id=${String(e.id).padEnd(4)}` +
  ` retry=${String(e.retry).padEnd(4)} data=${e.data}`);

console.log("first connection (canceled after the first two events):");
const first = await consumeStream(0, 2);
first.forEach(log);

const lastReceivedId = first.filter((e) => e.id !== null).at(-1).id;
console.log(`last received id: ${lastReceivedId}`);

console.log("reconnection (with Last-Event-ID):");
const second = await consumeStream(lastReceivedId, 99);
second.forEach(log);

// --- Applying events to the store: repeated ids are ignored -----------------
const store = new Map();
const seen = new Set();
let skipped = 0;
for (const event of [...first, ...second, ...first]) {  // the first batch repeats on purpose
  if (event.data === null || event.name !== "measurement") continue;
  if (seen.has(event.id)) { skipped++; continue; }
  seen.add(event.id);
  const m = JSON.parse(event.data);
  store.set(m.code, m);
}
console.log("events processed:", seen.size, "| duplicates skipped:", skipped);
console.log("final state     :", JSON.stringify([...store.values()]));

server.close();
```

```
first connection (canceled after the first two events):
  name=message     id=null retry=3000 data=null
  name=measurement id=1    retry=null data={"id":1,"code":"NS-01","value":-4.2}
  name=measurement id=2    retry=null data={"id":2,"code":"NS-02","value":-6.1}
last received id: 2
reconnection (with Last-Event-ID):
  name=message     id=null retry=3000 data=null
  name=measurement id=3    retry=null data={"id":3,"code":"NS-01","value":-4.4}
  name=measurement id=4    retry=null data={"id":4,"code":"NS-02","value":-5.9}
events processed: 4 | duplicates skipped: 2
final state     : [{"id":3,"code":"NS-01","value":-4.4},{"id":4,"code":"NS-02","value":-5.9}]
```

## Accepting the Disconnect

A long-lived connection drops. The wireless network changes, an intervening proxy
closes an idle connection, a laptop wakes from sleep. This is not an error but the normal
condition of stream-based work; the design is not aimed at preventing the drop but at
resuming from the right place afterward.

The field that enables resumption is `id`. The client stores the last id it received,
sends it back with the `Last-Event-ID` header when reconnecting, and the server
publishes everything after that id. The output shows this flow: the first connection was
cut after two events, the second connection started from three and four. For this
mechanism to work, the server has to **keep the events it published for a while**; if it
does not, events from the gap are lost and the client has to pull the full state again
after reconnecting.

The `retry` field suggests a reconnection interval in milliseconds. Code that writes its
own parser should honor this value and apply a growing wait instead of a fixed interval;
if every client comes back at the same interval when the server crashes, it knocks down
the server that is trying to come back up. This calculation itself is the subject of the
next lesson.

The last two lines show a third rule: **the same event can arrive twice**. During
reconnection the server may interpret the last id inclusively, or the client may drop
before it saves the id it received. Processing code therefore has to be **idempotent**:
applying the same event twice must give the same result as applying it once. In the
example this is achieved by keeping seen ids in a set; the first batch's repeat was
ignored and the final state did not change.

The second route to idempotence is writing the event as an **announcement**, not a
command. "Increment the measurement counter by one" breaks when applied twice; "station
NS-01's last measurement is `m-114`, value −4.4" gives the same result applied twice.
Events carried over a stream are designed in the second form whenever possible.

## The Two-Way Socket's Differences

A socket starts with an ordinary HTTP request, and if the server accepts, the connection
switches protocol. From that moment HTTP semantics — method, status code, headers — no
longer apply; the channel carries text or binary messages. The application has to define
its own message format; which field says the message type, where the sequence number
sits, is left to an agreement between client and server.

It departs from the event stream on four points. **Reconnection is not built in**;
detecting a drop and rebuilding the connection is the application's job. **A drop is not
always noticed**; when transmission is cut silently, the connection can keep looking
open, so both sides send a poll message at regular intervals and wait for a reply,
treating the connection as dead if none arrives. **Queuing responsibility sits with the
client**: messages meant to be sent while the connection is down wait in a queue,
drained once the connection is up — or deliberately dropped. **Authentication happens at
the initial handshake**; no header can be sent once the channel is open, so the
session's validity for the channel's lifetime is a separate problem, covered in the
Session Renewal lesson.

## Applying the Stream to State

An incoming event has no meaning by itself; its meaning forms from the state it is
applied to. Three problems come up in every application.

**Meeting the initial state with the stream.** When a screen opens, the naive approach
fetches the full list first, then listens to the stream. Events produced in the interval
between the two steps are missed. The right order is reversed: open the stream first and
buffer incoming events, then fetch the full list, apply the buffered events that come
after the list's timestamp, and drain the buffer.

**Out-of-order arrival.** If events arrive by different routes — one from the stream,
one from the response to the user's own action — an older value can overwrite a newer
one. The fix is for every record to carry a version or timestamp, and for the processing
code to accept **only a newer stamp**.

**The hidden tab.** Keeping the stream open in a background tab consumes a connection on
the server and produces work on the client for no benefit. The stream is closed when
visibility changes and reopened from the last id once the tab comes back to the front.
This is also a natural instance of using the missed-event recovery path.

## Summary

- The client always opens the connection; transports differ in how long it lasts and how
  many directions it carries. Polling is the cheapest setup, and its latency equals the
  interval.
- An event stream is one-way and an ordinary HTTP response; a socket is two-way and set
  up by switching protocol, leaving reconnection and liveness checking to the
  application in exchange.
- The stream format is line-based: `event`, `data`, `id`, and `retry` fields, events
  completed by a blank line, a heartbeat sent as a comment line.
- Disconnecting is normal; the last event id is stored and reported at reconnection with
  `Last-Event-ID`. If the server does not keep its published events for a while,
  resumption is not possible.
- The same event can arrive twice; processing code must be idempotent, and events should
  be designed as final-state announcements rather than increment commands.
- When meeting the initial list with the stream, the stream is opened first and events
  are buffered; out-of-order events are filtered out with a version stamp.

## Next Step

The last three lessons built the ways of getting data; what they share is that data is,
for a while, **not yet there**. The screen has to show something in the meantime: an
empty frame, a loading indicator, stale data, or an error. Keeping these states as
separate logical flags — loading, has error, has data — quickly produces combinations
that contradict each other. The next lesson reduces the states a view can be in to a
finite set and tests the transitions with a table.