Streaming Responses

All four approaches in this topic carried an event from server to client: a status changed, a queue advanced, stock was updated. One case remains. Sometimes what needs to be carried is not an event, it is the response itself.

A monthly report scanning thirty thousand loan records can be sent line by line before the result is complete. This requires no new protocol; it is done by writing the HTTP response body in pieces, and in exchange, the time the client waits to see the first data shrinks significantly.

Two Forms

A response body can be sent in two forms. If its length is known, it is announced in the Content-Length header, and the receiver knows from the start how many bytes to expect. If its length is not known, chunked transfer encoding is used: the body is split into chunks that each carry their own length, and the last chunk announces the end of the stream with a length of zero.

The server below serves the same report in three ways: /full, which accumulates the whole thing before sending it; /streaming, which writes each line the moment it is produced; and /cut, which abandons production partway through.

// chunked.mjs — serves the monthly loan report in two forms
//   /full        all lines are accumulated in memory, sent as a single piece
//   /streaming   each line is written the moment it is produced: chunked transfer
//   /cut         production fails after the third line
import { createServer } from "node:http";

const LINES = 10;
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const makeLine = (i) =>
  `${String(i).padStart(2, "0")};branch-${1 + (i % 3)};loan=${40 + i * 3}\n`;

createServer(async (request, response) => {
  response.sendDate = false;
  const path = new URL(request.url, "http://local").pathname;

  if (path === "/full") {                          // length known: single piece
    const lines = [];
    for (let i = 1; i <= LINES; i++) { await wait(100); lines.push(makeLine(i)); }
    const body = lines.join("");
    response.writeHead(200, { "content-type": "text/csv; charset=utf-8",
                              "content-length": Buffer.byteLength(body) });
    return response.end(body);
  }

  if (path === "/streaming" || path === "/cut") {  // length unknown: chunked transfer
    response.writeHead(200, { "content-type": "text/csv; charset=utf-8" });
    for (let i = 1; i <= LINES; i++) {
      await wait(100);
      if (response.writableEnded) return;          // client disconnected
      if (path === "/cut" && i === 4) {
        // Headers have already been sent: the status code can no longer be changed.
        response.end("#error;report cut off midway;line=3\n");
        return;
      }
      if (!response.write(makeLine(i))) await new Promise((c) => response.once("drain", c));
    }
    return response.end();
  }
  response.writeHead(404).end();
}).listen(8401, "127.0.0.1", () => console.log("listening: 127.0.0.1:8401"));

Chunked transfer does not need to be requested explicitly. When Content-Length is not written and the body is not finished in a single call, the runtime does the framing itself. The only thing written is the line itself.

The streaming response checking the writableEnded state before every write is the same rule as in server-sent events: continuing production after the client has disconnected means wasted processor time and an error writing to a closed socket.

Measurement

The script below compares the two forms in terms of headers, latency, and chunk arrival times. The measuring script timer.mjs writes each chunk’s arrival moment in hundred-millisecond slices.

// timer.mjs — writes the arrival moment of each chunk of the streaming response as a 100 ms slice
const start = performance.now();
let response;
try {
  response = await fetch("http://127.0.0.1:8401/streaming");
} catch {
  console.log("could not connect to server");
  process.exit(0);
}
for await (const chunk of response.body) {
  const slice = Math.round((performance.now() - start) / 100);
  process.stdout.write(`${String(slice).padStart(2)} slice  ${Buffer.from(chunk)}`);
}

Port 8401 is arbitrary and must be free.

#!/usr/bin/env bash
# Starts chunked.mjs; measures headers, latencies, and chunk arrivals.
node chunked.mjs > /dev/null & server=$!
sleep 1
A=http://127.0.0.1:8401

echo "--- response headers ---"
for path in full streaming; do
  printf '%-10s ' "$path"
  curl -sS -o /dev/null -D - "$A/$path" | grep -i -E 'content-length|transfer-encoding' | tr -d '\r' | paste -sd' ' -
done

echo "--- first byte and total duration (100 ms slices) ---"
for path in full streaming; do
  curl -sS -o /dev/null -w "$path %{time_starttransfer} %{time_total}\n" "$A/$path"
done | awk '{ printf "%-10s first-byte=%2d slice  total=%2d slice\n", $1, int($2*10), int($3*10) }'

echo "--- arrival times of the streaming response's chunks ---"
node timer.mjs

echo "--- raw chunked framing (first three chunks) ---"
curl -sS --raw -N "$A/streaming" 2>/dev/null | head -c 63 | od -c | head -4

echo "--- if production is cut off midway ---"
curl -sS -w 'status=%{http_code}\n' "$A/cut"
kill "$server"

--- response headers ---
full       content-length: 200
streaming  Transfer-Encoding: chunked
--- first byte and total duration (100 ms slices) ---
full       first-byte=10 slice  total=10 slice
streaming  first-byte= 1 slice  total=10 slice
--- arrival times of the streaming response's chunks ---
 1 slice  01;branch-2;loan=43
 2 slice  02;branch-3;loan=46
 3 slice  03;branch-1;loan=49
 4 slice  04;branch-2;loan=52
 5 slice  05;branch-3;loan=55
 6 slice  06;branch-1;loan=58
 7 slice  07;branch-2;loan=61
 8 slice  08;branch-3;loan=64
 9 slice  09;branch-1;loan=67
10 slice  10;branch-2;loan=70
--- raw chunked framing (first three chunks) ---
0000000    1   4  \r  \n   0   1   ;   b   r   a   n   c   h   -   2   ;
0000020    l   o   a   n   =   4   3  \n  \r  \n   1   4  \r  \n   0   2
0000040    ;   b   r   a   n   c   h   -   3   ;   l   o   a   n   =   4
0000060    6  \n  \r  \n   1   4  \r  \n   0   3   ;   b   r   a   n    
--- if production is cut off midway ---
01;branch-2;loan=43
02;branch-3;loan=46
03;branch-1;loan=49
#error;report cut off midway;line=3
status=200

Three results stand out.

The headers give the format away: /full announces a length, /streaming uses chunked transfer. The two cannot coexist.

The total duration is the same in both forms: ten slices. Streaming does not speed up the work, because the report is still produced in the same amount of time. The only thing that changes is the time to first byte: it drops from ten slices to one. The user sees the report’s first lines without waiting for the whole thing to finish. This is the form the shared gain across this entire real-time data topic takes here.

The chunk arrivals confirm the first result: the lines arrive one slice apart, not all together at the end.

Framing

The raw output shows the format of chunked transfer. Each chunk begins with a line that writes its length in hexadecimal: 14 hex, i.e. twenty bytes — the length of one report line. Then comes \r\n, then the data itself, then \r\n again.

This framing solves the question of how the receiver knows where the body ends when the length is not known up front: the last chunk is written with a length of zero, and the stream ends there. It is also technically possible to end a body without announcing a length by closing the connection, but then the receiver cannot tell whether the response finished or was cut off.

After Headers Have Been Sent

The last measurement section shows the most important limit of streaming responses. At the /cut endpoint, report production fails at the fourth line, but the status code stays 200. Because the headers were already sent with the first chunk, and in HTTP, headers cannot be taken back.

This has three consequences. The error must now be reported inside the body; the output format must have a place for this — here, a notification line starting with #. Second, the receiving side must read this line; if it does not, it will mistake the incomplete report for a complete one. Third, writing a completion marker that shows the report finished is safer than writing an error marker: if the connection drops, the error line cannot be written either, but the absence of the completion marker is noticed in every case.

HTTP also defines trailers for this; they allow a header to be sent after the body. In practice, intermediary support is inconsistent, so an in-body marker is the more common and safer choice.

Backpressure in a Streaming Response

Checking the return value of response.write(...) in the code is this lesson’s counterpart to the rule measured in the backpressure lesson. If the network client is slow, the lines being written pile up in the server’s buffer; if the return value is ignored, the buffer grows without bound.

When the report is ten lines long, this is invisible. But a server streaming a thirty-thousand-line report to a slow connection, if it does not listen to the return value, ends up holding the entire report in memory — losing the very memory gain that was the point of streaming. The value of a streaming response lies in production speed being coupled to consumption speed.

When to Stream

Streaming is not right for every response. The gain materializes when production takes a long time and the first piece of the result is meaningful on its own: report lines, search results, log entries. For responses that are small and meaningful only as a single piece — the detail of one loan record — streaming only adds framing overhead.

There are two practical obstacles. Intermediaries can buffer the response and turn the stream back into a single piece; in that case the measured time to first byte shows no gain. And compression can delay streaming, because it produces no output until its buffer fills; for streaming responses, compression must be configured to flush at chunk boundaries.

Summary

• A response body of unknown length is sent with chunked transfer encoding; each chunk writes its length in hexadecimal, and a zero-length chunk ends the stream.
• Streaming does not shorten the total duration; in the measurement, both forms took ten slices, but the time to first byte dropped from ten slices to one.
• Because headers are sent with the first chunk, the status code cannot be changed afterward; the error is reported inside the body, and writing a completion marker is safer than writing an error marker.
• If the write’s return value is not listened to in a streaming response, the entire body piles up in the server’s buffer, and streaming’s memory gain disappears.
• Streaming’s gain depends on a long production time and the first piece being meaningful on its own; intermediary buffering and compression can make that gain invisible.

Course Wrap-Up

This course built four separate layers for shortening the path a request is answered on, and for making the work moved outside that path reliable.

Caching covered reusing a computed result: the client, edge, server, and database layers; the cache-aside, write-through, and write-behind strategies; time-based, event-based, and version-based invalidation; key design with namespace and tenant separation; preventing hot keys and cache stampedes; HTTP validators and cache-control directives; and measuring hit ratio. Its shared lesson: a cache is not an accelerator, it is the controlled acceptance of staleness.

Messaging built taking work out of the request path: the case for asynchronous processing, the producer–consumer queue, publish–subscribe, the difference between a stream and a queue, at-most-once, at-least-once, and exactly-once delivery semantics, the cost of ordering and partitioning, the dead-letter queue, poison messages, and backoff. Its shared lesson: delivery guarantees are not free; every guarantee trades away something from ordering, volume, or latency.

Background Jobs treated the side that processes the message as a unit of scaling: worker processes and supervision, recurring task definitions with a lease lock preventing overlap, progress reporting and cooperative cancellation for long-running jobs, backpressure where production speed exceeds consumption speed, and load shedding to enable controlled rejection under overload. Its shared lesson: a rate mismatch cannot be eliminated, only where it gets written can be chosen — to memory, to loss, or to the producer’s speed.

Real-Time Data built the result reaching the client: the request cost of short and long polling, server-sent events and reconnection by event ID, the lifecycle of a WebSocket connection built with its own handshake, channel distribution in a multi-instance deployment, and responses produced piece by piece. Its shared lesson: an open connection is a state; the price of the latency gain it carries is managing that state.

There was an assumption held silently throughout the course. The cache, the queue, the worker, and the stream were always treated as parts of a single application; all of them inside the same deployment unit, in the same codebase, under the same team’s responsibility. This assumption was necessary for learning the concepts, but it does not last long in real systems.

The next course, Service Architectures, deals with splitting exactly this unit: making a criteria-based choice among monolithic, modular, and distributed architectures, drawing service boundaries according to the domain model, and applying inter-service communication and consistency patterns. None of the concepts measured in this course become invalid there; all of them cross to the other side of a boundary and become more expensive there. A delivery guarantee is no longer a queue’s contract but two teams’. An idempotency key must also handle duplicates arriving over the network. Backpressure spreads not within a process but between services. Channel distribution happens not between instances but between services with different ownership. Every number measured in this course becomes a tool for reading the larger-scale counterpart of the same questions there.