File Upload and Storage

The previous two lessons dealt with data flowing out of the server: a file on disk, a document rendered from a template. In the library service’s operation, data also flows in the opposite direction. A donated book’s cover image, a photo added to a member’s record, a count file coming from a branch — these arrive at the server from the outside.

Incoming content differs from content leaving the server in three ways: its size is not known in advance, its declared type may not match what it actually is, and where it gets written is the server’s decision. This lesson builds the upload endpoint as the answer to these three questions. Every item it takes up is a defense: a rule the server applies before it accepts anything.

The Size Limit Has Two Layers

The size of an upload body can be learned from two separate places, and neither is sufficient on its own.

The first is the declared length: the content-length value in the request header. This value is visible before the body is read at all, so an upload above the limit can be rejected before its first byte ever reaches memory. It is a cheap, early defense.

The second is the stream counter: counting the bytes that actually arrive as the body is read. The declared length is part of the request, and the side that built the request may not send it, or may send it wrong; in chunked transfer the header is not present at all. The counter enforces the limit regardless of what was declared.

The two layers work together: the declaration enables an early rejection, and the counter guarantees that the rejection cannot be bypassed. With only the first, the limit disappears the moment it is not declared; with only the second, a large payload gets read all the way to the limit every single time.

Getting the Rejection to the Other Side

Closing the connection on an over-limit request the instant it is detected sounds reasonable, but the outcome is not what one would expect: the other side keeps writing the body, so the connection breaks with an error, and the response the server wrote is never read. What the sender sees is “connection reset”; the fact that a limit was exceeded, what the limit was, and how to fix the request are all lost.

For the rejection to be readable, the server has to keep draining the incoming bytes: not accumulating them, but reading them. Draining also has an upper bound of its own; once that bound is exceeded too, the connection is dropped, because reading forever would erase the point of rejecting in the first place.

This is the detail most often skipped in the upload endpoint, and it shows up directly in the measurement: without draining, the client sees a connection error instead of a 413 response.

Type Is Determined from Content, Not the Extension

Two pieces of information reach the server about the uploaded file’s type: the content-type header in the request, and the file name’s extension. Both are text the sending side wrote, and neither is required to match the content. When they do not, what results is an inconsistency before it is an attack: the record is marked as an image, served as an image, and is not an image.

The trustworthy source is the content itself. Common binary formats carry a fixed byte sequence at the start of the file; this sequence is called a magic number. The server knows the signatures of the types it accepts and compares the body’s first bytes against that list. If there is no match, the upload is rejected.

Signature checking is an allowlist: the server counts what it accepts, not what it rejects. Everything outside the list is rejected because it is unrecognized. A rule built in the opposite direction — banning specific types — silently accepts every new format that never made it onto the list.

The limit of signature checking must also be understood plainly: the signature shows the file’s format, not that the rest of it is valid. If an image’s dimensions and content need to be checked as well, that is the job of a step that decodes the format.

Storage Location and the Generated Name

The question of where an uploaded file gets written continues the document-root decision from the first lesson. The static server serves every file under the document root. If uploaded content were written into that same directory, byte sequences coming from outside would be inserted directly into the server’s served area. That is why the store directory is kept separate from the served directory, and uploaded content is only ever served through the application’s own endpoint, after its record has been looked up.

The file name is not taken from outside either. The sent name is stored only as metadata to be shown to the user; the name written to disk is generated on the server. The generated name has three properties: it never collides, it carries no meaning in the directory structure, and its extension comes not from the sent name but from the detected type. This prevents two simultaneous uploads from overwriting each other, and it prevents the name from being interpreted as a file path.

Two more headers are added on the serving side. The content type is read from the record, not from what was declared, and type sniffing is turned off, so the receiving side never steps outside the declared type and mistakes the content for something else.

The Endpoint in Full

The endpoint below takes the body directly as file bytes, instead of decoding the multipart/form-data encoding that browser forms use. This is so the limit logic is not hidden behind part parsing; in multipart encoding, the same three defenses are applied separately to each part.

// upload.mjs — cover image upload endpoint: size limit, content check, separate store
import { createServer } from "node:http";
import { randomUUID } from "node:crypto";
import { mkdir, writeFile, readFile } from "node:fs/promises";
import { join } from "node:path";

const LIMIT = 64 * 1024;              // largest accepted cover image
const DRAIN_LIMIT = 1024 * 1024;      // most bytes still read from a rejected body
const STORE = join(process.cwd(), "data", "covers"); // outside the served directory
const records = new Map();            // generated name -> { type, bytes }

// Type is determined from the body's first bytes, not the declared header or file name.
const SIGNATURES = [
  { extension: "png", type: "image/png", signature: [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a] },
  { extension: "jpg", type: "image/jpeg", signature: [0xff, 0xd8, 0xff] },
  { extension: "gif", type: "image/gif", signature: [0x47, 0x49, 0x46, 0x38] },
];
const detectType = (body) =>
  SIGNATURES.find((s) => s.signature.every((b, i) => body[i] === b)) ?? null;

// Reads the body. If the limit is exceeded the bytes are not accumulated, but the stream
// keeps being read up to the drain limit: without draining, the written response never
// reaches the client before the connection breaks.
const readBody = (request, preRejected) =>
  new Promise((resolve) => {
    const chunks = [];
    let total = 0;
    let exceeded = preRejected;
    request.on("data", (chunk) => {
      total += chunk.length;
      if (!exceeded && total > LIMIT) { exceeded = true; chunks.length = 0; }
      if (exceeded) { if (total > DRAIN_LIMIT) request.destroy(); return; }
      chunks.push(chunk);
    });
    request.on("end", () => resolve(exceeded ? null : Buffer.concat(chunks)));
    request.on("error", () => resolve(null));
    request.on("close", () => resolve(null));
  });

const json = (response, status, body) => {
  const text = JSON.stringify(body);
  response.setHeader("Content-Type", "application/json; charset=utf-8");
  response.setHeader("Content-Length", Buffer.byteLength(text));
  response.writeHead(status).end(text);
};

await mkdir(STORE, { recursive: true });

createServer(async (request, response) => {
  response.sendDate = false;

  if (request.method === "POST" && request.url === "/cover") {
    // Defense 1: if the declared length is above the limit, the body is never accumulated.
    // Defense 2: the bytes that actually arrive are counted; a wrong declaration still hits the limit.
    const declared = Number(request.headers["content-length"] ?? 0);
    const stage = declared > LIMIT ? "declared_length" : "stream_counter";
    const body = await readBody(request, declared > LIMIT);
    if (body === null)
      return json(response, 413, { error: "limit_exceeded", stage, limit: LIMIT });

    // Defense 3: type is determined by the content, not the sent header or file name.
    const type = detectType(body);
    if (!type)
      return json(response, 415, {
        error: "type_not_accepted",
        declared: request.headers["content-type"] ?? null,
      });

    // Defense 4: the name is generated on the server; the extension comes from the detected type.
    const name = `${randomUUID()}.${type.extension}`;
    await writeFile(join(STORE, name), body);
    records.set(name, { type: type.type, bytes: body.length });

    response.setHeader("Location", `/cover/${name}`);
    return json(response, 201, { name, type: type.type, bytes: body.length });
  }

  // Defense 5: serving goes through recorded names; the request path never turns into a file path.
  if (request.method === "GET" && request.url.startsWith("/cover/")) {
    const name = request.url.slice("/cover/".length);
    const record = records.get(name);
    if (!record) return json(response, 404, { error: "not_found" });
    const body = await readFile(join(STORE, name));
    response.setHeader("Content-Type", record.type);
    response.setHeader("Content-Length", body.length);
    response.setHeader("Content-Disposition", "inline");
    response.setHeader("X-Content-Type-Options", "nosniff");
    return response.writeHead(200).end(body);
  }

  return json(response, 404, { error: "not_found" });
}).listen(8312, "127.0.0.1", () => console.log("listening: 127.0.0.1:8312"));

Keeping records in memory is only within this lesson’s scope; when the server restarts, the records are lost and the files on disk become unreachable. Writing metadata to a persistent store is the subject of the Data Access Layer and Business Logic course.

Measurement

The measurement uses three files: a valid image, a text file whose name ends with an image extension, and a file above the limit. The block below generates all three.

// generate-samples.mjs — generates three files for testing: valid PNG, PNG-named text, large PNG
import { crc32, deflateSync } from "node:zlib";
import { statSync, writeFileSync } from "node:fs";

const chunk = (tag, data) => {
  const length = Buffer.alloc(4);
  length.writeUInt32BE(data.length);
  const body = Buffer.concat([Buffer.from(tag, "ascii"), data]);
  const crc = Buffer.alloc(4);
  crc.writeUInt32BE(crc32(body) >>> 0);
  return Buffer.concat([length, body, crc]);
};

const png = (width, height) => {
  const ihdr = Buffer.alloc(13);
  ihdr.writeUInt32BE(width, 0); ihdr.writeUInt32BE(height, 4);
  ihdr[8] = 8; ihdr[9] = 2; // 8-bit depth, true color
  const row = Buffer.alloc(1 + width * 3); // filter byte + RGB
  const raw = Buffer.concat(Array.from({ length: height }, () => row));
  return Buffer.concat([
    Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
    chunk("IHDR", ihdr),
    chunk("IDAT", deflateSync(raw)),
    chunk("IEND", Buffer.alloc(0)),
  ]);
};

writeFileSync("cover.png", png(8, 8));
writeFileSync("fake.png", Buffer.from("This is a text file; its name ends with .png.\n"));
writeFileSync("large.png", Buffer.concat([png(1200, 1200), Buffer.alloc(120 * 1024, 0x41)]));
for (const f of ["cover.png", "fake.png", "large.png"])
  console.log(f, statSync(f).size, "B");

#!/usr/bin/env bash
# Starts upload.mjs, tests four upload attempts, lists the store, then stops it.
rm -rf data
node generate-samples.mjs
node upload.mjs > /dev/null &
server=$!
sleep 1
A=http://127.0.0.1:8312

echo "--- 1) valid cover (69 B) ---"
NAME=$(curl -sS -X POST --data-binary @cover.png -H 'Content-Type: image/png' "$A/cover" \
     | tee /dev/stderr | sed 's/.*"name":"\([^"]*\)".*/\1/')
echo
echo "--- 2) serving the uploaded cover ---"
curl -sS -D - -o /dev/null "$A/cover/$NAME"

echo "--- 3) text file named .png, declared as image/png ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @fake.png \
     -H 'Content-Type: image/png' "$A/cover"

echo "--- 4) above the limit, with declared length ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @large.png \
     -H 'Content-Type: image/png' "$A/cover"

echo "--- 5) above the limit, without declared length (chunked transfer) ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @large.png \
     -H 'Content-Type: image/png' -H 'Transfer-Encoding: chunked' "$A/cover"

echo "--- written to the store ---"
ls data/covers | sed 's/^/  /'
echo "  total files: $(ls data/covers | wc -l | tr -d ' ')"

kill "$server"

cover.png 69 B
fake.png 46 B
large.png 127159 B
--- 1) valid cover (69 B) ---
{"name":"9397da79-072c-46bb-bbec-49e9afddcc59.png","type":"image/png","bytes":69}
--- 2) serving the uploaded cover ---
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 69
Content-Disposition: inline
X-Content-Type-Options: nosniff
Connection: keep-alive
Keep-Alive: timeout=5

--- 3) text file named .png, declared as image/png ---
{"error":"type_not_accepted","declared":"image/png"}  <- HTTP 415
--- 4) above the limit, with declared length ---
{"error":"limit_exceeded","stage":"declared_length","limit":65536}  <- HTTP 413
--- 5) above the limit, without declared length (chunked transfer) ---
{"error":"limit_exceeded","stage":"stream_counter","limit":65536}  <- HTTP 413
--- written to the store ---
  9397da79-072c-46bb-bbec-49e9afddcc59.png
  total files: 1

The generated name changes on every run; connection headers depend on the runtime’s default settings. Port 8312 is arbitrary and must be free.

Four results are worth reading. Type checking did not look at the declaration: in the third attempt the request declared the type as an image, but because the content was text, the upload was rejected with 415. Both limit layers worked independently: in the fourth attempt the rejection happened at the declared length, and in the fifth — where the length was never declared — it happened at the stream counter. The stage field in the response says which layer engaged. No rejected upload was written to disk: close to two hundred kilobytes of data were sent, yet the store holds exactly one file, and it is the valid cover.

Summary

• The size limit consists of two layers: the declared length enables an early, cheap rejection, while the stream counter enforces the limit when the declaration is wrong or missing.
• If an over-limit request is cut off instantly, the written response never reaches the other side; the rejection is communicated by draining the incoming bytes up to their own upper bound.
• The file type is determined from the signature at the start of the body, not from the declared header or the extension, and the accepted types are counted with an allowlist.
• The store directory is kept separate from static serving’s document root; uploaded content is served only after its record is looked up, through the application’s own endpoint.
• The name written to disk is generated on the server and its extension comes from the detected type; the sent name is stored only as metadata to be displayed.

Next Step

This lesson brought out the first persistent data the server writes to disk, and it also made a gap visible: because records live in memory, they are lost when the server restarts. In a real setup, those records live in a database, the cover files live in an object store, and search lives in a separate service. The application is no longer a single process running on its own; it needs other services alongside it. The course’s last lesson takes up the question of bringing these dependencies up on the developer’s machine: in what order do the services start, how does the application know they are ready, and how does the same environment get built identically on two machines?