---
title: 'File Upload and Storage'
source: 'https://academia.sh/en/courses/server-fundamentals/file-upload-and-storage'
course: 'Server-Side Fundamentals'
language: en
updated: '2026-08-19T05:19:38+00:00'
license: 'CC BY-SA 4.0'
---

# File Upload and Storage

Accepting a file coming from outside within limits: the two-layer size limit formed by the declared length and the stream counter, getting the rejection to the other side, determining the type from content instead of the extension, separating the store directory from the served directory, and generating the file name on the server.

The previous two lessons dealt with data flowing out of the server: a file on disk, a
document rendered from a template. In the library service's operation, data also flows in
the opposite direction. A donated book's cover image, a photo added to a member's record,
a count file coming from a branch — these arrive at the server from the outside.

Incoming content differs from content leaving the server in three ways: its size is not
known in advance, its declared type may not match what it actually is, and where it gets
written is the server's decision. This lesson builds the upload endpoint as the answer to
these three questions. Every item it takes up is a **defense**: a rule the server applies
before it accepts anything.

## The Size Limit Has Two Layers

The size of an upload body can be learned from two separate places, and neither is
sufficient on its own.

The first is the **declared length**: the content-length value in the request header. This
value is visible before the body is read at all, so an upload above the limit can be
rejected before its first byte ever reaches memory. It is a cheap, early defense.

The second is the **stream counter**: counting the bytes that actually arrive as the body
is read. The declared length is part of the request, and the side that built the request
may not send it, or may send it wrong; in chunked transfer the header is not present at
all. The counter enforces the limit regardless of what was declared.

The two layers work together: the declaration enables an early rejection, and the counter
guarantees that the rejection cannot be bypassed. With only the first, the limit disappears
the moment it is not declared; with only the second, a large payload gets read all the way
to the limit every single time.

## Getting the Rejection to the Other Side

Closing the connection on an over-limit request the instant it is detected sounds
reasonable, but the outcome is not what one would expect: the other side keeps writing the
body, so the connection breaks with an error, and the response the server wrote is never
read. What the sender sees is "connection reset"; the fact that a limit was exceeded, what
the limit was, and how to fix the request are all lost.

For the rejection to be readable, the server has to keep **draining** the incoming bytes:
not accumulating them, but reading them. Draining also has an upper bound of its own; once
that bound is exceeded too, the connection is dropped, because reading forever would erase
the point of rejecting in the first place.

This is the detail most often skipped in the upload endpoint, and it shows up directly in
the measurement: without draining, the client sees a connection error instead of a 413
response.

## Type Is Determined from Content, Not the Extension

Two pieces of information reach the server about the uploaded file's type: the
content-type header in the request, and the file name's extension. Both are text the
sending side wrote, and neither is required to match the content. When they do not, what
results is an inconsistency before it is an attack: the record is marked as an image,
served as an image, and is not an image.

The trustworthy source is the content itself. Common binary formats carry a fixed byte
sequence at the start of the file; this sequence is called a **magic number**. The server
knows the signatures of the types it accepts and compares the body's first bytes against
that list. If there is no match, the upload is rejected.

Signature checking is an allowlist: the server counts what it accepts, not what it
rejects. Everything outside the list is rejected because it is unrecognized. A rule built
in the opposite direction — banning specific types — silently accepts every new format
that never made it onto the list.

The limit of signature checking must also be understood plainly: the signature shows the
file's format, not that the rest of it is valid. If an image's dimensions and content need
to be checked as well, that is the job of a step that decodes the format.

## Storage Location and the Generated Name

The question of where an uploaded file gets written continues the document-root decision
from the first lesson. The static server serves every file under the document root. If
uploaded content were written into that same directory, byte sequences coming from
outside would be inserted directly into the server's served area. That is why **the store
directory is kept separate from the served directory**, and uploaded content is only ever
served through the application's own endpoint, after its record has been looked up.

The file name is not taken from outside either. The sent name is stored only as metadata
to be shown to the user; the name written to disk is **generated** on the server. The
generated name has three properties: it never collides, it carries no meaning in the
directory structure, and its extension comes not from the sent name but from the
**detected type**. This prevents two simultaneous uploads from overwriting each other, and
it prevents the name from being interpreted as a file path.

Two more headers are added on the serving side. The content type is read from the record,
not from what was declared, and type sniffing is turned off, so the receiving side never
steps outside the declared type and mistakes the content for something else.

## The Endpoint in Full

The endpoint below takes the body directly as file bytes, instead of decoding the
**multipart/form-data** encoding that browser forms use. This is so the limit logic is not
hidden behind part parsing; in multipart encoding, the same three defenses are applied
separately to each part.

```js
// upload.mjs — cover image upload endpoint: size limit, content check, separate store
import { createServer } from "node:http";
import { randomUUID } from "node:crypto";
import { mkdir, writeFile, readFile } from "node:fs/promises";
import { join } from "node:path";

const LIMIT = 64 * 1024;              // largest accepted cover image
const DRAIN_LIMIT = 1024 * 1024;      // most bytes still read from a rejected body
const STORE = join(process.cwd(), "data", "covers"); // outside the served directory
const records = new Map();            // generated name -> { type, bytes }

// Type is determined from the body's first bytes, not the declared header or file name.
const SIGNATURES = [
  { extension: "png", type: "image/png", signature: [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a] },
  { extension: "jpg", type: "image/jpeg", signature: [0xff, 0xd8, 0xff] },
  { extension: "gif", type: "image/gif", signature: [0x47, 0x49, 0x46, 0x38] },
];
const detectType = (body) =>
  SIGNATURES.find((s) => s.signature.every((b, i) => body[i] === b)) ?? null;

// Reads the body. If the limit is exceeded the bytes are not accumulated, but the stream
// keeps being read up to the drain limit: without draining, the written response never
// reaches the client before the connection breaks.
const readBody = (request, preRejected) =>
  new Promise((resolve) => {
    const chunks = [];
    let total = 0;
    let exceeded = preRejected;
    request.on("data", (chunk) => {
      total += chunk.length;
      if (!exceeded && total > LIMIT) { exceeded = true; chunks.length = 0; }
      if (exceeded) { if (total > DRAIN_LIMIT) request.destroy(); return; }
      chunks.push(chunk);
    });
    request.on("end", () => resolve(exceeded ? null : Buffer.concat(chunks)));
    request.on("error", () => resolve(null));
    request.on("close", () => resolve(null));
  });

const json = (response, status, body) => {
  const text = JSON.stringify(body);
  response.setHeader("Content-Type", "application/json; charset=utf-8");
  response.setHeader("Content-Length", Buffer.byteLength(text));
  response.writeHead(status).end(text);
};

await mkdir(STORE, { recursive: true });

createServer(async (request, response) => {
  response.sendDate = false;

  if (request.method === "POST" && request.url === "/cover") {
    // Defense 1: if the declared length is above the limit, the body is never accumulated.
    // Defense 2: the bytes that actually arrive are counted; a wrong declaration still hits the limit.
    const declared = Number(request.headers["content-length"] ?? 0);
    const stage = declared > LIMIT ? "declared_length" : "stream_counter";
    const body = await readBody(request, declared > LIMIT);
    if (body === null)
      return json(response, 413, { error: "limit_exceeded", stage, limit: LIMIT });

    // Defense 3: type is determined by the content, not the sent header or file name.
    const type = detectType(body);
    if (!type)
      return json(response, 415, {
        error: "type_not_accepted",
        declared: request.headers["content-type"] ?? null,
      });

    // Defense 4: the name is generated on the server; the extension comes from the detected type.
    const name = `${randomUUID()}.${type.extension}`;
    await writeFile(join(STORE, name), body);
    records.set(name, { type: type.type, bytes: body.length });

    response.setHeader("Location", `/cover/${name}`);
    return json(response, 201, { name, type: type.type, bytes: body.length });
  }

  // Defense 5: serving goes through recorded names; the request path never turns into a file path.
  if (request.method === "GET" && request.url.startsWith("/cover/")) {
    const name = request.url.slice("/cover/".length);
    const record = records.get(name);
    if (!record) return json(response, 404, { error: "not_found" });
    const body = await readFile(join(STORE, name));
    response.setHeader("Content-Type", record.type);
    response.setHeader("Content-Length", body.length);
    response.setHeader("Content-Disposition", "inline");
    response.setHeader("X-Content-Type-Options", "nosniff");
    return response.writeHead(200).end(body);
  }

  return json(response, 404, { error: "not_found" });
}).listen(8312, "127.0.0.1", () => console.log("listening: 127.0.0.1:8312"));
```

Keeping records in memory is only within this lesson's scope; when the server restarts,
the records are lost and the files on disk become unreachable. Writing metadata to a
persistent store is the subject of the Data Access Layer and Business Logic course.

## Measurement

The measurement uses three files: a valid image, a text file whose name ends with an
image extension, and a file above the limit. The block below generates all three.

```js
// generate-samples.mjs — generates three files for testing: valid PNG, PNG-named text, large PNG
import { crc32, deflateSync } from "node:zlib";
import { statSync, writeFileSync } from "node:fs";

const chunk = (tag, data) => {
  const length = Buffer.alloc(4);
  length.writeUInt32BE(data.length);
  const body = Buffer.concat([Buffer.from(tag, "ascii"), data]);
  const crc = Buffer.alloc(4);
  crc.writeUInt32BE(crc32(body) >>> 0);
  return Buffer.concat([length, body, crc]);
};

const png = (width, height) => {
  const ihdr = Buffer.alloc(13);
  ihdr.writeUInt32BE(width, 0); ihdr.writeUInt32BE(height, 4);
  ihdr[8] = 8; ihdr[9] = 2; // 8-bit depth, true color
  const row = Buffer.alloc(1 + width * 3); // filter byte + RGB
  const raw = Buffer.concat(Array.from({ length: height }, () => row));
  return Buffer.concat([
    Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
    chunk("IHDR", ihdr),
    chunk("IDAT", deflateSync(raw)),
    chunk("IEND", Buffer.alloc(0)),
  ]);
};

writeFileSync("cover.png", png(8, 8));
writeFileSync("fake.png", Buffer.from("This is a text file; its name ends with .png.\n"));
writeFileSync("large.png", Buffer.concat([png(1200, 1200), Buffer.alloc(120 * 1024, 0x41)]));
for (const f of ["cover.png", "fake.png", "large.png"])
  console.log(f, statSync(f).size, "B");
```

```bash
#!/usr/bin/env bash
# Starts upload.mjs, tests four upload attempts, lists the store, then stops it.
rm -rf data
node generate-samples.mjs
node upload.mjs > /dev/null &
server=$!
sleep 1
A=http://127.0.0.1:8312

echo "--- 1) valid cover (69 B) ---"
NAME=$(curl -sS -X POST --data-binary @cover.png -H 'Content-Type: image/png' "$A/cover" \
     | tee /dev/stderr | sed 's/.*"name":"\([^"]*\)".*/\1/')
echo
echo "--- 2) serving the uploaded cover ---"
curl -sS -D - -o /dev/null "$A/cover/$NAME"

echo "--- 3) text file named .png, declared as image/png ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @fake.png \
     -H 'Content-Type: image/png' "$A/cover"

echo "--- 4) above the limit, with declared length ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @large.png \
     -H 'Content-Type: image/png' "$A/cover"

echo "--- 5) above the limit, without declared length (chunked transfer) ---"
curl -sS -o - -w '  <- HTTP %{http_code}\n' -X POST --data-binary @large.png \
     -H 'Content-Type: image/png' -H 'Transfer-Encoding: chunked' "$A/cover"

echo "--- written to the store ---"
ls data/covers | sed 's/^/  /'
echo "  total files: $(ls data/covers | wc -l | tr -d ' ')"

kill "$server"
```

```
cover.png 69 B
fake.png 46 B
large.png 127159 B
--- 1) valid cover (69 B) ---
{"name":"9397da79-072c-46bb-bbec-49e9afddcc59.png","type":"image/png","bytes":69}
--- 2) serving the uploaded cover ---
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 69
Content-Disposition: inline
X-Content-Type-Options: nosniff
Connection: keep-alive
Keep-Alive: timeout=5

--- 3) text file named .png, declared as image/png ---
{"error":"type_not_accepted","declared":"image/png"}  <- HTTP 415
--- 4) above the limit, with declared length ---
{"error":"limit_exceeded","stage":"declared_length","limit":65536}  <- HTTP 413
--- 5) above the limit, without declared length (chunked transfer) ---
{"error":"limit_exceeded","stage":"stream_counter","limit":65536}  <- HTTP 413
--- written to the store ---
  9397da79-072c-46bb-bbec-49e9afddcc59.png
  total files: 1
```

The generated name changes on every run; connection headers depend on the runtime's
default settings. Port 8312 is arbitrary and must be free.

Four results are worth reading. **Type checking did not look at the declaration**: in the
third attempt the request declared the type as an image, but because the content was
text, the upload was rejected with 415. **Both limit layers worked independently**: in the
fourth attempt the rejection happened at the declared length, and in the fifth — where the
length was never declared — it happened at the stream counter. The stage field in the
response says which layer engaged. **No rejected upload was written to disk**: close to
two hundred kilobytes of data were sent, yet the store holds exactly one file, and it is
the valid cover.

## Summary

- The size limit consists of two layers: the declared length enables an early, cheap
  rejection, while the stream counter enforces the limit when the declaration is wrong or
  missing.
- If an over-limit request is cut off instantly, the written response never reaches the
  other side; the rejection is communicated by draining the incoming bytes up to their own
  upper bound.
- The file type is determined from the signature at the start of the body, not from the
  declared header or the extension, and the accepted types are counted with an allowlist.
- The store directory is kept separate from static serving's document root; uploaded
  content is served only after its record is looked up, through the application's own
  endpoint.
- The name written to disk is generated on the server and its extension comes from the
  detected type; the sent name is stored only as metadata to be displayed.

## Next Step

This lesson brought out the first persistent data the server writes to disk, and it also
made a gap visible: because records live in memory, they are lost when the server
restarts. In a real setup, those records live in a database, the cover files live in an
object store, and search lives in a separate service. The application is no longer a
single process running on its own; it needs other services alongside it. The course's last
lesson takes up the question of bringing these dependencies up on the developer's machine:
in what order do the services start, how does the application know they are ready, and how
does the same environment get built identically on two machines?