File System

The runtime model topic measured how blocking work stops the event loop. The file system interface is the direct application ground of that measurement: the same read operation is offered in three separate call forms, and the three behave differently toward the event loop.

This lesson compares the three forms, determines which one is right in which situation, and writes the measurement collector’s first real step: reading a file and producing its summary.

Three Interfaces

The built-in file system module offers the same operations in three separate forms. The sync form lives in the node:fs module with a Sync suffix, the callback form in the same module without a suffix, and the promise-based form in the node:fs/promises module.

// interfaces.mjs
import { readFileSync, readFile } from 'node:fs';
import { readFile as readFilePromise } from 'node:fs/promises';

const file = 'measurements.ndjson';
const countLines = (text) => text.trim().split('\n').length;

// 1. Sync
console.log('sync    :', countLines(readFileSync(file, 'utf8')));

// 2. Callback
readFile(file, 'utf8', (error, text) => {
  if (error) throw error;
  console.log('callback:', countLines(text));
});

// 3. Promise
readFilePromise(file, 'utf8').then((text) => {
  console.log('promise :', countLines(text));
});

console.log('synchronous flow continued here');

node interfaces.mjs

sync    : 12
synchronous flow continued here
callback: 12
promise : 12

The order of the output shows the difference between the three forms. The sync call stopped the process’s progress until it returned a result; it wrote its result before the second line. The other two started the work and returned right away, so the line synchronous flow continued here appeared before their results.

The callback form’s signature carries a contract: the first parameter is the error, the result is the second parameter. If there is no error, the first parameter comes as null. This arrangement was the only way to do asynchronous work before promises entered the language, and it still exists in most built-in modules.

The promise-based form makes the same work readable in sequence with async/await. It is the default choice throughout the course.

The selection rule is this:

• The sync form is used while the event loop is not doing any work yet: an initial configuration read, the body of a one-shot script, a final sync write at shutdown. It is not used while handling a request.
• The promise-based form is the default for service code.
• The callback form is used on hot paths where the cost of a promise wrapper has to be avoided, and in calls with no promise-based counterpart.

Where the Asynchrony Comes From

For network sockets, the kernel’s notification mechanism takes over the waiting; the process sleeps in the poll phase and wakes when data arrives. The file system is different: on common operating systems there is no comparable notification mechanism for ordinary files.

The runtime fills this gap with a thread pool. File read, write, and directory listing calls are handed to a thread in the pool; that thread makes the blocking system call, and once the work is done the result is placed on the main thread’s queue. The main thread keeps handling other requests in the meantime.

This has two consequences. First, file operations genuinely run in parallel; second, the pool has a fixed width. If more file operations than the pool’s width are started at once, the excess queues up. The pool width can be adjusted with an environment variable; changing it without measuring first does no good.

Branching on Error Codes

File system errors are distinguished not by their message but by their code field. The message text is for readability and can change; the code corresponds to the name the operating system’s system call returned.

// errors.mjs
import { readFile } from 'node:fs/promises';

async function safeRead(path) {
  try {
    return await readFile(path, 'utf8');
  } catch (error) {
    if (error.code === 'ENOENT') return '';       // file missing: empty data
    if (error.code === 'EACCES') {                 // no permission: leave to caller
      throw new Error(`no read permission: ${path}`, { cause: error });
    }
    throw error;
  }
}

console.log('existing:', (await safeRead('measurements.ndjson')).length, 'characters');
console.log('missing :', (await safeRead('missing.ndjson')).length, 'characters');

try {
  await readFile('missing.ndjson', 'utf8');
} catch (error) {
  console.log('code:', error.code, '| syscall:', error.syscall, '| path:', error.path);
}

node errors.mjs

existing: 1008 characters
missing : 0 characters
code: ENOENT | syscall: open | path: missing.ndjson

ENOENT (does not exist), EACCES (no permission), EISDIR (a directory was given), ENOSPC (disk full), and EMFILE (open file limit exceeded) are the most frequently encountered. The permission model from the Introduction to Linux course determines when EACCES arrives: the process’s user id and group membership are compared against the file’s permission bits.

It matters that every caught error is not swallowed. The function above handles only the two cases it knows about, and rethrows the rest as is. Silently turning an unknown code into empty data hides the failure.

Directory Operations

Creating, listing, and removing a directory live in the same module. The block below sets up an example tree itself, then lists it:

// directories.mjs
import { mkdir, writeFile, readdir, stat, rm, readFile } from 'node:fs/promises';

// This block sets up the example tree itself
await rm('data', { recursive: true, force: true });
await mkdir('data/2024-02-07/raw', { recursive: true });   // also opens intermediate directories
await writeFile('data/2024-02-07/measurements.ndjson', await readFile('measurements.ndjson'));
await writeFile('data/2024-02-07/note.txt', 'edge-03 node has been hot since morning\n');

const entries = await readdir('data/2024-02-07', { withFileTypes: true });
for (const entry of entries.sort((a, b) => a.name.localeCompare(b.name))) {
  if (entry.isDirectory()) {
    console.log(`directory - ${entry.name}`);
  } else {
    const { size } = await stat(`data/2024-02-07/${entry.name}`);
    console.log(`file  ${String(size).padStart(5)} ${entry.name}`);
  }
}

node directories.mjs

file   1008 measurements.ndjson
file     40 note.txt
directory - raw

The recursive option appears in two places and does two different jobs. In the mkdir call it opens intermediate directories and does not error if the directory already exists; this makes the operation idempotent. In the rm call it removes the entire subtree — together with force, it also does not error if the target does not exist.

An rm call cannot be undone. The rule stated for the shell in the Introduction to Linux course holds here too: if the path to be removed comes from a variable, verify before removing that the path stays under the root you expect. How to write that verification is the subject of the next lesson.

The withFileTypes option on the readdir call also brings type information along with the names; without it, a separate stat call would be needed for each name. The order of the returned entries depends on the file system and cannot be assumed to be alphabetical — that is why the example sorts them.

Writing Without Leaving a Half-Written File

The measurement collector’s first output is a file producing a summary per node and metric. Writing directly over the target file carries a risk: if the process ends mid-write, the file is left half-written and the program reading it sees corrupt data.

The fix is writing to a temporary name and putting it in place once complete. Within the same file system, a rename operation is atomic: readers see either the entirety of the old file or the entirety of the new one.

// summarize.mjs
import { readFile, writeFile, rename, stat } from 'node:fs/promises';

async function summarize(source, destination) {
  const text = await readFile(source, 'utf8');
  const buckets = new Map();

  for (const line of text.split('\n')) {
    if (line.trim() === '') continue;
    const { node, metric, value } = JSON.parse(line);
    const key = `${node}/${metric}`;
    const record = buckets.get(key) ?? { count: 0, total: 0 };
    record.count += 1;
    record.total += value;
    buckets.set(key, record);
  }

  const summary = [...buckets].map(([key, { count, total }]) => ({
    key,
    count,
    average: Number((total / count).toFixed(2)),
  }));

  // To avoid leaving a half-written file, write to a temp name first, then rename into place
  const temp = `${destination}.temp`;
  await writeFile(temp, JSON.stringify(summary, null, 2) + '\n');
  await rename(temp, destination);

  const info = await stat(destination);
  return { recordCount: summary.length, bytes: info.size };
}

const result = await summarize('measurements.ndjson', 'summary.json');
console.log(`${result.recordCount} summary records, ${result.bytes} bytes written`);

node summarize.mjs

6 summary records, 471 bytes written

The temporary file has to be created in the same directory as the target. A rename spanning different file systems is not atomic; the kernel carries it out as a copy-then-delete, and the guarantee disappears.

This approach has a limit too: the data becomes visible only after the rename operation, not right after the process finishes, but exactly when it is really written to disk depends on the operating system’s cache. If resilience against a power outage is needed, a sync call is made through the file descriptor; ordinary services usually do not take on this extra cost.

The produced summary.json file is the content the service will serve throughout the course; in later lessons the same summary is produced first through streams, then over HTTP.

Summary

• File system operations are offered through three interfaces: sync, callback, and promise-based. The sync form is used only while the event loop is doing no work.
• File operations’ asynchrony comes from the thread pool; the pool has a fixed width, and excess concurrent requests queue up.
• Errors are distinguished by their code field; branching based on message text is fragile.
• The recursive option in an mkdir call makes the operation idempotent; the order readdir returns depends on the file system.
• When an output file is updated by writing to a temporary name and renaming within the same directory, readers never see half-written content.

Next Step

Every path in this lesson was written relative to the working directory; when the program is invoked from a different directory, it cannot find the file. Writing the separator character by hand also ties the tool to a single platform. The next lesson covers building paths: the rules for joining parts, making a relative path absolute, and verifying that a piece of user input does not escape the root directory.