--- title: 'Modules, Tooling and the Ecosystem' source: 'https://academia.sh/en/courses/javascript-ecosystem' language: en updated: '2026-08-17T04:41:29+00:00' license: 'CC BY-SA 4.0' lessons: 14 outcomes: - 'Explains module systems and how packages are resolved' - 'Manages dependencies with semantic versioning rules' - 'Configures build and static analysis tools to the needs of a project' --- # Modules, Tooling and the Ecosystem How module systems work, package resolution and version management, the control of dependency risk, and the classes of build, static analysis, and task automation tools. ## Module Systems The move from global scope pollution to module scope, the two module systems' binding and loading models, interoperability, and dynamic import. ### The Need for Modules Source: https://academia.sh/en/courses/javascript-ecosystem/need-for-modules The name-collision problem born from sharing the global scope, the namespace pattern built with closures, and module scope's answer to that problem. The Asynchronous JavaScript and the Runtime course covered how a single program runs and how its problems are diagnosed. This course moves up one level: to the question of how the pieces find one another, and who can see what, once a program no longer fits in one file. The question looks innocent. Splitting code into two files is assumed to be nothing more than loading the files one after another. Seeing where this assumption breaks also explains why the concept of a module was added to the language. #### The Global Scope Is a Shared Resource The language's first unit of execution was the **script**: source text is parsed, and its top-level declarations are written to the global scope. Two separate files, loaded one after another in the same runtime, do not each build a separate world; both write to the **same** global object. This behavior can be observed directly through the runtime's script-evaluation interface: ```javascript // file: scripts.mjs import vm from 'node:vm'; // Two separate "scripts", run in the same global scope. vm.runInThisContext('var version = "chart-1.0"; function format(d) { return "chart: " + d; }'); vm.runInThisContext('var version = "report-2.0"; function format(d) { return "report: " + d; }'); console.log(globalThis.version); console.log(globalThis.format('x')); ``` ``` $ node scripts.mjs report-2.0 report: x ``` The first script's `version` variable and `format` function vanished. No error, no warning: the second declaration wrote over the first. The program keeps running and produces the **wrong** result. The problem's size grows not with the number of files but with the **number of names**. If twenty files each define five global names, a shared pool of a hundred names forms, and keeping that pool collision-free depends on nothing but the authors' care. The scope concept established in the Programming Fundamentals course turns here into an operational problem: a shared, writable namespace is an **unisolated resource**. The second problem is **order**. If one file uses a function another defines, load order matters — but that order is not written in the code itself; whoever loads it has to know it separately. Dependency information sits **outside** the source, not inside it. #### The Namespace Built with Closures Before the language had module syntax, the answer to these two problems was to use closures' scope-isolation property. A function is defined and called immediately, its inner names stay inside that function's scope, and only a chosen object is handed outward: ```javascript // file: iife.mjs var Measurer = (function () { var separator = /\s+/; function words(text) { return text.trim().split(separator); } return { words: words }; })(); console.log(Measurer.words('one two three')); console.log(typeof separator); ``` ``` $ node iife.mjs [ 'one', 'two', 'three' ] undefined ``` `separator` is not visible from outside; only the name `Measurer` is written to the global scope. This pattern is a direct application of the immediately invoked function pattern from the Objects and Functions in JavaScript course, and it delivers a real gain: **one** global name instead of a hundred. It also loses as much as it gains. The namespace object itself is still in the global scope, and it can still collide. Dependencies are still carried by convention: if `Measurer` expects some other namespace to already be loaded, it has no written way to declare that. Beyond that, finding where a given name comes from means scanning every file, not reading the source. #### Module Scope A **module** solves these problems at the language's parsing level. When a source text is parsed as a **module**, its top-level declarations are written not to the global scope but to a scope specific to that module: ```javascript // file: scope.mjs const version = 'text-measurer-1.0'; function format(d) { return 'measurer: ' + d; } console.log(typeof globalThis.version); // the module-scope name is not on the global object console.log(typeof globalThis.format); console.log(this); // this at the top level of a module ``` ``` $ node scope.mjs undefined undefined undefined ``` All three lines of output say the same thing: a module's top level is **not** the global scope. The names `version` and `format` are not visible from outside; the top-level `this` is bound not to the global object but to `undefined`. The difference from the namespace pattern is **where the isolation comes from**. In the closure pattern, isolation was a structure the programmer built, and it could break; in a module, isolation is the parser's decision, and it cannot break. #### The Script/Module Distinction The same source text can be parsed under two different rule sets. The distinction is not limited to scope alone: | Criterion | Script | Module | |---|---|---| | Top-level declarations | Written to the global scope | Stay in module scope | | Top-level `this` | The global object | `undefined` | | Strict mode | Must be requested explicitly | Always on | | Import/export syntax | Invalid | Valid | | Evaluation timing | The moment it is encountered | After dependencies are resolved | The last row is decisive and will be detailed in the next lesson. Which rule a text gets parsed under is decided by a marker the runtime reads: the file extension, the `type` field in the manifest, or type information given by whoever loads the script. The same character sequence takes on a different meaning depending on that marker. #### What Is Expected of a Module System The problems a module system has to solve can be drawn straight from the two experiments above: **Isolation.** A module's internal names should not be visible from outside; what it hands outward should be the module's own choice. **Explicit interface.** What a module depends on and what it offers should be written in the source itself. Dependency information should live in the code, not in the loader's memory. **Deterministic resolution.** A dependency name should map to the same file in every environment. The name-resolution rule should be written down and predictable. **Single evaluation.** When the same module is requested from more than one place, it should be evaluated once; the state it holds should be the same for every consumer. This guarantee is the language-level counterpart of the symbol resolution introduced in the linking-and-loading lesson of the How Computers Work course: a name binds to exactly one definition. These four guarantees are the criterion for every mechanism examined throughout this course. Package resolution is an implementation of the third; caching is an implementation of the fourth. #### The Course's Example One concrete example will be built throughout the course: a small library package called `text-measurer`. Its job is computing a given text's word and sentence count, average word length, and longest word. It starts as a single-file script; it gets split into modules, its export surface gets defined, it gains a dependency, its version gets tagged, it becomes part of a workspace, and it finally gets wired into build and analysis tools. Its first form is a single file, and it exports nothing: ```javascript // file: meter.mjs const WORD_SEPARATOR = /[^\p{L}\p{N}]+/u; const SENTENCE_SEPARATOR = /[.?]+/; function splitWords(text) { return text.split(WORD_SEPARATOR).filter((s) => s.length > 0); } function splitSentences(text) { return text.split(SENTENCE_SEPARATOR).map((c) => c.trim()).filter((c) => c.length > 0); } function averageLength(words) { if (words.length === 0) return 0; return words.reduce((t, s) => t + s.length, 0) / words.length; } function measure(text) { const words = splitWords(text); return { wordCount: words.length, sentenceCount: splitSentences(text).length, averageLength: averageLength(words), }; } console.log(measure('A module carries its own scope. A script runs in the global scope.')); ``` ``` $ node meter.mjs { wordCount: 13, sentenceCount: 2, averageLength: 4 } ``` Three separate responsibilities are interleaved in this file: splitting the text, computing numeric measures, and presenting the result. Since all three share the same scope, they see each other's names directly. As the file grows, this visibility stops being a design decision and turns into a source of accidents: the `WORD_SEPARATOR` constant is exposed to the entire file even though only the splitting functions care about it. The next lesson splits this file along its responsibilities and examines the effect of that split on the export surface. Splitting's real question is not which code goes into which file, but **which name stays visible from outside.** #### Summary - A source parsed as a script writes its top-level declarations to the global scope; when two files define the same name, the second silently overrides the first. - The namespace pattern built with closures reduces the number of global names, but leaves isolation to the programmer's discipline and does not declare dependencies in the code. - A source parsed as a module keeps its top-level declarations in module scope; strict mode is always on, and top-level `this` is `undefined`. - A module system provides four guarantees: isolation, an explicit interface, deterministic resolution, and single evaluation. #### Next Step Module scope keeps names from leaking out — but we have not yet defined what a module deliberately hands to another module. The next lesson takes up the language's standard module syntax: the forms of export, the binding behavior of import, and why this syntax's static structure is not just a matter of writing style. ### ES Modules Source: https://academia.sh/en/courses/javascript-ecosystem/es-modules Named and default export, import's live-binding behavior, binding coming before evaluation, and the opportunity static structure gives to tree shaking. The previous lesson left the single-file measurement script behind and raised the real question: when a file gets split, who decides which name stays visible from outside? In the language's standard module syntax, the module itself makes that decision, and declares it in writing, inside the source. This lesson takes up that syntax and the mechanism behind it. The syntax is short; the mechanism is the source of several behaviors that will be used throughout the course. #### The Export Surface The measurement script splits into three responsibilities. Splitting the text goes to one module, numeric measurement to a second, and the entry point that combines the two to a third. ```javascript // file: src/splitting.mjs const WORD_SEPARATOR = /[^\p{L}\p{N}]+/u; const SENTENCE_SEPARATOR = /[.?]+/; export function splitWords(text) { return text.split(WORD_SEPARATOR).filter((s) => s.length > 0); } export function splitSentences(text) { return text.split(SENTENCE_SEPARATOR).map((c) => c.trim()).filter((c) => c.length > 0); } ``` ```javascript // file: src/statistics.mjs export function averageLength(words) { if (words.length === 0) return 0; const total = words.reduce((t, s) => t + s.length, 0); return total / words.length; } export function longestWord(words) { return words.reduce((longest, s) => (s.length > longest.length ? s : longest), ''); } ``` ```javascript // file: src/index.mjs import { splitSentences, splitWords } from './splitting.mjs'; import { longestWord, averageLength } from './statistics.mjs'; export function measure(text) { const words = splitWords(text); return { wordCount: words.length, sentenceCount: splitSentences(text).length, averageLength: averageLength(words), longest: longestWord(words), }; } export { splitSentences, splitWords }; ``` ```javascript // file: try.mjs import { measure } from './src/index.mjs'; console.log(measure('A module carries its own scope. A script runs in the global scope.')); ``` ``` $ node try.mjs { wordCount: 13, sentenceCount: 2, averageLength: 4, longest: 'carries' } ``` Two details matter. First, the `WORD_SEPARATOR` constant is now visible only in its own module; since it never receives `export`, it cannot be reached from outside. Visibility is a property of the **declaration**, not of the file boundary. Second, `index.mjs`'s last line **re-exports** two names. The entry point carries part of the sub-modules' surface onto its own, gathering the interface the package offers outward in one place. `averageLength` and `longestWord` were not added to this list: they stay internal computation detail. The export surface is a design decision, and it sets how much the library is free to change later. #### Named and Default Export There are two forms. A **named export** places a name on the surface; the consumer takes it by writing that name. A **default export** offers a single, nameless value; the consumer gives it whatever name it wants. ```javascript export function measure(text) { /* ... */ } // named export default function measure(text) { /* ... */ } // default ``` The distinction is not just a matter of writing convenience. In a named export, the name is part of the module's contract: a misspelled name is caught at the binding stage. In a default export, the name is the consumer's choice; two files can call the same value by different names, and automated tools cannot track that name. In library interfaces, the named form builds a stronger contract for cases where the name has to stay stable. #### An Imported Name Is a Binding Importing does not copy a value; it establishes a **live binding** to the declaration in the exporting module. When the exporting module changes the value, the importing side sees the new one. ```javascript // file: record.mjs console.log('record.mjs body ran'); export let callCount = 0; export function count() { callCount += 1; } ``` ```javascript // file: main.mjs console.log('first line of main.mjs body'); import { callCount, count } from './record.mjs'; console.log('imported value:', callCount); count(); count(); console.log('after two calls:', callCount); ``` ``` $ node main.mjs record.mjs body ran first line of main.mjs body imported value: 0 after two calls: 2 ``` `callCount` is a number — an immutable primitive — and yet the second read gave `2`. What is being read is not the value, it is the **binding**. The binding is **one-directional**. The importing side cannot write to it: ```javascript // file: assignment.mjs — record.mjs is the file above import { callCount } from './record.mjs'; callCount = 5; ``` ``` $ node assignment.mjs 2>&1 | grep -E 'Error' | head -1 TypeError: Assignment to constant variable. ``` This restriction guarantees that only a module's own code can change its state. If an outside change is genuinely needed, the module has to export a function for it. #### Binding Comes Before Evaluation The order of the output lines stands out: `record.mjs`'s body ran **before** `main.mjs`'s own console line — even though the import statement came after that line in the source. The reason is that module loading runs in three separate phases: 1. **Parsing and resolution.** The source is read, import specifiers are converted into files, and the process repeats for every file found. The result is a module graph. 2. **Binding.** Every imported name is matched against a declaration in the exporting module. A name with no match throws here. 3. **Evaluation.** Module bodies run once each, dependency by dependency. This was the linker's job in the How Computers Work course's linking-and-loading lesson: resolving symbols and reporting missing ones. The module system's second phase does the same job at the source level. The result is similar too — a missing name is caught before any code runs: ```javascript // file: broken.mjs — record.mjs exports only callCount and count import { callCount, counter } from './record.mjs'; console.log('this line never runs', callCount, counter); ``` ``` $ node broken.mjs 2>&1 | grep 'SyntaxError' SyntaxError: The requested module './record.mjs' does not provide an export named 'counter' ``` The error type stands out: a **syntax error**. The module graph's consistency is a property checked before the program starts. The same holds for a missing file: ```javascript // file: missing-dependency.mjs console.log('this line does not run'); import './nonexistent-module.mjs'; ``` ``` $ node missing-dependency.mjs 2>&1 | grep '^Error' | sed "s|$(pwd -P)/||g" Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'nonexistent-module.mjs' imported from missing-dependency.mjs ``` Nothing was ever printed to the console. The `sed` call is only there to shorten the absolute directory path in the message; the path varies depending on the directory you run it from. #### The Namespace Object All of a module's exports can also be gathered into a single object: ```javascript // file: namespace.mjs — src/index.mjs is the entry point defined above import * as meter from './src/index.mjs'; console.log(Object.keys(meter)); console.log(Object.isFrozen(meter)); console.log(meter[Symbol.toStringTag]); ``` ``` $ node namespace.mjs [ 'measure', 'splitSentences', 'splitWords' ] false Module ``` The keys' order is not the source's writing order, it is **alphabetical order**: the namespace object's content is produced deterministically, independent of the source text's layout. The object is not frozen, but it is not extensible either; no new name can be added to it, and no existing name can be deleted. #### The Consequence of Static Structure Import and export statements can only appear at the top level, and their specifiers have to be string literals. At first glance, this restriction looks like rigidity that reduces flexibility; in exchange, it becomes possible to resolve the code **without running it**. The concrete gain is that unused exports can be dropped from the output; this is called **tree shaking**. A tool can look at the source and see that `index.mjs` uses only the name `measure`, and leave the `splitSentences` function out of the output entirely. Making the same inference in a system where names are computed at runtime is, in general, impossible. The second gain is that tools can build the module graph without running the program. Every bundler, analyzer, and editor feature you will see throughout this course rests on this property. #### Summary - Export is decided at the declaration level; a name with no `export` cannot be reached from outside the file, and the entry point gathers the interface in one place through re-export. - In a named export, the name is part of the contract and a misspelling is caught at the binding stage; in a default export, the consumer chooses the name. - An imported name is not a copy of a value; it is a live binding to the declaration in the exporting module, and it cannot be written to from the importing side. - Loading has three phases: resolution, binding, evaluation. Missing-name and missing-file errors surface in the first two phases, before any body runs. - The syntax being static is what makes tree shaking possible, along with building the graph without running the code. #### Next Step It would be misleading to think this model is the only option. The language has a second, older module system, still commonly encountered, where importing is not a statement but a **function call**, and every behavior branches off from that. The next lesson examines that system, how it lives alongside this one, and how the two diverge on circular dependencies. ### CommonJS Source: https://academia.sh/en/courses/javascript-ecosystem/commonjs The call-based loading model, the module wrapper and cache, the difference between a value copy and a live binding, the two systems' behavior on circular dependencies, and interoperability. The previous lesson established a model where import is a statement and binding completes before evaluation. The language has a second module system that became widespread earlier, and it still carries a large share of the code that runs. The two existing side by side is not a transitional accident, it is a permanent state: in a single project, files from both systems sit next to each other and call one another. This lesson's question is what the second system does differently, and where the boundary between the two models sits. #### Loading Is a Function Call In this system, obtaining a module means calling a function named `require`; exporting means writing to an object named `module.exports`. Neither is syntax — both are ordinary runtime operations. This means the source text is never run directly. The runtime wraps the file's content in a function body and calls that function with five arguments: ```javascript // file: wrapper.cjs console.log(typeof require, typeof module, typeof exports); console.log(__filename.endsWith('wrapper.cjs'), typeof __dirname); console.log(module.exports === exports); ``` ``` $ node wrapper.cjs function object object true string true ``` These five names are not part of the language; they are supplied by the **module wrapper**. Module scope here is the same isolation obtained at the parsing level in the previous lesson, obtained instead through an ordinary function scope. The result is identical: top-level declarations do not leak into the global scope. The last line also shows a trap: `exports` starts out as the same object as `module.exports`. Writing `exports.name = ...` affects both, but writing `exports = ...` changes only the local name and does not touch the export surface. To replace the surface entirely, `module.exports` has to be assigned. A direct consequence of being call-based is that loading can be **conditional and computed**: ```javascript // file: format-plain.cjs module.exports = (measurement) => `word: ${measurement.wordCount}`; ``` ```javascript // file: choice.cjs — format-plain.cjs is the file above const mode = process.argv[2] ?? 'plain'; const format = require(`./format-${mode}.cjs`); console.log(format({ wordCount: 13, sentenceCount: 2 })); ``` ``` $ node choice.cjs word: 13 ``` Since the specifier is computed at runtime, which file gets loaded cannot be known by reading the source. This is the price of that flexibility: tools that do static analysis cannot see this dependency, and tree shaking cannot be applied. #### The Cache and Single Evaluation When the same file is requested more than once, its body runs exactly once. The runtime keeps a cache keyed by the **resolved file path**: ```javascript // file: counter.cjs console.log('counter.cjs body ran'); let count = 0; module.exports = { increment() { count += 1; return count; }, get value() { return count; }, }; ``` ```javascript // file: cache.cjs — counter.cjs is the file above const a = require('./counter.cjs'); const b = require('./counter.cjs'); console.log('same object:', a === b); a.increment(); console.log('read through b:', b.value); console.log('registered in cache:', require.resolve('./counter.cjs') in require.cache); ``` ``` $ node cache.cjs counter.cjs body ran same object: true read through b: 1 registered in cache: true ``` The body ran once, and the two calls gave the same object. The fourth of the four guarantees listed in the previous lesson — single evaluation — is realized here through an observable table. The cache's key is the **resolved absolute path**; if the same file is reached through different paths, the entry differs too. #### Copy and Live Binding `module.exports` is an ordinary object; importing is an ordinary read. This means the previous lesson's live-binding behavior does not exist here: ```javascript // file: record.cjs let callCount = 0; function count() { callCount += 1; } module.exports = { callCount, count }; ``` ```javascript // file: copy.cjs — record.cjs is the file above const { callCount, count } = require('./record.cjs'); const whole = require('./record.cjs'); count(); count(); console.log('taken by destructuring:', callCount); console.log('through the object:', whole.callCount); ``` ``` $ node copy.cjs taken by destructuring: 0 through the object: 0 ``` Both reads gave zero. When the `module.exports` object was built, `callCount`'s value **at that moment** was copied into the object; the increments that followed changed the local variable but not the object. The same scenario gave `2` in standard module syntax. This difference is a source of silent bugs in code moved between the two systems. If a changing counter or flag is being exported, the call-based system needs a **getter computed at read time** — this is exactly what the `get value` definition in the `counter.cjs` example was for. #### Circular Dependency When two modules import each other, the graph contains a cycle. No system can **resolve** this cycle; what they can do is **break** it at a specific point. Where they break it differs between the two systems. In the call-based system, `require` finds a **partially filled** export object in the cache and returns it: ```javascript // file: a.cjs console.log('a: body started'); exports.name = 'A'; const b = require('./b.cjs'); console.log('a: b.name =', b.name, '| b.define =', typeof b.define); exports.define = function () { return 'A description'; }; console.log('a: body finished'); ``` ```javascript // file: b.cjs console.log('b: body started'); const a = require('./a.cjs'); console.log('b: a.name =', a.name, '| a.define =', typeof a.define); exports.name = 'B'; exports.define = function () { return 'B description'; }; console.log('b: body finished'); ``` ``` $ node a.cjs a: body started b: body started b: a.name = A | a.define = undefined b: body finished a: b.name = B | b.define = function a: body finished (node:4526) Warning: Accessing non-existent property 'define' of module exports inside circular dependency (Use `node --trace-warnings ...` to show where the warning was created) ``` `b.cjs` saw `a.cjs` in a **half-run** state: `name` was defined, `define` was not. The program did not crash; it gave a missing value. The process number in the warning line changes on every run. In standard module syntax, since binding completes before evaluation, the names already exist; the problem is that a name's **value** has not been assigned yet. Function declarations are ready before the body runs, so they can be used without trouble: ```javascript // file: a.mjs import { bName, bDefine } from './b.mjs'; console.log('a: body started'); export const aName = 'A'; export function aDefine() { return 'A description'; } console.log('a: b.bName =', bName, '| bDefine() =', bDefine()); ``` ```javascript // file: b.mjs import { aName, aDefine } from './a.mjs'; console.log('b: body started'); console.log('b: aDefine() =', aDefine()); export const bName = 'B'; export function bDefine() { return 'B description'; } ``` ``` $ node a.mjs b: body started b: aDefine() = A description a: body started a: b.bName = B | bDefine() = B description ``` `b.mjs` was able to call a function from `a.mjs`, even though `a.mjs`'s body had not run yet. In the same mechanism, an early access to a value defined with `const` does not give a silent `undefined`, it throws: ```javascript // file: a2.mjs import { bName } from './b2.mjs'; export const aName = 'A'; console.log('a2: bName =', bName); ``` ```javascript // file: b2.mjs import { aName } from './a2.mjs'; console.log('b2: aName =', aName); export const bName = 'B'; ``` ``` $ node a2.mjs 2>&1 | grep '^ReferenceError' ReferenceError: Cannot access 'aName' before initialization ``` The distinction can be summed up: one lets missing data pass silently, the other stops the access. In both, the real fix is the same — a circular dependency is a **design smell**, and it is resolved by extracting the shared part into a third module. #### Interoperability When the two systems coexist in the same project, which file is read under which rule is decided by a marker: the `.mjs` extension forces the standard module, `.cjs` forces the call-based module; for files with a `.js` extension, the `type` field in the nearest manifest is consulted. Access from the standard module to the call-based module is direct: ```javascript // file: legacy-format.cjs function format(number) { return number.toFixed(2); } module.exports = { format, version: '1.0.0' }; ``` ```javascript // file: consumer.mjs — legacy-format.cjs is the file above import legacy from './legacy-format.cjs'; import { format } from './legacy-format.cjs'; import { createRequire } from 'node:module'; const require = createRequire(import.meta.url); const manual = require('./legacy-format.cjs'); console.log('default export:', legacy); console.log('named import:', format(3.14159)); console.log('with createRequire:', manual === legacy); ``` ``` $ node consumer.mjs default export: { format: [Function: format], version: '1.0.0' } named import: 3.14 with createRequire: true ``` The `module.exports` object arrives as the default export. Whether named import works depends on an **inference**: the runtime scans the source and guesses which names got assigned. The scan performs no scope analysis, so it counts assignments inside a conditional or a loop as names too, even ones that never run. Where it does fail is computed keys: an assignment whose name is computed at runtime cannot be scanned, and there only the default export can be used. The reverse direction is more restricted. `require` is a **synchronous** call: it has to return its value immediately. A standard module graph, though, can contain **top-level await**, meaning its evaluation can be asynchronous. A graph like that cannot be loaded with a synchronous call: ```javascript // file: pending.mjs const delay = await new Promise((settle) => setTimeout(() => settle('ready'), 10)); export const status = delay; ``` ```javascript // file: pending-require.cjs — pending.mjs is the file above const m = require('./pending.mjs'); console.log(m.status); ``` ``` $ node pending-require.cjs 2>&1 | grep '^Error' Error [ERR_REQUIRE_ASYNC_MODULE]: require() cannot be used on an ESM graph with top-level await. Use import() instead. To see where the top-level await comes from, use --experimental-print-required-tla. ``` This boundary is a technical necessity, not a shortcoming: a synchronous call cannot wait for work whose completion spreads out over time. The solution the error message points to is the next lesson's subject. #### Summary - In the call-based system, importing is a function call, exporting is writing to an object; the specifier can be computed at runtime, which is why static analysis cannot be done. - The module wrapper provides scope isolation through an ordinary function scope, and supplies the names `require`, `module`, `exports` from outside. - Loaded modules are cached by resolved path; the body runs once, and every call gets the same object. - Exported primitive values are copies; a getter has to be defined for live-binding behavior. - On a circular dependency, the call-based system gives a partially filled object; the standard system stops early access with an error. - The extension and the type field decide which rule applies; a synchronous call cannot load a graph that contains top-level await. #### Next Step There is a form that does what a synchronous call cannot: an import that returns a promise, can be called at runtime, and can have a computed specifier. The next lesson takes up that form, the code-splitting and lazy-loading patterns, and the module registry's role in those patterns. ### Dynamic Imports Source: https://academia.sh/en/courses/javascript-ecosystem/dynamic-imports The promise-returning import expression, conditional loading with a computed specifier, the module registry's single-evaluation guarantee, and the cost balance of code splitting. The previous lesson showed that a synchronous call cannot wait for a loading operation that spreads out over time, and left the solution for later. This lesson takes up that solution: an import form that returns a promise, can be used in expression position, and can have its specifier computed at runtime. The flexibility this form provides combines properties of both module systems: the standard form's deterministic resolution and the call-based form's conditionality. In exchange, it brings a new responsibility — loading now spreads out over time, and it can fail. #### Import as an Expression Dynamic import is an expression whose result is a promise. It can be used inside a function body, a condition, a loop — anywhere in the program — and called from either module system. The value the promise resolves to is the namespace object introduced in the previous lesson. ```javascript // file: pending.mjs const delay = await new Promise((settle) => setTimeout(() => settle('ready'), 10)); export const status = delay; ``` ```javascript // file: pending-import.cjs — pending.mjs is the file above import('./pending.mjs').then((m) => { console.log('async load result:', m.status); }); console.log('call returned immediately'); ``` ``` $ node pending-import.cjs call returned immediately async load result: ready ``` The module that could not be loaded with a synchronous call in the previous lesson was loaded with this form. The output order explains the mechanism: the call returned immediately, the body kept running, and the callback ran once the module was ready. The promise model from the Asynchronous JavaScript and the Runtime course applies here too. The syntax resembles a function call, but it is not a function: it cannot be assigned to another name or called without an argument. Its specifier, though, does not have to be constant. #### A Computed Specifier The measurement library can present its result in different formats. Each format is a separate module, and which one gets used is decided at runtime: ```javascript // file: formatters/plain.mjs console.log('plain.mjs loaded'); export function format(measurement) { return `word: ${measurement.wordCount}, sentence: ${measurement.sentenceCount}`; } ``` ```javascript // file: formatters/table.mjs console.log('table.mjs loaded'); export function format(measurement) { return Object.entries(measurement) .map(([key, value]) => `${key.padEnd(16)}${value}`) .join('\n'); } ``` ```javascript // file: report.mjs — uses the two formatter files above const measurement = { wordCount: 8, sentenceCount: 2 }; async function report(style) { const module = await import(`./formatters/${style}.mjs`); return module.format(measurement); } console.log(await report('plain')); console.log('---'); console.log(await report('table')); console.log('---'); console.log(await report('plain')); ``` ``` $ node report.mjs plain.mjs loaded word: 8, sentence: 2 --- table.mjs loaded wordCount 8 sentenceCount 2 --- word: 8, sentence: 2 ``` On the third call, `plain.mjs` was not loaded again. The runtime keeps every resolved specifier in a **module registry**; the second request returns the already-evaluated module from the registry. The single-evaluation guarantee from the previous two lessons holds for dynamic loading too. The registry's key is the resolved address: even if the same file is reached through two different specifiers, only one entry is created. The price of this flexibility is the same one from the call-based system. When the specifier is built from a template string, which file gets loaded cannot be known by reading the source. For a tool doing static analysis, the graph breaks: it either bundles the entire directory into the output, or none of it. For this reason, when the choice is among a finite set, the form that keeps the specifier constant is preferred: ```javascript // file: fixed.mjs — uses the two formatter files above const condition = process.argv[2] === 'table'; const module = condition ? await import('./formatters/table.mjs') : await import('./formatters/plain.mjs'); console.log(module.format({ wordCount: 8, sentenceCount: 2 })); ``` ``` $ node fixed.mjs plain.mjs loaded word: 8, sentence: 2 ``` The behavior is the same, but both specifiers are constant. The tool can see both files and mark them as chunks. #### Loading Can Fail In the statement form, a missing module threw an error before the program started. In the dynamic form, loading happens at runtime, so failure is a **catchable rejection**: ```javascript // file: missing.mjs try { await import('./formatters/missing.mjs'); } catch (error) { console.log('caught:', error.code); } console.log('program continues'); ``` ``` $ node missing.mjs caught: ERR_MODULE_NOT_FOUND program continues ``` The program did not crash. This is useful for loading optional capabilities: when a plugin cannot be found, the main function can keep running. The same behavior also creates a responsibility — if a required dependency is loaded dynamically, its absence goes unnoticed until runtime. The relationship between binding time and the time an error surfaces is the one built in the How Computers Work course's linking-and-loading lesson. For a module loaded over the network, a missing file is not the only cause of rejection: a dropped connection, a timeout, and a bad response all lead to the same path. This is why dynamic-loading calls are places for the retry and backoff patterns from the Asynchronous JavaScript and the Runtime course. #### The Structure of the Resolved Value The namespace object the promise resolves to is the same as in the statement form; the default export sits there under the name `default`. When a call-based module is loaded dynamically, this object's content is the result of the name inference from the previous lesson: ```javascript // file: legacy-format.cjs function format(number) { return number.toFixed(2); } module.exports = { format, version: '1.0.0' }; ``` ```javascript // file: dynamic-cjs.mjs — legacy-format.cjs is the file above const m = await import('./legacy-format.cjs'); console.log('namespace keys:', Object.keys(m)); console.log('default:', m.default.version, '| named:', m.version); ``` ``` $ node dynamic-cjs.mjs namespace keys: [ 'default', 'format', 'module.exports' ] default: 1.0.0 | named: undefined ``` The inference found the name `format`, but not `version`: one was tied to a declaration through shorthand property syntax, the other was a direct string value. The only reliable way to take a name from a call-based module is through `default`. The third name in the key list is a convenience the runtime adds, and it is not portable. The loading promise can also be bound to a value, giving the **lazy-initialization pattern**: it loads a module the first time it is needed and shares the same promise on later calls: ```javascript // file: lazy.mjs — formatters/table.mjs is the module defined above let pending; function getFormatter() { pending ??= import('./formatters/table.mjs'); return pending; } const [a, b] = await Promise.all([getFormatter(), getFormatter()]); console.log('same promise:', getFormatter() === pending); console.log('same namespace:', a === b); console.log(a.format({ wordCount: 8 })); ``` ``` $ node lazy.mjs table.mjs loaded same promise: true same namespace: true wordCount 8 ``` #### Code Splitting The tool-level counterpart of dynamic import is **code splitting**. Wherever a bundler sees a dynamic import, it cuts the graph there and puts that branch into a separate output file — a **chunk**. The main output carries only the chunk's address; the file is requested the first time it is needed. The gain is a smaller initial-load cost: a part of an application that is rarely opened is never downloaded. The table formatter in the measurement library is an example of this — most calls use the plain format. The cost has three items: **Extra round trip.** The chunk is requested separately, at the moment it is needed. The moment the user is waiting is exactly the most expensive moment to measure. **Duplicated code.** If two chunks use the same helper module, the tool either copies the code into both chunks or produces a third, shared chunk. Both add complexity. **Failure path.** Every chunk is a point of failure; every dynamic-loading call needs an error handler. This is why splitting is not the automatic result of every dynamic import, but a measurement-based decision. The time it saves has to exceed the cost of the extra request. Splitting a frequently used code path hurts performance. #### When Not to Use It The dynamic form does not replace the statement form. When a module is needed on every run and immediately, the statement form is the right choice: the dependency is written into the source, its absence is caught before the program starts, and tools can see the graph. The dynamic form fits three situations: the module is only needed under a specific condition, which module will be needed is decided at runtime, or its loading can be deferred past the initial startup. Used outside these cases, it hides the graph from tools without providing any gain. #### Summary - Dynamic import is a promise-returning expression; it can be called in any position and from either module system, and the value the promise resolves to is a namespace object. - The specifier can be computed at runtime; since a computed specifier makes static analysis impossible, the constant-specifier form is preferred when the choice is finite. - The module registry preserves the single-evaluation-per-resolved-address guarantee in dynamic loading too. - Loading failure is a catchable rejection; this means a gain for optional capabilities and a delayed error for required dependencies. - Code splitting lowers the initial-load cost; in exchange it brings an extra request, duplicated code, and new failure paths, which is why it is a measurement-based decision. #### Next Step Up to this point, every specifier has been a relative file path. In real projects, though, most imports use bare names like `text-measurer`, and the file system carries no record of which file such a name corresponds to. The next topic takes up that resolution: a package's manifest file, the rule for looking up a bare name, and pinning the installed tree down with a lock file. ## Package Management The manifest and lock file, resolving bare specifiers, semantic versioning rules, conflicts in the dependency tree, workspaces, and managing dependency risk. ### Package Registries and Dependencies Source: https://academia.sh/en/courses/javascript-ecosystem/package-registries-and-dependencies The fields of the manifest file, resolving a bare specifier by walking the directory tree, encapsulation through the exports map, dependency types, and what the lock file does. Every specifier in the module systems topic was a relative file path. In real projects, most imports are bare names like `format-helper`: no extension, no directory information. What links this name to a file is a filesystem convention and the package's own declaration. This lesson first defines what a package is, then follows step by step how a bare name is searched for. #### The Manifest File A **package** is a directory that carries its own declaration. The **manifest file (manifest)** defines the package's identity, entry points, and dependencies. This is the language-level counterpart of the metadata concept introduced in the package manager lesson of the Introduction to Linux course; the role of the fields is similar too. The measurement library's manifest — `package.json` at the project root: ```json { "name": "text-measurer", "version": "0.1.0", "type": "module", "exports": { ".": "./src/index.mjs", "./segmentation": "./src/segmentation.mjs" }, "files": ["src"], "dependencies": { "format-helper": "^1.4.0" } } ``` The decisions carried by the fields: | Field | Decision | |---|---| | `name` | The package's unique identity within the registry | | `version` | The version of this release; the second part of the identity | | `type` | Which module system files with a `.js` extension are read as | | `exports` | Which paths can be imported from outside | | `files` | Which files go into the published archive | | `dependencies` | The packages needed to run, and the accepted version ranges | The last three fields are design decisions, not technical requirements. It is the package's author who decides which file is visible from outside, which is published, and which versions are accepted. #### Adding a Dependency The measurement library depends on a small helper to print the average length with a fixed number of digits. The helper package is a directory carrying its own manifest — `node_modules/format-helper/package.json`: ```json { "name": "format-helper", "version": "1.4.0", "type": "module", "main": "index.mjs" } ``` ```javascript // file: node_modules/format-helper/index.mjs export function decimal(value, digits = 2) { return value.toFixed(digits); } ``` The report module added to the library imports this package by its bare name. `src/index.mjs` below, along with `src/segmentation.mjs` and `src/statistics.mjs` that it uses, was written in this topic's ES Modules lesson; the library keeps growing on the same tree: ```javascript // file: src/report.mjs -- src/index.mjs is the entry point that exports the measure function import { decimal } from 'format-helper'; import { measure } from './index.mjs'; export function report(text) { const measurement = measure(text); return `${measurement.wordCount} words, average ${decimal(measurement.averageLength)} characters`; } ``` ```javascript // file: try.mjs import { report } from './src/report.mjs'; console.log(report('A module carries its own scope. A script runs in the global scope.')); console.log(import.meta.resolve('format-helper').split('/node_modules/')[1]); ``` ``` $ node try.mjs 13 words, average 4.00 characters format-helper/index.mjs ``` There are two separate facts here that do not require each other: the package being **written in the manifest** and the package **being present in the directory**. A package written in the manifest but not present in the directory produces a resolution error: ```javascript // file: missing-package.mjs import { decimal } from 'nonexistent-package'; console.log(decimal(1.5)); ``` ``` $ node missing-package.mjs 2>&1 | grep '^Error' | sed "s|$(pwd -P)/||g" Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'nonexistent-package' imported from missing-package.mjs ``` The reverse case is sneakier: a package present in the directory but not written in the manifest runs without a hitch locally. Resolution only looks at the filesystem, not the manifest. Installed somewhere else, that file may not be there, and the error only surfaces on first use. Every used dependency has to be written in the manifest too — the assumption underlying the tree arrangements covered later. #### How a Bare Specifier Is Resolved A bare name is not a file path; it is a name that has to be searched for. The rule fits in one sentence: start from the directory of the file doing the importing, look for this name inside a `node_modules` subdirectory at every level, and if not found, move up one directory. A script that applies the rule directly shows that the search really does proceed this way: ```javascript // file: src/deep/search.mjs import { dirname, join } from 'node:path'; import { existsSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; // Directories searched for the bare specifier: from the current directory toward the root. let dir = dirname(fileURLToPath(import.meta.url)); while (true) { const candidate = join(dir, 'node_modules', 'format-helper'); console.log(`${existsSync(candidate) ? 'found ' : 'missing'} ${candidate}`); if (existsSync(candidate) || dir === '/') break; dir = dirname(dir); } console.log('resolved by the runtime:', import.meta.resolve('format-helper')); ``` ``` $ node src/deep/search.mjs | sed "s|$(pwd -P)|.|" missing ./src/deep/node_modules/format-helper missing ./src/node_modules/format-helper found ./node_modules/format-helper resolved by the runtime: file://./node_modules/format-helper/index.mjs ``` The `sed` call shortens the absolute path in the output; the last line gets the same shortening. The search itself reveals four properties: **It is location-dependent.** The same name can resolve to a different package for two files in different directories. This is the basis of the version-conflict resolution covered in a later lesson. **It requires no registry.** Resolution is entirely local; it never reaches the network. The network is used while packages are being placed into the directory, not during resolution. **The package itself speaks.** Once the directory is found, it is the package's own manifest that says which file gets loaded. **It is deterministic.** The same specifier in the same file tree always resolves to the same file. This is the third guarantee listed in the first lesson of module systems. #### The Exports Map Once the package directory is found, it is the map in the manifest that determines which paths are accessible from outside. A path not listed in the map cannot be imported even if it exists in the directory. The following two files sit in a separate consumer project that installs the `text-measurer` package, and run inside that project's directory. The consumer's own manifest — `consumer/package.json`: ```json { "name": "consumer", "version": "1.0.0", "type": "module", "dependencies": { "text-measurer": "^0.1.0" } } ``` ```javascript // file: consumer/use.mjs import { measure } from 'text-measurer'; import { splitWords } from 'text-measurer/segmentation'; console.log(measure('One sentence. Two sentences.').wordCount); console.log(splitWords('one two three').length); ``` ``` $ node use.mjs 4 3 ``` ```javascript // file: consumer/deep.mjs -- tries to reach an inner file not listed in the map import { averageLength } from 'text-measurer/src/statistics.mjs'; console.log(averageLength(['one', 'two'])); ``` ``` $ node deep.mjs 2>&1 | grep -E '^Error' | sed "s|$(pwd -P)/||g" Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './src/statistics.mjs' is not defined by "exports" in node_modules/text-measurer/package.json imported from deep.mjs ``` This is encapsulation at the package level. With no map, every inner file is open to the outside; consumers bind themselves to inner files, and reorganizing them turns into a breaking change. The map separates which paths are contract and which files can be freely changed. #### Dependency Types In the manifest, dependencies are not a single list; each list answers a different question. **Runtime dependencies.** Packages needed while the package runs. Everyone who installs the package installs these too. **Development dependencies.** Needed only while developing the package: a test runner, a linter, a build tool. The party consuming the package does not install these. The point of the split is to keep the tree the consumer downloads small. **Peer dependencies.** Dependencies the package needs to run, but which the package itself must not install. If a plugin has to use the **same instance** of the library it extends, it is written to this list; that way the party installing the library is the application itself, and there is a single copy in the tree. Why a duplicate copy is a problem is shown in the dependency tree lesson. **Optional dependencies.** Dependencies whose failed installation does not stop the process. The package has to carry a fallback path against their absence; if it does not, this list has been used incorrectly. #### The Package Registry and the Lock File A **package registry** is a directory that maps a name–version pair to an archive. The content of a published name–version pair is considered immutable: the same pair should always give the same bytes. This immutability is the foundation all later guarantees rest on. A range like `^1.4.0` in the manifest denotes not a single version but a set. A choice is made from the set when install time comes. Two installs done at two different times can make different choices as the set grows; the result is two trees that run differently from the same source. The **lock file** removes this uncertainty. It records three pieces of information: 1. **The exact version chosen.** Not a range, a single version number. 2. **The source address.** Where the archive was fetched from. 3. **The integrity digest.** A checksum of the archive's content. The shape of the tree — which package sits at which level — is also recorded. Once a lock file exists, installing is not a search, it is a reproduction: the written tree is installed exactly as is. This split comes from the same place as the Introduction to Linux course's framing of dependency resolution as a constraint problem. The solution the resolver produces has to be storable without recomputing it; the lock file is that record. This is why it is added to version control: a second developer cloning a repository, and a build server, both depend on installing the same tree. #### Summary - A package is a directory carrying a manifest file; the manifest defines its identity, entry points, files to publish, and dependencies. - A bare specifier is searched for in the `node_modules` directory at every level, from the importing file's directory toward the root; resolution is local, location- dependent, and deterministic. - The exports map provides encapsulation at the package level; inner files not in the map cannot be imported from outside. - Dependency lists answer different questions: runtime, development, peer, and optional. - A version range denotes a set; the lock file makes installation reproducible by recording the choice made from the set, the source, and the integrity digest. #### Next Step It was said that ranges denote a set, but not what draws the set's boundaries. Behind a manifest that writes `^1.4.0` accepting version `1.9.0` and rejecting `2.0.0` is a contract that assigns meaning to the parts of a version number. The next lesson builds that contract and its comparison rules with running code. ### Semantic Versioning Source: https://academia.sh/en/courses/javascript-ecosystem/semantic-versioning The promise carried by the three parts of a version number, comparison and precedence rules, pre-release tags, the bounds of version ranges, and the definition of a breaking change. The previous lesson said that a range like `^1.4.0` in the manifest denotes not a single version but a set. What draws the set's boundaries is a contract that assigns meaning to the parts of a version number. This contract is called **semantic versioning**. It is not a technical mechanism, it is a way of making a promise: the party raising the number tells the consumer what it did. The part that is a mechanism — comparison and range arithmetic — is built on top of the contract. #### The Promise the Three Parts Make A version number consists of three numbers separated by dots: **major**, **minor**, and **patch**. Which part gets raised when a release is prepared depends on the kind of change made: | Change | Part raised | Promise given to the consumer | |---|---|---| | Bug fix, internal improvement | Patch | Your code runs unchanged | | Backward-compatible capability addition | Minor | Your code runs unchanged, and there is a new capability | | An edit that changes existing behavior | Major | Your code may change; read the migration notes | The subject of the promise is the package's **public interface**: the paths listed in the exports map and the names reachable from those paths. Renaming an inner file not in the map does not raise the major version — this is the counterpart of encapsulation. This is the package-level contract that the backward compatibility concept introduced in the Programming Fundamentals course turns into. The contract itself enforces nothing. Raising the number correctly is the author's responsibility; on the consumer side, the limit of the trust placed in the number is what the next section's explanation of why ranges are vulnerable is about. #### Comparison and Precedence The precondition for range arithmetic is that two versions can be compared. Comparison cannot be done by text order: version `1.10.0` is greater than `1.9.0` but looks smaller as text. The same problem was described for system packages in the package manager lesson of the Introduction to Linux course; here the rule is narrower and entirely written down. The rule is three steps: the three numbers are compared numerically left to right; on equality, a version carrying a pre-release tag is smaller than one without; if both sides are pre-release, the tag parts are compared one by one. ```javascript // file: version.mjs // Splits a version string into the core triple and pre-release parts; build metadata is discarded. export function parse(string) { const [unversioned] = string.split('+'); const [core, preRelease = ''] = unversioned.split('-', 2); const [major, minor, patch] = core.split('.').map(Number); return { major, minor, patch, preRelease: preRelease === '' ? [] : preRelease.split('.'), }; } function comparePart(a, b) { const aNumber = /^\d+$/.test(a); const bNumber = /^\d+$/.test(b); if (aNumber && bNumber) return Number(a) - Number(b); if (aNumber) return -1; // numeric part is smaller than a lettered part if (bNumber) return 1; return a < b ? -1 : a > b ? 1 : 0; } export function compare(x, y) { const a = parse(x); const b = parse(y); for (const field of ['major', 'minor', 'patch']) { if (a[field] !== b[field]) return a[field] - b[field]; } if (a.preRelease.length === 0 && b.preRelease.length > 0) return 1; if (a.preRelease.length > 0 && b.preRelease.length === 0) return -1; const length = Math.min(a.preRelease.length, b.preRelease.length); for (let i = 0; i < length; i += 1) { const result = comparePart(a.preRelease[i], b.preRelease[i]); if (result !== 0) return result; } return a.preRelease.length - b.preRelease.length; } ``` ```javascript // file: sort.mjs -- version.mjs is the file above import { compare } from './version.mjs'; const versions = ['1.10.0', '1.9.0', '2.0.0-rc.2', '2.0.0-rc.10', '2.0.0', '2.0.0-alpha', '1.9.0+build.7']; console.log(versions.sort(compare).join(' < ')); ``` ``` $ node sort.mjs 1.9.0 < 1.9.0+build.7 < 1.10.0 < 2.0.0-alpha < 2.0.0-rc.2 < 2.0.0-rc.10 < 2.0.0 ``` Four observations in the output summarize the whole rule. Version `1.10.0` came after `1.9.0` — the parts were compared as numbers. Version `2.0.0-alpha` came **before** `2.0.0`. A version carrying a pre-release tag is smaller than an untagged version with the same core triple. The tag means "this version is not that version yet." Version `2.0.0-rc.2` came before `2.0.0-rc.10`. The tag's numeric parts are compared as numbers; if text order were used, the order would come out reversed. `1.9.0` and `1.9.0+build.7` came out **equal** in the comparison: everything after the plus sign is **build metadata** and does not enter precedence. Because the sort is stable, the two kept their original order in the list. Equality means the two archives are counted as the same version; this is why build metadata cannot be used to tell published versions apart. #### Version Ranges The range in the manifest specifies the accepted set of versions. The two most commonly used markers rely on two different levels of the contract: one asks the major version to stay fixed, the other asks the minor version to stay fixed. ```javascript // file: range.mjs -- version.mjs is the file from the previous section import { compare, parse } from './version.mjs'; // Converts a '^' or '~' range into a [lower, upper) pair. export function bounds(range) { const marker = range[0] === '^' || range[0] === '~' ? range[0] : '='; const base = marker === '=' ? range : range.slice(1); const { major, minor, patch } = parse(base); if (marker === '=') return [base, `${major}.${minor}.${patch + 1}`]; if (marker === '~') return [base, `${major}.${minor + 1}.0`]; if (major > 0) return [base, `${major + 1}.0.0`]; if (minor > 0) return [base, `0.${minor + 1}.0`]; return [base, `0.0.${patch + 1}`]; } export function satisfies(version, range) { const [lower, upper] = bounds(range); const isPreRelease = parse(version).preRelease.length > 0; const basePreRelease = parse(lower).preRelease.length > 0; if (isPreRelease && !basePreRelease) return false; return compare(version, lower) >= 0 && compare(version, upper) < 0; } ``` ```javascript // file: trial.mjs -- range.mjs is the file above import { bounds, satisfies } from './range.mjs'; for (const range of ['^1.4.0', '~1.4.0', '^0.4.2', '^0.0.3', '1.4.0']) { const [lower, upper] = bounds(range); console.log(`${range.padEnd(8)} => ${lower} <= v < ${upper}`); } console.log(''); const trials = [ ['1.4.9', '^1.4.0'], ['1.9.0', '^1.4.0'], ['2.0.0', '^1.4.0'], ['1.5.0', '~1.4.0'], ['0.4.9', '^0.4.2'], ['0.5.0', '^0.4.2'], ['2.0.0-rc.1', '^1.4.0'], ]; for (const [version, range] of trials) { console.log(`${version.padEnd(11)} ${range.padEnd(8)} ${satisfies(version, range) ? 'satisfies' : 'fails'}`); } ``` ``` $ node trial.mjs ^1.4.0 => 1.4.0 <= v < 2.0.0 ~1.4.0 => 1.4.0 <= v < 1.5.0 ^0.4.2 => 0.4.2 <= v < 0.5.0 ^0.0.3 => 0.0.3 <= v < 0.0.4 1.4.0 => 1.4.0 <= v < 1.4.1 1.4.9 ^1.4.0 satisfies 1.9.0 ^1.4.0 satisfies 2.0.0 ^1.4.0 fails 1.5.0 ~1.4.0 fails 0.4.9 ^0.4.2 satisfies 0.5.0 ^0.4.2 fails 2.0.0-rc.1 ^1.4.0 fails ``` The caret says "let the major version not change" and accepts minor upgrades. The tilde says "let the minor version not change either" and accepts only patch upgrades. Unmarked notation pins a single version. The third and fourth lines show a special rule. Packages with a major version of zero are not considered stable; the contract is not yet in effect. This is why the caret shifts one digit to the right at major version zero and keeps the minor version fixed. At minor version zero it shifts one more digit. This shift keeps unstable packages from slipping into a project through silent upgrades. The last line shows the rule about pre-releases: version `2.0.0-rc.1`, even though it is numerically smaller than `2.0.0`, does not fall inside range `^1.4.0`. Pre-releases are not included in any range unless the range's base is itself already a pre-release. Otherwise a stable range would pull in versions that count as unpublished. #### The Limits of the Contract The convenience the range markers give — picking up minor upgrades automatically — rests on an assumption: that the publisher keeps the minor-version promise. This assumption weakens in three places. **The definition of a breaking change is ambiguous.** A consumer relying on undocumented behavior breaks when that behavior changes; the publisher, meanwhile, believes it did not change the public interface. If the boundary of documented behavior is not written down, disagreement is inevitable. **A bug fix can break things too.** A patch release that fixes a wrong result breaks code written around that wrong result. The contract still counts this as a patch. **Human error exists.** The wrong part can be raised; a breaking change can be published as a minor release. For these three reasons, ranges are a convenience, not a guarantee. What gives reproducibility is not the range but the lock file introduced in the previous lesson. The range says "which upgrades I am prepared to accept"; the lock file records the exact state of the installed tree. The two work together: the upgrade decision is made as a deliberate action, the lock file is updated, and the tests are run. The tool that makes the contract enforceable on the publisher's side is the **deprecation** step: a capability slated for removal is first marked in a minor release with a warning, the replacement path is documented, and only then removed in the next major release. Skipping this step means the major version number announces the upgrade but does not make the migration possible. #### Summary - The three parts of a version number carry a promise: patch and minor upgrades announce that the code runs unchanged, a major upgrade that it may change. - The subject of the promise is the package's public interface; a change to an encapsulated inner file does not raise the major version. - Comparison is done numerically over the parts; a pre-release-tagged version is smaller than an untagged one, and build metadata does not enter precedence. - The caret keeps the major version fixed, the tilde keeps the minor version fixed; for packages with a major version of zero these bounds shift one digit to the right. - Ranges are a convenience, not a guarantee: the definition of a breaking change is ambiguous, and patch fixes can break things too. The lock file provides reproducibility. #### Next Step Up to now, a single package's single range was examined. In a real project, dozens of packages request the same dependency with different ranges, and these requests do not always intersect. The next lesson shows what happens when they do not: duplicate versions in the tree, two separate instances of the same library, and the identity problems this duality produces. ### Dependency Trees and Conflicts Source: https://academia.sh/en/courses/javascript-ecosystem/dependency-trees-and-conflicts How non-intersecting version requests are resolved by nested installation, the identity and state problems produced by duplicate copies, flattening the tree, and dependencies that require singularity. The previous lesson examined a single range. A project holds dozens of packages, and most of them carry their own dependencies. It is ordinary for the same package to be requested from two different places with two different ranges. If the requests intersect, there is no problem: a version is chosen from the intersection and everyone uses it. What happens when they do not intersect? That is this lesson's subject — the solution to that case, and its cost. #### Non-Intersecting Requests The measurement project requests the second major version of a package named `shared`. The `translator` package the project uses, however, depends on the first major version of that same package. The intersection of the sets `^2.0.0` and `^1.0.0` is empty. In system package managers, as described in the package manager lesson of the Introduction to Linux course, this results in a conflict report: only one version of a library can be installed system-wide. Module resolution, being location-dependent, offers a different way out — two versions of the same package can sit at two separate levels of the tree. Two manifests show the conflicting requests in writing: ```json { "name": "project", "version": "1.0.0", "type": "module", "dependencies": { "translator": "^3.1.0", "shared": "^2.0.0" } } ``` ```json { "name": "translator", "version": "3.1.0", "type": "module", "main": "index.mjs", "dependencies": { "shared": "^1.0.0" } } ``` The shape of the installed tree is this: ``` $ node tree.mjs shared@2.0.0 translator@3.1.0 shared@1.0.0 duplicate: shared -> 2.0.0, 1.0.0 ``` The script that produces this output applies the search rule from the previous lesson in reverse: it walks every `node_modules` directory and prints the manifests it finds. ```javascript // file: tree.mjs import { readdir, readFile } from 'node:fs/promises'; import { join } from 'node:path'; async function walk(dir, depth = 0, findings = new Map()) { const moduleDir = join(dir, 'node_modules'); let entries; try { entries = await readdir(moduleDir); } catch { return findings; } for (const name of entries.sort()) { const packageDir = join(moduleDir, name); const manifest = JSON.parse(await readFile(join(packageDir, 'package.json'), 'utf8')); console.log(`${' '.repeat(depth)}${manifest.name}@${manifest.version}`); findings.set(manifest.name, (findings.get(manifest.name) ?? new Set()).add(manifest.version)); await walk(packageDir, depth + 1, findings); } return findings; } const findings = await walk('.'); for (const [name, versions] of findings) { if (versions.size > 1) console.log(`duplicate: ${name} -> ${[...versions].join(', ')}`); } ``` The placement in the tree is this: the second version of `shared` sits at the root directory, the first version sits inside `translator`'s own `node_modules` directory. By the search rule, an import made from inside `translator` looks in its own directory first and finds the first version; an import made from the root finds the second version. Both requests are satisfied. #### The Cost of Two Copies The conflict is resolved, but the result is two separate package instances. The single evaluation guarantee established in the module systems topic holds **per resolved file path, not per package**; two different files mean two different modules. ```javascript // file: node_modules/shared/index.mjs -- the second version export const version = '2.0.0'; export class Record { constructor(name) { this.name = name; } } const records = []; export function add(record) { records.push(record); return records.length; } ``` ```javascript // file: node_modules/translator/index.mjs -- uses the first version from its own directory import { add, Record, version } from 'shared'; export function use(name) { const record = new Record(name); return { seenVersion: version, recordCount: add(record), record }; } ``` ```javascript // file: main.mjs -- uses the two packages above import { add, Record, version } from 'shared'; import { use } from 'translator'; const ours = new Record('root'); console.log('version seen at root:', version, '| record count:', add(ours)); const theirs = use('inside-translator'); console.log('version seen by translator:', theirs.seenVersion, '| record count:', theirs.recordCount); console.log('is the record translator produced an instance of our class:', theirs.record instanceof Record); ``` ``` $ node main.mjs version seen at root: 2.0.0 | record count: 1 version seen by translator: 1.0.0 | record count: 1 is the record translator produced an instance of our class: false ``` Three lines, three separate problems. **Version visibility depends on who is asking.** Two different `shared` versions are running in the same program. Which one runs depends on where the question is asked from. **State is not shared.** The two record lists are separate; both contain a single element. Anything of the cache or counter kind that the package holds is split. If a library offers something like "list all records," it only sees its own copy's records. **Identity checks fail.** The object `translator` produces is not an instance of the root's class. Even though the classes come from the same source, they are the product of two separate evaluations; object identity does not know this. The same problem shows up with symbols, error classes, and brand checks. These three problems are a frequent cause of the "my program runs but behaves strangely somewhere" kind of hard-to-diagnose failure. Looking for a duplicate version in the dependency tree is the first step in diagnosing it. #### Flattening Nested installation is not always necessary. If the requests intersect, a single copy satisfies all of them, and that copy is placed at the root of the tree. This placement is called **flattening**. Flattening has two justifications. The first is space and time: a single copy is installed instead of dozens of copies of the same package. The second is preventing the three problems just listed. A side effect of flattening is that it sets the stage for the sneaky situation touched on in the previous lesson. A package hoisted to the root becomes visible even from modules that do not write it in their own manifest. The code runs locally; when the tree is built a different way — say another installation tool prefers nested placement — the package disappears. Hence the explicit rule: every used package has to be written in the manifest of the package using it. #### Dependencies Requiring Singularity For some packages, two copies are not merely wasteful, they are an outright bug. Libraries that keep a record, manage global state, or rely on object identity have to be a single instance. The mechanism that provides this is the **peer dependency** list introduced in the first lesson. If a plugin writes the library it extends as a runtime dependency, it installs its own copy into its own directory and duality is created. Written as a peer dependency instead, it does not install the library; it only says "a version in this range has to be present in the environment." Installing becomes the application's responsibility, and a single copy remains in the tree. The peer dependency range has to be chosen deliberately. Too narrow a range needlessly shrinks the set of versions the plugin can be used with and produces non-intersecting requests; too wide a range lets installation appear to succeed on versions where the plugin does not actually work. #### When a Conflict Is Not Resolved Nested installation does not resolve every conflict. In three situations a single copy is mandatory and a decision has to be made: **When peer dependencies do not intersect.** If two plugins want incompatible major versions of the same library, the two cannot be used together. The fix is to upgrade or replace one of the plugins. **When values pass between packages.** If the object one package produces is given as an argument to another package, and the receiving side does an identity check, two copies do not work. **When output size is decisive.** Carrying two versions of the same library in output sent to the browser doubles the bytes downloaded. In these situations, installation tools offer a manifest field that pins a transitive dependency's version from outside. The field is useful but dangerous: forcing a package to run on a version other than the one it expects deliberately violates the version contract. When used it has to be justified with a comment and treated as temporary; the lasting fix is upgrading the incompatible package. #### Summary - Non-intersecting version requests are satisfied by installing two copies of the same package at different levels of the tree; the search rule is what makes each consumer find its own copy. - The single evaluation guarantee holds per resolved file path, not per package; two copies keep separate state and fail object identity checks. - Intersecting requests are satisfied with a single copy, hoisted to the root; this flattening also makes packages not written in the manifest become visible. - Libraries requiring singularity are declared as peer dependencies; the responsibility of installing passes to the application and a single copy remains in the tree. - Pinning a transitive dependency's version from outside violates the contract; it should be treated as temporary. #### Next Step Up to here, every package came from outside. A team's own packages sit on the same tree too, and the dependencies among them cannot wait for a publish on every change. The next lesson covers the arrangement where multiple packages are kept in a single repository, how that arrangement changes resolution, and how the build order across packages is determined. ### Workspaces Source: https://academia.sh/en/courses/javascript-ecosystem/workspaces Keeping multiple packages in a single repository, the workspace declaration, resolving local packages through links, topological build order, and versioning strategies. In the previous lessons, every package came from outside: requested with a range in the manifest, installed into a directory, resolved from there. For packages a team writes itself, this cycle is heavy — every fix made in a library has to be published before it can be tried in the application that uses it. The measurement library has reached this point. The formatting helper was pulled out into its own package, and a report tool that presents the measurement result was added. All three are the same team's product and change together. This lesson covers how the three are kept in a single repository. #### The Multi-Package Repository The arrangement that keeps multiple packages in a single version-control repository is called a **monorepo**. The problems it solves are concrete: **Split changes.** Adding a parameter to a library and updating the tools that use it is a multi-step operation across separate repositories: publish, wait, update the range, install. In a single repository, this is a single commit. **Version lag.** In separate repositories, there is always a gap between the library's latest state and the state the application is using. In a single repository, this gap is zero. **Duplicated configuration.** Linter rules, formatting settings, and the test setup are kept in one place. There is a cost too. The repository grows; which packages are affected by each change has to be computed; test time tends toward the sum of all packages; versioning and release decisions get more complex. Not worth paying if the packages do not change together. The rule of thumb: a single repository suits packages updated together in the same change; separate repositories suit packages that evolve independently. #### The Workspace Declaration The manifest at the root says which subdirectories are packages, and marks itself as not to be published: ```json { "name": "measurement-repo", "private": true, "workspaces": ["packages/*"] } ``` The `private` field is a safeguard: the root directory is not a library, and publishing it by accident is not wanted. The `workspaces` field tells the installation tool to treat every directory matching the listed pattern as a **workspace**. The three packages in the repository are these: ```json { "name": "format-helper", "version": "1.4.0", "type": "module", "main": "index.mjs" } ``` ```json { "name": "text-measurer", "version": "0.1.0", "type": "module", "exports": { ".": "./src/index.mjs" }, "dependencies": { "format-helper": "^1.4.0" } } ``` ```json { "name": "report-tool", "version": "0.2.0", "type": "module", "main": "index.mjs", "dependencies": { "text-measurer": "^0.1.0" } } ``` The point worth noting: the dependencies use no special notation stating they are local. Package `text-measurer` requests `format-helper` by its bare name and a range, the same as any package coming from outside. Locality is a decision made by installation, not the manifest: if the installer sees the requested name is a workspace in the repository and its version fits the range, it does not go to the registry. #### How the Links Are Installed A local package is placed not by copying it into the tree, but by a symbolic link: ``` $ for l in node_modules/*; do printf '%s -> %s\n' "$l" "$(readlink "$l")"; done node_modules/format-helper -> ../packages/format node_modules/report-tool -> ../packages/report node_modules/text-measurer -> ../packages/measurer ``` The symbolic link concept was defined in the links lesson of the Introduction to Linux course: an entry that points at a target without being the target itself. The consequence here is decisive. The resolution rule looks for the name in `node_modules` and does not check whether the entry it finds is a link; the directory it finds is the source tree itself. A change made in the source is visible to the consumer without a reinstall. ```javascript // file: packages/format/index.mjs export function decimal(value, digits = 2) { return value.toFixed(digits); } ``` ```javascript // file: packages/measurer/src/index.mjs -- segmentation and statistics modules sit in the same directory import { decimal } from 'format-helper'; import { splitSentences, splitWords } from './segmentation.mjs'; import { averageLength, longestWord } from './statistics.mjs'; export function measure(text) { const words = splitWords(text); return { wordCount: words.length, sentenceCount: splitSentences(text).length, averageLength: decimal(averageLength(words)), longest: longestWord(words), }; } ``` ```javascript // file: packages/report/index.mjs import { measure } from 'text-measurer'; export function report(text) { const m = measure(text); return `${m.wordCount} words / ${m.sentenceCount} sentences / avg. ${m.averageLength}`; } ``` ```javascript // file: try.mjs -- repo root import { report } from 'report-tool'; console.log(report('A module carries its own scope. A script runs in the global scope.')); ``` ``` $ node try.mjs 13 words / 2 sentences / avg. 4.00 ``` Three packages found each other by their bare names without any of them being published. This arrangement has a silent trap: an import that works locally may not work once published. The directory reached through the link is the entire source tree; the published archive carries only the files listed in the manifest. If the exports map or file list is missing something, the error only shows up after publishing — which is why checking the archive's contents before publishing has to be a habit in multi-package repositories. #### Build Order Since the packages depend on one another, bulk operations performed on them — build, test, publish — cannot run in an arbitrary order. Testing `report-tool` requires `text-measurer` to be ready first. The correct order is the topological sort of the dependency graph. This sort, introduced in the Data Structures course, applies directly here: ```javascript // file: order.mjs -- repo root, reads the packages/ directory import { readdir, readFile } from 'node:fs/promises'; import { join } from 'node:path'; const root = 'packages'; const packages = new Map(); for (const dir of (await readdir(root)).sort()) { const manifest = JSON.parse(await readFile(join(root, dir, 'package.json'), 'utf8')); packages.set(manifest.name, Object.keys(manifest.dependencies ?? {})); } // Topological sort by depth-first search: dependencies are published first. const result = []; const state = new Map(); function visit(name) { if (state.get(name) === 'done') return; if (state.get(name) === 'visiting') throw new Error(`cycle: ${name}`); state.set(name, 'visiting'); for (const dependency of packages.get(name) ?? []) { if (packages.has(dependency)) visit(dependency); } state.set(name, 'done'); result.push(name); } for (const name of packages.keys()) visit(name); console.log(result.join(' -> ')); ``` ``` $ node order.mjs format-helper -> text-measurer -> report-tool ``` Two details of the script matter. Dependencies not present in the repository are skipped: the order of external packages does not concern us, they are already installed. Second, visit state has three values; reaching a "visiting" node again means a cycle, and an error is raised. Unlike a circular dependency between modules, one between packages cannot be resolved at all. The same graph answers a second question too: which packages are affected by a change made in one package? The answer is found by walking the graph in the reverse direction, and it lets only the affected packages be tested. This is the method that keeps test time manageable in large repositories. #### Versioning Strategy The versions of packages in the repository are managed in two ways. **Unified versioning.** All packages carry the same number and are published together. The publish decision is single and easy to track; in exchange, a package that has not changed at all also gets a new version, forcing consumers into an unnecessary upgrade. **Independent versioning.** Each package carries its own number and is published only when it changes — honest toward the consumer; in exchange, it requires tracking which package is at which version and keeping in-repo ranges up to date. The choice depends on who the consumer is. If the packages are only used within the repository, unified versioning makes work easier. If they are published outward, independent versioning is the only choice faithful to the contract established in the previous lesson: a version number carries meaning only when it announces a real change. #### Summary - A multi-package repository lets packages that change together be updated in a single commit and share a single configuration; the cost goes unpaid if the packages evolve independently. - The workspace list in the root manifest states which directories are packages; dependencies use no special notation to mark themselves as local. - Local packages are placed in the tree with a symbolic link; a change made in the source is visible without requiring a reinstall. - Because the entire source tree is reachable through the link, files missing from the published archive are only noticed after publishing. - Bulk operations on packages run in the topological order of the dependency graph; walking in the reverse direction gives the packages affected by a change. #### Next Step Most of the packages in the tree were not chosen directly; they are there because they are some other package's dependency. These packages run code too, and they can execute scripts during installation. The next lesson measures the size of this transitive mass, and covers the defensive sides of dependency selection, locking, and verification. ### Dependency Risk Source: https://academia.sh/en/courses/javascript-ecosystem/dependency-risk Measuring the transitive dependency mass, what code running at install time means, integrity verification, locking discipline, package selection criteria, and managing abandoned packages. The previous lessons treated dependencies as a technical resolution problem: which version gets chosen, where it gets installed, who sees it. This lesson asks a different question — how much is the installed code as a whole trusted, and how is that trust narrowed? The question is a management one, but the answer is entirely technical, starting with a measurable quantity: how many packages a project actually runs. #### The Transitive Mass The packages written in the manifest are a small fraction of the installed packages. Every dependency brings its own dependencies; this relationship continues until it closes. The entire set is called the **dependency closure**. The closure is computed with a breadth-first search over the manifests: ```javascript // file: closure.mjs -- reads registry//package.json files import { readFile } from 'node:fs/promises'; async function manifest(name) { return JSON.parse(await readFile(`registry/${name}/package.json`, 'utf8')); } const root = await manifest('text-measurer'); const direct = Object.keys(root.dependencies); const closure = new Map(); const queue = [...direct]; while (queue.length > 0) { const name = queue.shift(); if (closure.has(name)) continue; const pkg = await manifest(name); closure.set(name, pkg.version); queue.push(...Object.keys(pkg.dependencies ?? {})); } console.log('direct dependencies:', direct.length, '->', direct.join(', ')); console.log('transitive closure:', closure.size); for (const [name, version] of closure) console.log(` ${name}@${version}`); ``` In a small sample registry — a six-package directory — the result is this: ``` $ node closure.mjs direct dependencies: 2 -> format-helper, string-tool transitive closure: 5 format-helper@1.4.0 string-tool@2.1.3 number-core@1.0.7 unicode-table@3.2.0 data-package@1.2.9 ``` Two deliberate choices brought in five packages. The ratio is not due to the example's small size: there is a multiplier effect at every level, and in real projects a double-digit count of direct dependencies can produce four-digit closures. What this number means: the code of every package in the closure runs inside your program, with your program's privileges. You made the decision for the packages you chose directly; for the rest, you trusted indirectly through the packages that chose them. Closure size is the measure of that indirect trust. #### Code That Runs at Install Time The trust boundary does not begin the moment you call the library. The manifest file can define scripts to be run during installation: ```json { "name": "sample-package", "version": "1.0.0", "scripts": { "postinstall": "node setup/prepare.mjs" } } ``` This field exists for legitimate work: local compilation, cache preparation, compatibility checks. The consequence, though — installing a package is equivalent to **running** that package's code. Without ever importing the package, just by running the install command, execution rights go to the install script of every package in the closure. Scripts run with the user's privileges and can reach the filesystem and the network. The defense is three steps: **Turn off install scripts by default.** Installation tools offer this as an option. With them off, packages whose installation fails are examined one by one, and an exception is granted only if genuinely necessary. **Install in an isolated environment.** Installing not on the developer's machine but in a limited-privilege, discard-afterward environment narrows the area a script can reach — the principle of least privilege from the Introduction to Linux course applies directly here. **Review changes that update the lock file.** A changed lock line means the archive to be installed has changed, and should be reviewed as carefully as source code. #### Integrity Verification The third piece of information the lock file records — the integrity digest — answers a specific question: is the downloaded archive identical to the archive at the moment it was locked? The digest is a cryptographic checksum computed over the archive's bytes. Changing a single byte changes the digest entirely: ```javascript // file: digest.mjs import { createHash } from 'node:crypto'; import { readFile } from 'node:fs/promises'; const content = await readFile(process.argv[2]); const digest = createHash('sha512').update(content).digest('base64'); console.log(`bytes: ${content.length}`); console.log(`sha512-${digest.slice(0, 44)}...`); ``` ```javascript // file: package.mjs export function decimal(value, digits = 2) { return value.toFixed(digits); } ``` ``` $ node digest.mjs package.mjs bytes: 100 sha512-+J2oBbbq0tFWSv0c1SoceVACG6lMhXEwdJ/46hX89Uru... ``` With a single character in the file — the default digit count — changed to `3` and the same command run again: ``` $ node digest.mjs package.mjs bytes: 100 sha512-AuXc8NVQcn4vfnfQvKDNnEHicRcsEPDkRoSygM0ma2DA... ``` The size stayed the same, the digest changed completely. The script prints the digest truncated; the real digest is longer and is stored in full in the lock file. The surface it closes is clear: the same name and version served with different content. The surface it does not close is equally clear: publishing a **new version** changes the digest anyway. The integrity digest proves the content has not changed; it does not prove the content is safe. #### Upgrade Discipline In a project with a lock file, the install command that follows the lock exactly installs the tree written in it; an ordinary install may update the lock from the manifest. This turns upgrading from an accident into a deliberate process. A working routine includes these steps: 1. An upgrade is made as a separate change; it is not merged into the same commit as another edit. 2. The changed lock lines are reviewed: which packages, from which version to which. 3. The tests are run. 4. The reason for the upgrade is recorded. Delaying the upgrade is a risk too: staying on an old version means continuing to carry fixed defects. The balance is regular, small upgrades; accumulated upgrades done all at once make review impractical and, on failure, make it hard to tell which package is responsible. #### Package Selection Criteria Adding a dependency is a decision, and it can be evaluated with criteria that do not depend on time: **Closure size.** How many packages are in the package's own dependency closure? One that brings a large closure for a small function is expensive. **Interface surface.** How many of the package's functions will actually be used? A large library pulled in for a single function makes both tree shaking and review harder. **Replaceability.** What replaces the package if it is dropped? One with a narrow, standard interface is replaceable; one that imposes its own concepts on the project is not. **Maintenance signals.** Are reported defects answered, are there tests, how many contributors? Single-person maintenance ends when that person loses interest. **License.** Do the usage and distribution terms fit the project? **The cost of writing it yourself.** If the function can be written in a few lines, the dependency's installation, update, and trust cost may not be worth it. The same measure applies in reverse too: in high-detail domains like cryptography, date computation, or text encoding, writing your own implementation is riskier than using a mature package. #### Name Verification and Abandoned Packages Package names are unique within a registry, but similarity is unrestricted. A name with a letter added to a popular package, or a hyphen moved, can be picked unnoticed when typed by hand. The defense is spelling discipline: the name is copied from documentation or the manifest, never typed from memory; after adding it, the manifest line and the package's source repository link are checked. An abandoned package is a separate problem, diagnosed by observation: defects have piled up, no releases are being made, its own transitive dependencies are unmaintained too. The options: **Replace it.** Swap in a maintained package doing the same job. Workable if the interface surface is narrow. **Vendor it.** Copy the source into the project and take over maintenance. Suitable if the code is small and can be considered frozen; its closure has to be carried along too. **Take over maintenance.** Fork it and publish under your own name. Responsibility increases; in exchange, control is entirely yours. In all three, the first step is the same: asking again whether the package is really needed. The safest dependency is the one removed. #### Summary - The dependency closure is far larger than the packages chosen directly; every package in the closure runs with the program's privileges. - Install scripts run code without the package ever being imported; the defenses are turning scripts off, installing in an isolated environment, and reviewing lock changes. - The integrity digest catches the same name and version being served with different content; it does not prove the content is safe. - An upgrade should be a separate, regular action; delaying it is a risk too. - Package selection is evaluated by closure size, interface surface, replaceability, maintenance signals, license, and the cost of writing it yourself. - There are three paths for an abandoned package — replace it, vendor it, take over maintenance — and before all three, the package's necessity is questioned. #### Next Step Dependencies have been resolved, locked, and audited. What remains is how these packages and your own modules get delivered to the target environment. The next topic covers the class of tools that take the module graph as input and produce a runnable output; its first lesson shows why this production is needed and how it works with a running example. ## Build Tools Producing output from the module graph, adapting the source to the target environment, tracing back from generated output to the source, rule-based analysis and formatting, and task automation. ### The Bundler Concept Source: https://academia.sh/en/courses/javascript-ecosystem/bundler-concept The rationale for producing a single output from a module graph, the three steps of a working bundler, the shape of the output, and how tree shaking is limited by side effects. The module systems and package management topics established how code is organized in the source tree and how dependencies are found. The source layout does not have to be the same layout the target environment wants. A **bundler** is a class of tool that takes a module graph as input and produces one or a few files that run in the target environment. Rather than describe what the tool does, this lesson builds a small implementation of one; after that, what real tools add on top of this skeleton becomes easy to see. #### Why a Single Output The source layout consists of hundreds of small files. Using this layout directly in the target environment creates three problems. **Per-file cost.** Every file loaded over the network is a separate request. Request latency can exceed the time spent transferring bytes. The per-request latency introduced in the How the Internet Works course becomes the dominant cost once file count grows large. **Resolution difference.** Bare specifiers are resolved by a lookup rule in the file system. That rule is not present in every environment; a browser cannot turn a name like `format-helper` into a file on its own. A bundler replaces these names with their resolved forms, so the lookup work described in the package management topic happens once, at build time, instead of on every load. **Language and format difference.** The source may contain syntax the target environment does not understand, or a mix of two different module systems. The output is produced in a single, uniform format. None of these apply to every project. A library that only ever runs at build time and is never distributed over the network can be used without bundling. Bundling is not a requirement, it is the answer to specific problems. #### Three Steps A bundler's core is three steps: extract the graph starting from an entry point, wrap each module in a function, and write a small runtime that links the modules together. Take the call-based version of the measurement library as input: ```javascript // file: source/splitting.js const WORD_SPLITTER = /[^\p{L}\p{N}]+/u; exports.splitWords = function (text) { return text.split(WORD_SPLITTER).filter((s) => s.length > 0); }; ``` ```javascript // file: source/statistics.js exports.averageLength = function (words) { if (words.length === 0) return 0; return words.reduce((t, s) => t + s.length, 0) / words.length; }; exports.longestWord = function (words) { return words.reduce((longest, s) => (s.length > longest.length ? s : longest), ''); }; ``` ```javascript // file: source/index.js const { splitWords } = require('./splitting.js'); const { longestWord, averageLength } = require('./statistics.js'); exports.measure = function (text) { const words = splitWords(text); return { wordCount: words.length, averageLength: averageLength(words), longest: longestWord(words), }; }; ``` ```javascript // file: source/entry.js const { measure } = require('./index.js'); console.log(measure('A module carries its own scope. A script runs in the global scope.')); ``` A bundler turns these four files into a single output: ```javascript // file: bundle.mjs import { readFile, writeFile } from 'node:fs/promises'; import { dirname, join } from 'node:path'; const REQUIRE_PATTERN = /require\(['"]([^'"]+)['"]\)/g; async function extractGraph(entry) { const modules = new Map(); const queue = [entry]; while (queue.length > 0) { const path = queue.shift(); if (modules.has(path)) continue; const source = await readFile(path, 'utf8'); const map = {}; for (const [, request] of source.matchAll(REQUIRE_PATTERN)) { const resolved = join(dirname(path), request); map[request] = resolved; queue.push(resolved); } modules.set(path, { source, map }); } return modules; } function generate(modules, entry) { const registry = [...modules] .map(([path, m]) => `${JSON.stringify(path)}: [function (require, module, exports) {\n${m.source}}, ${JSON.stringify(m.map)}]`) .join(',\n'); return `(function (registry, entry) { const cache = {}; function run(path) { if (cache[path]) return cache[path].exports; const [factory, map] = registry[path]; const mod = { exports: {} }; cache[path] = mod; factory((request) => run(map[request]), mod, mod.exports); return mod.exports; } run(entry); })({\n${registry}\n}, ${JSON.stringify(entry)}); `; } const entry = 'source/entry.js'; const modules = await extractGraph(entry); const output = generate(modules, entry); await writeFile('bundle.js', output); console.log('module count:', modules.size); console.log('graph:', [...modules.keys()].join(', ')); console.log('output size:', output.length, 'bytes'); ``` ``` $ node bundle.mjs module count: 4 graph: source/entry.js, source/index.js, source/splitting.js, source/statistics.js output size: 1716 bytes ``` ``` $ node bundle.js { wordCount: 13, averageLength: 4, longest: 'carries' } ``` Four files became one, and the output produced the same result the source did. #### The Shape of the Output The start of the generated file shows the entire runtime: ```javascript (function (registry, entry) { const cache = {}; function run(path) { if (cache[path]) return cache[path].exports; const [factory, map] = registry[path]; const mod = { exports: {} }; cache[path] = mod; factory((request) => run(map[request]), mod, mod.exports); return mod.exports; } run(entry); })({ "source/entry.js": [function (require, module, exports) { // file: source/entry.js ``` The shape is a direct match for what the module systems topic described. Every module is wrapped in a function — the written-out form of the module wrapper. The `cache` object provides the single-evaluation guarantee. The `map` object turns the specifiers found in the source into keys in the output; resolution was done at build time, not left to run time. The `registry` object is the graph itself, written out as data: one entry per module, each holding the wrapped factory and that module's own specifier map. Nothing in `run` needs to know how many modules there are or how they are connected — it only follows whatever map the current module carries. Its limits are just as visible, and they mark the difference from real tools. Finding specifiers with a regular expression is not enough: a `require`-looking string that happens to appear inside a string literal matches too. Real tools parse the source and work over an abstract syntax tree — the structure introduced in the compilation stages lesson of the How Computers Work course. There is also no extension completion, directory index resolution, or bare-name lookup here; all of that is a real resolver's job. #### Scope Hoisting Wrapping every module in its own function carries a cost: a function object, a closure, and a call for every module. As module count grows, this cost becomes measurable. **Scope hoisting** is the transform that merges interlinked modules into a single scope instead of wrapping each in a separate function. Names are renamed to avoid collisions; the result is a single body that runs with no intermediate layer. The transform can only be applied if the relationships between modules are known at build time. In the call-based system this information is missing, because a `require` call can be computed at run time; in the standard module syntax it is complete. This is the second concrete payoff of static structure. The renaming step is what makes the merge safe: two modules can each declare a local variable named `separator` without conflict as long as they stay in separate function scopes, so merging them into one scope has to rewrite every name that would otherwise collide. #### The Limits of Tree Shaking Dropping unused code from the output is not done at all in the example bundler. Its effect is measurable: adding a function to the statistics module that nothing calls ```javascript // added to the end of source/statistics.js // An export that nothing calls, but that stays in the module. exports.letterDistribution = function (words) { const counts = new Map(); for (const word of words) { for (const letter of word.toLowerCase()) { counts.set(letter, (counts.get(letter) ?? 0) + 1); } } return [...counts.entries()].sort((a, b) => b[1] - a[1]); }; ``` grows the output while behavior stays the same: ``` $ node bundle.mjs module count: 4 graph: source/entry.js, source/index.js, source/splitting.js, source/statistics.js output size: 2063 bytes ``` ``` $ node bundle.js { wordCount: 13, averageLength: 4, longest: 'carries' } ``` Three hundred forty-seven bytes were paid for code that never runs. Real tools can drop this function. The precondition is the same as for scope hoisting: relationships between modules must be known at build time, that is, standard module syntax. Once that holds, three further conditions apply. **Usage must be visible.** It must be possible to statically trace which name an export is used under and from where. If access is computed, it cannot be traced. **There must be no side effect.** If a module's top level does something independent of its exports — writing to a registry, mutating a built-in object — the whole module cannot be dropped. A tool is cautious with modules that have not declared themselves free of side effects, which is why packages mark side-effect information in their manifest. **Access must be direct.** If the entire namespace object is bound to a value and iterated over, which names are used cannot be known. These conditions place a concrete responsibility on the library author: do no work at the top level, export names directly, and mark side-effect status correctly in the manifest. #### Externalized Dependencies Not every dependency needs to join the output. Modules the runtime provides on its own, or libraries already present in the target environment, are marked as **external**: the bundler leaves them out of the graph and leaves the call as it is. A framework a page already loads from a shared location, or a runtime module like `node:fs`, are typical candidates — bundling them again would only duplicate bytes the environment already has. The distinction also holds between a library and an application. An application's output must carry everything it needs to run. A library's output should not embed its dependencies; if it does, the consumer carries two copies of the same package, and the duplicate-version problems described in the Dependency Trees and Conflicts lesson show up. #### Summary - A bundler takes a module graph as input and produces output that runs in the target environment; its rationale is per-file network cost, resolution difference, and format difference. - The core is three steps: extract the graph, wrap each module in a function, write a small runtime that links the modules. - The output's cache carries the single-evaluation guarantee; its specifier map carries resolution done at build time. - Scope hoisting removes the per-module wrapper cost and can only be applied when relationships are known at build time. - Tree shaking requires static module syntax first, then three conditions: traceable usage, freedom from side effects, and direct access. Side-effect information is marked in the manifest. #### Next Step The bundler merged modules, but it assumed the syntax the source was written in would be recognized in the target environment. That assumption does not always hold: the target may not be able to parse the syntax the source uses, or it may not provide the capability it calls for. The next lesson covers adapting the source to the target, and where that adaptation stops. ### Transpilation and Targeting Source: https://academia.sh/en/courses/javascript-ecosystem/transpilation-and-targeting The split between syntax downleveling and capability polyfilling, determining a target set, what transpilation cannot solve, and cases where the source language differs from the target language. The previous lesson merged modules into a single output and assumed the syntax the source was written in would be recognized in the target environment. That assumption does not always hold. The transform that turns the source into a form the target understands is called **transpilation**. Transpilation looks like one job; it actually solves two separate problems, and solving one does not solve the other. #### Two Separate Problems The **syntax problem** sits at the parsing level. If the target does not recognize the notation the source uses, it cannot even read the file; the error appears before a single line runs. The fix is expressing the same meaning with constructs the target does recognize. The **capability problem** sits at the run-time level. The target can parse the file, but a function it calls or an object it accesses may not exist there. The error appears once execution reaches that line. The fix is defining the missing capability at run time. The practical consequence of this split is that the first problem is solved at build time, the second at run time. One tool can do both, but what each produces is different — one rewrites the source, the other adds code to the output. #### Syntax Downleveling **Downleveling** is re-expressing a construct with the same meaning in an older notation. A class from the measurement library uses three separate features: a private field, an accessor, and an arrow function defined in field position. ```javascript // file: source.mjs export class Measurement { #words; constructor(words) { this.#words = words; } get count() { return this.#words.length; } longest = () => this.#words.reduce((longest, s) => (s.length > longest.length ? s : longest), ''); } const measurement = new Measurement(['one', 'second', 'third-word']); console.log(measurement.count, measurement.longest()); ``` The same meaning, in a form that uses none of these three features: ```javascript // file: target.js 'use strict'; var _words = new WeakMap(); function Measurement(words) { var self = this; _words.set(this, words); this.longest = function () { return _words.get(self).reduce(function (longest, s) { return s.length > longest.length ? s : longest; }, ''); }; } Object.defineProperty(Measurement.prototype, 'count', { get: function () { return _words.get(this).length; }, }); var measurement = new Measurement(['one', 'second', 'third-word']); console.log(measurement.count, measurement.longest()); ``` ``` $ node source.mjs 3 third-word $ node target.js 3 third-word $ wc -c source.mjs target.js 377 source.mjs 543 target.js 920 total ``` The behavior is the same, the size is nearly half again as large. Downleveling is not free, and its cost has three items. **Size.** Every downleveled construct is written out as a longer equivalent. **Performance.** Faking a private field with a weak map is more expensive than direct field access. The downleveled equivalents of some class-syntax constructs fall outside the paths the engine optimizes. **Semantic approximation.** The imitation is not exact. In the equivalent above, the `#words` field is not actually inaccessible; code that reaches the map named `_words` can read it. The private-field guarantee introduced in the Objects and Functions in JavaScript course does not survive downleveling. #### Capability Polyfilling The fix for the second problem is defining the missing capability at run time; this is called a **polyfill**. The correct pattern is testing for the capability first and defining it only when it is absent. Actually testing the pattern needs an environment where the capability is missing. The runtime's isolated-context facility is useful for producing exactly such an environment, by removing a capability from it: ```javascript // file: polyfill.mjs import vm from 'node:vm'; const oldEnvironment = vm.createContext({ console }); // We simulate an old runtime by removing the capability from the context. vm.runInContext('delete Array.prototype.at;', oldEnvironment); vm.runInContext('console.log("before polyfill:", typeof [].at);', oldEnvironment); const polyfill = ` if (typeof Array.prototype.at !== 'function') { Object.defineProperty(Array.prototype, 'at', { value: function (index) { const n = Math.trunc(index) || 0; return this[n < 0 ? this.length + n : n]; }, writable: true, configurable: true, enumerable: false, }); } `; vm.runInContext(polyfill, oldEnvironment); vm.runInContext('console.log("after polyfill:", typeof [].at, [10, 20, 30].at(-1));', oldEnvironment); // The main context was not affected. console.log('main context:', typeof [].at, [10, 20, 30].at(-1)); ``` ``` $ node polyfill.mjs before polyfill: undefined after polyfill: function 30 main context: function 30 ``` Three details are part of the pattern. The test is done with **the capability itself**, not a version query: what is asked is whether the function being looked for exists, not what the environment's identity is. The definition uses a property descriptor instead of a plain assignment, so enumerability stays off and a built-in method's behavior is imitated. Finally, because a polyfill modifies a built-in object, it affects the whole program — which is why polyfills are loaded once, at the entry point, not scattered through library modules. A library should not modify the built-ins of the environment that consumes it. The correct behavior for a library is meeting the missing capability with a helper function of its own; the decision to modify a built-in belongs to the application. #### The Target Set What decides which constructs get downleveled and which polyfills get added is the **target set**: the list of environments the output is expected to run in. As the set widens, more downleveling happens, the output grows, and performance drops. Two mistakes are common when determining the set. The first is keeping the set wider than it needs to be: the size and performance paid for environments no one actually uses is a cost borne by real users. The second is not writing the set down at all — a default list may not match the project's actual users. The right approach is keeping the set written down and justified. The set is a contract that states which environments the output will run in; the output is not expected to work outside that contract. Library packages should document this information, because a consumer's target set has to cover the library's own set. #### What Transpilation Cannot Do Transpilation's limits are targeting's limits. **Engine primitives cannot be imitated.** A new primitive type, a structure tied to the garbage collector, or a capability that needs a particular memory layout cannot be written in library code. A polyfill for such a capability is either approximate or does not exist at all. **Semantic differences remain.** As seen in the private-field example, the downleveled equivalent may not give the same observable behavior. The differences usually show up in edge cases and may not be caught by tests. **Performance does not come back.** Downleveled code cannot draw on the target's native support. These three limits show that targeting is a trade-off: a wider set means more users, and a larger, slower output. The decision should be made by measurement. #### When the Source Language Differs Transpilation's scope is not limited to version differences. The source can be a different language from the target altogether; the tool reads that language and produces output the target can run. In that case the transform does two jobs: meeting the constructs specific to the source language, and stripping the extra information the source language carries out of the output. The most common example is type information: types written in the source are checked at build time and never written to the output. This is the cheapest form of the transform — the output is just the source with its type annotations stripped. This is where the connection lands in the course's last lesson. The divergence between source and output is the real problem this lesson leaves open. When an error produces a stack trace, the line numbers in that trace belong to the output, not the source. The more extensive the transpilation, the farther apart these two files are. #### Summary - Transpilation solves two separate problems: syntax the target does not recognize and a capability absent from the target; the first is solved at build time, the second at run time. - Syntax downleveling grows the output, lowers performance, and turns some semantic guarantees into approximations. - A polyfill is conditioned on the capability's presence, not a version query; because it modifies built-ins, it belongs at the application's entry point, not in libraries. - The target set is a written contract stating which environments the output will run in; keeping it wider than needed costs real users. - Engine primitives cannot be imitated, semantic differences remain, and downleveled code cannot reach native support's performance. #### Next Step The output is now a file different from the source: its lines have shifted, its constructs have changed, its names may have been rewritten. When an error occurs, the line number the developer sees belongs to the generated file. The next lesson covers the mapping file that carries a generated position back to its source position, and how that file gets produced. ### Source Maps Source: https://academia.sh/en/courses/javascript-ecosystem/source-maps Mapping from a generated position to a source position, the map file's fields, how the mapping string is encoded, hand-building a map, and the decision to publish it in production. The previous lesson stopped at the point where transpilation separates source from output. The output's lines may have shifted, its constructs may have changed. When an error occurs, the line number in the stack trace belongs to the generated file and does not match the source the developer has open. A **source map** is a data file that carries the position mapping between these two files. It is not code and is not executed; it holds only the information that "this line and column of the generated file correspond to this line and column of the source file." #### What the Mapping Covers The mapping works at the level of **position**, not file. Given a generated position, the corresponding source file, line, and column are found. This level of detail is necessary: a single generated line can come from more than one source line. Three operations use the mapping: resolving stack traces back to the source, placing breakpoints in the debugger at the source file, and writing performance-measurement durations back to source functions. All three let the tools introduced in the debugging and profiling lessons of the Asynchronous JavaScript course work against generated code. #### The Fields of the Map File The map is an object in JSON form: | Field | Content | |---|---| | `version` | The format's version number | | `file` | The name of the generated file the map belongs to | | `sources` | The paths of the source files | | `sourcesContent` | The content of the source files; optional | | `names` | The original names of identifiers that were renamed | | `mappings` | The encoded form of the position mappings | The `sourcesContent` field lets the source be embedded inside the map file itself. If it is embedded, a tool does not have to look for the source elsewhere; if it is not, the tool tries to read from the `sources` paths and cannot display the source if it fails to find it. The generated file is linked to its map by a comment on its last line. This comment tells the runtime and debugging tools where to look for the map. #### The Shape of the Mapping String The `mappings` field carries every mapping in a single string. The structure has three levels: - A semicolon separates **generated lines**. Two semicolons in a row mean that line has no mapping at all. - A comma separates the **segments** on the same line. - Each segment that carries a mapping consists of four or five numbers: generated column, source file index, source line, source column, and a name index if one applies. The numbers are not written directly. Two compressions apply: each number is kept as a **delta** relative to the previous one, and the deltas are written with **variable-length quantity** encoding. The encoding splits the number into five-bit groups, puts a sign bit in the lowest bit, and marks continuing groups with the sixth bit; each group is written as one letter from a 64-character alphabet. The result is a string that is small but not human-readable. A thousand-line file's mapping, written out without compression, would run to hundreds of thousands of characters; this encoding brings it down to a tenth of that. #### Producing the Map The shortest way to understand the encoding is to write a working generator. The transform below is deliberately plain: it adds a three-line banner to the top of the source. By the end of the transform, every source line has shifted down by three lines, and that shift cannot be tracked without a map. ```javascript // file: source/measure.mjs export function measure(text) { if (typeof text !== 'string') { throw new TypeError('expected a string'); } return text.split(/\s+/).length; } console.log(measure('one two three')); console.log(measure(42)); ``` ```javascript // file: generate.mjs import { mkdir, readFile, writeFile } from 'node:fs/promises'; const ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; // Variable-length quantity encoding: sign bit in the lowest bit, 5-bit groups. function vlq(number) { let value = number < 0 ? (-number << 1) | 1 : number << 1; let output = ''; do { let chunk = value & 0b11111; value >>>= 5; if (value > 0) chunk |= 0b100000; output += ALPHABET[chunk]; } while (value > 0); return output; } await mkdir('output', { recursive: true }); const sourceName = 'source/measure.mjs'; const source = await readFile(sourceName, 'utf8'); const lines = source.split('\n'); const banner = ['/* generated file */', '/* do not edit by hand */', '']; const output = [...banner, ...lines]; // Banner lines have no mapping; every generated line looks at its source counterpart. const segments = banner.map(() => ''); let previousSourceLine = 0; for (let i = 0; i < lines.length; i += 1) { const indent = lines[i].length - lines[i].trimStart().length; segments.push([vlq(indent), vlq(0), vlq(i - previousSourceLine), vlq(indent)].join('')); previousSourceLine = i; } const map = { version: 3, file: 'measure.mjs', sources: ['../source/measure.mjs'], sourcesContent: [source], names: [], mappings: segments.join(';'), }; await writeFile('output/measure.mjs', output.join('\n') + '\n//# sourceMappingURL=measure.mjs.map\n'); await writeFile('output/measure.mjs.map', JSON.stringify(map)); console.log('mappings:', map.mappings.slice(0, 60)); ``` ``` $ node generate.mjs mappings: ;;;AAAA;AACA;EACE;IACI;EACE;EACE;AACA;AACA;AACA;AACA;AACA ``` The three semicolons at the start of the string say the banner lines have no mapping. The fourth line's segment is `AAAA` — four zeros: generated column 0, source file 0, source line delta 0, source column 0. The fifth line's segment reads `AACA`: the third field is no longer zero, a delta showing the source line advanced by one. On indented lines the first and last fields also move off zero; the segment `EACE` means generated column 2 and source column 2. The generated file's first and last lines show how the link is established: ``` $ head -5 output/measure.mjs /* generated file */ /* do not edit by hand */ // file: source/measure.mjs export function measure(text) { $ tail -2 output/measure.mjs //# sourceMappingURL=measure.mjs.map ``` #### Tracing Back When the generated file is run, the error reports the generated position: ``` $ node output/measure.mjs 2>&1 | grep 'at measure' | sed "s|file://||; s|$(pwd -P)/||" at measure (output/measure.mjs:7:11) ``` The same file, run with the runtime's source-map support turned on: ``` $ node --enable-source-maps output/measure.mjs 2>&1 | grep 'at measure' | sed "s|$(pwd -P)/||" at measure (source/measure.mjs:4:7) ``` The stack trace now shows the source file and the source line. The line number is exact; the column number is approximate, because this generator wrote one segment per line. Real tools write a segment for every token, and the column becomes exact too. The `sed` calls only shorten the absolute path in the output. #### Chaining and Integrity A source can go through more than one transform: transpiled first, then bundled, then minified. Each step produces its own map, and the last one looks back at the one before it. The tools have to merge this chain into a single map; a step that does not merge makes the whole chain unusable. This explains the following observation: a stack trace sometimes lands not on the source but on an intermediate output. The reason is usually that the map was not carried through one link of the chain. #### The Map in Production If the map is published, the entire source is effectively published along with it — especially if the `sourcesContent` field is filled in. The decision is between three options. **Do not publish.** The map is produced but not published alongside the output. The source stays protected; in exchange, stack traces coming from production have to be resolved by hand. **Publish.** The map sits next to the output. Debugging is at its easiest; the source code is open to everyone. **Publish somewhere separate.** The map is uploaded to an error-tracking system; it is not present next to the output. Stack traces are resolved inside that system, the source is not exposed. This needs extra infrastructure. The choice depends on how private the source is. If the source is already open, there is no reason not to publish the map. If it is closed, the third option strikes the balance between the other two. Whichever option is used, producing the map is not optional: a map that was never produced cannot be obtained afterward. #### Summary - A source map is a data file that maps positions in the generated file to positions in the source file; it contains no code and is not executed. - Mappings are held in a single string, separated into lines and segments, with each number kept as a delta and compressed with variable-length quantity encoding. - The generated file is linked to its map by a comment on its last line; the source text can be embedded in the map. - Once the runtime's source-map support is turned on, stack traces show source positions; column precision depends on how often segments are written. - In a multi-step transform, each step has to merge its map with the one before it; a step that does not merge breaks the chain. - Publishing the map means publishing the source too; uploading to a separate error-tracking system balances that trade-off. #### Next Step The tools covered so far transformed code. Another class of tool does not transform code at all — it only reads it and judges it against a rule set: which constructs are a sign of a defect, which formatting is inconsistent. The next lesson covers this check performed without running the code, and why formatting is kept separate from it. ### Static Analysis and Formatting Source: https://academia.sh/en/courses/javascript-ecosystem/static-analysis-and-formatting The limit of parse-level checking, the rule set and severity levels, the discipline of suppressing a warning, keeping formatting separate from analysis, and resilience under repetition. The tools covered so far transformed code: they merged modules, downleveled syntax, produced position mappings. This lesson covers a class of tool that does not transform code — **static analysis** tools that read the source and judge it against a rule set. The Shell Programming course introduced the same class of tool for shell scripts; the criteria and the discipline are the same. Here the language's own features and package-level layout are added on top. The common thread across both is that the tool never executes the program it examines — its judgment comes entirely from reading structure, which is also the source of its blind spots. #### Checking at the Parsing Level The most basic check is whether the source can be parsed at all. Runtimes offer an option that does this without running the code: ```javascript // file: broken.js function measure(text) { return text.split(/\s+/.length; } ``` ``` $ node --check broken.js 2>&1 | sed "s|$(pwd -P)/||" | head -6 broken.js:3 return text.split(/\s+/.length; ^^^^^^ SyntaxError: missing ) after argument list at wrapSafe (node:internal/modules/cjs/loader:1804:18) ``` The exit code is nonzero; this means the check can be placed in a script. The internal frames on the message's last line vary with the runtime's version; the first four lines are stable. Running this check costs almost nothing next to actually executing the file — no module is loaded, no side effect runs — which is why it belongs early in a pipeline, before slower steps that assume the input already parses. This check's limit is parsing itself. Every source that can be parsed passes — whatever it means: ```javascript // file: suspicious.js function exceedsThreshold(number, threshold) { if (number = threshold) { return true; } return false; } console.log(exceedsThreshold(3, 5)); ``` ``` $ node --check suspicious.js && echo "syntax check: passed" syntax check: passed $ node suspicious.js true ``` The condition performed an assignment, not a comparison. The program parsed, ran, and gave the wrong answer: 3 does not exceed 5. The language considers this notation valid because assignment is an expression and produces a value. Catching it needs a judgment beyond parsing — this is where a rule set comes in. #### The Rule Set and Severity Levels An analyzer parses the source and runs rules over the resulting structure. Each rule looks for a pattern, produces a finding when it matches, and the finding carries a **severity level**. The severity is what turns a finding into a decision: two rules can report the exact same pattern and still lead to different outcomes, one blocking the build and the other only noted for later. A small analyzer is enough to see how rules work. The implementation below is line-based; real tools work over an abstract syntax tree, but the rule, severity, and suppression structure is the same. ```javascript // file: analyze.mjs import { readFile } from 'node:fs/promises'; const RULES = [ { name: 'loose-equality', severity: 'error', test: (s) => /[^=!<>]==[^=]|[^!]!=[^=]/.test(s), message: 'use strict comparison instead of loose comparison' }, { name: 'empty-catch', severity: 'error', test: (s) => /catch\s*(\([^)]*\))?\s*\{\s*\}/.test(s), message: 'an empty catch block hides the error' }, { name: 'console', severity: 'warning', test: (s) => /console\.(log|debug)\(/.test(s), message: 'direct console output in library code' }, ]; const IGNORE = /\/\/\s*analyze-ignore:\s*([\w-]+)/; export async function analyze(file) { const lines = (await readFile(file, 'utf8')).split('\n'); const findings = []; lines.forEach((line, i) => { const ignored = line.match(IGNORE)?.[1]; for (const rule of RULES) { if (rule.name === ignored) continue; if (rule.test(line)) { findings.push({ file, line: i + 1, ...rule }); } } }); return findings; } const findings = await analyze(process.argv[2]); for (const f of findings) { console.log(`${f.file}:${f.line} ${f.severity.padEnd(6)} ${f.name.padEnd(16)} ${f.message}`); } const hasError = findings.some((f) => f.severity === 'error'); console.log(`${findings.length} findings`); process.exit(hasError ? 1 : 0); ``` Suppose the measurement library's statistics module is written so that it triggers all three rules: ```javascript // file: src/statistics.mjs export function averageLength(words) { if (words.length == 0) return 0; return words.reduce((t, s) => t + s.length, 0) / words.length; } export function longestWord(words) { console.log('longestWord called'); // analyze-ignore: console try { return words.reduce((longest, s) => (s.length > longest.length ? s : longest), ''); } catch {} } ``` ``` $ node analyze.mjs src/statistics.mjs; echo "exit code: $?" src/statistics.mjs:3 error loose-equality use strict comparison instead of loose comparison src/statistics.mjs:11 error empty-catch an empty catch block hides the error 2 findings exit code: 1 ``` The console write on the eighth line produced no finding: the end-of-line comment suppressed that rule. The exit code follows the same exit-code contract as the Shell Programming course: a nonzero code lets the calling script stop the process. #### Classifying Rules Not every rule carries the same weight. Three classes are distinguished. **Correctness rules.** The code is very likely misbehaving: an unused assignment, an assignment in a condition, unreachable code, a dropped promise. These are worth marking at error level. **Suspicion rules.** The code may not be wrong, but the pattern is a sign of a defect: an empty catch, a coercing comparison, an overly broad variable. Warning level is appropriate; deliberate uses need suppression. **Formatting rules.** These have no effect on behavior: indentation, quote style, semicolons. These are not an analyzer's job; they are the next section's subject. The practical importance of this split is that error level carries a cost: every rule at error level has the authority to stop the workflow. When that authority is given to formatting preferences, the team starts seeing the analyzer as an obstacle, and suppression comments multiply. The three classes also age differently: a correctness rule earns its place once and rarely needs revisiting, while a suspicion rule's value depends on how often its warning turns out to matter in this particular codebase, and is worth reviewing periodically. #### The Discipline of Suppression A suppression comment is a necessary mechanism: no rule set can anticipate every legitimate use. Misusing it is just as easy. Three rules keep it working. Suppression must be **narrow**: one line, not the whole file; one rule, not all rules. Suppression must be **justified**: the comment next to it must say why it is suppressed. Suppression must be **countable**: if the number of suppressions is measured, a rise is a signal. If a rule keeps getting suppressed, the problem is the rule itself. Either it does not fit the project and should be turned off, or it is defined wrong and should be narrowed. #### Formatting Is a Separate Job A **formatter** is a tool that puts a source's notation into a single arrangement without changing its meaning. It differs from an analyzer in two ways: it does not report findings, it writes directly; and it does not choose what is correct, it chooses what is **the one** form. The property a correct formatter has to carry is **idempotence**: applying it again to output it has already produced must change nothing. Without this property, the tool produces a change on every run and drowns version control in noise. ```javascript // file: format.mjs import { readFile, writeFile } from 'node:fs/promises'; export function format(source) { const lines = source .replace(/\t/g, ' ') .split('\n') .map((s) => s.replace(/\s+$/, '')); while (lines.length > 0 && lines.at(-1) === '') lines.pop(); return lines.join('\n') + '\n'; } const file = process.argv[2]; const before = await readFile(file, 'utf8'); const once = format(before); const twice = format(once); await writeFile(file, once); console.log('changed:', before !== once); console.log('idempotent:', once === twice); ``` Suppose the input is a file containing a tab character and trailing whitespace. The `cat -et` call shows tabs as `^I` and line ends as `$`: ``` $ printf 'export function measure(text) {\n\treturn text.split(/\\s+/).length; \n}\n' > unformatted.mjs $ cat -et unformatted.mjs export function measure(text) {$ ^Ireturn text.split(/\s+/).length; $ }$ $ node format.mjs unformatted.mjs changed: true idempotent: true $ cat -et unformatted.mjs export function measure(text) {$ return text.split(/\s+/).length;$ }$ $ node format.mjs unformatted.mjs changed: false idempotent: true ``` The second run changed nothing. The example formatter only performs two transforms; real tools parse the source and rewrite it from the tree, also deciding line length, brace placement, and alignment. The result of the rewrite approach is that the output is **entirely independent** of the input's original notation. The rationale for the split is a cost calculation. Formatting debates consume a measurable share of review time and settle nothing technical. Handing the decision to a tool zeroes out that time. In exchange, the team gives up individual formatting preferences. #### Automation and Its Limits Both tools are only useful if they actually run. The usual placement is: format on save in the editor, check before commit, a final check in the build pipeline. Without that last link the others stay optional; if only the last link exists, feedback arrives late. The earlier a check runs, the cheaper the fix is — a warning shown while a line is still being typed costs a keystroke; the same warning caught only in the build pipeline costs a new commit, a second review pass, and a wait for the pipeline to run again. Static analysis's limit is a matter of principle: a rule set knows how the code is written, not what it does. False findings are unavoidable, and so are missed defects. An analyzer does not replace tests; the two catch different classes of defect. A clean report from an analyzer does not say the code is correct — it says it avoids known defect patterns. #### Summary - A syntax check tests parseability and gives a result without running the code; it does not catch semantic defects such as an assignment in a condition. - A static analyzer runs a rule set over parsed source; every finding carries a severity level, and the exit code can stop the workflow. - Rules split into correctness, suspicion, and formatting; error level is given only to correctness rules. - Suppression must be narrow, justified, and countable; a rule that is suppressed constantly is defined wrong. - A formatter applies a single notation without changing meaning and must be idempotent; handing the decision to a tool frees review time from formatting debates. - Static analysis does not replace tests; a clean report reports the absence of known defect patterns, not correctness. #### Next Step Every tool set up in this course is a program invoked from the command line: the bundler, the analyzer, the formatter, the test runner. Every developer having to remember to invoke them with the same options should not be a memory exercise. The course's last lesson covers how these invocations are named and stored in a project, and how a reproducible set of commands is set up. ### Task Automation Source: https://academia.sh/en/courses/javascript-ecosystem/task-automation Storing script definitions in the manifest file, what a script runner does, lifecycle hooks, the exit-code contract, and a reproducible command set. Every tool set up over the course is a program invoked from the command line: the bundler, the analyzer, the formatter, the test runner. Each has its own options, its own input paths. Keeping these commands in the developer's memory causes two problems. Commands differ from one person to the next, and the command that runs on the build server ends up different from the one the developer ran. This lesson covers writing the commands into the project itself and binding their execution to a single contract. #### Script Definitions The manifest file's scripts field stores the project's commands under names: ```json { "name": "text-measurer", "version": "0.1.0", "type": "module", "exports": { ".": "./src/index.mjs" }, "scripts": { "pretest": "node --check src/index.mjs", "test": "node --test", "measure": "node tools/measure.mjs", "broken": "node --check src/broken.mjs" } } ``` Keeping definitions in the manifest has three consequences. Commands enter version control and their changes can be reviewed. A developer cloning the repository learns which commands exist by reading it. The same names are invoked both locally and in the build pipeline; the difference between the two environments disappears. The naming convention matters too. A name should describe **the work being done**, not the tool. The test command carries the name `test`; which test runner is used sits on the definition's right side. When the tool changes, the callers do not — command names are the project's interface, tool choice is the implementation. #### What a Script Runner Does The tool that runs scripts is called a **script runner**, and its job fits into four items. A small implementation of one makes all four visible: ```javascript // file: run.mjs import { spawnSync } from 'node:child_process'; import { readFileSync } from 'node:fs'; import { delimiter, resolve } from 'node:path'; const manifest = JSON.parse(readFileSync('package.json', 'utf8')); const scripts = manifest.scripts ?? {}; const [requested, ...extraArgs] = process.argv.slice(2); function runScript(name, args = []) { const command = [scripts[name], ...args].join(' '); console.log(`> ${name}: ${command}`); const result = spawnSync(command, { shell: true, stdio: 'inherit', env: { ...process.env, PATH: resolve('node_modules/.bin') + delimiter + process.env.PATH }, }); return result.status ?? 1; } if (scripts[requested] === undefined) { console.error(`no such script: ${requested}`); process.exit(1); } for (const name of [`pre${requested}`, requested, `post${requested}`]) { if (scripts[name] === undefined) continue; const code = runScript(name, name === requested ? extraArgs : []); if (code !== 0) { console.error(`stopped: script ${name} exited with code ${code}`); process.exit(code); } } ``` **It finds the command.** The name is read from the table in the manifest; if it is missing, an error is raised. **It prepares the environment.** Installed packages' executables sit in the `node_modules/.bin` directory, and that directory is added to the front of the search path. The result is that calling a tool by its command name does not require a system-wide install; the project's own copy is found. The lookup order introduced in the command path lesson of the Introduction to Linux course becomes a deliberate tool here. **It calls the hooks.** Before the requested script, a same-named script with a `pre` prefix runs if it exists; after it, one with a `post` prefix runs if it exists. **It carries the exit code.** The subprocess's code becomes the runner's own code. When the test script runs, two of the four show up in a single output: ``` $ node ../run.mjs test 2>&1 | grep -vE 'duration_ms|ms\)|^$' > pretest: node --check src/index.mjs > test: node --test ℹ tests 2 ℹ suites 0 ℹ pass 2 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0 ``` The hook ran on its own, then the actual script ran. The filter drops the duration measurements from the output; durations change on every run. The test file uses the runtime's built-in test interface: ```javascript // file: test/measurement.test.mjs import assert from 'node:assert/strict'; import test from 'node:test'; import { measure } from '../src/index.mjs'; test('word and sentence count', () => { const measurement = measure('One sentence. Two sentence.'); assert.equal(measurement.wordCount, 4); assert.equal(measurement.sentenceCount, 2); }); test('empty text returns zero', () => { assert.equal(measure('').wordCount, 0); }); ``` #### Passing Arguments Some scripts take variable input. The runner appends the arguments given after the script name to the end of the command: ```javascript // file: tools/measure.mjs import { measure } from '../src/index.mjs'; const text = process.argv.slice(2).join(' '); if (text === '') { console.error('usage: node tools/measure.mjs '); process.exit(2); } const measurement = measure(text); console.log(`${measurement.wordCount} words, ${measurement.sentenceCount} sentences, longest: ${measurement.longest}`); ``` ``` $ node ../run.mjs measure "A module carries its own scope. A script runs in the global scope." > measure: node tools/measure.mjs A module carries its own scope. A script runs in the global scope. 13 words, 2 sentences, longest: carries ``` Notice that the arguments get reinterpreted by the shell: because the command runs inside a shell, spaces and special characters are subject to shell rules. The quoting rules from the Shell Programming course apply here; arguments carrying variable content must be quoted. #### The Failure Contract All of automation rests on a single contract: when a command fails, it returns a nonzero exit code. When the contract holds, the chain stops on its own: ```javascript // file: src/broken.mjs export function missing(text) { return text.split(/\s+/.length; } ``` ``` $ node ../run.mjs broken 2>&1 | grep -E '^>|^stopped' > broken: node --check src/broken.mjs stopped: script broken exited with code 1 $ node ../run.mjs broken > /dev/null 2>&1; echo "exit code: $?" exit code: 1 ``` The syntax check failed, the runner stopped, and set its own exit code to 1. This code lets a layer above it — the build pipeline — that calls it also stop. Two common mistakes break the contract. The first is a script that returns zero on failure; if a pipeline's last command succeeds, the shell returns zero and the real failure is hidden. The second is a script that only prints an error to the screen without setting the exit code; automation does not read the screen, only the code. #### Reproducibility The same command giving the same result depends on four conditions. **Dependencies are installed from the lock.** If a lock file exists, installation is reproduction, not a search. The build pipeline should use the install form that conforms to the lock rather than updating it; if the lock conflicts with the manifest, the process should stop. **Tools are installed in the project.** The analyzer, the formatter, and the build tools are declared as development dependencies. Relying on a system-wide tool leaves its version unchecked. **Commands are defined in one place.** The build pipeline's configuration does not rewrite commands; it calls script names. This way, when a command's content changes, there are not two places to update. **Outputs are determined by inputs.** If a build's output depends on something other than the source and the configuration — time, machine name, network access — it cannot be reproduced. Once these four conditions hold, the build pipeline turns into nothing more than an environment that runs the same commands the developer runs locally. If an error shows up in the build pipeline but not locally, one of the conditions is not holding; the condition that broke should be looked for before the error itself. #### Summary - Script definitions are kept in the manifest; they enter version control, serve as documentation, and remove the command difference between local and the build pipeline. - Script names describe the work done, not the tool; when the tool changes, the callers do not. - The runner finds the command, adds the installed tools' directory to the front of the search path, calls the `pre` and `post` hooks, and carries the exit code. - Automation rests on the exit-code contract: if a failure is not reported with a nonzero code, the chain keeps going incorrectly. - Reproducibility depends on four conditions: installation from the lock, tools installed in the project, commands defined in one place, and outputs determined by inputs. #### Course Wrap-Up The course began with a single-file measurement script and moved forward by turning that script into a package. Along the way, four questions were answered. **How is code split?** Module scope blocks names from leaking out at the parsing level. The export surface is a design decision: static syntax settles binding before evaluation, the call-based system trades that guarantee away for flexibility, dynamic import opens a path between the two. **How are dependencies managed?** A bare name is resolved by a deterministic rule walking the directory tree. Version ranges are a convenience, a lock file is a guarantee. Duplicate copies in the tree produce silent failures; the transitive closure is the measure of the code mass being trusted. **How does the source reach the target?** The bundler reduces the module graph to a single output, transpilation adapts syntax and capabilities to the target, a source map carries a generated position back to its source. Every step has a measurable cost. **How is quality kept?** Static analysis catches defect patterns without running the code, formatting removes the argument, task automation makes these tools run the same way for everyone. The principle the course carries from end to end is the same in all four: **write the information into the code.** The dependency lives in the manifest, the version contract in the number, the installed tree in the lock, the source mapping in the map, the commands in the script table. Any information left to memory or habit gets lost on someone else's machine. One kind of information still lives outside the code: what type of value a function expects and what it returns. The measurement library's functions expect text, but that expectation is only tested at run time — a function called with a number only reports the error once execution reaches it. The next course covers writing this information into the source and checking it at build time: type inference, narrowing, generics, and the deliberate choice of how strict the compiler should be.