Asset Processing

For three lessons, the graph was built only from script files. The North Slope Measurement Station interface’s output is not limited to that: the measurement list’s style file, the icon in front of every row in the list, the drawing of the station’s elevation curve, the interface’s font, and a handful of fixed-name files search engines read are also published.

These files enter the output through two separate routes. Some are linked from code or a style file; the link is an edge in the graph, and the asset becomes a node. Some are linked from nowhere but must still be published. This lesson separates the two routes and runs the pipeline linked assets pass through.

An Asset Is a Graph Node

For an asset to enter the graph, its link must be statically readable. Two standard syntaxes provide this.

A url() declaration in a style file gives a relative path as its address. The path in the source is resolved against the directory the style file sits in.

A new URL("./x.svg", import.meta.url) expression in code does the same job on the script side. The second argument is the module’s own address; the expression produces an address resolved against the module’s location. This syntax is part of the standard and is not tied to any tool; it works with no tool present, and can also be caught at build time when a tool is present.

A syntax that cannot be caught is one where the address is assembled at runtime. A concatenation in the form "/asset/" + name + ".svg" does not say which file it points to during the build. Such a link does not enter the pipeline: the file is not copied, its name is not rewritten, and a request goes to an address absent from the output. The rule is clear — asset addresses computed at runtime are served from the directly copied directory.

Two Flows

The processed flow covers linked assets. Depending on their size, these assets are either written to the output as a separate file or embedded into the body of the file that links them; either way, the link in the source is translated into an address in the output.

The directly copied flow is for files that must be published with their name and location unchanged: the file carrying crawler rules, domain verification files, data files downloaded from a fixed address. These files are not scanned, their names are preserved, and their content does not change. The distinguishing criterion is a single question: is the build free to determine the address, or is it pinned from outside?

The Asset Tree

The block below produces assets deterministically. A filler file of the same size stands in for a font body; because the pipeline’s decisions depend on a file’s size, not its content, this does not change the result.

The block below adds asset links to the source/ tree set up in the previous lessons; those lessons’ setup blocks must run in this directory first.

#!/usr/bin/env bash
mkdir -p asset public style

cat > asset/icon.svg <<'EOF'
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" fill="none" stroke="#243">
<path d="M2 13 L6 5 L9 10 L11 7 L14 13 Z"/><circle cx="12" cy="3" r="2"/>
</svg>
EOF

node --input-type=module -e '
import { writeFileSync } from "node:fs";
// Station map: a 60-point elevation curve, deterministic generation.
const point = Array.from({ length: 60 }, (_, i) =>
  `${(i * 8).toFixed(0)} ${(120 - 40 * Math.sin(i / 6) - i / 3).toFixed(2)}`);
writeFileSync("asset/station.svg",
  `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 480 160">\n` +
  `<path fill="none" stroke="#1b3a2f" stroke-width="2" d="M${point.join(" L")}"/>\n` +
  point.filter((_, i) => i % 6 === 0).map((n) => `<circle cx="${n.split(" ")[0]}" cy="${n.split(" ")[1]}" r="3"/>`).join("\n") +
  `\n</svg>\n`);
// A deterministic binary file of the same size stands in for a font body.
writeFileSync("asset/measurement.woff2", Buffer.alloc(4096, 0x77));
'

cat > style/station.css <<'EOF'
@font-face {
  font-family: "Measurement";
  src: url("../asset/measurement.woff2") format("woff2");
  font-display: swap;
}
.measurement li::before {
  content: "";
  background-image: url("../asset/icon.svg");
  width: 16px;
  height: 16px;
}
.station-map {
  background-image: url("../asset/station.svg");
  min-height: 160px;
}
EOF

cat > source/asset-links.js <<'EOF'
export const iconUrl = new URL("../asset/icon.svg", import.meta.url);
export const mapUrl = new URL("../asset/station.svg", import.meta.url);
EOF

printf 'User-agent: *\nDisallow: /panel/\n' > public/robots.txt
printf '{ "station": "north-slope", "elevation_m": 1840 }\n' > public/station.json

The Pipeline

The pipeline is four steps: scan for links, decide embedding for each asset, replace the reference with its output address, and write the mapping between source and output into a manifest.

// asset.mjs — asset pipeline: scanning, embed decision, rewriting, manifest generation.
import { readFileSync, writeFileSync, mkdirSync, cpSync, statSync, rmSync } from "node:fs";
import { basename, dirname, join, relative } from "node:path";

const EMBED_THRESHOLD = 1024;                    // bytes
const OUTPUT = "output";
const TYPE = { ".svg": ["image/svg+xml", "text"], ".css": ["text/css", "text"],
              ".woff2": ["font/woff2", "binary"], ".png": ["image/png", "binary"] };
const extension = (p) => p.slice(p.lastIndexOf("."));

const SOURCES = ["style/station.css", "source/asset-links.js"];
const LINK = [
  /url\(\s*["']?([^"')]+)["']?\s*\)/g,                          // link inside a style file
  /new URL\(\s*["']([^"']+)["']\s*,\s*import\.meta\.url\s*\)/g, // link inside code
];

function dataUrl(filePath) {                     // percent-encoding for text assets, base64 for binary
  const [mime, kind] = TYPE[extension(filePath)], body = readFileSync(filePath);
  return kind === "text"
    ? "data:" + mime + "," + encodeURIComponent(body.toString("utf8")).replace(/%20/g, " ")
    : "data:" + mime + ";base64," + body.toString("base64");
}

rmSync(OUTPUT, { recursive: true, force: true });
mkdirSync(join(OUTPUT, "asset"), { recursive: true });

const assets = new Map();                        // path -> { size, refs, embedded, target, outputSize }
const manifest = {};

for (const sourcePath of SOURCES) {
  let text = readFileSync(sourcePath, "utf8");
  for (const pattern of LINK) {
    text = text.replace(pattern, (full, link) => {
      const filePath = relative(".", join(dirname(sourcePath), link));
      if (!assets.has(filePath)) {
        const size = statSync(filePath).size;
        const embedded = size <= EMBED_THRESHOLD;
        const target = embedded ? dataUrl(filePath) : "/asset/" + basename(filePath);
        if (!embedded) cpSync(filePath, join(OUTPUT, target));
        assets.set(filePath, { size, refs: 0, embedded, target,
          outputSize: embedded ? Buffer.byteLength(target) : size });
      }
      const a = assets.get(filePath);
      a.refs++;
      return full.startsWith("url") ? 'url("' + a.target + '")'
        : 'new URL("' + a.target + '", import.meta.url)';
    });
  }
  const outputPath = join(OUTPUT, basename(sourcePath));
  writeFileSync(outputPath, text);
  manifest["/" + basename(sourcePath)] = { source: sourcePath, size: statSync(outputPath).size };
}

for (const [filePath, a] of assets)
  if (!a.embedded) manifest[a.target] = { source: filePath, size: a.size };
cpSync("public", OUTPUT, { recursive: true });   // directly copied directory
writeFileSync(join(OUTPUT, "asset-manifest.json"), JSON.stringify(manifest, null, 2) + "\n");

const base64Length = (filePath) => Math.ceil(statSync(filePath).size / 3) * 4;
console.log("asset".padEnd(20) + "size".padStart(6) + "  ref  decision    output  if embedded");
for (const [filePath, a] of assets)
  console.log(basename(filePath).padEnd(20) + String(a.size).padStart(6) + "   " + a.refs +
    "   " + (a.embedded ? "embedded  " : "separate  ") + String(a.outputSize).padStart(7) +
    String(base64Length(filePath)).padStart(12));

console.log("-- rewritten style file (long values truncated) --");
for (const s of readFileSync(join(OUTPUT, "station.css"), "utf8").trimEnd().split("\n"))
  console.log("  " + (s.length > 64 ? s.slice(0, 64) + "[...]" : s));

console.log("-- manifest keys --");
for (const [url, k] of Object.entries(manifest))
  console.log("  " + url.padEnd(24) + String(k.size).padStart(5) + " B  <- " + k.source);

const separate = [...assets.values()].filter((a) => !a.embedded);
console.log("-- total --");
console.log("  style file      : " + manifest["/station.css"].size + " B  (source was " +
  statSync("style/station.css").size + " B)");
console.log("  separate asset  : " + separate.length + " file, " +
  separate.reduce((t, a) => t + a.size, 0) + " B");
console.log("  direct copy     : robots.txt, station.json (name and content unchanged)");

$ node asset.mjs
asset                 size  ref  decision    output  if embedded
measurement.woff2     4096   1   separate     4096        5464
icon.svg               168   2   embedded      275         224
station.svg           1162   2   separate     1162        1552
-- rewritten style file (long values truncated) --
  @font-face {
    font-family: "Measurement";
    src: url("/asset/measurement.woff2") format("woff2");
    font-display: swap;
  }
  .measurement li::before {
    content: "";
    background-image: url("data:image/svg+xml,%3Csvg xmlns%3D%22ht[...]
    width: 16px;
    height: 16px;
  }
  .station-map {
    background-image: url("/asset/station.svg");
    min-height: 160px;
  }
-- manifest keys --
  /station.css              586 B  <- style/station.css
  /asset-links.js           398 B  <- source/asset-links.js
  /asset/measurement.woff2 4096 B  <- asset/measurement.woff2
  /asset/station.svg       1162 B  <- asset/station.svg
-- total --
  style file      : 586 B  (source was 332 B)
  separate asset  : 2 file, 5258 B
  direct copy     : robots.txt, station.json (name and content unchanged)

In the rewritten style file, all three links have changed: two into a root-relative address, one into the body itself. The ../asset/ relative path in the source never appears in the output — output layout is independent of source layout.

The ref column exposes a detail: the icon and the map are linked from both the style file and the code. The map, written as a separate file, is copied once, and both references point at the same address. The embedded icon, on the other hand, is rewritten at every reference; the 168-byte file totals 550 bytes across two outputs.

The Embed Threshold

The embed decision is made with a threshold, and the threshold balances two opposing costs.

Embedding’s gain is one request. Every asset written as a separate file brings a request’s latency alongside the byte it carries. For small assets, this latency is more expensive than the file itself.

Embedding’s cost has three items. First is encoding expansion: binary data grows by roughly a third when converted to base64 — a 4096-byte font body becomes 5464 characters. Text assets can use percent-encoding, but it is not always cheaper: the example’s icon is 275 bytes with percent-encoding and 224 with base64, because the proportion of characters that need escaping is high. Second is duplication: an embedded asset sits separately in every output that links it. Third is caching — an embedded asset has no separate address, so it cannot be cached separately; it downloads again with the file that links it every time that file changes.

This third item explains why the threshold is kept small. Embedding a rarely changing image inside a frequently changing style file makes that image download again with every release. The next lesson will make this connection measurable.

Decisions Specific to Asset Classes

The threshold alone is not a sufficient policy; each asset class has its own constraints.

Fonts are written as separate files regardless of size. A font body is shared across multiple pages and cached long-lived; embedding it destroys that sharing. The two settings established in the Font Loading lesson of the Visual Presentation with CSS course apply here as well: display behavior is declared with font-display, and the body can be requested with an early loading hint inside the document. A third thing the build side can add is subsetting — keeping only the used character ranges from the body shrinks the file noticeably, and the decision depends on the targeted language set.

Images carry two extra pieces of metadata. First are dimensions: when width and height are read during the build and written into the markup, the content shift defined in the Web Fundamentals and HTML course is prevented. Second are derivatives: the candidate set in the Layout Systems and Responsive Design course’s Responsive Images lesson asks for several widths of the same image; the pipeline is what produces those versions. Format choice is made by class, not by product name — drawings that need lossless compression and photographs suited to lossy compression are handled separately.

Style files are both asset and source: the links inside them are scanned, and they themselves are written to the output. This is why they appear twice in the pipeline.

The Asset Manifest

The pipeline’s final output is a mapping: which source file corresponds to which address in the output. This file is called the asset manifest, and it is the contract between the build and everything outside it.

At least three parties read the manifest. The document template takes the address of the style file and the entry chunk from it. A runtime rendering on the server writes the same addresses into the markup it produces. A service worker running offline generates the list of files it will precache from it — the Caching Strategies lesson in the Browser and the Web Platform course showed how fragile writing that list by hand is; the manifest lets the build produce the list instead.

Name similarity invites confusing three separate files. The package manifest carries dependencies and package metadata; the application manifest carries installability information; the asset manifest is only the source-to-output mapping. The three are read at different times, by different parties.

Summary

• Assets enter the graph through two standard links: a url() declaration in a style file and a new URL(...) expression in code; addresses assembled at runtime cannot enter the pipeline.
• Linked assets pass through the processed flow; files with a fixed name pass through the directly copied flow — the criterion is whether the build is free to determine the address.
• The pipeline is four steps: scan for links, decide embedding, replace the reference with the output address, write the mapping into the manifest.
• Embedding buys one request; in exchange it brings encoding expansion, duplication at every reference, and the loss of separate caching — a 4096-byte body becomes 5464 characters in base64.
• Fonts are written as separate files regardless of size; images are produced together with dimension metadata and width derivatives.
• The asset manifest is the shared contract for the document template, the server runtime, and the service worker.

Next Step

The files in the output still carry their source names for now. This leaves both of two questions unresolved. If a published file is cached long-lived, the user keeps getting the old version once its content changes; if it is cached short-lived, revalidation happens on every visit and the cache’s value is lost. The way to avoid both extremes is binding the name to the content: when a file’s name carries a digest of its content, the name does not change unless the content changes, and the cache stays valid as long as the name does not change. The next lesson produces these names, measures how names change in a chain across interdependent chunks, and builds the mechanism that breaks that chain.