Browser Tasks

The previous lesson established that markup is a declaration layer and showed that a document is made of text meant to be read plus the structure that describes it. What is left is the question of what the browser does with that declaration.

The What the Browser Does lesson in the How the Internet Works course covered the chain from resolving the address, to establishing a connection, to requesting subresources. This lesson opens one link of that chain: what happens once the response body arrives. The stages are linked by an input–output relationship, and each one’s output is written in the next one’s language.

The Chain of Stages

Once the document arrives, six transformations are applied in sequence.

Stage	Input	Output
Decoding	byte sequence	character sequence
Tokenization	character sequence	token stream
Tree building	token stream	node tree
Style computation	node tree + style rules	styled tree
Layout	styled tree	boxes’ positions and sizes
Paint	boxes	pixels

Every link in the chain carries a separate responsibility. The tokenizer does not know where an element belongs; the tree builder does not know the character encoding; layout does not know what the tokens were. This separation is what lets any one stage be changed independently.

Decoding: From Byte to Character

What comes over the network is bytes. As established in the Character Encodings lesson of the How Computers Work course, turning a byte sequence into a character sequence requires a rule; the bytes themselves do not say which rule they were written with.

// tokens.mjs — byte sequence to token stream
const bytes = Buffer.from('<p class="measurement">Temperature: <b>-4</b>°C</p>', "utf8");
console.log("byte count      :", bytes.length);

const text = bytes.toString("utf8"); // the charset declaration decides this step
console.log("char count      :", text.length);
console.log("--- tokens ---");

let i = 0;
while (i < text.length) {
  const k = text.indexOf("<", i);
  if (k === -1) { console.log("TEXT    ", JSON.stringify(text.slice(i))); break; }
  if (k > i) console.log("TEXT    ", JSON.stringify(text.slice(i, k)));
  const end = text.indexOf(">", k);
  const inner = text.slice(k + 1, end);
  if (inner.startsWith("/")) {
    console.log("CLOSE   " + inner.slice(1));
  } else {
    const space = inner.search(/[\s/]/);
    const name = space === -1 ? inner : inner.slice(0, space);
    const rest = space === -1 ? "" : inner.slice(space).trim();
    console.log("OPEN    " + name + (rest ? "  [" + rest + "]" : ""));
  }
  i = end + 1;
}

byte count      : 52
char count      : 51
--- tokens ---
OPEN    p  [class="measurement"]
TEXT     "Temperature: "
OPEN    b
TEXT     "-4"
CLOSE   b
TEXT     "°C"
CLOSE   p

The first two lines show why decoding is a separate stage at all: 52 bytes correspond to 51 characters. The difference of one comes from the ° sign taking two bytes rather than one in this encoding. This distinction between byte count and character count matters in every computation built on a document’s length.

Which rule decoding follows is declared inside the document itself. We will look at this declaration’s form and what order it is looked for in, in the Meta Tags lesson.

Tokenization: From Character to Token

The second block of the output above is this stage’s product. The unit this stage produces is called a token; the concept was defined under the same name in the Compilation Stages lesson of the How Computers Work course. The tokenizer reads the character sequence from left to right and produces three kinds of unit: an opening tag, a closing tag, and text. There are separate token kinds for comments and the document type declaration too.

What the tokenizer does not do is worth noting. It does not say that the b element is inside the p element; it only gives the tokens’ sequence. Nesting information is a result to be built from the sequence; it is not in the token itself. In the same way, the tokenizer does not treat the p element being unclosed as a problem either: if the closing token never comes, it is not produced at all.

The tokenizer is a state machine: how a given character is interpreted depends on the state it is currently in. While in text state, the character < is taken as the start of a tag; while in tag-name state, whitespace announces the name has ended; while in attribute-value state, a < character inside quotes is an ordinary character. This is why “parsing HTML with a regular expression” does not give a correct result in the general case: regular expressions cannot build this state-to-state behavior. The script above only works correctly on an input as narrow as this example.

Tree Building

A token stream is sequential and flat; a document is nested. The tree builder establishes this nesting by pushing opening tokens onto a stack and popping the stack on closing tokens. Its details are the next lesson’s subject.

Style Computation and Box Generation

Once the tree is built, what each element is is known, but not what it will look like. The style computation stage calculates the style values to apply to every element. Part of these values comes from rules attached to the document from outside, part from the browser’s own default rules.

The existence of default styles matters: a document with no style file attached at all still looks structured — headings are large and bold, lists are indented, links are distinguishable. This look does not come from the elements’ nature, it comes from the default rule set the browser carries for every element. Calling elements “the bold one” or “the indented one” is therefore misleading; these are the presentation layer’s defaults, and they can be changed.

Style computation’s output is a box for every visible element. The box’s kind depends on the element’s style value: some boxes stretch across a line, some flow inside text. Some elements produce no box at all; content in the head section and hidden elements are like this.

Layout and Paint

Layout computes the boxes’ position and size relative to one another. This computation starts from the available width and proceeds down the tree: every box knows the space allotted to it and distributes it to its children. The output is a position and a size for every box.

Paint turns the computed boxes into colors. Backgrounds, borders, text, and images are drawn in sequence. The paint order is separate from the elements’ order of appearance in the document: one element appearing on top of another is the style layer’s decision.

Separating these two stages also separates the cost of redoing work. When an element’s color changes, only paint is repeated; when its width changes, layout is repeated too, and layout is more expensive because the changed box also affects its neighbors and children. This distinction is the foundation for the performance subject in later courses.

Boxes Do Not Map One to One onto Elements

The number of elements in the document and the number of boxes on screen are not equal, and this is the reason layout is a separate stage at all. There are three kinds of divergence.

An element may produce no box at all. Content in the head section, comment nodes, and elements hidden by the presentation layer do not enter layout. These elements stay in the tree, can be queried, and can be changed by scripts; they are just not painted.

An element may produce more than one box. The text of a paragraph that spans two lines is split into two line boxes; if a link’s text falls at a line break, it splits into two pieces and each piece gets its own position and size. The element itself is single; its box is not.

A box may correspond to no element at all. The presentation layer can generate content that is not in the document; list markers and generated content are of this kind. These boxes correspond to no node in the tree.

These three divergences make concrete the difference between “changing the document” and “changing the appearance.” The tree carries the document’s meaning; boxes are the appearance that meaning takes on at a particular width, with a particular style.

The Stages Overlap

The chain does not work in a way where the second stage waits for the first to finish entirely. Parsing begins before the whole document has arrived; once part of the tree is built, an initial layout computation can be done and an initial paint can happen. That a user sees the first screen of a long document before the whole document has downloaded is a result of this overlap.

There are places where the overlap breaks: some resources halt parsing, some hold up paint. This is the subject of this topic’s fourth lesson.

Summary

• A document passes through six transformations in sequence: decoding, tokenization, tree building, style computation, layout, paint. Each stage’s output is the next one’s input.
• The decoding stage turns a byte sequence into a character sequence; byte count and character count are not equal, and the rule for the conversion is declared in the document.
• The tokenizer is a state machine and only produces a sequence; it carries no nesting information. This is why markup cannot be parsed with a regular expression in the general case.
• Elements’ default appearance comes not from their nature, but from the browser’s default style rules.
• Layout computes boxes’ position and size, paint computes their appearance; which stage a change reruns decides its cost.
• The stages overlap: parsing can begin before the whole document has arrived, and paint can begin before the whole tree has been built.

Next Step

This lesson passed over the tree-building stage in one sentence: a token stream is turned into a tree. Yet that tree is the foundation for everything after it — style is applied to it, layout is computed over it, scripts change it. The next lesson shows step by step how the tree is built: how the stack works, how elements that are not in the source text come into being, and why two different ways of writing produce the same tree.