Audio and Video

Visual content is in place. Content that flows through time brings separate problems. Once an image gets shown, the reader looks at it as long as wanted; once a video plays, though, the flow may not be in the user’s control, its sound can interrupt other work, and without a text counterpart of the speech, the content stays closed to those who cannot hear.

The audio and video elements carry this content. Both share the same attribute set; video additionally declares the image dimensions and a cover image.

Basic Markup

<video controls preload="metadata" width="1280" height="720" poster="cover.jpg">
  <source src="briefing.webm" type="video/webm">
  <source src="briefing.mp4" type="video/mp4">
  <track kind="captions" src="briefing-en.vtt" srclang="en"
         label="English captions" default>
  <p>The video cannot be played.
     <a href="briefing.mp4">Download the file</a>.</p>
</video>

The controls attribute shows the playback controls. If it does not get written, the user cannot stop the content, lower its volume, or seek forward; it therefore gets written on every media the user needs to control.

preload declares how much data gets downloaded ahead of time: none nothing, metadata only duration and size information, auto as much as the browser sees fit. For media whose playback is uncertain, metadata shows the duration and waits without downloading the data.

The p block inside the element is fallback for a client that does not process media; it gets written after the source and track elements.

Source Selection

The source elements get tried in order, and the first supported source gets chosen. The type attribute carries the container format and codec information, letting the browser decide whether it can play the file without downloading it.

// media.mjs — trying source elements in order
const sources = [
  { src: "briefing.av1.mp4", type: 'video/mp4; codecs="av01.0.05M.08"' },
  { src: "briefing.vp9.webm", type: 'video/webm; codecs="vp9"' },
  { src: "briefing.h264.mp4", type: 'video/mp4; codecs="avc1.42E01E"' },
];

function pick(supported) {
  for (const source of sources) {
    const codec = source.type.match(/codecs="([^"]+)"/)?.[1] ?? "";
    const container = source.type.split(";")[0].trim();
    if (supported.containers.includes(container) && supported.codecs.some((c) => codec.startsWith(c))) {
      return source.src;
    }
  }
  return null;
}

const clients = [
  { name: "A", containers: ["video/mp4", "video/webm"], codecs: ["av01", "vp9", "avc1"] },
  { name: "B", containers: ["video/mp4"], codecs: ["avc1"] },
  { name: "C", containers: ["video/webm"], codecs: ["vp9"] },
  { name: "D", containers: ["video/ogg"], codecs: ["theora"] },
];

for (const client of clients) {
  console.log("client " + client.name + ":", pick(client) ?? "no source can play");
}

console.log("--- track kind values ---");
const tracks = [
  { kind: "captions", srclang: "en", label: "English captions", default: true },
  { kind: "subtitles", srclang: "es", label: "Spanish subtitles" },
  { kind: "descriptions", srclang: "en", label: "Audio description" },
];
for (const track of tracks) {
  const line = track.kind.padEnd(14) + track.srclang.padEnd(4)
    + track.label.padEnd(20) + (track.default ? "default" : "");
  console.log(line.trimEnd());
}

client A: briefing.av1.mp4
client B: briefing.h264.mp4
client C: briefing.vp9.webm
client D: no source can play
--- track kind values ---
captions      en  English captions    default
subtitles     es  Spanish subtitles
descriptions  en  Audio description

The output shows three things. Order is decisive: client A can play all three sources and picks the first on the list, which is why the most efficient format gets written first. Clients B and C fall to the first suitable source on the list. Client D cannot play any of them and sees the fallback content.

Container format and codec differ: the same container can carry different codecs, and a client may support the container without supporting the codec inside it. Writing only the container in the type declaration hides this distinction and leads the browser to choose the wrong source and fail.

Text Tracks

The track element ties timed text to the media; kind declares the track’s function, and functions are not interchangeable.

captions writes out meaningful sounds alongside speech — a door sound, a warning signal — and is for the user who cannot hear. subtitles only translates the speech into another language, for the user who can hear but does not know the language. The two look alike visually, but their content differs, and neither substitutes for the other.

descriptions narrates information in the picture not covered in the speech, and is for the user who cannot see. chapters divides the media into sections; the user can jump from a chapter list.

The default attribute declares the track that starts on absent another choice. srclang is the track’s language, label is the name shown to the user.

Captions are not an enhancement added to the media afterward. If a piece of audio content has no text counterpart, that content does not exist for users who cannot hear it; this is the same situation as an image with no alternative text.

Cover Image and Allotted Space

The poster attribute declares the image shown before the video plays. If it does not get written, the browser waits for the media’s first frame; with preload="none", no frame downloads and the space stays empty.

The width and height attributes do the same job here as with images: as in the Images lesson, they declare the aspect ratio and get space allotted before download, keeping content below from shifting. The cover image’s ratio gets kept the same as the video’s; if it differs, a jump appears once playback starts.

Linking to a Time Range

A specific moment in the media can get linked to with the address’s fragment part. As in the Links lesson, the fragment part does not go to the server; here the player interprets it.

<p>The removal step begins <a href="maintenance.mp4#t=90">at second 90</a>;
   <a href="maintenance.mp4#t=90,145">the 90–145 second range</a> shows only that step.</p>

The value is in seconds; when two values get written separated by a comma, a start and an end get specified. This lets text refer to a specific part of a long recording, saving the user from seeking by hand. When unsupported, the link opens the media from the start; the failure cost is low.

Autoplay

The autoplay attribute requests that the media start without the user asking for it. Browsers restrict this request for the user’s sake: unexpected sound drowns out screen-reader output, causes disruption in shared spaces, and costs money on a metered data connection.

The shape of the restriction varies by browser; the common point: media starting with sound on does not play without user interaction. The restriction is looser for media that start muted. This cannot get tested by feature detection; the start request returns a promise that can get rejected.

In practice, two consequences follow. Autoplay does not get assumed to work; the media itself must look meaningful even when unstarted by the user. Writing autoplay without muted does nothing in most contexts.

The loop attribute repeats the media. Repeating motion with no time limit is a problem for users with motion sensitivity; repeating content longer than five seconds needs a way to stop it.

In the Station Document

<figure>
  <video controls preload="metadata" width="1280" height="720" poster="maintenance-cover.jpg">
    <source src="maintenance.webm" type='video/webm; codecs="vp9"'>
    <source src="maintenance.mp4" type='video/mp4; codecs="avc1.42E01E"'>
    <track kind="captions" src="maintenance-en.vtt" srclang="en" label="English captions" default>
    <track kind="descriptions" src="maintenance-description.vtt" srclang="en" label="Audio description">
    <p>The video cannot be played. <a href="maintenance.mp4">Download the file</a>.</p>
  </video>
  <figcaption>Figure 2. Reading the humidity sensor without removing it from its mount.</figcaption>
</figure>

The video is a figure: it can get detached from the text’s flow and gets referenced in the flow. Controls are on, caption and description tracks are declared, and autoplay has not been requested.

Summary

• If controls does not get written, the user cannot stop the media; it gets written on every media that needs control.
• source elements get tried in order and the first supported one gets chosen; the type declaration carries container and codec information together.
• captions is for the user who cannot hear, subtitles for the one who does not know the language, descriptions for the one who cannot see; one does not substitute for another.
• Audio content with no text counterpart gets treated as nonexistent for users who cannot hear it.
• An autoplay request can get rejected; audible autoplay does not happen in most contexts, and the document gets written for the case where it does not.

Next Step

Audio and video were the document’s own resources. Some content, though, is the whole of another document: a map, a player, a section from another site. Embedding these hands part of the control outside. The next lesson takes up embedding a frame and the isolation attributes restricting the embedded document’s capabilities.