Accessibility

A user who cannot see the screen uses the interface with a screen reader. The reader looks not at pixels but at the semantics tree the platform derives from the UI tree. This tree carries what each item is, what it says and what it can do. If a row has an icon, a product name and a price, the screen reader is expected to read a single sentence instead of passing through three separate stops: product name, price.

This merging is a move that has a direction on the tree. The labels of the children join the label of the merging ancestor. The measure of this lesson is the direction of that move.

// Direction.kt — classifies which direction a value travels in the UI tree
class Tree(private val parentOf: Map<String, String>) {
    fun contains(node: String): Boolean = node in parentOf.keys || node in parentOf.values
    fun parent(node: String): String? = parentOf[node]
    fun ancestors(node: String): List<String> {
        val chain = mutableListOf<String>()
        var current = parentOf[node]
        while (current != null) {
            chain.add(current)
            current = parentOf[current]
        }
        return chain
    }
}

class Move(val label: String, val from: String, val to: String, tree: Tree) {
    val direction: String = when {
        !tree.contains(from) || !tree.contains(to) -> "outside"
        to in tree.ancestors(from) -> "up"
        tree.parent(to) == from -> "down"
        from in tree.ancestors(to) -> "deep"
        else -> error("unrelated: $from and $to")
    }

    fun describe() {
        println("$label: from=$from | to=$to -> $direction")
    }
}

fun countDirections(moves: List<Move>) {
    for (name in listOf("down", "up", "deep", "outside")) {
        println("$name: ${moves.count { it.direction == name }}")
    }
}

The Semantics Tree and Merging

Every UI item is a node in the semantics tree. A node can carry a content description, a role and actions. By default every node is presented as a separate stop. When an ancestor node wants to merge its subtree, the descriptions of its children join it and the screen reader reads the ancestor node as a single stop.

// Platform code: merging a row semantically
@Composable
fun CartRow(name: String, price: String) {
    Row(Modifier.semantics(mergeDescendants = true) {}) {
        Icon(Icons.Default.ShoppingCart, contentDescription = null)
        Text(name)
        Text(price)
    }
}

mergeDescendants = true joins the descriptions of the nodes under this row to the row. The icon is given contentDescription = null: reading a decorative item adds nothing. The screen reader reads the row at once as “product name, price”. To drop the merging and give its own description in place of the whole subtree, clearAndSetSemantics is used. Prebuilt UI components pad the touch target up to a minimum size when they can take an action; on a hand-written touchable item the developer provides this size. Like the size and spacing relationship in the layout lesson, this size is a platform rule and is not measured in this lesson.

The model captures this platform rule; the model is not the platform itself. The helper file builds the tree of the merging row and computes which nodes carry labels and the text of the merged label. Two functions also supply the ends of the measurements: screenReader returns the name of the screen reader, and mergeOwner reads the row’s parent from the tree. In this lesson one end of a move is found from the tree (mergeOwner); the other end is a source the model names that is not in the tree (screenReader returns a fixed name).

// Semantics.kt — the semantics tree and the merged label (the model's calculation)
fun semanticsTree(): Tree = Tree(
    mapOf(
        "Screen" to "App",
        "Row" to "Screen",
        "Icon" to "Row",
        "Title" to "Row",
        "Wrap" to "Row",
        "Price" to "Wrap",
    )
)

fun screenReader(): String = "ScreenReader"

fun mergeOwner(tree: Tree): String = tree.parent("Row") ?: error("row is the root")

fun labelHolders(labels: Map<String, String>): List<String> = labels.keys.sorted()

fun mergedDescription(labels: Map<String, String>): String =
    labelHolders(labels).joinToString(", ") { labels.getValue(it) }

Labels Go Up to the Ancestor

In the first measurement two nodes carry labels: the product name is a direct child of the row, while the price is under a wrapper that steps in between, that is, a grandchild of the row. The icon has no label because it is decorative.

// Main.kt — the merging ancestor of the nodes that carry labels
fun main() {
    val tree = semanticsTree()
    val labels = mapOf("Title" to "Pen", "Price" to "Cost")
    println("merged label: ${mergedDescription(labels)}")
    val moves = labelHolders(labels).map { Move("label-$it", it, "Row", tree) }
    for (move in moves) {
        move.describe()
    }
    countDirections(moves)
}

merged label: Cost, Pen
label-Price: from=Price | to=Row -> up
label-Title: from=Title | to=Row -> up
down: 0
up: 2
deep: 0
outside: 0

Both moves are up. The direct child’s label went up one level, the grandchild’s label two. The deep direction is only toward below; in going up to an ancestor it makes no difference whether it is a child or a grandchild, the move is up. The wrapper that steps in neither carries nor changes the label.

LC1. The model takes the labels as single words and joins them in name order. The order in which the platform reads labels, its adding role and action information, and the conditions of merging (for example a clickable node keeping its own boundary) are not in the model.

The Label Goes to the Screen Reader

The merged label does not stay in the row; it is passed to the screen reader. The screen reader is not a node of the UI tree. It is the platform’s accessibility service and it reads the tree from outside.

// Main.kt — the merged label's trip to the screen reader
fun main() {
    val tree = semanticsTree()
    val labels = mapOf("Title" to "Pen", "Price" to "Cost")
    println("read: ${mergedDescription(labels)}")
    Move("reading", "Row", screenReader(), tree).describe()
}

read: Cost, Pen
reading: from=Row | to=ScreenReader -> outside

Outside. Because the screen reader is not in the tree, one end of the move is outside the tree. The row does not give the label to the reader; the reader takes the label from the row. There is a chain of two separate directions: the labels go up in the tree, and the merged label waiting at the top of the tree is read from outside.

LC2. The model builds the reader as a name that is not in the tree. The reader’s focus order, which node the user touches and when the label is read are not in the model.

The Bounding Measurement

The lesson’s direction comes from the labels going up to the ancestor. The label reaching the reader is outside, while the opening of the merging by the ancestor is down: the screen gives the row the merge flag. The measurement below shows all four together.

// Main.kt — the accessibility moves
fun main() {
    val tree = semanticsTree()
    val labels = mapOf("Title" to "Pen", "Price" to "Cost")
    val moves = labelHolders(labels).map { Move("label-$it", it, "Row", tree) } + listOf(
        Move("merge-flag", mergeOwner(tree), "Row", tree),
        Move("reading", "Row", screenReader(), tree),
    )
    for (move in moves) {
        move.describe()
    }
    countDirections(moves)
}

label-Price: from=Price | to=Row -> up
label-Title: from=Title | to=Row -> up
merge-flag: from=Screen | to=Row -> down
reading: from=Row | to=ScreenReader -> outside
down: 1
up: 2
deep: 0
outside: 1

Two up, one down, one outside. The merge flag descends from the screen to the row; the labels go up to the row; the merged label reaches the reader from outside. The separation of directions overlaps the separation of functions: the decision is made from top to bottom, the information is gathered from bottom to top, and the reading takes place outside. The lesson’s direction: up.

LC3. The model builds merging in a single row. A merging ancestor sitting inside another merging ancestor, and how nested merging would join the labels, were not measured in this lesson.

Summary

• This lesson’s direction: up. Labels go up from children and grandchildren to the merging ancestor.
• The screen reader looks not at pixels but at the semantics tree derived from the UI tree.
• The merging ancestor gathers the labels of its children and grandchildren; the labels go up to the ancestor.
• The label of a direct child and of a grandchild go in the same direction, up.
• The merged label going to the screen reader is a move outside the tree, an outside move.

Next Step

The tree of this course is the tree of the declarative interface. Many applications, though, still carry screens written with the classic view system. The next lesson measures using the two worlds together and how the direction is named when data crosses the boundary of the tree.