UI Tests

In the previous lesson every test started the screen from scratch, and the measurement counted the cost of the setup. Once the screen is up, the test has to talk to it: it finds a node, taps it, and reads the result. This conversation has a cost too, and it is the subject of this lesson. The course’s question is the same: in how many samples does this measurement exceed its budget? The four answers do not change: within, minority, majority, all. The unit of work measured is the step taken to find one node.

The core is the same as in the previous lessons: a budget, the label of a measurement and its list of samples, and the number of samples that strictly exceed the limit.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

How the Screen Looks to a Test

A UI test reads the meaning of the screen, not its pixels. In a component-based UI, this meaningful form of the screen is called the semantics tree. Each node in the tree carries a text, a role, or an action: a heading, a button, the name of a product. Accessibility services build on the same tree, although they apply their own rules for merging nodes. The job of the test is to find a node in this tree, apply an action to it, and verify the node’s later state.

There are two ways to find a node. The first is to look it up by its text: you find the node by the words the user sees. This is a natural way, because the test looks at what the user looks at. The second is to look it up by its test tag: the developer gives the node a tag, and the test asks for that tag. The tag is invisible to the user; it serves only the test. On the platform the two look like this.

// Platform code: CartComposeTest
class CartComposeTest {
    @get:Rule
    val composeRule = createComposeRule()

    @Test
    fun deleteButtonIsShown() {
        composeRule.setContent { CartScreen() }
        composeRule.onNodeWithTag("delete-notebook").assertIsDisplayed()
        composeRule.onNodeWithText("Delete notebook").assertIsDisplayed()
    }
}

setContent sets up the screen, onNodeWithTag finds the node by tag, and onNodeWithText finds it by text. In the classic view system the same job is done with a different tool.

// Platform code: CartViewTest
class CartViewTest {
    @Test
    fun deleteButtonIsShownInViews() {
        onView(withId(R.id.deleteNotebook)).check(matches(isDisplayed()))
    }
}

onView finds a view in the view tree by its id or its text. Both pieces belong to platform code. The model builds the rule of this measurement; the model is not the tool itself. In the model, the tree is a list of nodes, and a lookup is counted by the steps taken in this list.

TE1. The model does not run the UI testing library; there is no screen or tree in this environment. The semantics tree is represented by a flat list, and a lookup walks the list from the start. The real tool walks the tree in depth order; the model reduces this order to the order of the list.

Looking a Node Up by Its Text

The tree of a shopping cart screen has twelve nodes: the heading, the back button, two products with a quantity and a delete button for each, the total, the coupon code, the apply-coupon button, and the proceed-to-checkout button. Each node has a text and a unique tag. The helper file builds the tree and defines the lookup by text and the lookup by tag.

// Semantics.kt — the semantics tree of the cart screen, lookup by text and by tag
class Node(val text: String, val tag: String)

fun cartTree(extra: Int = 0): List<Node> {
    val head = listOf(
        Node("My cart", "title"),
        Node("Back", "back"),
        Node("Notebook", "item-notebook"),
        Node("Notebook quantity", "quantity-notebook"),
        Node("Delete notebook", "delete-notebook"),
        Node("Pencil", "item-pencil"),
        Node("Pencil quantity", "quantity-pencil"),
        Node("Delete pencil", "delete-pencil"),
    )
    val offers = (1..extra).map { Node("Offer $it", "offer-$it") }
    val tail = listOf(
        Node("Total", "total"),
        Node("Coupon code", "coupon"),
        Node("Apply coupon", "apply-coupon"),
        Node("Proceed to checkout", "checkout"),
    )
    return head + offers + tail
}

fun pickByTags(tree: List<Node>, tags: List<String>): List<Node> =
    tags.map { tag -> tree.first { it.tag == tag } }

fun findSteps(tree: List<Node>, text: String): Int {
    var steps = 0
    for (node in tree) {
        steps++
        if (node.text == text) return steps
    }
    return steps
}

fun tagSteps(byTag: Map<String, Node>, tag: String): Int =
    if (byTag.containsKey(tag)) 1 else 0

findSteps walks the tree from the start and stops when the text matches. The steps taken equal the node’s position in the tree: the heading is found in 1 step, the proceed-to-checkout button in 12. cartTree can optionally add offer nodes to the tree; for now we add none. Eight tests will look up eight nodes: back, delete notebook, pencil, pencil quantity, total, apply coupon, proceed to checkout, and the heading. The budget is 5 steps: this is the model’s counterpart of a team decision that wants a node to be found within five nodes from the top of the screen, and it is not a platform threshold.

// Main.kt
fun main() {
    val tree = cartTree()
    val picks = pickByTags(
        tree,
        listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title"),
    )
    val steps = picks.map { findSteps(tree, it.text) }
    println("steps: $steps")

    val overrun = Overrun("lookup-by-text", Budget("step", 5), steps)
    overrun.line()
    countOverruns(listOf(overrun))
}

steps: [2, 5, 6, 7, 9, 11, 12, 1]
lookup-by-text: budget=step<=5 | over=5/8 -> majority
within: 0
minority: 0
majority: 1
all: 0

Five of the eight lookups exceeded the budget: the pencil, pencil quantity, total, apply-coupon, and proceed-to-checkout nodes sit beyond the first five nodes of the screen. The measurement is majority. The heading, the back button, and the delete-notebook button stayed within the budget because they are at the start of the tree. The overrun comes from the position of the node, not from its text: if a node with the same text were at the start of the tree, the lookup would be short.

The rule misses two things in this measurement as well. It does not see the magnitude: the proceed-to-checkout node exceeds the budget by seven steps and the pencil by one, and both count as one. It does not see the cause: the overrun comes from the walking order of the tree and from how the nodes are arranged for the user.

The Nodes of the Semantics Tree

Each node in the semantics tree is the summary of an element on the screen, as it appears to tests and to accessibility services. A node can carry a text, declare a role, offer an action, and receive a test tag. Several elements nested on the screen, for example the heading and the description inside a card, can by default merge into a single node in a component-based UI. The test looks up this merged node; the tree before merging can be requested separately. This detail changes how many steps a lookup takes, because the merged tree carries fewer nodes.

In our model there are twelve nodes, and each is a separate element. On a real screen, the number of nodes can be far smaller or far larger depending on the structure of the screen. What the measurement teaches is not the number but the relationship: if the lookup depends on the position of the node, the overrun grows as the screen grows.

Reading the Platform Pieces Line by Line

In the CartComposeTest piece, the @get:Rule annotation makes the declarative UI’s test rule be set up before every test and shut down afterward. createComposeRule() produces this rule. The line composeRule.setContent { CartScreen() } loads the screen under test into the rule; the screen is set up inside an empty activity, so it is not necessary to start the app’s own screen as in the previous lesson. onNodeWithTag("delete-notebook") finds the node whose tag is delete-notebook. assertIsDisplayed() verifies that the node is visible on the screen. onNodeWithText("Delete notebook") finds the same node by its text.

In the CartViewTest piece, onView(withId(R.id.deleteNotebook)) finds the view with the given id in the classic view system. check(matches(isDisplayed())) verifies that the view is visible. An id is a label that stands in for the text: it is defined once in a resource file and is independent of translation.

If we want to add an interaction, an action such as performClick() is added to the same lookup chain. The first link of the chain is always the node lookup; the action and the verification work on the node that was found. For this reason the cost of the lookup is added to the cost of every interaction in the test.

The Fragility of Lookup by Text

Lookup by text is not only slow; it breaks when the text changes. If the label of the button turns from “Delete notebook” into “Delete”, every test that looks it up by text breaks. If the app is published in several languages, the test passes in only one language and cannot find the node in the others. A tag has neither problem, because a tag is independent of the user’s language.

On the other hand, the text is what the user really sees. A test that looks up by text implicitly checks that the button appears to the user with the right words. A test that looks up by tag does not check this, and the difference is a choice: the team decides whether it wants the fast lookup or the correctness of the visible text.

Naming the Tags

The value of a tag lies in its staying stable. The model chose the form action-element for the tags: delete-notebook, quantity-pencil, apply-coupon. This name is tied to the function of the element, not to the layout of the screen. Even if the button is moved to another place on the screen, the tag stays the same. A name that describes the layout, for example second-row-right-button, becomes misleading when the screen changes.

A tag also has to be unique. If every row in a list carries the same tag, a lookup that expects a single node finds multiple matches and the test fails with an error. For this reason, in lists a value that tells the rows apart, for example the product code, is added to the tag.

The Relationship with Accessibility

The semantics tree is a structure used not only by tests but also by screen readers. For this reason a test that looks up by text searches for a text close to the one the screen reader will read; a test that cannot find a node whose text is wrong also catches an accessibility problem indirectly. A tag, on the other hand, serves only tests and may be invisible to the screen reader. This is the reason for using the two lookups together: the tag finds quickly and stably, and the text verifies what the user hears.

Common Misconceptions

The first misconception is the thought that “putting a tag on every node makes the test more robust”. Tags add a test-only burden to production code, and a forgotten tag breaks the test. A tag is put only on nodes whose text cannot be trusted: repeated texts, dynamic texts, and icons.

The second misconception is the thought that “lookup by text is always slow”. In the measurement the slowness came from the order of the node. If the node being sought is at the start of the tree, lookup by text is short as well: the heading was found in one step and the back button in two. The overrun depends on the position, not on the form of the lookup.

The third misconception is the thought that “when the screen grows, the test has to be rewritten”. In tests that look up by tag, growth did not change the cost of the lookup. What has to be rewritten on a growing screen is the lookup of the nodes that remain without a tag.

The Bounding Measurement

The bounding measurement here is the lookup by tag. The reasoning is this: in the first measurement the overrun came from the lookup depending on the position in the tree. A lookup that finds the node from a map removes the position. If the nodes, the budget, and the number of samples stay the same, only the form of the lookup explains the difference in the result.

Look up the same screen, this time by test tag. The tag is unique, so it is found in a map with a single glance. The map itself has to be built once: the tree is walked once and each node is put into the map under its tag. This work is paid once per screen.

// Main.kt — the same screen, lookup by tag
fun main() {
    val tree = cartTree()
    val picks = pickByTags(
        tree,
        listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title"),
    )
    val byTag = tree.associateBy { it.tag }
    println("map setup: ${tree.size} steps, once per screen")

    val steps = picks.map { tagSteps(byTag, it.tag) }
    println("steps: $steps")

    val overrun = Overrun("lookup-by-tag", Budget("step", 5), steps)
    overrun.line()
    countOverruns(listOf(overrun))
}

map setup: 12 steps, once per screen
steps: [1, 1, 1, 1, 1, 1, 1, 1]
lookup-by-tag: budget=step<=5 | over=0/8 -> within
within: 1
minority: 0
majority: 0
all: 0

The same screen, the same eight nodes, the same budget. Every lookup finished in one step, and the measurement moved from majority to within. What changed is not the screen under test but the way the node is reached.

Lookup by tag gives up one thing as well: the tag is invisible to the user. If the label on a button is wrong, a test that looks up by text catches the defect and a test that looks up by tag does not. On the platform, the two ways are used together: the node is found by tag, and the text is verified separately.

A third measurement shows what happens as the screen grows. Add eight offer nodes before the total. Set the budget this time to the old tree’s length, 12 steps: on the old screen, lookup by text finds every node within this limit.

// Main.kt — when the screen grows, lookup by text gets more expensive and lookup by tag stays the same
fun main() {
    val tags = listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title")
    val small = cartTree()
    val large = cartTree(8)
    val smallPicks = pickByTags(small, tags)
    val largePicks = pickByTags(large, tags)
    val byTag = large.associateBy { it.tag }

    val smallText = smallPicks.map { findSteps(small, it.text) }
    val largeText = largePicks.map { findSteps(large, it.text) }
    val largeTag = largePicks.map { tagSteps(byTag, it.tag) }
    println("old screen, text: $smallText")
    println("large screen, text: $largeText")
    println("large screen, tag: $largeTag")

    val limit = Budget("step", 12)
    val overruns = listOf(
        Overrun("text-old-screen", limit, smallText),
        Overrun("text-large-screen", limit, largeText),
        Overrun("tag-large-screen", limit, largeTag),
    )
    overruns.forEach { it.line() }
    countOverruns(overruns)
}

old screen, text: [2, 5, 6, 7, 9, 11, 12, 1]
large screen, text: [2, 5, 6, 7, 17, 19, 20, 1]
large screen, tag: [1, 1, 1, 1, 1, 1, 1, 1]
text-old-screen: budget=step<=12 | over=0/8 -> within
text-large-screen: budget=step<=12 | over=3/8 -> minority
tag-large-screen: budget=step<=12 | over=0/8 -> within
within: 2
minority: 1
majority: 0
all: 0

On the old screen, lookup by text was within the budget of 12 steps. When the screen grew by eight nodes, the total, apply-coupon, and proceed-to-checkout nodes moved beyond the limit, and the measurement fell to minority. Lookup by tag is still one step on the large screen. As the screen grows, the overrun of lookup by text increases; the overrun of lookup by tag is independent of position.

TE2. The model places the offer nodes in front of the total; on a real screen, the place of new nodes in the tree depends on the screen design. It is also assumed that the map covers every node: if two nodes carry the same tag, the real tool fails with an error, and the model does not.

Summary

• This lesson’s overrun: majority. The test that looked up nodes by text went past the first five steps of the tree in five of the eight lookups.
• The lesson’s criterion, in how many lookups the steps taken to find a node exceed the budget: the overrun comes not from the text of the node but from its position in the tree.
• Lookup through a test tag and a map is one step in every sample; on the same screen the measurement moved from majority to within. The map is built once per screen.
• When the screen grew by eight nodes, lookup by text went from within to minority on the budget of 12 steps, and lookup by tag stayed within.
• The rule counts steps and does not look at whether a node appears correctly to the user; a test that looks up by tag does not catch a wrongly written button text.

Next Step

In this lesson the screen stood still, and the lookup only counted steps. A real screen, however, may not be ready when the test starts looking for a node. The next lesson, Test Stability, measures in how many runs a test that waits for the ready signal exceeds its waiting budget.