UI Tests

A unit test calls a function and looks at the value that comes back. A user does not call functions; a user taps a button. Even if the right code runs behind the button, the button may not be wired to the screen. In the previous lesson (Unit Tests) we saw that a boundary fault was caught only by the check that tried the boundary value. In this lesson the fault is somewhere else: the “Confirm Order” button is visible, it can be tapped, but it does not send an order.

The measurement axis is separation again: we apply the same check to the correct version and to the faulty version. A separating check passes on the correct version and fails on the faulty one. A blind check passes on both. A noisy check fails on both. An inverted check fails on the correct version and passes on the faulty one. We will see the four names in the output as ASCII. The core is the same as the core in the previous lesson and is written out again, because each lesson runs on its own.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

Contrast holds the output of the two versions and the check’s verdict on those outputs; separation derives the label only from the two truth values. The contrast function applies the same closure to both outputs, and countSeparations counts the four categories. The reason for the rule: a check’s value lies not in passing but in telling the two versions apart.

Driving the App

A UI test does not call the app’s code. The test target launches the app as a separate process and touches the screen from outside, like a user. So from inside the test the app’s types and functions cannot be reached; only the elements visible on the screen can. The driver is the XCUIApplication object.

// order confirmation UI test — SOURCE
import XCTest

final class CheckoutUITests: XCTestCase {
    override func setUpWithError() throws {
        continueAfterFailure = false
    }

    func testConfirmButtonExists() {
        let app = XCUIApplication()
        app.launchArguments = ["--reset-orders"]
        app.launch()
        XCTAssertTrue(app.buttons["confirmButton"].exists)
    }

    func testConfirmAddsOrder() {
        let app = XCUIApplication()
        app.launchArguments = ["--reset-orders"]
        app.launch()
        app.buttons["confirmButton"].tap()
        XCTAssertTrue(app.tables["orderList"].cells.element(boundBy: 0).waitForExistence(timeout: 5))
    }
}

This fragment is not run; it needs a simulator or a physical device. continueAfterFailure = false stops the test when the first assertion does not hold; otherwise a test that keeps tapping an empty screen produces meaningless errors one after another. launchArguments tells the app the starting state of the test: here, a flag that empties the order list. If each test does not start from the same state, the record left by the previous one changes the results.

Elements are found in three ways: by visible text (app.buttons["Confirm Order"]), by index (element(boundBy:)) or by accessibility identifier. The identifier is a fixed name given to the element in the app code, and it stays the same even if the text changes. In the fragment above the button is found with the confirmButton identifier.

Waiting

The screen does not change instantly. When the button is tapped, the order is sent, the response arrives and the list refreshes; this chain takes some time. waitForExistence(timeout:) waits for the element to appear on the screen for at most the given time and returns a Bool. Reading exists directly without waiting risks getting the wrong answer while the element has not been drawn yet.

In this lesson we will not measure durations and we will not show how close a test comes to its timeout. Timing varies from machine to machine; printing a number that depends on it would not be repeatable. Instead, in the model the wait is split into two states: the screen right after the button is tapped and the state of the screen after it has settled. In a real test, how many milliseconds pass between these two moments is not measured; in the model there is only “before” and “after”.

The Confirmation Screen Model

The model below sets up what the checks say to the two versions; it is not the UI test framework itself. The screen is a struct. It holds the button’s title, the pending orders and the orders listed on the screen. tapConfirm represents tapping the button and settle represents the screen settling. In the correct version the tap adds an order; in the faulty version tapping the button does nothing.

// Checkout.swift — state of the confirmation screen, correct and faulty versions
struct CheckoutScreen {
    let confirmTitle = "Confirm Order"
    private(set) var pending: [String] = []
    private(set) var orders: [String] = []

    mutating func tapConfirm(sending: Bool) {
        if sending { pending.append("order-1") }
    }

    mutating func settle() {
        orders += pending
        pending = []
    }
}

func screenAfterTap(sending: Bool, settled: Bool) -> CheckoutScreen {
    var screen = CheckoutScreen()
    screen.tapConfirm(sending: sending)
    if settled { screen.settle() }
    return screen
}

Three checks are applied to the settled state. The first is bound to text: it compares the button’s title with the spelling in the app’s old version (“Confirm order”). The text was later changed with a typo fix to a capital letter, and both versions carry the new text. The second is bound to presence: is the button on the screen. The third is bound to outcome: after the button is tapped, has the order list grown by one order.

// main.swift — three checks, two versions
let correct = screenAfterTap(sending: true, settled: true)
let faulty = screenAfterTap(sending: false, settled: true)

let oldTitle = contrast("old-title-text", correct: correct.confirmTitle, faulty: faulty.confirmTitle, check: { $0 == "Confirm order" }).describe()
let exists = contrast("button-exists", correct: !correct.confirmTitle.isEmpty, faulty: !faulty.confirmTitle.isEmpty, check: { $0 }).describe()
let listGrew = contrast("order-list-grew", correct: correct.orders.count, faulty: faulty.orders.count, check: { $0 == 1 }).describe()

countSeparations([oldTitle, exists, listGrew])

old-title-text | correct=Confirm Order/fail faulty=Confirm Order/fail -> noisy
button-exists | correct=true/pass faulty=true/pass -> blind
order-list-grew | correct=1/pass faulty=0/fail -> separating
separating: 1
blind: 1
noisy: 1
inverted: 0

old-title-text is noisy: both versions carry the “Confirm Order” title and the check expects the old spelling, so it rejects both. The button works, and the test turns red. button-exists is blind: the button is on the screen in both versions. order-list-grew is separating: in the correct version the list holds one order, in the faulty version it holds zero.

All three of these checks can be written in a UI test, and all three look the same in a test report. One breaks for no reason when the copy in the app changes, one stays green even if the button does not work at all, and one says the button does its job. The difference is what the check looks at: text, presence or outcome.

TP1. The screen model adds a single order. Multiple orders, deletion or cancellation must be measured separately.

TP2. What makes the old-title-text check noisy is that both versions carry the new text. If the text had changed only in the faulty version, the same check could have been inverted or separating.

The Check Bound to Text

A check bound to text finds the button by its visible text. This is convenient, because whatever is written on the screen is also in the test. But the text is in the hands of marketing, translation or typo fixes. When the title changes, the app keeps working correctly and the test turns red. This kind of breakage does not point to a fault; it creates a maintenance burden.

Moreover, in a multilingual app the text of the same button changes by language. A test bound to text works only in the language it was written in. When a second language configuration is added to the test plan, the tests bound to text fail in the second language; we will measure this in a later lesson of this topic. A button found with an identifier carries the same name in every language.

Throwing text out of the test altogether is not a solution either. Some tests check the text itself: when an order is confirmed, the screen showing “Your order has been received” is a behavior. The difference is this: using text to find an element brings a maintenance burden, while using it to verify a behavior does not.

The Check Bound to Outcome

order-list-grew separated because it looked at the effect the button produces. If the button really works, the list grows; if it does not, it does not grow. When writing a UI test, the first question should not be “is there a button” but “what should happen when the button is tapped”.

The check bound to outcome has a cost. If the place where the order list appears on the screen changes, if it moves to a new tab, or if the table becomes a collection, the test breaks again. This is why the identifier is kept fixed and, as the screen’s structure changes, the identifier is carried along. A check bound to outcome separates, but it also asks for maintenance.

UI tests are slow. Each one launches the app, drives the screen and finishes. So their number is kept small and their coverage wide: unit tests check the code’s logic, and UI tests check a few essential paths end to end. Writing a UI test for a boundary fault is the wrong tool; so is writing a unit test for a wiring fault.

The Bounding Measurement

The same check bound to outcome behaves differently depending on the moment of waiting. Below, the order-list-grew check is applied on two screens: the screen looked at right after the button is tapped, and the screen looked at after it has settled. We named the first immediate-look and the second after-wait.

// main.swift — same check, two moments of looking
let immediateCorrect = screenAfterTap(sending: true, settled: false)
let immediateFaulty = screenAfterTap(sending: false, settled: false)
let settledCorrect = screenAfterTap(sending: true, settled: true)
let settledFaulty = screenAfterTap(sending: false, settled: true)

let immediate = contrast("immediate-look", correct: immediateCorrect.orders.count, faulty: immediateFaulty.orders.count, check: { $0 == 1 }).describe()
let afterWait = contrast("after-wait", correct: settledCorrect.orders.count, faulty: settledFaulty.orders.count, check: { $0 == 1 }).describe()

countSeparations([immediate, afterWait])

immediate-look | correct=0/fail faulty=0/fail -> noisy
after-wait | correct=1/pass faulty=0/fail -> separating
separating: 1
blind: 0
noisy: 1
inverted: 0

The same check became noisy when applied before the screen had settled: even the correct version shows zero orders and the test fails. After waiting, the same check is separating. A check’s value depends not only on what it looks at but also on when it looks.

In the model the two moments are exact and deterministic: before and after. In a real app the moment of settling varies; in some runs the test catches up with the wait, in others it cannot. A test whose results are uncertain like this is called a flaky test, and it is not printed in this lesson’s model, because the model gives the same result on every run and does not represent uncertainty. To fix a flaky test, explicit waits such as waitForExistence(timeout:) are used; sleeping for an arbitrary time hides the problem.

TP3. The model counts settling as a single step. In reality the list can refresh in several steps and intermediate states can be seen.

TP4. The immediate-look check coming out noisy comes from the model counting both screens as empty at the same moment. In a real run the screen may sometimes be ready and the result changes.

Common Mistakes

• Finding the button only by its text. The test breaks when the title changes or when the app opens in another language.
• Checking only that the button is on the screen. It stays blind and misses a button that does nothing.
• Waiting by sleeping for a fixed time. If the time is short the test becomes flaky; if it is long it slows down for no reason.
• Starting every test from the state left over from the app’s previous run. The starting state must be reset with flags.
• Looking for a boundary fault with a UI test. A unit test is faster and more precise for that.

Summary

• A UI test drives the app as a separate process and reaches the screen only through visible elements.
• Elements are found by text, by index or by accessibility identifier; the identifier stays the same even when the text changes.
• Of three checks, one came out noisy, one blind and one separating; only the check bound to outcome showed that the button sends an order.
• The same check became noisy when applied before the screen settled; the moment of waiting changed the check’s category.
• Flaky tests, whose results are uncertain, are not printed in the model; their fix is an explicit wait, not an arbitrary duration.
• This lesson’s separation: noisy.

Next Step

In this lesson we wrote the checks one by one. In the next lesson, Test Plans and Coverage will gather all the tests under one plan, count which lines run, and measure that a line running does not say it is correct.