Test Plans and Coverage

In the previous lesson (UI Tests) we saw that a test checking the outcome of a button separated, while a test checking only its presence did not. This lesson steps up one layer: to the plan that brings hundreds of tests together and to the coverage report that says how much of the code those tests touch.

The measurement axis is separation: we apply the same check first to the correct version and then to the faulty one. A separating check passes on the correct version and fails on the faulty one. A blind check passes on both. A noisy check fails on both. An inverted check fails on the correct version and passes on the faulty one. The core is the same as in the previous lessons; we write it out again because each lesson runs on its own.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

Contrast holds the two outputs and the check’s verdict on each. The separation label derives from these two verdicts; contrast applies the same closure to both outputs; countSeparations counts the four categories. The reason for the rule: a check’s value lies not in passing but in telling the correct version from the faulty one.

The Test Plan

An app can have more than one test target: unit tests, UI tests, performance tests. A test plan gathers them in a single file. The plan says which test targets will run, which tests will be skipped and with which configurations the tests will run. A configuration is the same tests running a second time with a changed setting: screen language, region, coverage collection, random order and the like.

The plan is a JSON file, and the development environment shows it as a table. From the command line it is given to the xcodebuild test command with the -testPlan flag; this way the same plan runs the same tests both on the development machine and on the automated check server.

{
  "configurations": [
    { "name": "Turkish", "options": { "language": "tr", "region": "TR" } },
    { "name": "English", "options": { "language": "en", "region": "US" } }
  ],
  "defaultOptions": { "codeCoverage": true },
  "testTargets": [
    { "target": { "name": "ShopTests" } }
  ]
}

This is a configuration file; the lesson does not run it. There are two configurations: Turkish and English. While the codeCoverage option is on, which lines ran is recorded in every run. Every test in the plan runs in both configurations; that is, the same test is tried twice and can have two different results.

What Coverage Measures

Code coverage counts which lines or branches of the code were entered while a test suite ran. The report says “ran” or “did not run” line by line and gives a ratio. This information is valuable: a line that never ran has certainly not been tried.

But there is something coverage does not say. A line running does not say its result is correct. If a test calls a function and never looks at the returned value, the line counts as run and coverage goes up. Thinking that high coverage is assurance is a common misconception that arises from here. Low coverage points to a gap; high coverage does not show there is no gap.

The two tests below show this difference. The first runs the discount branch but discards the returned value. The second runs the same branch and compares the result with 180.

// discount tests — SOURCE
import XCTest

final class PricingTests: XCTestCase {
    func testDiscountBranchRuns() {
        _ = price(100, coupon: true)
        _ = price(100, coupon: false)
    }

    func testDiscountedTotal() {
        XCTAssertEqual(price(200, coupon: true), 180)
    }
}

Both tests show the discount line green in the coverage report. But only the second asks whether the discount is computed correctly. The first says nothing more than that the function did not crash.

The Discount Model

The model below sets up what the coverage count and the check say to the two versions; it is not the coverage tool itself. A real tool gets its line counts from the compiler. The model does the same job on a small scale: it puts a marker on each branch of the function and collects the markers that ran in a set. There are three markers: entry, discount and plain.

In the correct version, ten percent is taken off a cart with a coupon. In the faulty version the division has been changed to a hundred and only one percent is taken off. Both pass through the same branches; the difference is only in the calculation.

// Pricing.swift — correct and faulty versions of the discount rule, with a counter of lines run
func correctPrice(_ subtotal: Int, coupon: Bool, hits: inout Set<String>) -> Int {
    hits.insert("entry")
    if coupon {
        hits.insert("discount")
        return subtotal - subtotal / 10
    }
    hits.insert("plain")
    return subtotal
}

func faultyPrice(_ subtotal: Int, coupon: Bool, hits: inout Set<String>) -> Int {
    hits.insert("entry")
    if coupon {
        hits.insert("discount")
        return subtotal - subtotal / 100
    }
    hits.insert("plain")
    return subtotal
}

func executedCoverage(_ price: (Int, Bool, inout Set<String>) -> Int) -> String {
    var hits = Set<String>()
    _ = price(100, true, &hits)
    _ = price(100, false, &hits)
    return "\(hits.count) of 3 lines"
}

func discountedTotal(_ price: (Int, Bool, inout Set<String>) -> Int) -> Int {
    var hits = Set<String>()
    return price(200, true, &hits)
}

executedCoverage is the model of the first test: it makes the two calls, discards the result and writes how many markers ran. discountedTotal is the model of the second test: it runs the same branch and returns the result. This way we can put the two checks side by side. The coverage check is the question “did three of three lines run”; the outcome check is the question “is the total 180”.

// main.swift — coverage check and outcome check
let coverage = contrast("coverage-check", correct: executedCoverage(correctPrice), faulty: executedCoverage(faultyPrice), check: { $0 == "3 of 3 lines" }).describe()
let outcome = contrast("discount-result", correct: discountedTotal(correctPrice), faulty: discountedTotal(faultyPrice), check: { $0 == 180 }).describe()

countSeparations([coverage, outcome])

coverage-check | correct=3 of 3 lines/pass faulty=3 of 3 lines/pass -> blind
discount-result | correct=180/pass faulty=198/fail -> separating
separating: 1
blind: 1
noisy: 0
inverted: 0

coverage-check is blind: in both versions three of three lines ran and the coverage threshold passed. The faulty version’s coverage is the same as the correct version’s, because the fault is not in which line ran but in what that line computes. discount-result is separating: the correct version gives 180, the faulty version gives 198, and the check accepts only 180.

These two lines alone call into question a coverage threshold decision. When a team sets the rule “no merge without a hundred percent coverage”, the faulty version passes this rule. The rule catches missing tests but does not catch wrong tests.

TP1. The model counts a single function with three markers. Real code has thousands of lines and branches; but the difference in a single function is hidden in the same way in a large table.

TP2. The coverage check here looks at line markers. Branch coverage or condition coverage measures more finely, but they too do not measure the correctness of the calculation.

How to Use a Coverage Threshold

The useful way to use coverage is as a floor, not as a goal. A file with low coverage points to code that has never been tried; writing tests there is the beginning of adding outcome checks. High coverage, on the other hand, is not a result. Writing assertion-free tests to raise coverage spoils the measure: the number goes up, the assurance does not.

So when a test is reviewed, two questions are asked. The first: which lines does the test enter. The second: which results does the test check. Coverage answers only the first. For the second, one has to look at the same test applied to the faulty version; this lesson’s separation measure does exactly that.

There is also the problem of false confidence. Teams often put the coverage percentage on a dashboard like a quality indicator. As the indicator rises, the team believes the tests are sufficient and the effort to write new tests drops. The real function of coverage is to make visible the pieces of code the tests never touch.

Configurations and Expectation

The second function of the test plan is configuration. The same test runs in two languages. A money amount is written 180,00 in the Turkish region and 180.00 in the English region; the date format changes in a similar way. If a test wrote the expected text for only one language, even the correct version fails in the other configuration.

In the model, formatting is not a real formatter but a fixed rule for two languages: in Turkish the decimal separator is a comma, in English a dot. In a real app this job is left to the platform’s formatter.

The Bounding Measurement

The measurement below applies the text form of the discount-result check in two configurations. The expected text is written for the Turkish configuration as 180,00. Both versions are formatted separately in each configuration.

// main.swift — same expectation, two configurations
func formattedTotal(_ total: Int, language: String) -> String {
    language == "tr" ? "\(total),00" : "\(total).00"
}

let correctTotal = discountedTotal(correctPrice)
let faultyTotal = discountedTotal(faultyPrice)

let turkish = contrast("tr-configuration", correct: formattedTotal(correctTotal, language: "tr"), faulty: formattedTotal(faultyTotal, language: "tr"), check: { $0 == "180,00" }).describe()
let english = contrast("en-configuration", correct: formattedTotal(correctTotal, language: "en"), faulty: formattedTotal(faultyTotal, language: "en"), check: { $0 == "180,00" }).describe()

countSeparations([turkish, english])

tr-configuration | correct=180,00/pass faulty=198,00/fail -> separating
en-configuration | correct=180.00/fail faulty=198.00/fail -> noisy
separating: 1
blind: 0
noisy: 1
inverted: 0

In the Turkish configuration the check separates. In the English configuration the same check becomes noisy: the correct version also produces 180.00, so it fails. When the plan runs both configurations together, the report is red, and the cause of the red is not a fault. After a while the team gets used to this red and, when a real fault appears, passes over it with the same habit.

The solution is not to write the expectation tied to a language: either the expected text is produced according to the configuration’s language, or the check looks at the numeric value rather than the text. This measurement compares only two configurations; both run with the two versions that carry the same fault.

TP3. We did not measure the percentage of a real coverage report anywhere in this lesson. The model counts the markers that ran itself; a percentage is a product output and is not this lesson’s measure.

TP4. The formatting in the model is a fixed rule for two languages. A real formatter carries more region and currency detail.

Common Mistakes

• Counting the coverage percentage as a quality indicator. The percentage does not say whether the result is correct.
• Writing tests that do not check the result in order to raise coverage.
• Writing the expected text for one language and running the plan in two languages. The second configuration stays red all the time.
• Treating the red in a configuration as normal and not looking at it.
• Adding every test and every configuration to the plan. If the plan does not stay fast, the team does not run it.

Summary

• A test plan gathers test targets and configurations in one file; the same test runs separately in each configuration.
• Coverage counts which line ran; it does not ask whether the result is correct.
• Of two checks, one came out blind and one separating: the coverage check passed on both versions, the outcome check passed only on the correct version.
• When the expectation was written for the Turkish format, the same check became noisy in the second configuration.
• Coverage is a floor; it shows a missing test, it does not show a wrong test.
• This lesson’s separation: blind.

Next Step

Tests check the correctness of code; they do not check its speed. In the next lesson, Profiling Tools will count the work a list cell does on every draw and measure when a check that looks at the amount of work separates.