Continuous Integration

The idea that the same steps run in the same order every time a change is made, tied to nobody’s machine, is called continuous integration. The steps are written as a pipeline: build first, then test, then package. If the pipeline fails on a change, the change stops before it reaches the team. It is a frame that also holds the previous lesson’s signing checks; the signing step can sit at the end of the pipeline, and in this lesson it stays outside the pipeline model.

The measure of this course is called separation: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: separating, where the correct version passes and the faulty version fails; blind, where both pass; noisy, where both fail; and inverted, where the correct version fails and the faulty version passes. In the output they appear as separating, blind, noisy, and inverted. This lesson measures with the core file.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

This lesson’s fault is in the basket total. When the total of a three-item basket is computed, the last item is silently skipped: the correct version adds up all the items, and the faulty version leaves out the last one. The computation raises no error and does not crash; it only returns a number that is too small. What finds a fault of this kind is usually a test, and a test finds it only if it runs. The question is whether the pipeline actually runs the test.

The Pipeline Model

The model runs a pipeline step by step. Each step gets one of three results: pass, fail, or skipped. The pipeline tries the steps in order; it stops at the first step that gets fail, and all the remaining steps are marked skipped. This matches the behavior of real pipelines: if the build has failed, there is no point in running tests, and the pipeline does not spend time.

There are three steps. The build step looks at the target’s required setting; if the setting is missing, the build fails. The test step compares the basket total with the expected value. The package step checks that the total is greater than zero and stands here only as the last link of the pipeline. The settings are of the Settings type; its single field says whether the target setting is present.

// Pipeline.swift — a pipeline model that runs steps in order and stops at the first failing step
struct Settings {
    let hasTargetSetting: Bool
}

let pipelineSteps = ["build", "test", "package"]

func runStep(_ name: String, total: Int, expected: Int, settings: Settings) -> Bool {
    switch name {
    case "build": return settings.hasTargetSetting
    case "test": return total == expected
    case "package": return total > 0
    default: return false
    }
}

func runPipeline(total: Int, expected: Int, settings: Settings, steps: [String] = pipelineSteps) -> [String] {
    var outcomes: [String] = []
    var stopped = false
    for name in steps {
        if stopped {
            outcomes.append("\(name):skipped")
        } else if runStep(name, total: total, expected: expected, settings: settings) {
            outcomes.append("\(name):pass")
        } else {
            outcomes.append("\(name):fail")
            stopped = true
        }
    }
    return outcomes
}

func outcome(of step: String, in outcomes: [String]) -> String {
    for entry in outcomes where entry.hasPrefix(step + ":") {
        return String(entry.dropFirst(step.count + 1))
    }
    return "none"
}

func trace(_ outcomes: [String]) -> String {
    outcomes.joined(separator: " ")
}

func stepContrast(_ label: String, step: String, correct: [String], faulty: [String]) -> String {
    let onCorrect = outcome(of: step, in: correct)
    let onFaulty = outcome(of: step, in: faulty)
    if onCorrect == "skipped" || onFaulty == "skipped" {
        print("\(label) | correct=\(onCorrect) faulty=\(onFaulty) -> outside-category")
        return "outside-category"
    }
    return contrast(label, correct: onCorrect, faulty: onFaulty) { $0 == "pass" }.describe()
}

func countOutside(_ results: [String]) {
    print("outside-category: \(results.filter { $0 == "outside-category" }.count)")
}

stepContrast compares the result of a single step for the two versions, but it does not classify when the step did not run in one of the two versions. Why it is done this way is explained in a separate section; for now it is enough to know that the output says outside-category. This is the lesson’s own rule: the four categories of separation stay in the core, and outside-category is written only in this lesson, when a step did not run.

The basket and the expected value live in a separate file. The correct function adds up all the items, and the faulty function skips the last one. The expected total is the correct version’s value, and the test step takes it from there.

// Basket.swift — the lesson's basket and its two total functions
let basketPrices = [40, 25, 15]
let expectedTotal = 80

func basketTotal(_ prices: [Int]) -> Int {
    prices.reduce(0, +)
}

func faultyBasketTotal(_ prices: [Int]) -> Int {
    prices.dropLast().reduce(0, +)
}

DS4. This measurement models the pipeline as a single job that runs three steps in sequence. The behavior of parallel steps and of the dependency graph in real pipelines was not measured.

The Pipeline on a Clean Machine

The pipeline runs for the first time on a clean machine. A clean machine is an environment that carries none of the settings and local cache a developer’s machine has accumulated. On the developer’s machine, the build setting was made once and forgotten; it was never written into the pipeline definition. On the clean machine, that setting does not exist.

DS5. This measurement assumes that the clean environment carries no local cache or developer settings. Which settings a real pipeline environment brings along on its own was not measured.

The pipeline is applied to the two versions, and three checks run: that the pipeline as a whole is green, that the build step passes, and that the test step passes.

// main.swift — three checks on a clean machine
let cleanMachine = Settings(hasTargetSetting: false)
let correctRun = runPipeline(total: basketTotal(basketPrices), expected: expectedTotal, settings: cleanMachine)
let faultyRun = runPipeline(total: faultyBasketTotal(basketPrices), expected: expectedTotal, settings: cleanMachine)

let allGreen = contrast("pipeline-green", correct: trace(correctRun), faulty: trace(faultyRun)) { !$0.contains("fail") && !$0.contains("skipped") }.describe()
let buildStep = stepContrast("build-step-passed", step: "build", correct: correctRun, faulty: faultyRun)
let testStep = stepContrast("test-step-passed", step: "test", correct: correctRun, faulty: faultyRun)

countSeparations([allGreen, buildStep, testStep])
countOutside([allGreen, buildStep, testStep])

pipeline-green | correct=build:fail test:skipped package:skipped/fail faulty=build:fail test:skipped package:skipped/fail -> noisy
build-step-passed | correct=fail/fail faulty=fail/fail -> noisy
test-step-passed | correct=skipped faulty=skipped -> outside-category
separating: 0
blind: 0
noisy: 2
inverted: 0
outside-category: 1

On both versions the pipeline stops in the same place: build:fail. The pipeline-green check, which looks at the pipeline as a whole, fails on both versions, so it is noisy. The check that looks at the build step itself is the same: fail on both versions. The pipeline is red, and the fault has no share in that; a missing setting stopped the pipeline before the basket total was ever computed.

The practical consequence is harmful. When the pipeline turns red, the team first repairs the red, because no change can move forward while the pipeline is red. The cause of the red is the setting, and the repair is adding the setting. But during the time until the setting is added, nobody knows whether the pipeline would have seen the fault; the pipeline never reached the fault. A red pipeline is very easy to read as “the fault was caught”, while the measurement says the opposite: whether the fault was caught does not come out of this run.

The third line, test-step-passed, was not classified. The test step did not run on either version, because the pipeline stopped before it. The output says correct=skipped faulty=skipped and the result is outside-category. This line is not separating, not blind, not noisy, and not inverted; the check ran on neither version, and a check that did not run has no answer for the two versions.

Once the Setting Is Fixed

The setting is added to the pipeline definition, and the clean machine now passes the build. The same three checks, plus a fourth that looks at the package step, are applied to the same two versions.

// main.swift — four checks after the setting is fixed
let fixedMachine = Settings(hasTargetSetting: true)
let correctRun = runPipeline(total: basketTotal(basketPrices), expected: expectedTotal, settings: fixedMachine)
let faultyRun = runPipeline(total: faultyBasketTotal(basketPrices), expected: expectedTotal, settings: fixedMachine)

let allGreen = contrast("pipeline-green", correct: trace(correctRun), faulty: trace(faultyRun)) { !$0.contains("fail") && !$0.contains("skipped") }.describe()
let buildStep = stepContrast("build-step-passed", step: "build", correct: correctRun, faulty: faultyRun)
let testStep = stepContrast("test-step-passed", step: "test", correct: correctRun, faulty: faultyRun)
let packageStep = stepContrast("package-step-passed", step: "package", correct: correctRun, faulty: faultyRun)

countSeparations([allGreen, buildStep, testStep, packageStep])
countOutside([allGreen, buildStep, testStep, packageStep])

pipeline-green | correct=build:pass test:pass package:pass/pass faulty=build:pass test:fail package:skipped/fail -> separating
build-step-passed | correct=pass/pass faulty=pass/pass -> blind
test-step-passed | correct=pass/pass faulty=fail/fail -> separating
package-step-passed | correct=pass faulty=skipped -> outside-category
separating: 2
blind: 1
noisy: 0
inverted: 0
outside-category: 1

Once the setting is fixed, the table changes. test-step-passed passes on the correct version and fails on the faulty version: separating. pipeline-green, which looks at the pipeline as a whole, is also separating, because the pipeline is red when the test fails. build-step-passed passes on both versions, blind: the fault does not touch the build, and the build finishes clean on both versions. The fourth line is again outside-category.

The difference between the first measurement and the second is a single setting. The code is the same, the basket is the same, the test is the same; but the pipeline-green check moved from noisy to separating. This shows that separation is a function not only of the check or the code but also of the environment. The same pipeline, as long as the setting is missing, says nothing about the fault.

Why a Step That Did Not Run Is Not a Category

Returning to the fourth line: package-step-passed. On the correct version the packaging passed; on the faulty version it did not run. If the core classified this pair, pass and skipped, directly, it would count the check as separating: the correct version passed, the faulty version “could not pass”. But the packaging step never saw the fault. In the faulty version the basket total is too small, and the total > 0 check of the package step would have passed with the too-small total as well; the step did not run only because the pipeline stopped at the test.

This difference matters, because otherwise every step at the end of the pipeline would look separating. Packaging, signing, upload: each sits after the pipeline’s first failing step, so each would count as separating whenever the fault is caught earlier in the pipeline. But to separate is for a check to give a different answer to the faulty version on its own. A step that did not run has no answer; that answer is determined by the pipeline’s first failing step.

The distinction the lesson makes is this. If the step ran on both versions, the result falls into one of the four categories. If it did not run on one of them, outside-category is written and counted separately. The last line of the output gives this number; so “how many checks were classified” and “how many checks never ran” are not mixed up. If thirty of the forty steps of a green pipeline did not run, that pipeline is as silent as thirty checks.

The Pipeline Definition

A pipeline definition is a configuration file and usually lives in the repository like code. The notation below is a generic form; these are not the real key names of any service, and this lesson does not measure the pipeline definition. What is measured is the pipeline model.

steps:
  - name: build
    run: xcodebuild build -scheme Shop
  - name: test
    run: xcodebuild test -scheme Shop
  - name: package
    run: xcodebuild archive -scheme Shop

The definition looks like a three-line list and matches the model’s pipelineSteps array one to one. What matters is the order of the list and whether each step stops the pipeline on failure. Most pipeline definitions do this by default; some can be configured to continue even when a step fails. In a pipeline that continues there is no skipped, and what every step says about the two versions can be seen. This is a trade-off: the pipeline takes longer and produces more output, but the outside-category lines turn into lines that can be classified.

If a setting is missing from the pipeline definition, the gap becomes visible only on the first clean machine. That is why it is a good habit to run the pipeline’s first run with a deliberately faulty version as well: can the pipeline actually reach the test step, and if it does, does it fail on the faulty version. This single trial catches, while the pipeline is being built, a check problem that would otherwise stay inconclusive while the pipeline is green.

The Bounding Measurement

Separation also depends on the order of the steps. In the measurement below, the same clean machine, that is, the same missing setting, runs in two different orders. In the first order, the build comes first; in the second, the test. When the order changes, the test step is reached; but the missing setting still breaks the build step. The fifth check looks at the build step once the setting is fixed.

// main.swift — same machine, different step order, same check
let cleanMachine = Settings(hasTargetSetting: false)
let fixedMachine = Settings(hasTargetSetting: true)
let testFirst = ["test", "build", "package"]

let correctTotalNow = basketTotal(basketPrices)
let faultyTotalNow = faultyBasketTotal(basketPrices)

let buildFirstClean = (
    runPipeline(total: correctTotalNow, expected: expectedTotal, settings: cleanMachine),
    runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: cleanMachine))
let testFirstClean = (
    runPipeline(total: correctTotalNow, expected: expectedTotal, settings: cleanMachine, steps: testFirst),
    runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: cleanMachine, steps: testFirst))
let buildFirstFixed = (
    runPipeline(total: correctTotalNow, expected: expectedTotal, settings: fixedMachine),
    runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: fixedMachine))

let first = stepContrast("build-first-test", step: "test", correct: buildFirstClean.0, faulty: buildFirstClean.1)
let second = stepContrast("test-first-test", step: "test", correct: testFirstClean.0, faulty: testFirstClean.1)
let third = stepContrast("setting-fixed-test", step: "test", correct: buildFirstFixed.0, faulty: buildFirstFixed.1)
let fourth = contrast("test-first-pipeline", correct: trace(testFirstClean.0), faulty: trace(testFirstClean.1)) { !$0.contains("fail") }.describe()
let fifth = stepContrast("setting-fixed-build", step: "build", correct: buildFirstFixed.0, faulty: buildFirstFixed.1)

countSeparations([first, second, third, fourth, fifth])
countOutside([first, second, third, fourth, fifth])

build-first-test | correct=skipped faulty=skipped -> outside-category
test-first-test | correct=pass/pass faulty=fail/fail -> separating
setting-fixed-test | correct=pass/pass faulty=fail/fail -> separating
test-first-pipeline | correct=test:pass build:fail package:skipped/fail faulty=test:fail build:skipped package:skipped/fail -> noisy
setting-fixed-build | correct=pass/pass faulty=pass/pass -> blind
separating: 2
blind: 1
noisy: 1
inverted: 0
outside-category: 1

The same test-step check, the same fault, three different answers: build-first-test is not classified, while test-first-test and setting-fixed-test separate. Whether the test step is seen while the missing setting stands depends entirely on the step order. A pipeline that runs the test first catches the fault even when the setting is missing. But test-first-pipeline, which looks at the pipeline as a whole, still stays noisy: the build breaks in the next step, and the pipeline is red on both versions. To see that the test caught the fault, one has to look not at the pipeline result but at the step’s own result.

These two observations lean on each other. The order determines separation; but choosing the order for the sake of separation is a bad idea, because the pipeline cannot test without a build. In a real application, a test needs the built output, and the order change in this measurement is possible only in the model. The lesson’s takeaway is that when writing a pipeline definition, besides asking whether the step is “in a position to see the fault”, one must also ask whether the fault is reachable.

DS6. This measurement treats the order change as free in the model; the test step runs without depending on the build. The test step’s dependence on the build output in a real pipeline was not measured.

Summary

• Continuous integration runs the same steps in order in a clean environment for each change and stops at the first failing step.
• On a clean machine, a missing build setting breaks the pipeline on both versions. The checks that look at the pipeline as a whole and at the build step become noisy.
• Once the setting is fixed, the test step passes on the correct version and fails on the faulty version: separating. The build step passes on both versions: blind.
• Steps that did not run after the pipeline stopped are not classified; in this lesson they are counted separately and written as outside-category.
• Separation depends not only on the check but also on the environment and the order of the steps; a red pipeline does not mean the fault was caught.
• This lesson’s separation: noisy.

Next Step

A pipeline whose test step catches the fault also reaches the steps that upload a version to the store. In the next lesson, Release Automation, we will measure which checks separate the upload script that raises the version and build numbers.