Release Automation

Every step done by hand after a version is ready is a chance for a mistake: raising the number, producing the archive, signing, uploading. Release automation gathers these steps into a single script and runs them in the same order every time. If the script is written wrongly, the mistake is repeated every time as well; that is why the script itself is checked too.

The measure of this course is called separation: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: separating, where the correct version passes and the faulty version fails; blind, where both pass; noisy, where both fail; and inverted, where the correct version fails and the faulty version passes. In the output they appear as separating, blind, noisy, and inverted. This lesson measures with the core file.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

This lesson’s fault is a small oversight: the upload script does not raise the build number. The version number stays the same, which is normal, but the build number stays the same too. The script runs, the archive is produced, the signing passes, and the upload command runs. No step turns red. The problem surfaces only when the store sees the same number a second time, and by then the upload has already been made.

The Version Number and the Build Number

An application has two numbers. The version number (such as 2.4.0) is visible to the user and says which form of the product it carries. The build number tells apart the successive uploads of the same version: if three uploads were made while the version was 2.4.0, there are three different build numbers. Because the two numbers have different roles, the mistake shows up in different places. Someone who writes the version number wrongly notices at once, because it stands on the screen. Someone who forgets to raise the build number sees nothing.

The script below is the skeleton of an upload step. The tool names are kept generic: the first command is a small helper that raises the number, and the third is the uploader. The script is not measured; the lesson only shows what it does.

./bump-build-number.sh
xcodebuild archive -scheme Shop -archivePath Shop.xcarchive
./upload-to-store.sh Shop.xcarchive

The faulty script lacks the first line. The other two lines are identical. The eye reading the script finds the fault only by searching for the line that slipped away, and to search for the line that slipped away, one must first know that something slipped away. The job of a check is exactly this.

Who Raises the Number

The job of raising the build number can be placed in three spots, and each spot has its own mistake. If it is raised by hand, it is forgotten; this is a special case of the faulty script. If the script raises it, one must be sure the script raises it on every run: if the script runs again after a failed upload, it raises the number a second time and the numbers skip. If the number is derived from the pipeline’s counter, every pipeline run raises the number automatically, but this time the link between the number and the version history loosens: two uploads carry different numbers even if they came from the same change.

No option is free of mistakes, and the choice is not this lesson’s subject. The lesson’s subject is whether the chosen path is checked for whether it actually raised the number. A team that raises it by hand keeps the check as a checklist; a team that raises it with a script puts the check on the script’s output. In both cases the check asks the same question: is the upload number greater than the previous one.

Building the Model

The model holds an upload as two fields: the version number and the build number. prepareUpload imitates the work of the script: it takes the last number in the store, raises it by one in the correct script, and leaves it as it is in the faulty script. hasVersionShape checks that the version number has the x.y.z form, that is, three groups of digits separated by dots. Reading carries an upload together with the last number it is compared against; this will be needed in the last section of the lesson.

// Release.swift — a model of the upload script's version and build numbers
struct Upload: CustomStringConvertible {
    let version: String
    let buildNumber: Int
    var description: String { "\(version)(\(buildNumber))" }
}

struct Reading: CustomStringConvertible {
    let upload: Upload
    let recorded: Int
    var description: String { "\(upload)/recorded=\(recorded)" }
}

func hasVersionShape(_ version: String) -> Bool {
    let parts = version.split(separator: ".", omittingEmptySubsequences: false)
    return parts.count == 3 && parts.allSatisfy { !$0.isEmpty && $0.allSatisfy { $0.isNumber } }
}

func prepareUpload(version: String, lastUploaded: Int, bumpsNumber: Bool) -> Upload {
    Upload(version: version, buildNumber: bumpsNumber ? lastUploaded + 1 : lastUploaded)
}

let releaseVersion = "2.4.0"
let storeLast = 12

The correct script produces version 2.4.0 with build number 13. The faulty script produces 12 with the same version, which repeats the last number in the store.

DS7. This measurement takes as the lesson’s assumption that the store expects the build number for the same version to be greater than the previous one. Which number the store actually accepts or rejects was not measured.

Three Checks

The upload the script produces is tested with three checks. The first looks at whether the version number has the x.y.z form: it is the check most often written in upload scripts, because it is easy to write. The second looks at whether the build number is greater than the last number in the store. The third looks at whether the build number is positive.

// main.swift — three checks, two categories
let withBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: true)
let withoutBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: false)

let shape = contrast("shape-x-y-z", correct: withBump, faulty: withoutBump) { hasVersionShape($0.version) }.describe()
let greater = contrast("number-greater-than-last", correct: withBump, faulty: withoutBump) { $0.buildNumber > storeLast }.describe()
let positive = contrast("number-positive", correct: withBump, faulty: withoutBump) { $0.buildNumber > 0 }.describe()

countSeparations([shape, greater, positive])

shape-x-y-z | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind
number-greater-than-last | correct=2.4.0(13)/pass faulty=2.4.0(12)/fail -> separating
number-positive | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind
separating: 1
blind: 2
noisy: 0
inverted: 0

Three lines, two categories. shape-x-y-z and number-positive give the same answer on both uploads: the version number is 2.4.0 in both uploads, and the build number is positive in both. The fault does not touch these two values, because the fault is a comparison: the number is not wrong, it is equal to the last number in the store. Only number-greater-than-last makes this comparison. On the correct upload 13 > 12 passes, and on the faulty upload 12 > 12 fails: separating.

Why the Format Check Is Common and Why It Is Blind

The format check is common because it is easy to write and it catches many faults: a letter in the version number, a missing group, an extra dot. The faults it catches are real. But what the check looks at is the consistency of a single upload within itself; it does not compare the upload with another upload. A number that does not increase becomes visible only when read together with the previous upload. Looked at alone, 2.4.0(12) is a valid upload, and so is 2.4.0(13).

This is a trait common to blind checks: the check does not look at the value the fault changes. Here the fault changes the relation of the build number to the previous value, while the x.y.z format looks only at the version number. A green format check does not carry the sentence “the numbers are correct”; it carries only “the version number is well formed”. Writing “numbers are valid” in the script’s report without separating these two sentences puts the first in the place of the second.

The number-positive check is weaker still. It asks for the build number to be greater than zero; the faulty number is 12 as well. A weak check does not mean a bad check: against another fault that produces a zero or a negative number, this check would be separating. The distinction belongs to the pair of check and fault; when the fault changes, the table changes too.

The Data the Greater-Than Check Depends On

number-greater-than-last is the separating one, but in this measurement the storeLast value was taken as if it came directly from the store. In a real script, where the last number is read from is a separate question, and the check is as reliable as that source is accurate. There are three common sources. One is the store itself: the script asks for the last number before uploading. One is a local file: the team keeps the last uploaded number in a file and updates it with each upload. One is the version control history: the number is read from tags. Each ages in a different way.

The local file is the easiest and the one that causes the most trouble. The file is written in the step that raises the number. In the faulty script that step does not exist, so the file is not written either. The file holds a number left over from an earlier upload, older even than the one in the store. If the check compares against this old number, the faulty upload looks greater than the old number and passes. The correct upload, on the other hand, has updated the file itself; the new number is equal to itself, and > fails. This is the flipping of a separating check into its opposite.

DS8. This measurement assumes that the local record is updated only in the step that raises the number. When and how a real script writes the record was not measured.

The Bounding Measurement

The same check, the build number being greater than the last number, runs with three different sources of the last number. The first source is the store: the last number is 12 on both uploads. The second source is a file written after the upload: both scripts write the number they uploaded to the file, so the file carries its own number on each upload. The third source is a file updated only in the step that raises the number: the correct script writes 13, the faulty script does not write, and the file stays at 11.

// main.swift — the same greater-than check, three sources of the last number
let withBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: true)
let withoutBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: false)

let fromStore = contrast("last-number-store",
    correct: Reading(upload: withBump, recorded: storeLast),
    faulty: Reading(upload: withoutBump, recorded: storeLast)) { $0.upload.buildNumber > $0.recorded }.describe()
let afterUpload = contrast("last-number-file-after-upload",
    correct: Reading(upload: withBump, recorded: withBump.buildNumber),
    faulty: Reading(upload: withoutBump, recorded: withoutBump.buildNumber)) { $0.upload.buildNumber > $0.recorded }.describe()
let staleFile = contrast("last-number-file-in-bump-step",
    correct: Reading(upload: withBump, recorded: withBump.buildNumber),
    faulty: Reading(upload: withoutBump, recorded: 11)) { $0.upload.buildNumber > $0.recorded }.describe()
let shapeAgain = contrast("shape-x-y-z", correct: withBump, faulty: withoutBump) { hasVersionShape($0.version) }.describe()

countSeparations([fromStore, afterUpload, staleFile, shapeAgain])

last-number-store | correct=2.4.0(13)/recorded=12/pass faulty=2.4.0(12)/recorded=12/fail -> separating
last-number-file-after-upload | correct=2.4.0(13)/recorded=13/fail faulty=2.4.0(12)/recorded=12/fail -> noisy
last-number-file-in-bump-step | correct=2.4.0(13)/recorded=13/fail faulty=2.4.0(12)/recorded=11/pass -> inverted
shape-x-y-z | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind
separating: 1
blind: 1
noisy: 1
inverted: 1

The same comparison, buildNumber > recorded, fell into three different categories with three sources, and with the format check on the fourth line all four categories appear. In the first line the source tells the truth: separating. In the second line the file was written after the upload; since each upload equals its own number, both versions fail: noisy. A red check gives the impression that the script broke the upload, yet what broke it is the comparison itself. In the third line the correct upload fails and the faulty upload passes: inverted.

This third line explains why a team sometimes switches off a check it once trusted. The check is red on the correct upload, the team counts it as a false alarm, and the threshold is loosened: >= is written instead of >. That way both the faulty and the correct upload pass, and the check goes blind. Instead of loosening a check that seems to raise false alarms, one must examine its source; > was written correctly in this lesson, and the fault was in the source.

The limit of this measurement must be stated openly. The upload and the store’s rejection were not measured; the script was shown only as source, and the model compared not the result of an upload but the two numbers that lead to it. Whether the store really rejects a second identical number, with what message, and how quickly, remains outside this lesson.

DS9. In this measurement, the file’s 11 value is a previous record that the faulty script did not write, and it is older than the last number in the store. How old a stale record can be in a real script was not measured.

What the Script Output Says

The summary printed at the end of the script is the only place where the team reads about the upload. A script that writes “version 2.4.0, build 12, uploaded” in the summary shows the faulty upload as clean as the correct one. For the summary to carry separating information, it is enough to write the number together with the previous number: “build 12, previous 12”. A team that sees these two numbers side by side notices the number that does not increase even if the script ran no check at all.

The same principle holds elsewhere. A number alone has little meaning; if the number it is compared against is written down, the check becomes readable. For that reason the upload record should carry not only the latest state but also the previous one. A record that does not carry the previous state silently produces a wrong comparison when it is stale or unwritten, as in the last measurement of this lesson.

Finally, recall the question to ask when writing a new check: what does this check say about the upload the faulty script produces? To answer it, it is enough to break the script on purpose; deleting the first line is the first mutation test. If a check does not fail on the first attempt, it is blind or inverted, and, as the lesson showed, the cause usually lies in the data the check looks at.

Summary

• The version number is visible to the user, and the build number tells apart successive uploads of the same version; a build number that does not increase turns no step red.
• Checks that look only at the x.y.z format or at whether the number is positive pass on both uploads: blind.
• The check on whether the build number is greater than the last number in the store separates the faulty upload: separating.
• The same greater-than check becomes noisy if the last number is read from a file written after the upload, and inverted if it is read from a stale file updated only in the step that raises the number.
• Before loosening a red greater-than check, the source of the last number is examined.
• This lesson’s separation: separating.

Next Step

An upload whose number is raised correctly still does not have to be opened to everyone. In the next lesson, Beta Distribution, we will measure opening a new feature to a limited audience, and a flag that stays open outside that audience as well.