Continuous Integration and Delivery

The previous lesson gave the version identity by hand: an integer in a build script. Someone has to increase this number on every change, and someone has to produce and test every build. If the work is done by hand it gets skipped, and the skipped work is noticed only when a user runs into a problem. Continuous integration ties this work to a pipeline: every time a change arrives, the build and the tests run on their own. Continuous delivery extends the end of the same pipeline: the build that passes the tests waits as an output ready for distribution.

When the pipeline works well, a change comes back green or red and nobody asks what ran. The question appears when the pipeline grows and every change starts waiting a long time. So in this lesson we adapt the course’s question to the pipeline: in how many samples does this measurement exceed its budget? The measurement is the number of jobs that run for a change, the budget is the number of jobs the team accepts per change, and each change is a sample.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

Overrun counts the samples that strictly exceed the limit and gives how widespread the overrun is with four names: within, minority, majority, all. The rule sees how many changes exceed the limit; whether a change ran six jobs or seven is the same for every sample that overruns.

Pipeline, Job and Filter

A pipeline (pipeline) is built from jobs. A job (job) is a step with a single purpose: static analysis, the unit tests of a module, building the app, a test that runs on a device. Each job says which modules it depends on. The change filter (change filter) uses this link: if a change touched only the data module, a job unrelated to data does not need to run.

# Platform code: workflow definition
on:
  pull_request:
    paths:
      - "feature/**"
      - "core/**"
jobs:
  unit-feature:
    steps:
      - run: ./build.sh :feature:test

This fragment is a workflow definition; it is not run and we claim no output here. The model sets up the rule of this measurement; the model is not the tool itself. In the model the filter is a set intersection: a job runs if one of the modules it depends on has changed. A job that depends on no module, that is, one whose dependency list is empty, runs on every change.

In the fragment above, the paths field is the filter: only a change that touches files under feature or core starts this workflow. The filter has two faces. Its benefit is that it does not run the job at all for unrelated changes. Its risk is that the paths are written by hand: if a new module is added and not written into the paths, a change in that module triggers no job. In the model, this risk is carried by the modules set in the Job class; if the set is written incompletely the job does not run and the number drops.

// PipelineModel.kt — models the jobs, the changes and the change filter
class Job(val name: String, val modules: Set<String>)

class Change(val label: String, val modules: Set<String>)

fun runs(job: Job, change: Change): Boolean =
    job.modules.isEmpty() || job.modules.any { it in change.modules }

fun jobsForChange(jobs: List<Job>, change: Change): List<String> =
    jobs.filter { runs(it, change) }.map { it.name }

fun pipelineJobs(): List<Job> = listOf(
    Job("checks", emptySet()),
    Job("unit-core", setOf("core")),
    Job("unit-data", setOf("data")),
    Job("unit-feature", setOf("feature", "core")),
    Job("unit-app", setOf("app", "core")),
    Job("device-test", setOf("app", "feature")),
    Job("build", emptySet()),
)

fun sampleChanges(): List<Change> = listOf(
    Change("comment", emptySet()),
    Change("data", setOf("data")),
    Change("feature", setOf("feature")),
    Change("app", setOf("app")),
    Change("data-feature", setOf("data", "feature")),
    Change("app-data", setOf("app", "data")),
    Change("core", setOf("core")),
    Change("typo-fix", emptySet()),
)

There are seven jobs and eight changes. checks and build run on every change because their module list is empty; the other five jobs run only when the module they depend on changes. Two of the eight changes are only a comment and a typo fix, and touch no module.

Jobs per Change

We set the budget at four jobs: the team does not want a change to wait for more than four jobs. The measurement is the size of the jobsForChange result for each change.

// Main.kt — the number of jobs that run for each change is measured, the budget is four jobs
fun main() {
    val jobs = pipelineJobs()
    val changes = sampleChanges()
    for (change in changes) {
        println("${change.label}: ${jobsForChange(jobs, change)}")
    }
    val samples = changes.map { jobsForChange(jobs, it).size }
    println("running jobs: $samples")
    Overrun("jobs-per-change", Budget("job", 4), samples).line()
}

comment: [checks, build]
data: [checks, unit-data, build]
feature: [checks, unit-feature, device-test, build]
app: [checks, unit-app, device-test, build]
data-feature: [checks, unit-data, unit-feature, device-test, build]
app-data: [checks, unit-data, unit-app, device-test, build]
core: [checks, unit-core, unit-feature, unit-app, build]
typo-fix: [checks, build]
running jobs: [2, 3, 4, 4, 5, 5, 5, 2]
jobs-per-change: budget=job<=4 | over=3/8 -> minority

Three changes exceed four jobs: two changes that touch two modules and one change that touches the shared module. Because fewer than half of the eight exceeded, the answer is minority. This is the lesson’s overrun. The filter keeps most changes within the budget; the samples that overrun are the weak point of the pipeline.

The core line shows one more thing: only core changed, but core is in the dependencies of three jobs, so this change triggered the widest run even though it touched a single module. The shared module behaves like a junction inside the pipeline: the more modules depend on it, the more jobs a change in that module runs.

DI3. The model counts the number of jobs and weighs every job equally. In a real pipeline a unit test can take very little time and a device test very long; the model does not set up this difference. The module links are also an input: in reality the links come from the dependencies in the build script, and here they are written by hand.

Transitive Dependency

So far we have written by hand the modules each job depends on. In reality, when a module changes, the modules that depend on it are affected as well: when core changes, data and feature, which rely on core, and app, which relies on them, count as changed too. The links sit in a graph, and the set a change affects is derived from the graph. The measurement below counts the same eight changes in two ways. The first uses only the modules that changed directly. The second spreads the change to the dependent modules. This time the jobs are a separate unit test for each module; checks and build still run on every change.

// Main.kt — when the transitive dependency is taken into account, the same changes run more jobs
val dependsOn = mapOf(
    "data" to setOf("core"),
    "feature" to setOf("core"),
    "app" to setOf("feature", "data"),
)

fun affected(changed: Set<String>): Set<String> {
    var result = changed
    while (true) {
        val next = result + dependsOn.filter { (_, deps) -> deps.any { it in result } }.keys
        if (next == result) return result
        result = next
    }
}

fun main() {
    val jobs = listOf(
        Job("checks", emptySet()),
        Job("unit-core", setOf("core")),
        Job("unit-data", setOf("data")),
        Job("unit-feature", setOf("feature")),
        Job("unit-app", setOf("app")),
        Job("build", emptySet()),
    )
    val changes = sampleChanges()
    val direct = changes.map { jobsForChange(jobs, it).size }
    val spread = changes.map { jobsForChange(jobs, Change(it.label, affected(it.modules))).size }
    println("direct jobs: $direct")
    println("transitive jobs: $spread")
    val budget = Budget("job", 4)
    Overrun("direct", budget, direct).line()
    Overrun("transitive", budget, spread).line()
}

direct jobs: [2, 3, 3, 3, 4, 4, 3, 2]
transitive jobs: [2, 4, 4, 3, 5, 4, 6, 2]
direct: budget=job<=4 | over=0/8 -> within
transitive: budget=job<=4 | over=2/8 -> minority

The same changes under the same budget came out within with the direct count and minority with the transitive count. The difference is largest in the core line: three jobs in the direct count, six jobs in the transitive count, because when core changed, all four modules were affected. We said the shared module is a junction; this measurement derives its cost from the graph. The direct count makes the pipeline look cheaper than it really is.

DI5. The model takes the dependency graph as a table written by hand and assumes it contains no cycle. A real build system derives the graph from the scripts; there is no derivation here. The model also runs the tests of every affected module; a real pipeline may choose to run only in the modules whose interface changed, and the model does not set this up.

The Budget Is a Decision

We measure the same eight changes with four separate budgets. When the budget changes, the overrun changes; this is not a contradiction, it is the nature of the measurement.

// Main.kt — the same eight changes give four different overruns under four budgets
fun main() {
    val jobs = pipelineJobs()
    val samples = sampleChanges().map { jobsForChange(jobs, it).size }
    val roomy = Overrun("five-jobs", Budget("job", 5), samples)
    val normal = Overrun("four-jobs", Budget("job", 4), samples)
    val tight = Overrun("three-jobs", Budget("job", 3), samples)
    val minimal = Overrun("one-job", Budget("job", 1), samples)
    listOf(roomy, normal, tight, minimal).forEach { it.line() }
    countOverruns(listOf(roomy, normal, tight, minimal))
}

five-jobs: budget=job<=5 | over=0/8 -> within
four-jobs: budget=job<=4 | over=3/8 -> minority
three-jobs: budget=job<=3 | over=5/8 -> majority
one-job: budget=job<=1 | over=8/8 -> all
within: 1
minority: 1
majority: 1
all: 1

The four budgets gave four different categories. Under the budget of five jobs no change exceeded, because the widest run was five jobs. Under the budget of one job, all eight changes exceeded, because checks and build run on every change. A lesson follows from this: the number of jobs that run on every change is a floor the filter can never go below. If the budget is set below that floor, the answer is all no matter which filter is built.

The Delivery End

The delivery end of the pipeline asks a different question from integration: is this build ready for distribution? Here signing and packaging come into play. A signed build is an output signed with the release key; the signing key is a secret input of the pipeline and is kept apart from the jobs that run on ordinary changes. For this reason the delivery step runs not on every change but only when a release tag arrives.

// Platform code: signed build
android {
    signingConfigs {
        create("release") {
            storeFile = file(providers.gradleProperty("keystorePath").get())
            keyAlias = providers.gradleProperty("keyAlias").get()
        }
    }
}

The key path and alias are not embedded in the build script; the pipeline supplies them from outside. This fragment has no output; it only shows that the signature is a release input. The job at the delivery end does not enter the budget: a model that counts it works per release, not per change, and the lesson does not measure it.

The Bounding Measurement

This section measures the value of the filter by removing the filter altogether, so that we can see whether the budget is reachable. We remove the filter. All the jobs of the pipeline are written so that they cover all modules: every job runs on every change. This is the easiest pipeline to set up and the first form of most teams.

// Main.kt — an unfiltered pipeline runs every job on every change
fun main() {
    val everyModule = setOf("app", "core", "data", "feature")
    val unfiltered = pipelineJobs().map { Job(it.name, emptySet()) }
    val filtered = pipelineJobs()
    val changes = sampleChanges()
    val budget = Budget("job", 4)

    val withFilter = Overrun("filtered", budget, changes.map { jobsForChange(filtered, it).size })
    val noFilter = Overrun("unfiltered", budget, changes.map { jobsForChange(unfiltered, it).size })
    listOf(withFilter, noFilter).forEach { it.line() }
    println("module count: ${everyModule.size}, job count: ${filtered.size}")
}

filtered: budget=job<=4 | over=3/8 -> minority
unfiltered: budget=job<=4 | over=8/8 -> all
module count: 4, job count: 7

The filtered pipeline came out minority under the same budget and the unfiltered pipeline all. In the unfiltered pipeline all the jobs are written with an empty dependency list, so seven jobs run on every change, and eight of eight changes exceed four jobs. Even a comment fix runs seven jobs.

Here the limit of the rule appears. The rule does not know that the filter prevented a job from running; it sees only the number. If the filter is set up wrongly, that is, if a needed job does not run because a dependency was written incompletely, the number of jobs falls and the pipeline looks better, although a change that should have been tested was not tested. The number of overruns does not measure the correctness of the pipeline; it measures only its cost.

DI4. The model assumes the filter is set up correctly. The filter being set up wrongly, that is, a dependency being forgotten, is not in the model; to see it, it is necessary to verify separately, for each job, on which changes it should run.

Common Misconceptions

“Continuous integration and continuous delivery are the same thing.” Integration builds and tests every change. Delivery keeps the build that passes the tests ready for distribution. In this lesson the second was not measured, because the delivery step runs per release, not per change.

“If a filter is set up, the pipeline has become cheaper.” The filter only reduces the number of jobs that run. The difference between the direct count and the transitive count showed that a wrongly set up filter can make the pipeline look cheaper than it really is. The correctness of the filter is checked separately.

“Fewer jobs is always a better pipeline.” The rule counts the number of jobs; it does not see which job was skipped. If a needed job was skipped, the number drops but the pipeline gets worse.

“The faster the pipeline, the better.” This lesson did not measure time; it counted only the number of jobs that run on a change. The number of jobs does not give the time, because jobs can run in parallel and one job’s time can differ greatly from another’s. A claim about time needs a separate measurement.

“The dependency graph is derived once and does not change.” The dependencies between modules are born again with every change. If the graph is kept by hand it becomes stale and the transitive count is silently wrong; this lesson’s model took the graph as an input written by hand, and in a real pipeline the graph must be read from the configuration.

“Every change affects every module.” It does not. An edit that changes only one module does not force the modules that do not depend on it to run; the job of the filter is to make this distinction.

Summary

• This lesson’s overrun: minority. Three of eight changes exceeded the budget of four jobs; because fewer than half exceeded, the answer is minority.
• Continuous integration runs the build and the tests on every change, and continuous delivery extends the same pipeline to an output ready for distribution.
• The filter runs only the jobs whose module has changed; a change that touched the shared module triggered three jobs at once and gave the widest run.
• The same changes under the budgets of five, four, three and one job came out within, minority, majority and all in turn; the two jobs that run on every change set the floor.
• When the dependency graph was taken into account, the same changes ran more jobs: the direct count came out within and the transitive count minority; the direct count makes the pipeline look cheap.
• The unfiltered pipeline exceeded the budget of four jobs on eight of eight changes: all. The rule counts the cost; it cannot count the correctness of the filter.

Next Step

The pipeline tested a build and left it ready for distribution. Before a build reaches real users, there are paths it passes through: internal testing, closed testing, open testing. The next lesson, Testing Tracks, measures in how many steps a build passes through these paths.