Beta Distribution

A build can be given to a small audience before it is opened to everyone. This audience is the team itself, invited outside testers, or both. Through the platform’s beta distribution service, the build reaches the devices of the invited users, and they try it in real use. The goal is to find the problem before it reaches everyone.

A build reaching an audience is not the same thing as a feature being open to that audience. The build is the same for everyone; a new feature, however, can arrive switched off behind a feature flag and be turned on only for certain users. A flag opened beyond the beta audience removes the very reason beta distribution exists.

The measure of this course is called separation: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: separating, where the correct version passes and the faulty version fails; blind, where both pass; noisy, where both fail; and inverted, where the correct version fails and the faulty version passes. In the output they appear as separating, blind, noisy, and inverted. This lesson measures with the core file.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

This lesson’s fault is in the flag’s configuration. In the correct version, the new feature is on only for the users on an allowlist. In the faulty version, the configuration was left as “open to everyone” and the allowlist is never read. Nothing changes for beta testers: the feature is on for them and it works. The difference shows only for users outside the beta.

The Flag Model

The flag consists of two values: whether it is open to everyone, and an allowlist. isOn says whether the flag is on for a user: on if the flag is open to everyone, otherwise on if the user is in the allowlist. exposure writes the state of each user in a list into a single string, so the answers of the two versions can be read side by side.

The configuration usually sits in a file that looks like the one below. The key names are not those of a real service; the lesson only shows the form.

flags:
  new-checkout:
    enabled_for: beta
    allowlist: [u01, u02, u03]

In the faulty version the enabled_for value was left as everyone; the allowlist is in place but nobody looks at it. An eye that reads the configuration notices this, because it is a single value. For a check that does not read it and only looks at behavior, the difference can stay invisible.

// Beta.swift — the feature flag's allowlist and user states
struct Rollout {
    let everyone: Bool
    let allowlist: Set<String>
}

func isOn(_ user: String, rollout: Rollout) -> Bool {
    rollout.everyone || rollout.allowlist.contains(user)
}

func exposure(of users: [String], rollout: Rollout) -> String {
    users.map { "\($0):\(isOn($0, rollout: rollout) ? "on" : "off")" }.joined(separator: " ")
}

let allowlist: Set<String> = ["u01", "u02", "u03"]
let betaTesters = ["u01", "u02", "u03"]
let newTester = ["u04"]
let outsiders = ["u10", "u11"]
let enabledForBeta = Rollout(everyone: false, allowlist: allowlist)
let enabledForAll = Rollout(everyone: true, allowlist: allowlist)

The lesson’s users are written with short identifiers: three beta testers, one newly invited beta tester, and two non-beta users. The newly invited user exists in the beta service but has not yet been added to the allowlist; why this matters will show in the bounding measurement.

DS10. This measurement represents users with a small list of identifiers. The size of the list says nothing about the size of the audience, and how a real beta audience is chosen was not measured.

Three Checks

Three checks are applied to the flag. The first is the most natural: is the feature on for beta testers. The second looks at whether the feature is off for non-beta users. The third looks not at behavior but at configuration: is the flag not open to everyone.

// main.swift — three checks, two categories
let onBeta = contrast("on-for-beta-testers",
    correct: exposure(of: betaTesters, rollout: enabledForBeta),
    faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("off") }.describe()
let onOutsiders = contrast("off-for-non-beta-users",
    correct: exposure(of: outsiders, rollout: enabledForBeta),
    faulty: exposure(of: outsiders, rollout: enabledForAll)) { !$0.contains("on") }.describe()
let configured = contrast("configuration-not-open-to-everyone",
    correct: enabledForBeta.everyone,
    faulty: enabledForAll.everyone) { !$0 }.describe()

countSeparations([onBeta, onOutsiders, configured])

on-for-beta-testers | correct=u01:on u02:on u03:on/pass faulty=u01:on u02:on u03:on/pass -> blind
off-for-non-beta-users | correct=u10:off u11:off/pass faulty=u10:on u11:on/fail -> separating
configuration-not-open-to-everyone | correct=false/pass faulty=true/fail -> separating
separating: 2
blind: 1
noisy: 0
inverted: 0

All three checks pass on the correct version. On the faulty version the first one passes too: for beta testers the feature really is on, in the faulty version as well. This check correctly measures whether the feature works, and it says nothing about whom the flag is open to. It gives the same answer on both versions: blind.

The second check looks at the audience the fault changed. For a non-beta user the correct version shows off and the faulty version shows on, and the check fails only on the faulty version: separating. The third check is also separating, but for another reason: the fault sits directly in the configuration value, and the check looks at that value. The difference between the two separating checks is that the first measures behavior and the second reads the file. The check that measures behavior is also robust against a fault where the configuration changes by another route; the check that reads the configuration does not show whether the configuration is actually read in the application.

Invitation, Build, and Flag

Who sees what in beta distribution is determined by three separate layers, and each layer has its own list. The first is the invitation list: it says which users can download the build. The second is the build itself: it determines which code is in the downloaded package. The third is the flag’s allowlist: it says whom the feature inside the package is opened to. The three lists live in different places and are updated by different people.

This separation explains why the lesson’s fault stays silent. The invitation list can be right, the build can be packaged right, and the flag can still have been left open to everyone. Each layer is consistent within itself; the inconsistency lies between the layers. A test that checks one layer cannot see the fault of the others. The newly invited user example is a small form of this: the user is in the invitation list, is not in the allowlist, and the two lists do not look at each other.

Why the “On” Check Gets Written

The check for the feature being on for beta testers is common, because the purpose of beta distribution is to try the feature, and the team naturally asks “does the feature work”. The question is valid and must stand somewhere in the pipeline. But it is the question does the feature work, not the question does the feature work only for the right audience. The two questions catch different faults. The first finds a flag that arrives closed; the second finds a flag that is opened too much.

A flag opened too much is a less known risk, because nobody complains. Beta testers see the feature on and are satisfied, and non-beta users also see the feature on and do not complain, since the feature works. The mistake comes out only when the feature does something unexpected, or when the release owner looks at whom the feature is visible to. The blind check is a silent danger because it reports this silence as green output.

Completing the Blind Check

A blind check is not deleted; it is completed. Here the complementary check is the one that says off for non-beta users. Together they cover the two faces of the flag: on where it should be on, off where it should be off. Writing only one of them is acceptable provided one knows which face is being tested; a first check written without knowing gives the impression that writing the second is unnecessary.

The mutation test applies here as well. Deliberately setting enabled_for to everyone in the configuration and running the checks shows which check turns red. The lesson’s first measurement is the result of that attempt: of the three checks, one never turned red.

What Beta Feedback Does Not Say

At the end of the beta period, the team usually looks at feedback: how many bug reports came in, what the users wrote. Few reports give the impression that the build is sound. But feedback reflects only seen behavior and is silent about behavior that is not seen. A flag left open to non-beta users is a typical example of this silence: if the feature works, nobody writes.

For this reason, beta feedback does not take the place of a check. Feedback answers the question “did the users notice anything”; a check answers the question “is the flag open to the right audience”. The two run together in the same beta period, and neither closes the gap of the other. If the summary the team writes at the end of the beta carries two separate lines, feedback and flag state, the reader knows which one proves what.

The cost of a blind check is visible here too. The check saying on for beta testers is green in every beta period, and this green gives the team confidence that the beta period went without problems. If underneath the green lies a fault where the flag is open to everyone, the confidence is in the wrong place. The solution is not to delete the green but to put the check on the audience outside the beta next to it.

When the Audience Grows

The beta audience grows: another team is added, outside testers are invited. With every expansion the allowlist changes, and the definition of “non-beta” changes with it. A list of non-beta users written by hand can contain a beta tester after the expansion. A check like that fails on the correct version, because the feature is on for a user who is now a beta tester. The check raises a false alarm and the team switches it off; but switching off a check because the audience grew leaves the risk of the flag opening beyond the audience uncovered.

The solution is to derive the non-beta users from the invitation list instead of choosing them by hand: every user not in the invitation list is outside the beta. Because the definition stands in one place, expansion does not break the check. Unifying the data source the check reads in this way frees separation from depending on user selection.

When the Flag Is Rolled Back

If a problem appears during the beta, the flag is turned off; no new build needs to be uploaded. This is one of the main values of flags, and rollback is also a subject for checks. If, after the flag is turned off, it is checked that the feature is off for beta testers, that check shows that the turning off was actually applied. Here too the same question applies: is the check written to catch a flag that was not turned off? Writing the opposite of the check saying on for beta testers is not enough; one has to look at what answer it gives on which version.

The Bounding Measurement

The same checks fall into different categories with different user lists. The first list is three beta testers. The second list is a single beta tester who was newly invited and has not yet been added to the allowlist. The third and fourth lines apply the off-for-non-beta check to the list of beta testers and to a single non-beta user.

// main.swift — the same checks, different user lists
let open = contrast("on-three-beta-testers",
    correct: exposure(of: betaTesters, rollout: enabledForBeta),
    faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("off") }.describe()
let invited = contrast("on-newly-invited",
    correct: exposure(of: newTester, rollout: enabledForBeta),
    faulty: exposure(of: newTester, rollout: enabledForAll)) { !$0.contains("off") }.describe()
let closedOnBeta = contrast("off-on-beta-testers",
    correct: exposure(of: betaTesters, rollout: enabledForBeta),
    faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("on") }.describe()
let closedOnOutsider = contrast("off-one-non-beta-user",
    correct: exposure(of: ["u10"], rollout: enabledForBeta),
    faulty: exposure(of: ["u10"], rollout: enabledForAll)) { !$0.contains("on") }.describe()

countSeparations([open, invited, closedOnBeta, closedOnOutsider])

on-three-beta-testers | correct=u01:on u02:on u03:on/pass faulty=u01:on u02:on u03:on/pass -> blind
on-newly-invited | correct=u04:off/fail faulty=u04:on/pass -> inverted
off-on-beta-testers | correct=u01:on u02:on u03:on/fail faulty=u01:on u02:on u03:on/fail -> noisy
off-one-non-beta-user | correct=u10:off/pass faulty=u10:on/fail -> separating
separating: 1
blind: 1
noisy: 1
inverted: 1

Four lines, four categories. on-three-beta-testers is the same as in the first measurement: blind. on-newly-invited is inverted: since the newly invited user is not in the allowlist, the feature is off for them in the correct version and the check fails; in the faulty version the flag is open to everyone, the feature is on for them as well, and the check passes. So a team that forgets to update the allowlist gets an error on the correct version and sees no error at all on the faulty version. The red one is the correct version.

In the third line the check looks for “off” among beta testers and fails on both versions, noisy: the feature is on for beta testers in both versions. This line shows the design of a bad check: the check’s expectation and the users it looks at do not match. The fourth line, with a single non-beta user, is again separating. So finding the separating check does not require a large user list; it is enough that the right user is in the list.

This measurement shows the importance of user selection, but it also has a limit. Each list is small, and in a real audience the allowlist also has gaps, misspelled identifiers, and old accounts. The audience size and proportions were not measured.

DS11. This measurement assumes that the allowlist is kept in a place separate from the beta invitations. How the invitation and the list are kept in sync in a real pipeline was not measured.

DS12. This measurement does not model the time the beta service takes to deliver the build to the user. When the invitation arrived or when the build reached the device was not measured.

Summary

• Beta distribution delivers the build to a limited audience; the feature flag determines whom the feature is open to.
• A check that the feature is on for beta testers passes on both versions when the flag is also open outside the audience: blind.
• A check that the feature is off for non-beta users separates the fault: separating.
• The same “on” check becomes inverted for a newly invited user who has not yet been added to the allowlist.
• The separation of a check depends on the user list it looks at; a blind check is not deleted but completed with a counterpart that checks what should be off.
• This lesson’s separation: blind.

Next Step

The build tried in the beta audience finally goes to the store’s review. In the next lesson, The Store Review Process, we will measure which items of a readiness checklist separate the fault and which fail on both versions.