The Store Listing

The application page in the store is the only thing a user sees before downloading the application: a few screenshots, a name, a subtitle, and a description. These parts are kept apart from the application’s code, are mostly uploaded by hand from the store console, and for that reason do not pass through the code’s checks. When a mismatch appears between the page and the application, the first to notice it is often a user.

The measure of this course is called separation. We apply the same check to the correct version and to the faulty version; if the check tells the two versions apart it is separating, if it passes both it is blind, if it fails both it is noisy, and if it fails the correct one and passes the faulty one it is inverted. In the output these four names are written as separating, blind, noisy, and inverted. This lesson also measures with the same core file.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

This lesson’s fault is in the page’s screenshot: the screenshot shows the old interface, not the new one. The application has been renewed, the page has not. The user sees an old interface, downloads, and meets a completely different application. The subtitle, on the other hand, is the same in both versions and stays within the limit the lesson sets.

The Listing Model

The model extracts two things from a build: which interface the screenshot shows, and the text of the subtitle. listing(of:) collects the two in a Listing value; Listing summarizes itself in one line, so the output shows what the screen displays and how many characters the subtitle has.

There are three checks. matchesReference compares the screenshot with a recorded reference; it is the simplified form of the equality check teams perform with image comparison. subtitleFits looks at whether the character count of the subtitle stays within the limit. showsNewInterface tests directly that the screenshot shows the new interface.

// Listing.swift — a listing from a build, three checks
struct Build {
    let interface: String
    let subtitle: String
}

struct Listing: CustomStringConvertible {
    let screenshot: String
    let subtitle: String

    var description: String {
        "screen=\(screenshot) subtitle=\(subtitle.count)"
    }
}

func listing(of build: Build) -> Listing {
    Listing(screenshot: "screen-\(build.interface)", subtitle: build.subtitle)
}

func matchesReference(_ listing: Listing, reference: String) -> Bool {
    listing.screenshot == reference
}

func subtitleFits(_ listing: Listing, limit: Int) -> Bool {
    listing.subtitle.count <= limit
}

func showsNewInterface(_ listing: Listing) -> Bool {
    listing.screenshot == "screen-new"
}

let subtitleLimit = 30
let correctBuild = Build(interface: "new", subtitle: "Track all orders in one place")
let faultyBuild = Build(interface: "old", subtitle: "Track all orders in one place")

DS16. The subtitle limit is thirty characters. This number is the lesson’s rule; the store’s real limit was not measured, and nothing measured depends on it. The lesson asks only this: does a check that looks at the subtitle not exceeding the limit see this fault.

Three Checks

The reference is recorded the first time a screenshot is uploaded to the page. The person who takes the record has taken the image from the build in their hands that day. In this measurement that build is faulty: the screenshot was recorded as screen-old. The equality check works against this record.

// main.swift — equality with the recorded reference, subtitle limit, direct test
let correctListing = listing(of: correctBuild)
let faultyListing = listing(of: faultyBuild)
let recordedFromFaulty = faultyListing.screenshot

let equality = contrast("screen-equals-reference", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: recordedFromFaulty) }.describe()
let subtitle = contrast("subtitle-within-limit", correct: correctListing, faulty: faultyListing) { subtitleFits($0, limit: subtitleLimit) }.describe()
let direct = contrast("screen-shows-new-interface", correct: correctListing, faulty: faultyListing) { showsNewInterface($0) }.describe()

countSeparations([equality, subtitle, direct])

screen-equals-reference | correct=screen=screen-new subtitle=29/fail faulty=screen=screen-old subtitle=29/pass -> inverted
subtitle-within-limit | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/pass -> blind
screen-shows-new-interface | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating
separating: 1
blind: 1
noisy: 0
inverted: 1

The equality check came out inverted. The correct version differs from the recorded image, and the check fails. The faulty version is identical to the record, and the check passes. The check works correctly in itself: it looks at whether two things are equal. What is wrong is what it should be equal to. Because the reference was recorded from a faulty build, the check confirms that the fault is preserved.

This is the most insidious form of being inverted. A noisy item turns red and draws attention; a blind item passes silently. An inverted item is green on the faulty version, and the team reads this green less as a warning than as a confirmation: we checked the page, it is identical to the record. When the correct version arrives, the check turns red, and the team may deem the new, correct image wrong and try to protect the old record.

The subtitle check stayed blind. The subtitle is the same in both versions and within the limit; the check measures something correct, but this fault has nothing to do with the subtitle. The direct test, on the other hand, came out separating: the claim that the screenshot shows the new interface is tied to the content, not to a reference.

Where the Reference Comes From

This is the common weakness of equality checks. The check assumes the reference is correct, and nothing verifies the reference. If the build was faulty when the reference was taken, the equality check locks in the fault. If the record was taken from the correct build, the same check becomes separating; this will be seen in the second part of the measurement.

Four situations are told apart. If the reference was taken from the correct build, the check separates. If the reference was taken from the faulty build, the check works backwards. If no reference was ever taken, the check fails on both versions and is noisy. If the reference is not updated between versions, even a correct new interface stays against the old record; this too is a kind of inverted behavior.

Teams usually do not notice this weakness, because the person who takes the reference and the person who writes the check are different. The first uploads the image and moves on; the second finds a file to compare the image with and writes the equality. Nobody writes down where the reference came from.

DS17. This lesson did not measure the search visibility of the store page, the download rate, or what the user reads on the page. How much the screenshot and the subtitle influence the user is not the subject of this lesson; the lesson measures only the separation of the checks that look at whether the page and the application agree.

When Screenshots Are Updated

Once the reference problem is solved, a second question appears: whenever the interface changes, the reference must change too. These two changes often happen in different hands and on different days. The person who changes the interface ships the application’s code; the person who will update the page goes on with the old images without hearing about it.

An equality check can catch this separation, but only if the reference has been updated as well. When a new interface is shipped without updating the reference, the check fails the new interface; the team should read this inverted behavior not as the interface being broken but as a reminder to renew the reference. This reading needs a separate discipline: a red check can have one of two causes, either the image is wrong or the reference is old. To say which, a direct test must stand next to the check.

The direct test thus provides two-way evidence. If the equality check is red and the direct test is green, the problem is in the reference. If the equality check is green and the direct test is red, the problem is in the record the equality check rests on. If both are red, the problem is in the image itself. A single check cannot tell these three situations apart.

The Place of the Subtitle

The subtitle check came out blind in this lesson, and coming out blind was natural. Still, it does not need to be deleted. The subtitle limit looks at a real fault: a subtitle that exceeds the limit breaks the store’s character rule. This fault did not exist in this lesson, so the check could not separate.

Whether to keep a blind check depends on whether the fault it was written to guard against is still possible. It is possible in a team that writes the subtitle by hand; it is even more possible for a subtitle machine-translated from another language, because a translation often comes out longer than the original text. A blind check turns into noise only when the fault it guards against can no longer occur at all.

This distinction also explains the difference between blind and noisy. A blind check is useless today and may be useful tomorrow. A noisy check gives the same answer today and tomorrow and has no tie left to any fault. The subtitle check is blind in this measurement; it is not noisy like the old size.

The Store Console

The listing information lives not in the application’s repository but in the store’s console. In the console there is a name, a subtitle, a description, and sets of screenshots for each language. Teams often keep a copy of this information in the repository as a configuration file; this way a change passes a review and is carried to the console in one step.

# listing.yaml — listing configuration carried to the console (SOURCE, not measured)
language: en
subtitle: Track all orders in one place
screenshots:
  - file: screens/home.png
  - file: screens/orders.png
reference:
  recorded_from: ""

This configuration was not measured; it is a yaml example, and nothing the lesson runs depends on it. What matters is the recorded_from field. Writing which build the reference was taken from makes the weakness of the equality check visible. When the field is left empty, the source of the reference is not on record and nobody can check it.

DS18. The store console was described in this lesson with prose and a configuration example. The console’s real fields, limits, and upload steps were not measured; neither was how everything in the console affects the page.

The Bounding Measurement

This time the reference is taken from the correct build. The separation returns with the same equality check and the same two versions. Next to it are put the case where the reference was never recorded and the subtitle check that stays the same on both versions.

// main.swift — the same checks when the source of the reference changes
let correctListing = listing(of: correctBuild)
let faultyListing = listing(of: faultyBuild)
let recordedFromCorrect = correctListing.screenshot

let fromCorrect = contrast("reference-from-correct-build", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: recordedFromCorrect) }.describe()
let missing = contrast("reference-never-recorded", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: "") }.describe()
let withinLimit = contrast("subtitle-within-thirty", correct: correctListing, faulty: faultyListing) { subtitleFits($0, limit: subtitleLimit) }.describe()
let seesNew = contrast("direct-new-interface-check", correct: correctListing, faulty: faultyListing) { showsNewInterface($0) }.describe()

countSeparations([fromCorrect, missing, withinLimit, seesNew])

reference-from-correct-build | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating
reference-never-recorded | correct=screen=screen-new subtitle=29/fail faulty=screen=screen-old subtitle=29/fail -> noisy
subtitle-within-thirty | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/pass -> blind
direct-new-interface-check | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating
separating: 2
blind: 1
noisy: 1
inverted: 0

When the reference is taken from the correct build, the equality check becomes separating; the check’s code did not change at all, only the source of the record changed. When there is no reference, the check fails on both versions and is noisy. The subtitle check is blind again; the page’s subtitle is not the carrier of this fault. Since the direct test needs no reference, it is separating in both measurements.

The limits are clear. This lesson did not measure how visible the page is in search results. The character limit of the subtitle is the lesson’s rule, not the store’s. The store console was described only with a configuration example and prose. What was measured is whether the code that checks the listing separates the fault; what the user sees on the page or understands from it is outside this measurement.

Verifying the Reference

An equality check against a reference becomes separating only if the reference is verified. Three habits provide this. The first is to write, when recording the reference, which build the record was taken from; the recorded_from field in the configuration example is for this. The second is for the person who takes the record to compare the screen with the application with their own eyes; the human approval covers only that record and is repeated every time the reference is renewed. The third is to put next to the equality check a direct test that needs no reference. If the direct test and the equality check diverge, one of the two rests on a wrong reference.

This third habit is the cheapest inference of the lesson. When showsNewInterface and matchesReference were applied to the same two versions, they gave different answers: one separated, the other worked backwards. The divergence between the two checks is a warning even without a measurement.

A Closing Thought

At the start of the lesson it was said that the listing does not pass through the code’s checks. The measurement showed that this has a price: the person who writes the check chooses what the check works against, and this choice may be wrong. In code checks, the correct version is written in the test input. In a listing, the correct version is an image, and once an image has been recorded nobody goes back and looks at it.

Summary

• The store page stands apart from the application’s code; for this reason the code’s checks do not cover the page.
• An equality check against a recorded reference fails the correct version and passes the faulty version when the reference was taken from a faulty build; that is, it works backwards.
• The check on whether the subtitle stays within the limit gives the same answer on both versions; it is blind on this fault.
• When the reference is taken from the correct build, the same equality check becomes separating; when there is no reference, it becomes noisy.
• Writing down the source of the reference and putting a direct test next to the equality check bring the separation back.
• This lesson’s separation: inverted.

Next Step

When the page and the submission are ready, one question remains: when the released version crashes for a user, how do you find out? In the next lesson, Crash Reporting and Analytics, we will measure whether a fault that crashes only in one language configuration is separated by a total counter and by a per-configuration counter.