--- title: 'Architecture, Testing and Release' source: 'https://academia.sh/en/courses/architecture-testing-and-release' language: en updated: '2026-10-21T05:03:06+00:00' license: 'CC BY-SA 4.0' lessons: 19 outcomes: - 'Chooses an app architecture by the criterion of testability' - 'Automates unit and UI tests' - 'Carries out signing and the store release process' - 'Distinguishes whether a check tells the correct version from the faulty one, or is blind, noisy or inverted' - "Measures a check's separation against a single fault and a single input, and names the fault it does not measure" --- # Architecture, Testing and Release A unit test looks at a return value, a UI test looks at a piece of text on screen, a coverage report counts the lines that ran, a continuous integration pipeline says the build passed, a store review applies a list of rules. The course measures every mechanism with a single question — does this check tell the correct version from the faulty one? Is the check separating, catching the fault; blind, looking at something the fault does not touch; noisy, failing on both versions; or inverted, rejecting the correct one and accepting the faulty one? Architectural patterns and the coordinator, unidirectional data flow, dependency injection and modules, unit and UI tests, coverage, profiling, memory leaks and performance, code quality tools, signing, continuous integration, release automation, beta distribution, store review and listing, and crash reporting all pass through the same frame. ## Architecture Variants of the view–model separation, the coordinator that separates navigation responsibility, unidirectional data flow with state, actions and a reducer, dependency injection for testability, and modularization with package boundaries. ### Architectural Patterns Source: https://academia.sh/en/courses/architecture-testing-and-release/architectural-patterns Three places the calculation can live relative to the screen, and six checks applied to the same correct and faulty version and measured in four separations. In the earlier courses we measured the screen, the network, and concurrency against their own questions. This course builds on them and looks at how an app is organized, how it is tested, and how it is released. The three jobs share one thing: each of them places a check. A test looks at a return value, a UI test looks at text on the screen, a release pipeline looks at whether the build passed. Every check says either "pass" or "fail". But a pass does not show that the app is correct. Whether a check works only becomes visible when the same check is applied to a faulty version. So the course has a single question: **does this check tell the correct version from the faulty one?** We set up two versions, one check, and two answers. Four combinations give four separations: - **separating:** passes on the correct version, fails on the faulty version. The check catches the fault. - **blind:** passes on both versions. The check looks at something the fault does not touch. - **noisy:** fails on both versions. The check fails for a reason unrelated to the fault. - **inverted:** fails on the correct version, passes on the faulty version. The check's expectation was taken from the faulty version. In the output the four names are written in ASCII: `separating`, `blind`, `noisy`, `inverted`. The lesson never writes a category name by hand. A rule gives the category: the core compares the two answers the check gives for the two versions. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `contrast` applies the same check, the `check` closure, to the output of both versions. The check is a single closure and it is identical for both versions; only the input differs. `Contrast` asks two questions: does the check pass on the correct version (`passesCorrect`), and does it pass on the faulty version (`passesFaulty`). `Output` can be any type that conforms to `CustomStringConvertible`. `countSeparations` counts the four names. One limit of this measure should be known from the start: separation is measured against one fault and one input. A check coming out "separating" does not show that it catches every fault, only that it catches the fault the lesson set up. Each lesson will name the fault it sets up. #### Separating the Calculation from the Screen To measure separation, we first need to run one version in isolation. If the calculation lives inside the screen, that is not possible. This is the first job of architectural patterns in this course: where the calculation lives determines what it costs to check it. The first variant keeps the calculation in the view controller. The cart total, the discount, and the receipt text are all computed inside the class that draws the screen. ```swift import UIKit // Calculation inside the screen — KAYNAK final class CartViewController: UIViewController { private let totalLabel = UILabel() private var lines: [(name: String, price: Int)] = [] override func viewDidLoad() { super.viewDidLoad() let sum = lines.reduce(0) { $0 + $1.price } let total = sum - sum * 10 / 100 totalLabel.text = "Total: \(total) TL" } } ``` This fragment is not run; the view controller depends on the platform's lifecycle and the lesson claims no output for it. What matters is its structure. To check `total`, `viewDidLoad` has to run, which means a screen has to be built. To check the faulty version (the one that applies the discount twice), you would need two separate copies of the screen. Checking the calculation becomes the same job as checking the screen. The second variant moves the calculation into the view model. The view model produces the values the screen needs to know, and the screen only displays them. ```swift // Calculation in the view model — KAYNAK struct CartViewModel { let lines: [Line] let percent: Int var total: Int { correctTotal(lines, percent: percent) } var receiptText: String { receipt(total) } } ``` Because the view model is a plain value, it can be called without building a screen. This lesson's measurement rests on this variant: `correctTotal` and `faultyTotal` below are screenless copies of the calculation in the view controller. Separating the calculation makes it cheap to give the same check to two versions. The third variant leaves the calculation to the server layer: the app does not compute the total itself, it fetches it with a request. Checking the calculation then requires a network call. A check that depends on the network fails when there is no network, and when there is one, it depends on the server's version at that moment. We will measure this problem in the fourth lesson, when we cover handing a dependency in from outside. None of the three variants is right on its own. Which one to pick depends on what each check costs. This lesson fixes only one thing: the more a check depends on the screen, the more expensive it is to compare two versions. #### Six Checks, Four Separations The fault is this: the discount is applied twice. The correct version gives 90 TL for a 100 TL cart with a ten percent discount, and the faulty version gives 81 TL. Six checks are applied to the same two versions. The names of the checks and the values they verify are the names and expected values of the tests that would be written in a test target. ```swift // main.swift — six checks, four separations struct Line { let name: String let price: Int } func correctTotal(_ lines: [Line], percent: Int) -> Int { let sum = lines.reduce(0) { $0 + $1.price } return sum - sum * percent / 100 } func faultyTotal(_ lines: [Line], percent: Int) -> Int { let sum = lines.reduce(0) { $0 + $1.price } let discounted = sum - sum * percent / 100 return discounted - discounted * percent / 100 } func receipt(_ total: Int) -> String { "Total: \(total) TL" } let cart = [Line(name: "notebook", price: 40), Line(name: "pen", price: 25), Line(name: "eraser", price: 35)] let good = correctTotal(cart, percent: 10) let bad = faultyTotal(cart, percent: 10) let exact = contrast("total-equals", correct: good, faulty: bad) { $0 == 90 }.describe() let discount = contrast("discount-amount", correct: 100 - good, faulty: 100 - bad) { $0 == 10 }.describe() let positive = contrast("positive-total", correct: good, faulty: bad) { $0 > 0 }.describe() let rows = contrast("row-count", correct: cart.count, faulty: cart.count) { $0 == 3 }.describe() let text = contrast("receipt-text", correct: receipt(good), faulty: receipt(bad)) { $0 == "Total: 90.00 TL" }.describe() let golden = contrast("recorded-sample", correct: good, faulty: bad) { $0 == 81 }.describe() countSeparations([exact, discount, positive, rows, text, golden]) ``` ```text total-equals | correct=90/pass faulty=81/fail -> separating discount-amount | correct=10/pass faulty=19/fail -> separating positive-total | correct=90/pass faulty=81/pass -> blind row-count | correct=3/pass faulty=3/pass -> blind receipt-text | correct=Total: 90 TL/fail faulty=Total: 81 TL/fail -> noisy recorded-sample | correct=90/fail faulty=81/pass -> inverted separating: 2 blind: 2 noisy: 1 inverted: 1 ``` The output was run three times in the sandbox on Swift 6.4 and is identical in all three. The six checks produced all four separations: 2 separating, 2 blind, 1 noisy, 1 inverted. `total-equals` and `discount-amount` look at the value the fault changes. The first expects the total to be 90, the second expects the discount to be 10; the faulty version gives 81 and 19, so both fail. These are separating. `positive-total` looks at a property the fault does not break: 81 is also greater than zero. `row-count` looks at a value the fault never touches: the discount does not change the row count. Both pass on both versions, so both are blind. A blind check misleads because it looks green. A code coverage report would count both of these checks as executed lines; the number in the report does not show the difference between separating and blind, only the separation does. `receipt-text` wrote its expected text in the old format, `90.00`. The receipt prints `Total: 90 TL`, so the check fails on the correct version as well. It fails on the faulty version too. The check does not catch the fault; it fails not because of the fault but because of the check itself. This is noisy. A noisy check is eventually silenced with "this test is always red", and a silenced check cannot catch a real fault later either. `recorded-sample` recorded its expected value from the faulty version: 81. The correct version gives 90 and the check rejects it; the faulty version gives 81 and the check accepts it. This is an inverted check, and it is the most dangerous of the four. The faulty version passes green while the correct version goes red. The typical cause is that the recorded sample (a golden file, a snapshot) was taken from a faulty version; whoever recorded it looked at the output and said "that looks right". This separation has one more cost: noisy and inverted are the fault of whoever wrote the check, not of the code. Looking for the fault in the code is looking in the wrong place. #### Where the Expected Value Comes From Four of the six checks failed at the same place: the source of the expected value. `positive-total` and `row-count` took their expectation from a property the fault does not break, `receipt-text` from an old format, and `recorded-sample` from the faulty version's own output. The two separating checks took their expectation not from the code but from the rule: a 100 TL cart with a ten percent discount comes to 90 TL. This difference looks small at first, but it has large consequences in practice. An expectation taken from the rule is independent of the code; changing the code does not affect it, and breaking the code breaks it. An expectation taken from the code breaks together with the code. So the first question to ask when writing a check should be "what should this value be", not "what value does the code give". The second turns the check into a mirror of the code, as in the recorded sample, and the mirror reflects the fault too. Mutation testing, which asks whether a test suite catches deliberately broken code, is the closest relative of this measure. The difference is this: mutation testing only asks "was it caught". Here the result on the correct version is measured too, so noisy and inverted checks show up separately from the blind check. A check can be green or red in all three cases, but the reason behind each is different and so is each remedy: a blind check gets a second input, a noisy check gets its expectation corrected, an inverted check has its recording retaken from the correct version. #### Assumptions **AR1.** `correctTotal` and `faultyTotal` are screenless copies of the calculation in the view controller. This lesson does not measure whether the real screen gives the same result; the screen stays KAYNAK. **AR2.** Amounts are whole TL and division rounds down. Fractions of a lira, the rounding mode, and a currency type do not enter the model; the noise in the `receipt-text` check therefore comes only from the text format. **AR3.** The checks do not run in a real test framework. Each check is a closure that stands for the work a test does; test names and expected values are kept the same as the tests in the framework. #### The Bounding Measurement The same check can be blind on one input and separating on another against the same fault. Here the fault is different: the percentage is applied rounded down to a multiple of ten. ```swift func correctPrice(_ sum: Int, percent: Int) -> Int { sum - sum * percent / 100 } func roundedPrice(_ sum: Int, percent: Int) -> Int { sum - sum * (percent / 10 * 10) / 100 } let ten = contrast("ten-percent", correct: correctPrice(100, percent: 10), faulty: roundedPrice(100, percent: 10)) { $0 == 90 }.describe() let fifteen = contrast("fifteen-percent", correct: correctPrice(100, percent: 15), faulty: roundedPrice(100, percent: 15)) { $0 == 85 }.describe() countSeparations([ten, fifteen]) ``` ```text ten-percent | correct=90/pass faulty=90/pass -> blind fifteen-percent | correct=85/pass faulty=90/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` The output is identical across three runs. Ten percent is the input where the fault is invisible, because rounding 10 down to a multiple of ten gives 10. On this input the same check stays blind. At fifteen percent the fault shows: 15 drops to 10 and the check fails. The check itself did not change; only the input did. What this measurement says is that whether a check is "blind" or "separating" is a property of the input as much as of the check. A test written with a single input may never have seen the input where the fault lives. #### Summary - If the calculation is in the view controller, checking it requires building a screen; once it moves to the view model, the two versions are compared without a screen. A calculation in the server layer depends on the network; we cover that in the fourth lesson. - A check's value comes from the two answers it gives to the correct and faulty versions: whether it passes says nothing by itself. - Six checks produced four separations: 2 separating, 2 blind, 1 noisy, 1 inverted. - A blind check looks green and gives false confidence; an inverted check lets the faulty version pass and breaks the correct one. - The same check can be blind on one input and separating on another: blind at ten percent, separating at fifteen percent. - This lesson's separation: **all four (2/2/1/1)**. #### Next Step Separating the calculation from the screen only helps the calculation; the transitions between screens may still live inside the screens. In the second lesson, The Coordinator Pattern takes navigation away from the views and gives it to a separate object. There the check will look not at a value on one screen but at which screen opens after payment, and against a fault that opens the wrong screen after payment, two checks will answer differently. ### The Coordinator Pattern Source: https://academia.sh/en/courses/architecture-testing-and-release/coordinator-pattern Handing navigation to a coordinator separate from the views, and two checks that answer differently to a fault that opens the wrong screen after payment. In the first lesson we separated the calculation from the screen and saw that six checks produced four separations. Once the calculation is separated, the screen keeps another responsibility: deciding which screen comes next. The cart screen opens the address screen, the address screen opens the payment screen, the payment screen opens the confirmation screen. If every view controller knows the next one, the flow cannot be read in one place, and changing one link of the flow touches three screens. The coordinator pattern takes this decision away from the screens and gives it to a separate object. The screen only says "payment finished"; the coordinator picks the next screen. This lesson's question is the course's question: **does this check tell the correct version from the faulty one?** The fault is this: when payment completes, the coordinator sends the user back to the cart instead of opening the confirmation screen. The core comes in verbatim, as in the first lesson. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` The core is the same as the core in the first lesson; it is not redefined in this lesson, only used. The separation names do not change: `separating`, `blind`, `noisy`, `inverted`. #### Taking Navigation Out of the Screen In a flow without a coordinator, the navigation decision lives inside the view controller. The payment screen builds the confirmation screen itself and opens it itself. ```swift import UIKit // Navigation inside the screen — KAYNAK final class PaymentViewController: UIViewController { func paymentDidFinish() { let next = ConfirmationViewController() navigationController?.pushViewController(next, animated: true) } } ``` This screen knows the confirmation screen. When you want to use the payment screen in another flow, for example renewing a subscription, the confirmation screen opens there too. Checking the flow also depends on the screen: to see what opens after payment, you have to build a navigation controller and a payment screen. With a coordinator, the same decision lives in a separate object and the screen only reports an event. ```swift import UIKit // Navigation in the coordinator — KAYNAK final class CheckoutFlowCoordinator { private let navigation: UINavigationController init(navigation: UINavigationController) { self.navigation = navigation } func paymentCompleted() { navigation.pushViewController(ConfirmationViewController(), animated: true) } } ``` Neither fragment is run; the navigation controller depends on the platform's lifecycle and the lesson claims no output for it. What sets the two apart is structure. In the second, the payment screen does not know the confirmation screen; the link between the screen and the flow is an event name. The whole flow reads in one class, and what opens after payment is visible in one function. This lesson's measurement looks at that function. #### The Coordinator's Model The platform's navigation is not measured, but the decision the coordinator makes is itself pure logic: an event arrives and the route sequence changes. The model builds this in three parts: route names, the payment screen's view model, and the coordinator that holds the route sequence. The fault is inside the coordinator: `backToCart` drops the sequence back to the cart after payment. `extraNotice` is the second fault, used in the bounding measurement. `Checkout.swift` is a helper file and prints no output. ```swift // Checkout.swift — model of the checkout flow: view model and coordinator enum Route: String { case cart, address, payment, notice, confirmation } enum Fault { case none, backToCart, extraNotice } struct PaymentViewModel { private(set) var done = false mutating func complete() { done = true } } struct CheckoutCoordinator { private(set) var path: [Route] = [.cart, .address, .payment] let fault: Fault mutating func paymentCompleted() { switch fault { case .none: path.append(.confirmation) case .backToCart: path = [.cart] case .extraNotice: path.append(contentsOf: [.notice, .confirmation]) } } } func runCheckout(_ fault: Fault) -> (done: Bool, path: [Route]) { var model = PaymentViewModel() var coordinator = CheckoutCoordinator(fault: fault) model.complete() coordinator.paymentCompleted() return (model.done, coordinator.path) } func trail(_ path: [Route]) -> String { path.map { $0.rawValue }.joined(separator: ">") } func finalRoute(_ path: [Route]) -> String { path.last?.rawValue ?? "none" } ``` `runCheckout` produces two things side by side: the view model's "payment completed" state and the coordinator's route sequence. The view model knows payment finished, but it does not know which screen opens afterward; that is the coordinator's job. `trail` turns the sequence into a readable string, and `finalRoute` gives only the last route. Four checks are applied to the same two versions. The correct version is `runCheckout(.none)`, the faulty version is `runCheckout(.backToCart)`. ```swift // main.swift — which screen opens after payment let good = runCheckout(.none) let bad = runCheckout(.backToCart) let paid = contrast("payment-completed", correct: good.done, faulty: bad.done) { $0 == true }.describe() let last = contrast("last-screen", correct: finalRoute(good.path), faulty: finalRoute(bad.path)) { $0 == "confirmation" }.describe() let first = contrast("route-start", correct: good.path.first?.rawValue ?? "none", faulty: bad.path.first?.rawValue ?? "none") { $0 == "cart" }.describe() let whole = contrast("whole-route", correct: trail(good.path), faulty: trail(bad.path)) { $0 == "cart>address>payment>confirmation" }.describe() countSeparations([paid, last, first, whole]) ``` ```text payment-completed | correct=true/pass faulty=true/pass -> blind last-screen | correct=confirmation/pass faulty=cart/fail -> separating route-start | correct=cart/pass faulty=cart/pass -> blind whole-route | correct=cart>address>payment>confirmation/pass faulty=cart/fail -> separating separating: 2 blind: 2 noisy: 0 inverted: 0 ``` The output was run three times in the sandbox on Swift 6.4 and is identical in all three. Of the four checks, two came out separating and two blind. `payment-completed` looks at the view model's state. Payment really did complete, so the state is `true` on both versions. The fault is in the screen the coordinator picks, and the view model takes no part in that decision. The check is blind, and it shows a consequence of separating the coordinator: a test that looks at a screen's value cannot see a navigation fault, because navigation is no longer the screen's value. If the fault moved to another object, the check must move to that object too. `route-start` is also blind. The faulty version makes the sequence `[.cart]`, and the first element of the sequence is still `cart`. The check looks at something correct, but at something the fault does not touch. `last-screen` looks at the last element of the route sequence. It comes out `confirmation` on the correct version and `cart` on the faulty version; the second does not meet the expectation. The check is separating. `whole-route` compares the entire sequence with the expected route and separates in the same way. What the two have in common is that both look at the route the coordinator produces. Handing navigation to the coordinator therefore changes not only the design but also the target of the check. #### What the Coordinator Solves and What It Does Not The coordinator's gain shows in three places. The first is readability: the whole flow sits in one class. None of the screens knows the screen after it, so a screen can be reused in another flow. The second is changeability: if you want a thank-you screen instead of the confirmation screen after payment, only `paymentCompleted` changes. The third is checkability, and it is the subject of this lesson: once the route decision comes down to pure logic, the same check is applied to two versions without building a screen. It has a cost too. The coordinator is an object that holds a reference to the screen, and the screen reports back to the coordinator. If the two hold each other with strong references, neither is ever released; this is a retain cycle and leads to a memory leak. The remedy is for the screen to hold the coordinator with a weak reference. The memory side will be covered in the Memory Leak Detection lesson; here, just note this: the coordinator's model does not show this cycle, because the model holds no screen. This is something the model deliberately ignores. Nested flows are a cost as well. If the payment flow is a child coordinator, it has to report its completion to the parent coordinator. If the notification is lost, the parent coordinator never learns that the child flow finished and the screen stays open. That behavior is also outside the model; the model looks at one event of one coordinator. #### What the Route Sequence Does Not Guarantee The check that looks at the whole route was the strongest check in this lesson, but being strong does not mean it catches everything. Even when the route sequence is correct, the real navigation controller can open a different screen: the coordinator produces a route called `confirmation`, but the code that builds the screen for that route may return the wrong screen. The route sequence tests the decision layer; the screen-building code tests the application layer. The mapping between them calls for a separate check, and because that check depends on the platform, it stays KAYNAK. This is the same limit as in the first lesson: a check catches only the fault it was set up for. The `whole-route` check caught the fault of opening the wrong route after payment, but it does not catch a fault that attaches the wrong screen to the right route. A check set up for that fault has to look at the type of the screen, and that lies in the layer that cannot be measured. So the lesson does not say "navigation was verified"; it says "the navigation decision was verified". Another limit is the event itself. The model assumes the `paymentCompleted` function is called. If the payment screen never reports the event, the coordinator does not run, the route sequence stays at the payment step, and the `whole-route` check fails, that is, it catches the fault as separating. But this says nothing about where the fault is: the check only knows the sequence is wrong; it does not know whether the wrongness is in the screen or in the coordinator. The question of why is not the check's job; finding the location of the fault is a separate step and requires looking at the source code. #### Assumptions **AR1.** The route sequence is the order of screens the coordinator will pass to the navigation controller. This lesson does not measure whether a real navigation controller's stack behaves exactly like this sequence; the navigation controller stays KAYNAK. **AR2.** What the user sees after payment, the animation, and the transition time do not enter the model. The model says only which route opens. **AR3.** The view model's "payment completed" state and the coordinator's event function are called at the same moment. In a real app a network response may come in between; that dependency is covered in the fourth lesson. #### The Bounding Measurement The same two checks can answer differently when the fault is somewhere else. This time the fault does not break the final screen; it breaks an intermediate step: after payment, before the confirmation screen, an unneeded notice screen opens. The correct version is `runCheckout(.none)`, the faulty version is `runCheckout(.extraNotice)`. ```swift // main.swift — the fault is in an intermediate step let goodNotice = runCheckout(.none) let badNotice = runCheckout(.extraNotice) let lastOnly = contrast("last-screen-notice", correct: finalRoute(goodNotice.path), faulty: finalRoute(badNotice.path)) { $0 == "confirmation" }.describe() let wholePath = contrast("whole-route-notice", correct: trail(goodNotice.path), faulty: trail(badNotice.path)) { $0 == "cart>address>payment>confirmation" }.describe() countSeparations([lastOnly, wholePath]) ``` ```text last-screen-notice | correct=confirmation/pass faulty=confirmation/pass -> blind whole-route-notice | correct=cart>address>payment>confirmation/pass faulty=cart>address>payment>notice>confirmation/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` The output is identical across three runs. The `last-screen` check, which was separating in the measurement above, is blind to this fault even though it is built on the same logic: both versions end with `confirmation`. The only check that sees the fault is the one that looks at the whole route. The check itself did not change; what changed is where the fault is. If a flow test checks only the end of the flow, every flow with a correct end passes green; unneeded or missing steps in between are never seen. #### Summary - The coordinator takes the navigation decision from the view controllers and gives it to a single object; the screen only reports an event. - When the place of the decision changes, the target of the check changes too: a navigation fault shows in the coordinator's route sequence, not in the view model's state. - A check that looks at the view model's "payment completed" state is blind to a faulty coordinator. - Checks that look at the final screen and at the whole route catch the same fault: of four checks, two are separating and two are blind. - When the fault moves to an intermediate step, the check that looks only at the final screen stays blind; the check that looks at the whole route stays separating. - This lesson's separation: **blind**. #### Next Step We separated navigation and the calculation; what remains is the state the screen depends on. In the third lesson, Unidirectional Data Flow will keep state in one place and let change arrive only through an action. There the fault will be in a reducer: a remove-product action will delete the line instead of decreasing the quantity, and a check that compares the resulting state for the same sequence of actions will catch the fault this time. ### Unidirectional Data Flow Source: https://academia.sh/en/courses/architecture-testing-and-release/unidirectional-data-flow Keeping state in a single central store that changes only through actions and a reducer, and measuring the separation of checks by feeding the same action sequence to two reducers. In the second lesson we took navigation out of the screen and saw that the target of the check changed. Now we look at the state the screen depends on. On a screen, the cart can change from several places: the add button, the remove button, a network response, a notification. If each place changes the state in its own way, you have to walk every path to say why a value is what it is. Unidirectional data flow reduces these paths to one. The state lives in a single central store. The screen reads the state and, when something happens, sends an action to the store; the store hands the action to a reducer, and the reducer produces the new state from the old one. The flow always goes in the same direction: state, screen, action, reducer, new state. This lesson's fault is in the reducer: the "remove product" action deletes the line completely instead of decreasing the quantity by one. The core comes in verbatim, as in the earlier lessons. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` In this lesson too, the core is only used. `contrast` applies the same check to the output of two versions and returns one of the four separations. #### State, Action, Reducer The central store is built from three parts. The **state** is the application's data at that moment: the items in the cart and their quantities. An **action** is a small value that says something happened: "add pen", "remove pen". The **reducer** is the function that takes the old state and an action and gives the new state. The reducer is pure: the same state and the same action always give the same new state, and it writes nothing to the outside. The platform-dependent part is a thin shell. The store holds the state, passes the action to the reducer and stores the result; the view reads the state and sends actions. ```swift import SwiftUI // Store and view — KAYNAK final class CartStore: ObservableObject { @Published private(set) var state = CartState() func send(_ action: CartAction) { state = reduceCart(state, action) } } struct CartView: View { @ObservedObject var store: CartStore var body: some View { Button("Add pen") { store.send(.add("pen")) } } } ``` This fragment is not run; the view and the published property depend on the platform, and the lesson claims no output for them. What matters is that the shell makes no decisions. `send` only calls the reducer, and the view only sends actions. All the decisions are in `reduceCart`, and that function is plain Swift. The measurement looks at that function. #### The Reducer's Model The model defines four things: the state, the action, the correct and faulty reducer, and `replayActions`, which applies a sequence of actions in order. The reducer is a single function; the `faulty` flag selects the faulty version. The fault is in the `remove` branch: the correct version decreases the quantity if it is greater than 1 and deletes the line otherwise; the faulty version always deletes. `Reducer.swift` is a helper file and prints no output. ```swift // Reducer.swift — cart state, actions and reducer struct CartItem: Equatable { let sku: String var quantity: Int } struct CartState: Equatable, CustomStringConvertible { var items: [CartItem] = [] var description: String { items.isEmpty ? "empty" : items.map { "\($0.sku)x\($0.quantity)" }.joined(separator: ",") } } enum CartAction { case add(String) case remove(String) } func reduceCart(_ state: CartState, _ action: CartAction, faulty: Bool = false) -> CartState { var next = state switch action { case .add(let sku): if let index = next.items.firstIndex(where: { $0.sku == sku }) { next.items[index].quantity += 1 } else { next.items.append(CartItem(sku: sku, quantity: 1)) } case .remove(let sku): guard let index = next.items.firstIndex(where: { $0.sku == sku }) else { return next } if !faulty && next.items[index].quantity > 1 { next.items[index].quantity -= 1 } else { next.items.remove(at: index) } } return next } func replayActions(_ actions: [CartAction], faulty: Bool) -> CartState { actions.reduce(CartState()) { reduceCart($0, $1, faulty: faulty) } } func quantity(of sku: String, in state: CartState) -> Int { state.items.first(where: { $0.sku == sku })?.quantity ?? 0 } ``` Four checks are applied to two action sequences. The first sequence adds twice and removes once: in the correct version one pen should remain, and in the faulty version none remains. The second sequence adds once and removes once: a product with a quantity of 1 is deleted completely in both versions. ```swift // main.swift — the same action sequence, two reducers let twoThenRemove: [CartAction] = [.add("pen"), .add("pen"), .remove("pen")] let oneThenRemove: [CartAction] = [.add("pen"), .remove("pen")] let goodState = replayActions(twoThenRemove, faulty: false) let badState = replayActions(twoThenRemove, faulty: true) let goodSingle = replayActions(oneThenRemove, faulty: false) let badSingle = replayActions(oneThenRemove, faulty: true) let sameState = contrast("compare-state", correct: goodState, faulty: badState) { $0 == CartState(items: [CartItem(sku: "pen", quantity: 1)]) }.describe() let leftover = contrast("remaining-quantity", correct: quantity(of: "pen", in: goodState), faulty: quantity(of: "pen", in: badState)) { $0 == 1 }.describe() let notNegative = contrast("quantity-not-negative", correct: quantity(of: "pen", in: goodState), faulty: quantity(of: "pen", in: badState)) { $0 >= 0 }.describe() let singleLine = contrast("single-quantity-item", correct: goodSingle.items.count, faulty: badSingle.items.count) { $0 == 0 }.describe() countSeparations([sameState, leftover, notNegative, singleLine]) ``` ```text compare-state | correct=penx1/pass faulty=empty/fail -> separating remaining-quantity | correct=1/pass faulty=0/fail -> separating quantity-not-negative | correct=1/pass faulty=0/pass -> blind single-quantity-item | correct=0/pass faulty=0/pass -> blind separating: 2 blind: 2 noisy: 0 inverted: 0 ``` The output was run three times in the sandbox on Swift 6.4 and is identical in all three. Of the four checks, two came out separating and two blind. `compare-state` compares the resulting state of the same action sequence with the expected state. The correct version gives `penx1`, the faulty version gives `empty`; the second does not meet the expectation. `remaining-quantity` reduces the same sequence to a single number: the quantity of the pen should be 1, and in the faulty version it comes out 0. Both are separating. Their common cause is this: both look at what the fault changes, that is, at the state the reducer produces. `quantity-not-negative` looks at a property the fault does not break. In the faulty version the pen is deleted and the quantity reads 0; 0 is not less than zero either. The check passes on both versions and is blind. `single-quantity-item`, on the other hand, shows that the fault never appears in the second sequence: removing a product with a quantity of 1 deletes the product in both versions, and while looking at the row count the check sees both give 0. The check is blind for this sequence, and it is blind because the input hides the fault. #### What the Flow Adds to Checking What makes this measurement easy is the flow itself. Because the reducer is pure, the same action sequence can be given to two versions and the two resulting states can be compared exactly. The screen or the network does not need to get involved. In an application that changes the state from many places, the same comparison would require driving each path separately. Also, because the state is a single value, what the check looks at can be chosen: the whole state, one field, or a derived number. Three of the four checks read the same two states in different ways; only two saw the fault. How strong a check is may depend on how much of the state it reads. `compare-state` reads the whole state, and for this reason it stays separating even when the fault shows up in every field, not just one. `remaining-quantity` reads only one field; it would have been blind if the fault had not touched that field. The action sequence is an input as well. The two sequences answered the same fault differently: the same check was in both, and only the sequence changed. This is the action-level counterpart of the ten percent and fifteen percent example in the first lesson. A good reducer test tries more than one sequence, and in particular it tries boundary sequences: an empty cart, a product with a quantity of one, consecutive additions of the same product. #### Assumptions **AR1.** The store and the view stay KAYNAK. Redrawing the view when a real store publishes its state is outside this lesson's scope; the model only compares the state the reducer produces. **AR2.** The reducer is assumed to be pure: it reads nothing from the outside and writes nothing to the outside. In a real application, side effects (a network request, saving) live in a separate layer outside the reducer, and this lesson does not model them. **AR3.** The product identifier (`sku`) is a string and maps to an item name. Stock, price and the error returned when a product is not found do not enter the model; a remove action for an item that does not exist leaves the state unchanged. #### The Bounding Measurement Taking the expected value from a recording is the easiest way to record the fault. Here the recorded sample is taken from the faulty reducer's result for the first sequence: `empty`. The same recorded sample is tried against two separate sequences. ```swift // main.swift — the recorded action and state sample was taken from the faulty reducer let recorded = "empty" let twoThenRemove: [CartAction] = [.add("pen"), .add("pen"), .remove("pen")] let oneThenRemove: [CartAction] = [.add("pen"), .remove("pen")] let onTwo = contrast("record-two-adds-one-remove", correct: replayActions(twoThenRemove, faulty: false), faulty: replayActions(twoThenRemove, faulty: true)) { $0.description == recorded }.describe() let onOne = contrast("record-one-add-one-remove", correct: replayActions(oneThenRemove, faulty: false), faulty: replayActions(oneThenRemove, faulty: true)) { $0.description == recorded }.describe() countSeparations([onTwo, onOne]) ``` ```text record-two-adds-one-remove | correct=penx1/fail faulty=empty/pass -> inverted record-one-add-one-remove | correct=empty/pass faulty=empty/pass -> blind separating: 0 blind: 1 noisy: 0 inverted: 1 ``` The output is identical across three runs. In the first sequence the recording was taken from the faulty version, so the correct version gives `penx1` and is rejected, while the faulty version gives `empty` and is accepted. This is an inverted check, and it lets the faulty reducer pass green. In the second sequence both versions give `empty`, so the same recording accepts both and the check stays blind. The same recorded sample is inverted on one sequence and blind on the other. A recorded sample is meaningful only when it is taken from the correct version; if the recording was once taken from a faulty output, then when the reducer is fixed every check turns red, and the person looking at the red may renew the recording instead of fixing the check, which makes the fault permanent. #### Summary - In unidirectional data flow the state lives in a central store, change arrives only through an action, and the reducer produces the new state from the old one. - The reducer is a pure function; the same action sequence can be given to two versions and the two resulting states can be compared. - Checks that compare the resulting state and the remaining quantity catch the faulty reducer: of four checks, two are separating and two are blind. - A check that looks at a property the fault does not touch, and a check that looks at a sequence in which the fault does not appear, stay blind. - If the recorded action and state sample is taken from the faulty reducer, the same check is inverted on one sequence and blind on the other. - This lesson's separation: **separating**. #### Next Step The reducer was easy to measure because it is pure; but in a real application part of the state comes from the outside. In the fourth lesson, Dependency Injection will cover giving external dependencies such as a network client to the screen or the store from outside. There the fault will be in a client: when an empty response arrives, it will clear the previous list instead of keeping it, and a check tied to the real network will stay behind in both versions. ### Dependency Injection Source: https://academia.sh/en/courses/architecture-testing-and-release/dependency-injection An object receiving the dependency it needs from outside instead of building it itself, and how what a fake dependency imitates determines the separation of a check. In the third lesson we gathered the state in one place and measured the reducer without a screen. The reducer was easy to measure because it takes nothing from the outside. Most objects are not like that: a product list view model gets its products from a network client, an order object reads its record from a store. If these dependencies are built inside the object, checking the object means running the dependency too. Dependency injection reverses this link: the object does not build the dependency it needs, it receives it from outside. That way, in a check, a fake dependency that resembles the real one can be passed in its place. This lesson's fault is in the product list: when the client gets an empty reply, it clears the list instead of keeping the previous one. The core comes in verbatim, as in the earlier lessons. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` In this lesson too, the core is only used. The separation names are the same; only what the checks do changes. #### Passing the Dependency from Outside A view model without injection builds its client itself. In this version there are two ways to check it, and both are bad: going out to the real network, or finding a way to replace the client that was built. ```swift import Foundation // Dependency built inside — KAYNAK final class ProductListViewModel { private let service = NetworkProductService(session: URLSession.shared) private(set) var items: [String] = [] func refresh() async { guard let reply = await service.fetch() else { return } items = reply } } ``` This fragment is not run; the session and the network depend on the platform, and the lesson claims no output for them. Its structure is enough: `service` is fixed and built inside. To check this model's `refresh` call, the network has to be up, the server has to give its reply at that moment, and the reply has to be what the check expects. None of the three is in the check's hands. There are three ways to pass a dependency from outside. The most direct is initializer injection: the dependency arrives as a parameter when the object is built. In property injection it is assigned after the object is built; in function parameter injection it is given only in the call to the function that needs it. This lesson uses the initializer, because the dependency does not change over the object's lifetime. ```swift import Foundation // Dependency passed from outside — KAYNAK protocol ProductService: Sendable { func fetch() async -> [String]? } struct NetworkProductService: ProductService { let session: URLSession func fetch() async -> [String]? { guard let url = URL(string: "https://example.com/products") else { return nil } guard let (data, _) = try? await session.data(from: url) else { return nil } return try? JSONDecoder().decode([String].self, from: data) } } final class InjectedListViewModel { private let service: any ProductService private(set) var items: [String] = [] init(service: any ProductService) { self.service = service } } ``` This fragment is not run either. What matters is the protocol: the view model no longer knows the real client, it knows the `ProductService` contract. The real client conforms to that contract, and the fake dependency in a check conforms to it too. The view model cannot tell the two apart. #### What a Fake Dependency Imitates The model builds the protocol and two fake dependencies in plain Swift. `OfflineService` imitates the case where there is no network: it returns `nil` on every call. `StubService` returns a reply given in advance; it is a stub that gives a fixed reply when called. The product list model takes one of three paths in `refresh`: if the request fails, the list is kept; if the reply is empty, the correct version also keeps the list and the faulty version clears it; if the reply has items, the list is replaced with the reply. The fault is only in the second path. `Catalog.swift` is a helper file and prints no output. ```swift // Catalog.swift — product list model and fake dependencies protocol ProductService: Sendable { func fetch() async -> [String]? } struct OfflineService: ProductService { func fetch() async -> [String]? { nil } } struct StubService: ProductService { let reply: [String] func fetch() async -> [String]? { reply } } struct ProductListModel { private(set) var items: [String] let service: any ProductService let faulty: Bool mutating func refresh() async { guard let reply = await service.fetch() else { return } if reply.isEmpty && !faulty { return } items = reply } } func refreshedItems(start: [String], service: any ProductService, faulty: Bool) async -> String { var model = ProductListModel(items: start, service: service, faulty: faulty) await model.refresh() return model.items.isEmpty ? "empty" : model.items.joined(separator: ",") } ``` Three checks are applied to the same two versions. The first stands in for a check tied to the real network client. Because the network is closed in the sandbox, the client in this check is modeled with `OfflineService`, and the check expects the server's list of three products. The second works with a stub that returns an empty reply. The third imitates a network error and expects the list to be kept. ```swift // main.swift — three checks, three different fake dependencies let start = ["notebook", "pen"] let goodReal = await refreshedItems(start: start, service: OfflineService(), faulty: false) let badReal = await refreshedItems(start: start, service: OfflineService(), faulty: true) let goodEmpty = await refreshedItems(start: start, service: StubService(reply: []), faulty: false) let badEmpty = await refreshedItems(start: start, service: StubService(reply: []), faulty: true) let real = contrast("real-network-client", correct: goodReal, faulty: badReal) { $0 == "notebook,pen,eraser" }.describe() let empty = contrast("empty-reply-stub", correct: goodEmpty, faulty: badEmpty) { $0 == "notebook,pen" }.describe() let failure = contrast("error-reply-stub", correct: goodReal, faulty: badReal) { $0 == "notebook,pen" }.describe() countSeparations([real, empty, failure]) ``` ```text real-network-client | correct=notebook,pen/fail faulty=notebook,pen/fail -> noisy empty-reply-stub | correct=notebook,pen/pass faulty=empty/fail -> separating error-reply-stub | correct=notebook,pen/pass faulty=notebook,pen/pass -> blind separating: 1 blind: 1 noisy: 1 inverted: 0 ``` The output was run three times in the sandbox on Swift 6.4 and is identical in all three. The three checks gave three different separations: one separating, one blind, one noisy. `real-network-client` expects the list of three products that will come from the server. Because there is no network in the sandbox, the client cannot give a reply, the list stays as it was, and `notebook,pen` comes out on both versions. The check fails on both versions, so it is noisy. It does not catch the fault; it fails not because of the fault but because the check is tied to the network. The same noise exists in a real environment too: if the server is slow, unreachable or gives a different reply, the check turns red, and the person looking at the red blames the network instead of the code. `empty-reply-stub` looks at whether the list is kept, and the stub returns an empty reply. The correct version gives `notebook,pen` and passes; the faulty version clears the list, gives `empty` and fails. The check is separating. The difference here is that the dependency is passed from outside: the path where the fault lives, the empty reply, can be triggered only by a check that controls the dependency. `error-reply-stub` imitates a network error and expects the list to be kept. In the error branch both versions keep the list, so the check passes on both versions and is blind. The fault is in the empty reply branch, not in the error branch. When a fake dependency imitates the wrong path, the check looks right and misses the fault. #### Assumptions **AR1.** The real network client is not measured. The network is closed in the sandbox; instead of the real client, the "no reply arrives" case is modeled with `OfflineService`, and the check for this case is noisy. How the same check would behave in a real environment depends on the server, and this lesson gives it no value. **AR2.** The model's version of the `ProductService` protocol is the same as the real client's contract: a list of replies, or `nil`. The real client has error types, authentication and retries; the model reduces them to a single `nil`. **AR3.** The reply is a list of product names. Paging, ordering and product fields do not enter the model. #### The Bounding Measurement What a fake dependency imitates determines the separation. If the stub returns the same full reply on every call, the faulty path never runs. The same two versions are tried with two stubs: one always returns a full reply of three products, the other returns an empty reply. ```swift // main.swift — the same two versions, two different stubs let start = ["notebook", "pen"] let full = ["notebook", "pen", "eraser"] let goodFull = await refreshedItems(start: start, service: StubService(reply: full), faulty: false) let badFull = await refreshedItems(start: start, service: StubService(reply: full), faulty: true) let goodNone = await refreshedItems(start: start, service: StubService(reply: []), faulty: false) let badNone = await refreshedItems(start: start, service: StubService(reply: []), faulty: true) let onFull = contrast("full-reply-stub", correct: goodFull, faulty: badFull) { $0 == "notebook,pen,eraser" }.describe() let onNone = contrast("empty-reply-stub", correct: goodNone, faulty: badNone) { $0 == "notebook,pen" }.describe() countSeparations([onFull, onNone]) ``` ```text full-reply-stub | correct=notebook,pen,eraser/pass faulty=notebook,pen,eraser/pass -> blind empty-reply-stub | correct=notebook,pen/pass faulty=empty/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` The output is identical across three runs. When the stub always returns a full reply, both versions replace the list with the reply and the check is blind; the path where the fault lives never runs. When the stub returns an empty reply, the same kind of check is separating. The check did not change; the reply the fake dependency imitates changed. This points to the most common mistake with fake dependencies: building the stub for the happy path. On the happy path nothing breaks, the check stays green, and the faulty path reaches the user with the next real empty reply. #### Summary - Dependency injection means an object receives the dependency it needs from outside; the most common way is to pass it through the initializer. - A check tied to the real network client depends on the network and the server; when the network gives no reply, it fails on both versions and is noisy. - A check with a stub that returns an empty reply triggers the path where the fault lives and is separating. - A stub that imitates an error reply is blind if the fault is in the empty reply branch: the three checks gave one separating, one blind and one noisy. - If the stub always returns a full reply, the faulty path does not run and the check stays blind. - This lesson's separation: **noisy**. #### Next Step Passing the dependency from outside separated the objects from each other; but if they all stand on the same target, every object can see every object. In the fifth lesson, Modularization will split the code into packages and put the question of which package may depend on which into a rule. There the fault will be one feature module binding directly to another feature module, and the check that looks at whether the build passes will stay blind to this fault. ### Modularization Source: https://academia.sh/en/courses/architecture-testing-and-release/modularization Splitting the application into modules along package boundaries, the rule that feature modules depend only on the core module, and the blindness of a build-passes check to a boundary violation. In the fourth lesson we learned to pass a dependency from outside: objects know each other through a contract instead of building each other. The same idea holds one step higher. If the whole application sits in a single target, every file can see every file; no rule prevents a view from calling the network layer directly, or one feature from reaching into another feature's internal types. Modularization splits the code into packages and makes each package show only the surface it opens to the outside. Which module may depend on which is a rule. This lesson's rule is this: feature modules depend only on the core module, and not on each other. The fault is the breaking of this rule: one feature module depends directly on another feature module. The core comes in verbatim, as in the earlier lessons. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` The core is not redefined in this lesson either. The separation names are the same. #### The Package Boundary A module is a target of the package manager. The dependencies between targets are written in the package definition file, and the package manager reads that file to decide which target may import which. The items of a module left `internal` are not visible from other modules; those opened with `public` or `open` cross the boundary; between modules of the same package, `package` is enough as well. ```swift // swift-tools-version: 6.0 import PackageDescription let package = Package( name: "Shop", targets: [ .target(name: "Core"), .target(name: "Feed", dependencies: ["Core"]), .target(name: "Profile", dependencies: ["Core"]), .target(name: "Settings", dependencies: ["Core"]), .target(name: "App", dependencies: ["Feed", "Profile", "Settings", "Core"]), ] ) ``` This package definition is not run; the package manager and the build are the platform's tools, and the lesson claims no output for them. Its structure shows the rule: `Feed`, `Profile` and `Settings` depend only on `Core`, and `App` brings them all together. In the faulty version a single line changes. ```swift .target(name: "Feed", dependencies: ["Core", "Profile"]), ``` `Feed` can now import `Profile` as well. The build does not reject this line: `Profile` exists as a target, and there is no cycle between them. This lesson's question lies exactly here: a dependency the compiler accepts may not be a dependency the rule accepts. #### The Dependency Graph The model reduces the package definition to a graph: the name of each module and the list of modules it depends on. The graph answers three questions: does every dependency really point to a module, is there a cycle in the graph, and which modules have to be rebuilt when a module changes. The third question is used in the bounding measurement. `Modules.swift` is a helper file and prints no output. ```swift // Modules.swift — module dependency graph struct ModuleGraph { let edges: [String: [String]] var resolves: Bool { edges.values.allSatisfy { deps in deps.allSatisfy { edges[$0] != nil } } } var isAcyclic: Bool { func visit(_ name: String, _ seen: Set) -> Bool { if seen.contains(name) { return false } return (edges[name] ?? []).allSatisfy { visit($0, seen.union([name])) } } return edges.keys.allSatisfy { visit($0, []) } } func dependents(of module: String) -> [String] { var found: Set = [] var queue = [module] while let current = queue.popLast() { for (name, deps) in edges where deps.contains(current) && !found.contains(name) { found.insert(name) queue.append(name) } } return found.sorted() } } func shopGraph(faulty: Bool) -> ModuleGraph { ModuleGraph(edges: [ "core": [], "feed": faulty ? ["core", "profile"] : ["core"], "profile": ["core"], "settings": ["core"], "app": ["feed", "profile", "settings", "core"], ]) } func featureViolations(_ graph: ModuleGraph) -> String { let broken = ["feed", "profile", "settings"].flatMap { feature in (graph.edges[feature] ?? []).filter { $0 != "core" }.map { "\(feature)->\($0)" } } return broken.isEmpty ? "none" : broken.joined(separator: ",") } ``` `resolves` says that every dependency is found in the graph as a module, `isAcyclic` says there is no cycle, and `featureViolations` applies the rule: it lists every module a feature module depends on other than the core in the form `feature->target`, and returns `none` if there are none. Three checks are applied to the same two graphs; the first two look at what the build would look at, and the third looks at the rule. ```swift // main.swift — the build or the rule let good = shopGraph(faulty: false) let bad = shopGraph(faulty: true) let compiles = contrast("build-passes", correct: good.resolves, faulty: bad.resolves) { $0 }.describe() let acyclic = contrast("no-cycle", correct: good.isAcyclic, faulty: bad.isAcyclic) { $0 }.describe() let rule = contrast("features-depend-only-on-core", correct: featureViolations(good), faulty: featureViolations(bad)) { $0 == "none" }.describe() countSeparations([compiles, acyclic, rule]) ``` ```text build-passes | correct=true/pass faulty=true/pass -> blind no-cycle | correct=true/pass faulty=true/pass -> blind features-depend-only-on-core | correct=none/pass faulty=feed->profile/fail -> separating separating: 1 blind: 2 noisy: 0 inverted: 0 ``` The output was run three times in the sandbox on Swift 6.4 and is identical in all three. Of the three checks, one came out separating and two blind. `build-passes` looks at whether the dependencies resolve. Because the `feed -> profile` link points to a real module, it resolves, and `true` comes out on both versions: the check is blind. `no-cycle` looks for a cycle. `feed -> profile` and `profile -> core` do not form a cycle, so this one also passes on both versions and is blind. Both are green and both are misleading: the fault is not in the validity of the graph but in the graph's conformance to the rule. `features-depend-only-on-core` applies the rule. In the correct version there is no violation and `none` comes out. In the faulty version `feed->profile` is listed and the check fails. This is separating, and it separates because it asks the rule's own question instead of a question as broad as the build. The build says "does this work"; the rule says "is this allowed". The value of modularization is in the second question. #### Assumptions **AR1.** The model reduces the package definition file to a dictionary. This lesson does not measure a real package manager's target resolution, version constraints or platform conditions; the package definition stays KAYNAK. **AR2.** The "build passes" check is modeled as the dependencies resolving and there being no cycle. A real build checks more than this (type errors, access control); those are outside this model. **AR3.** Build time is not measured. Time depends on the machine, the cache and the load; in this lesson a unit of work is counted instead: the number of dependents of the changed module that have to be rebuilt. #### The Bounding Measurement The same rule also has a cost in rebuilding. When a module changes, every module that depends on it directly or indirectly is rebuilt. We set a budget for each changed module: as this lesson's rule, when `profile` changes at most one module should be rebuilt, and when `core` changes at most four. The number is not time, it is a unit of work. ```swift // main.swift — dependents to rebuild when a module changes let good = shopGraph(faulty: false) let bad = shopGraph(faulty: true) let profileBudget = contrast("profile-budget", correct: good.dependents(of: "profile").count, faulty: bad.dependents(of: "profile").count) { $0 <= 1 }.describe() let coreBudget = contrast("core-budget", correct: good.dependents(of: "core").count, faulty: bad.dependents(of: "core").count) { $0 <= 4 }.describe() countSeparations([profileBudget, coreBudget]) ``` ```text profile-budget | correct=1/pass faulty=2/fail -> separating core-budget | correct=4/pass faulty=4/pass -> blind separating: 1 blind: 1 noisy: 0 inverted: 0 ``` The output is identical across three runs. In the correct version, when `profile` changes only `app` is rebuilt. In the faulty version `feed` also depends on `profile`, so `feed` and `app` are rebuilt and the budget is exceeded. The check is separating. `core`, on the other hand, has four dependents in both versions: `feed`, `profile`, `settings` and `app`. The fault does not change the number of dependents of the core, so the same kind of budget check is blind for `core`. The same check can be separating on one module and blind on another. The cost of a boundary violation is paid only around the module that contains it. #### Summary - Modularization splits the code along package boundaries; which module may depend on which is written in the package definition. - The lesson's rule: feature modules depend only on the core module and not on each other. - Checks that look at whether the build passes and whether there is no cycle are blind to a boundary violation; of three checks, two came out blind and one separating. - The check that applies the rule finds the violation by name as `feed->profile` and is separating. - The number of dependents to rebuild when a module changes is a unit of work and can be tied to a budget; it is separating on the `profile` budget and stays blind on the `core` budget. - This lesson's separation: **blind**. #### Next Step The architecture section is finished: the calculation was separated from the screen, navigation from the views, state from scatter, the dependency from the inside, and the features from each other. The next topic is tests. In the first lesson, Unit Tests will test one of these pieces in a test target. There the fault will be at a threshold: the shipping fee will be calculated wrongly when the cart total is exactly at the threshold, and a test written with a value below the threshold will stay blind to this fault. ## Testing and Performance Unit tests with test targets and assertions, UI tests with flow automation and stability, test plans and reading the coverage report, profiling tools, memory leaks through retain cycles, launch and scrolling performance, and formatting and static analysis tools. ### Unit Tests Source: https://academia.sh/en/courses/architecture-testing-and-release/unit-tests Setting up a unit test with a test target, an assertion and a test double, and measuring in four categories whether the same boundary check tells the correct version from the faulty one. In the previous lesson (Modularization) we looked at how a dependency that crosses a module's boundary becomes visible. This lesson moves to another kind of boundary: the one where a value changes the rule. If free shipping on a cart starts at a hundred units, the easiest place for the code to break is exactly the value one hundred. A unit test exists to try these places. The previous course (Data, Networking and Concurrency) compared two runs of an operation: does the result change if the operation is repeated or its order changes. This course asks something different. Here we look at a single **check** and apply it to two versions: the correct version of the code and a version into which we deliberately put a fault. If the check passes on the correct version and fails on the faulty one, it is doing its job. If it passes on both, it is telling us nothing. The measurement axis is **separation**. Four cases come out, and they appear in the output under their ASCII names. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both; it cannot see the fault. A **noisy** check fails on both; it rejects even the correct code. An **inverted** check fails on the correct version and passes on the faulty one; that is, it rewards the fault. | Category | Correct version | Faulty version | |---|---|---| | separating | passes | fails | | blind | passes | passes | | noisy | fails | fails | | inverted | fails | passes | The core below classifies these four cases. We do not assign the label ourselves; `separation` is derived only from the two truth values. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on those outputs. The `contrast` function applies the same `check` closure first to the correct output, then to the faulty one. `describe` writes the verdict on a single line. The reason for the rule: running a check on one version asks it nothing. A check is worth what it can tell apart between the two versions. #### Test Target, Test Class and Assertion A unit test has three parts. The **test target** is the package compiled separately from the app's code; the tests live there and import the app code. The **test class** is an `XCTestCase` subclass; each method in it whose name starts with `test` is a single experiment. An **assertion** is the call that compares the result of the experiment with the expected value; if it does not hold, the test fails. The example below checks the shipping fee of a cart. The rule is this: if the subtotal is one hundred or more, shipping is free; below that it costs fifteen units. ```swift // shipping fee test — SOURCE import XCTest final class ShippingFeeTests: XCTestCase { func testFeeAtThreshold() { XCTAssertEqual(shippingFee(for: 100), 0) } func testFeeFarAboveThreshold() { XCTAssertEqual(shippingFee(for: 500), 0) } func testFeeBelowThreshold() { XCTAssertEqual(shippingFee(for: 40), 15) } } ``` This fragment is not run; the test framework runs inside the test runner of the development environment. `XCTAssertEqual(_:_:)` compares two values. Each method runs independently; even if one method fails, the others still run. To prepare values, `setUp()` is called before each method and `tearDown()` after. The same three experiments can also be written in a second form. In this form a function marked with `@Test` is used in place of a method, and the `#expect(_:)` macro in place of an assertion. ```swift // shipping fee test, second form — SOURCE import Testing @Test func feeAtThreshold() { #expect(shippingFee(for: 100) == 0) } @Test func feeFarAboveThreshold() { #expect(shippingFee(for: 500) == 0) } ``` When a `#expect` assertion does not hold, it does not stop the test; the following assertions still run. If it needs to stop, `#require(_:)` is used. In the older form the counterpart of `#expect` is the `XCTAssertTrue(_:)` call, and the counterpart of `#require` is the `XCTUnwrap(_:)` call. Both forms ask the same question; in this lesson we measure only that shared question. #### The Shipping Fee Model The **model** below sets up what the check says to the two versions; it is not the test framework itself. In the model, `shippingFee(for:)` exists in two versions. The correct version returns zero at the threshold and above. In the faulty version the comparison is written as `>` instead of `>=`, so a subtotal of exactly one hundred is still charged. This is the best-known fault of unit tests: one side of a boundary is counted wrong. ```swift // Fees.swift — correct and faulty versions of the shipping fee, with a check helper func correctFee(for subtotal: Int) -> Int { subtotal >= 100 ? 0 : 15 } func faultyFee(for subtotal: Int) -> Int { subtotal > 100 ? 0 : 15 } func feeCheck(_ label: String, subtotal: Int, expected: Int) -> String { contrast(label, correct: correctFee(for: subtotal), faulty: faultyFee(for: subtotal), check: { $0 == expected }).describe() } ``` `feeCheck` is the model of one of the assertions above: it asks both versions for the fee at the same subtotal and, in place of `XCTAssertEqual(…, expected)`, performs the comparison `$0 == expected`. The labels are the ASCII kebab-case names of the tests. Next to the three real experiments we added a fourth: a test whose expected value is written wrong. This way three categories appear together. ```swift // main.swift — four checks, two versions let atThreshold = feeCheck("threshold-fee", subtotal: 100, expected: 0) let farAbove = feeCheck("far-above-fee", subtotal: 500, expected: 0) let below = feeCheck("below-fee", subtotal: 40, expected: 15) let wrongSpec = feeCheck("wrong-expected", subtotal: 40, expected: 10) countSeparations([atThreshold, farAbove, below, wrongSpec]) ``` ``` threshold-fee | correct=0/pass faulty=15/fail -> separating far-above-fee | correct=0/pass faulty=0/pass -> blind below-fee | correct=15/pass faulty=15/pass -> blind wrong-expected | correct=15/fail faulty=15/fail -> noisy separating: 1 blind: 2 noisy: 1 inverted: 0 ``` `threshold-fee` is separating: the correct version returns zero and the check passes; the faulty version returns fifteen and the check fails. `far-above-fee` and `below-fee` are blind: both pass on both versions, because the two versions behave the same far from the value one hundred. `wrong-expected` is noisy: since the expected value is ten, even the correct version fails. What stands out here is that all three tests could look green. Of the three real checks, two passed on the faulty version too. A test suite that does not try the boundary value can report code carrying the fault as green. This is the first sign that coverage and the number of tests alone give no guarantee; we will measure coverage separately in the third lesson. **TP1.** This measurement tried a single fault, `>` written in place of `>=`. If the code had broken in another way, for example if the threshold were ninety-nine instead of one hundred, which check separates would change. **TP2.** We know the model's `faultyFee` function because we put the fault there ourselves. In real code the location of the fault is not known in advance; this is why the separation measure is meaningful together with mutation testing (deliberately breaking the code and watching whether the test reacts), where the fault is known. #### Why the Boundary Value Separates The difference between a separating check and a blind check can be read as a rule: **try the value where the rule changes**. If the threshold is one hundred, the boundary values are ninety-nine, one hundred and one hundred one. One hundred carries the most information because it separates the two sides of the rule. A difference between `>=` and `>` appears only at this value; no other input can tell the two versions apart. Blind checks are not worthless. `below-fee` says the fee is fifteen, and that is a behavior contract; if someone changes the fee later, the test catches it. But as a check meant to find the fault, it is blind. The two roles are separate: a test can pin down behavior and still be unable to separate a particular fault. The separation measure looks at the second. The question to ask while writing a test is this: if I applied this check to the faulty version, would it fail. If the answer is yes, the check separates. If no, you write it knowing it is blind. The cheapest way to answer the question is to break the comparison in the code for a minute after writing the test and see whether the test turns red. #### Test Doubles Real cart code does not compute the threshold and the fee itself; it gets them from a pricing service, a clock or a network client. A unit test does not run these dependencies in their real form, because a test must be fast and deterministic. **Dependency injection** is giving the dependency from outside; the test also puts a **test double** in its place. A test double has three common forms. A **stub** returns a ready-made answer. A **spy** records that it was called, and the call is inspected afterwards. A **dummy** only fills a place and is never called. These three are collectively also called a **mock**. ```swift // test double — SOURCE protocol RateProviding { var freeShippingThreshold: Int { get } } struct StubRates: RateProviding { let freeShippingThreshold = 100 } final class SpyReceipt { private(set) var printedTotals: [Int] = [] func print(total: Int) { printedTotals.append(total) } } ``` `StubRates` is a stub: it gives the threshold as a fixed value, and the test is sure the threshold is one hundred. `SpyReceipt` is a spy: it keeps which totals were written to the receipt, and the test later looks at the `printedTotals` array. This fragment also stays SOURCE; it is only type declarations and has no output to measure. The limit of a test double is this: when a stub gives a fixed value, the behavior of the code with the real dependency is not measured. Because the threshold is pinned at one hundred, a fault where the threshold arrives wrong from the server does not come out of this test. A unit test measures the logic of the code; another test layer measures the components working together. #### The Bounding Measurement The sneakiest fault of a unit test is in the expected value itself. When a test is written, the expected value comes from a source called a **test oracle**: a specification, a hand calculation or another independent path. Sometimes there is no oracle. The author runs the code, sees the value that comes out and pastes it into the assertion. The value passes at that moment, because the assertion is the code's own output. The measurement below does this. The same `threshold-fee` assertion that tries the boundary value is taken, but the expected value is copied from the output of the faulty version. ```swift // main.swift — expected value copied from the faulty code let pasted = faultyFee(for: 100) let copied = feeCheck("pasted-expected", subtotal: 100, expected: pasted) countSeparations([copied]) ``` ``` pasted-expected | correct=0/fail faulty=15/pass -> inverted separating: 0 blind: 0 noisy: 0 inverted: 1 ``` The same assertion that tries the boundary value was separating in the previous measurement. Now it is **inverted**: the correct version fails, the faulty version passes. The check itself did not change; what changed is where the expected value comes from. When the output of the faulty code is made the oracle, the test approves the fault and later, when the code is fixed, turns red. The team blames the test, not the fault. This is why separation depends not only on what the check does but also on the source of the expected value. A good oracle exists before the code is read: the cart specification, a hand-calculated example or the documented output of the previous version. **TP3.** This measurement makes the copy from the faulty version. A value copied from the correct version would have made the assertion separating; but even then the value can be trusted only if its correctness is known beforehand. **TP4.** The measurement checks a single boundary, one hundred. All three boundary values would have given the same inverted result, but we did not measure that separately. #### Common Mistakes - Trying only values far from the threshold. Such checks stay blind. - Copying the expected value by running the code. If the oracle is the code itself, the test approves the fault. - Checking more than one behavior in a single test. When the test fails, it is unclear which behavior broke. - Making a test complicated by trying to make a test double resemble the real thing. A stub should behave only as much as the test needs. - Accepting a test as green without ever seeing it red. A test must have failed at least once on code that was deliberately broken. #### Summary - A unit test lives in the test target; each method in the test class is a single experiment, and an assertion compares the result with the expected value. - A check passing does not show the code is correct; separation is measured by whether the check tells the correct version from the faulty one. - Of four checks, one was separating, two blind and one noisy; the check that tried the boundary value was the only separating one. - When the expected value was copied from the faulty code, the same check became inverted: it rejected the correct version and accepted the faulty one. - Test doubles keep a test fast and deterministic but do not measure behavior with the real dependency. - This lesson's separation: **separating**. #### Next Step A unit test checks the logic of the code; it does not check the screen the user sees. In the next lesson, UI Tests will measure which checks of a driver that taps a button find the fault, and which break only when the text changes. ### UI Tests Source: https://academia.sh/en/courses/architecture-testing-and-release/ui-tests Setting up and waiting in UI tests that drive the app as a separate process and tap a button; measuring whether three checks and the moment of waiting tell the correct version from the faulty one. A unit test calls a function and looks at the value that comes back. A user does not call functions; a user taps a button. Even if the right code runs behind the button, the button may not be wired to the screen. In the previous lesson (Unit Tests) we saw that a boundary fault was caught only by the check that tried the boundary value. In this lesson the fault is somewhere else: the "Confirm Order" button is visible, it can be tapped, but it does not send an order. The measurement axis is **separation** again: we apply the same check to the correct version and to the faulty version. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. We will see the four names in the output as ASCII. The core is the same as the core in the previous lesson and is written out again, because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on those outputs; `separation` derives the label only from the two truth values. The `contrast` function applies the same closure to both outputs, and `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the two versions apart. #### Driving the App A UI test does not call the app's code. The test target launches the app as a separate process and touches the screen from outside, like a user. So from inside the test the app's types and functions cannot be reached; only the elements visible on the screen can. The driver is the `XCUIApplication` object. ```swift // order confirmation UI test — SOURCE import XCTest final class CheckoutUITests: XCTestCase { override func setUpWithError() throws { continueAfterFailure = false } func testConfirmButtonExists() { let app = XCUIApplication() app.launchArguments = ["--reset-orders"] app.launch() XCTAssertTrue(app.buttons["confirmButton"].exists) } func testConfirmAddsOrder() { let app = XCUIApplication() app.launchArguments = ["--reset-orders"] app.launch() app.buttons["confirmButton"].tap() XCTAssertTrue(app.tables["orderList"].cells.element(boundBy: 0).waitForExistence(timeout: 5)) } } ``` This fragment is not run; it needs a simulator or a physical device. `continueAfterFailure = false` stops the test when the first assertion does not hold; otherwise a test that keeps tapping an empty screen produces meaningless errors one after another. `launchArguments` tells the app the starting state of the test: here, a flag that empties the order list. If each test does not start from the same state, the record left by the previous one changes the results. Elements are found in three ways: by visible text (`app.buttons["Confirm Order"]`), by index (`element(boundBy:)`) or by **accessibility identifier**. The identifier is a fixed name given to the element in the app code, and it stays the same even if the text changes. In the fragment above the button is found with the `confirmButton` identifier. #### Waiting The screen does not change instantly. When the button is tapped, the order is sent, the response arrives and the list refreshes; this chain takes some time. `waitForExistence(timeout:)` waits for the element to appear on the screen for at most the given time and returns a `Bool`. Reading `exists` directly without waiting risks getting the wrong answer while the element has not been drawn yet. In this lesson we will not measure durations and we will not show how close a test comes to its timeout. Timing varies from machine to machine; printing a number that depends on it would not be repeatable. Instead, in the model the wait is split into two states: the screen right after the button is tapped and the state of the screen after it has settled. In a real test, how many milliseconds pass between these two moments is not measured; in the model there is only "before" and "after". #### The Confirmation Screen Model The **model** below sets up what the checks say to the two versions; it is not the UI test framework itself. The screen is a struct. It holds the button's title, the pending orders and the orders listed on the screen. `tapConfirm` represents tapping the button and `settle` represents the screen settling. In the correct version the tap adds an order; in the faulty version tapping the button does nothing. ```swift // Checkout.swift — state of the confirmation screen, correct and faulty versions struct CheckoutScreen { let confirmTitle = "Confirm Order" private(set) var pending: [String] = [] private(set) var orders: [String] = [] mutating func tapConfirm(sending: Bool) { if sending { pending.append("order-1") } } mutating func settle() { orders += pending pending = [] } } func screenAfterTap(sending: Bool, settled: Bool) -> CheckoutScreen { var screen = CheckoutScreen() screen.tapConfirm(sending: sending) if settled { screen.settle() } return screen } ``` Three checks are applied to the settled state. The first is **bound to text**: it compares the button's title with the spelling in the app's old version ("Confirm order"). The text was later changed with a typo fix to a capital letter, and both versions carry the new text. The second is **bound to presence**: is the button on the screen. The third is **bound to outcome**: after the button is tapped, has the order list grown by one order. ```swift // main.swift — three checks, two versions let correct = screenAfterTap(sending: true, settled: true) let faulty = screenAfterTap(sending: false, settled: true) let oldTitle = contrast("old-title-text", correct: correct.confirmTitle, faulty: faulty.confirmTitle, check: { $0 == "Confirm order" }).describe() let exists = contrast("button-exists", correct: !correct.confirmTitle.isEmpty, faulty: !faulty.confirmTitle.isEmpty, check: { $0 }).describe() let listGrew = contrast("order-list-grew", correct: correct.orders.count, faulty: faulty.orders.count, check: { $0 == 1 }).describe() countSeparations([oldTitle, exists, listGrew]) ``` ``` old-title-text | correct=Confirm Order/fail faulty=Confirm Order/fail -> noisy button-exists | correct=true/pass faulty=true/pass -> blind order-list-grew | correct=1/pass faulty=0/fail -> separating separating: 1 blind: 1 noisy: 1 inverted: 0 ``` `old-title-text` is noisy: both versions carry the "Confirm Order" title and the check expects the old spelling, so it rejects both. The button works, and the test turns red. `button-exists` is blind: the button is on the screen in both versions. `order-list-grew` is separating: in the correct version the list holds one order, in the faulty version it holds zero. All three of these checks can be written in a UI test, and all three look the same in a test report. One breaks for no reason when the copy in the app changes, one stays green even if the button does not work at all, and one says the button does its job. The difference is what the check looks at: text, presence or outcome. **TP1.** The screen model adds a single order. Multiple orders, deletion or cancellation must be measured separately. **TP2.** What makes the `old-title-text` check noisy is that both versions carry the new text. If the text had changed only in the faulty version, the same check could have been inverted or separating. #### The Check Bound to Text A check bound to text finds the button by its visible text. This is convenient, because whatever is written on the screen is also in the test. But the text is in the hands of marketing, translation or typo fixes. When the title changes, the app keeps working correctly and the test turns red. This kind of breakage does not point to a fault; it creates a maintenance burden. Moreover, in a multilingual app the text of the same button changes by language. A test bound to text works only in the language it was written in. When a second language configuration is added to the test plan, the tests bound to text fail in the second language; we will measure this in a later lesson of this topic. A button found with an identifier carries the same name in every language. Throwing text out of the test altogether is not a solution either. Some tests check the text itself: when an order is confirmed, the screen showing "Your order has been received" is a behavior. The difference is this: using text to find an element brings a maintenance burden, while using it to verify a behavior does not. #### The Check Bound to Outcome `order-list-grew` separated because it looked at the effect the button produces. If the button really works, the list grows; if it does not, it does not grow. When writing a UI test, the first question should not be "is there a button" but "what should happen when the button is tapped". The check bound to outcome has a cost. If the place where the order list appears on the screen changes, if it moves to a new tab, or if the table becomes a collection, the test breaks again. This is why the identifier is kept fixed and, as the screen's structure changes, the identifier is carried along. A check bound to outcome separates, but it also asks for maintenance. UI tests are slow. Each one launches the app, drives the screen and finishes. So their number is kept small and their coverage wide: unit tests check the code's logic, and UI tests check a few essential paths end to end. Writing a UI test for a boundary fault is the wrong tool; so is writing a unit test for a wiring fault. #### The Bounding Measurement The same check bound to outcome behaves differently depending on the moment of waiting. Below, the `order-list-grew` check is applied on two screens: the screen looked at right after the button is tapped, and the screen looked at after it has settled. We named the first `immediate-look` and the second `after-wait`. ```swift // main.swift — same check, two moments of looking let immediateCorrect = screenAfterTap(sending: true, settled: false) let immediateFaulty = screenAfterTap(sending: false, settled: false) let settledCorrect = screenAfterTap(sending: true, settled: true) let settledFaulty = screenAfterTap(sending: false, settled: true) let immediate = contrast("immediate-look", correct: immediateCorrect.orders.count, faulty: immediateFaulty.orders.count, check: { $0 == 1 }).describe() let afterWait = contrast("after-wait", correct: settledCorrect.orders.count, faulty: settledFaulty.orders.count, check: { $0 == 1 }).describe() countSeparations([immediate, afterWait]) ``` ``` immediate-look | correct=0/fail faulty=0/fail -> noisy after-wait | correct=1/pass faulty=0/fail -> separating separating: 1 blind: 0 noisy: 1 inverted: 0 ``` The same check became noisy when applied before the screen had settled: even the correct version shows zero orders and the test fails. After waiting, the same check is separating. A check's value depends not only on what it looks at but also on when it looks. In the model the two moments are exact and deterministic: before and after. In a real app the moment of settling varies; in some runs the test catches up with the wait, in others it cannot. A test whose results are uncertain like this is called a **flaky test**, and it is not printed in this lesson's model, because the model gives the same result on every run and does not represent uncertainty. To fix a flaky test, explicit waits such as `waitForExistence(timeout:)` are used; sleeping for an arbitrary time hides the problem. **TP3.** The model counts settling as a single step. In reality the list can refresh in several steps and intermediate states can be seen. **TP4.** The `immediate-look` check coming out noisy comes from the model counting both screens as empty at the same moment. In a real run the screen may sometimes be ready and the result changes. #### Common Mistakes - Finding the button only by its text. The test breaks when the title changes or when the app opens in another language. - Checking only that the button is on the screen. It stays blind and misses a button that does nothing. - Waiting by sleeping for a fixed time. If the time is short the test becomes flaky; if it is long it slows down for no reason. - Starting every test from the state left over from the app's previous run. The starting state must be reset with flags. - Looking for a boundary fault with a UI test. A unit test is faster and more precise for that. #### Summary - A UI test drives the app as a separate process and reaches the screen only through visible elements. - Elements are found by text, by index or by accessibility identifier; the identifier stays the same even when the text changes. - Of three checks, one came out noisy, one blind and one separating; only the check bound to outcome showed that the button sends an order. - The same check became noisy when applied before the screen settled; the moment of waiting changed the check's category. - Flaky tests, whose results are uncertain, are not printed in the model; their fix is an explicit wait, not an arbitrary duration. - This lesson's separation: **noisy**. #### Next Step In this lesson we wrote the checks one by one. In the next lesson, Test Plans and Coverage will gather all the tests under one plan, count which lines run, and measure that a line running does not say it is correct. ### Test Plans and Coverage Source: https://academia.sh/en/courses/architecture-testing-and-release/test-plans-and-coverage Reading test plan configurations and a coverage report; measuring whether a coverage rule that checks the number of lines run and two language configurations tell the correct version from the faulty one. In the previous lesson (UI Tests) we saw that a test checking the outcome of a button separated, while a test checking only its presence did not. This lesson steps up one layer: to the plan that brings hundreds of tests together and to the coverage report that says how much of the code those tests touch. The measurement axis is **separation**: we apply the same check first to the correct version and then to the faulty one. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. The core is the same as in the previous lessons; we write it out again because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the two outputs and the check's verdict on each. The `separation` label derives from these two verdicts; `contrast` applies the same closure to both outputs; `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the correct version from the faulty one. #### The Test Plan An app can have more than one test target: unit tests, UI tests, performance tests. A **test plan** gathers them in a single file. The plan says which test targets will run, which tests will be skipped and with which **configurations** the tests will run. A configuration is the same tests running a second time with a changed setting: screen language, region, coverage collection, random order and the like. The plan is a JSON file, and the development environment shows it as a table. From the command line it is given to the `xcodebuild test` command with the `-testPlan` flag; this way the same plan runs the same tests both on the development machine and on the automated check server. ```json { "configurations": [ { "name": "Turkish", "options": { "language": "tr", "region": "TR" } }, { "name": "English", "options": { "language": "en", "region": "US" } } ], "defaultOptions": { "codeCoverage": true }, "testTargets": [ { "target": { "name": "ShopTests" } } ] } ``` This is a configuration file; the lesson does not run it. There are two configurations: Turkish and English. While the `codeCoverage` option is on, which lines ran is recorded in every run. Every test in the plan runs in both configurations; that is, the same test is tried twice and can have two different results. #### What Coverage Measures **Code coverage** counts which lines or branches of the code were entered while a test suite ran. The report says "ran" or "did not run" line by line and gives a ratio. This information is valuable: a line that never ran has certainly not been tried. But there is something coverage does not say. A line running does not say its result is correct. If a test calls a function and never looks at the returned value, the line counts as run and coverage goes up. Thinking that high coverage is **assurance** is a common misconception that arises from here. Low coverage points to a gap; high coverage does not show there is no gap. The two tests below show this difference. The first runs the discount branch but discards the returned value. The second runs the same branch and compares the result with 180. ```swift // discount tests — SOURCE import XCTest final class PricingTests: XCTestCase { func testDiscountBranchRuns() { _ = price(100, coupon: true) _ = price(100, coupon: false) } func testDiscountedTotal() { XCTAssertEqual(price(200, coupon: true), 180) } } ``` Both tests show the discount line green in the coverage report. But only the second asks whether the discount is computed correctly. The first says nothing more than that the function did not crash. #### The Discount Model The **model** below sets up what the coverage count and the check say to the two versions; it is not the coverage tool itself. A real tool gets its line counts from the compiler. The model does the same job on a small scale: it puts a marker on each branch of the function and collects the markers that ran in a set. There are three markers: `entry`, `discount` and `plain`. In the correct version, ten percent is taken off a cart with a coupon. In the faulty version the division has been changed to a hundred and only one percent is taken off. Both pass through the same branches; the difference is only in the calculation. ```swift // Pricing.swift — correct and faulty versions of the discount rule, with a counter of lines run func correctPrice(_ subtotal: Int, coupon: Bool, hits: inout Set) -> Int { hits.insert("entry") if coupon { hits.insert("discount") return subtotal - subtotal / 10 } hits.insert("plain") return subtotal } func faultyPrice(_ subtotal: Int, coupon: Bool, hits: inout Set) -> Int { hits.insert("entry") if coupon { hits.insert("discount") return subtotal - subtotal / 100 } hits.insert("plain") return subtotal } func executedCoverage(_ price: (Int, Bool, inout Set) -> Int) -> String { var hits = Set() _ = price(100, true, &hits) _ = price(100, false, &hits) return "\(hits.count) of 3 lines" } func discountedTotal(_ price: (Int, Bool, inout Set) -> Int) -> Int { var hits = Set() return price(200, true, &hits) } ``` `executedCoverage` is the model of the first test: it makes the two calls, discards the result and writes how many markers ran. `discountedTotal` is the model of the second test: it runs the same branch and returns the result. This way we can put the two checks side by side. The coverage check is the question "did three of three lines run"; the outcome check is the question "is the total 180". ```swift // main.swift — coverage check and outcome check let coverage = contrast("coverage-check", correct: executedCoverage(correctPrice), faulty: executedCoverage(faultyPrice), check: { $0 == "3 of 3 lines" }).describe() let outcome = contrast("discount-result", correct: discountedTotal(correctPrice), faulty: discountedTotal(faultyPrice), check: { $0 == 180 }).describe() countSeparations([coverage, outcome]) ``` ``` coverage-check | correct=3 of 3 lines/pass faulty=3 of 3 lines/pass -> blind discount-result | correct=180/pass faulty=198/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` `coverage-check` is blind: in both versions three of three lines ran and the coverage threshold passed. The faulty version's coverage is the same as the correct version's, because the fault is not in which line ran but in what that line computes. `discount-result` is separating: the correct version gives 180, the faulty version gives 198, and the check accepts only 180. These two lines alone call into question a coverage threshold decision. When a team sets the rule "no merge without a hundred percent coverage", the faulty version passes this rule. The rule catches missing tests but does not catch wrong tests. **TP1.** The model counts a single function with three markers. Real code has thousands of lines and branches; but the difference in a single function is hidden in the same way in a large table. **TP2.** The coverage check here looks at line markers. Branch coverage or condition coverage measures more finely, but they too do not measure the correctness of the calculation. #### How to Use a Coverage Threshold The useful way to use coverage is as a floor, not as a goal. A file with low coverage points to code that has never been tried; writing tests there is the beginning of adding outcome checks. High coverage, on the other hand, is not a result. Writing assertion-free tests to raise coverage spoils the measure: the number goes up, the assurance does not. So when a test is reviewed, two questions are asked. The first: which lines does the test enter. The second: which results does the test check. Coverage answers only the first. For the second, one has to look at the same test applied to the faulty version; this lesson's separation measure does exactly that. There is also the problem of false confidence. Teams often put the coverage percentage on a dashboard like a quality indicator. As the indicator rises, the team believes the tests are sufficient and the effort to write new tests drops. The real function of coverage is to make visible the pieces of code the tests never touch. #### Configurations and Expectation The second function of the test plan is configuration. The same test runs in two languages. A money amount is written `180,00` in the Turkish region and `180.00` in the English region; the date format changes in a similar way. If a test wrote the expected text for only one language, even the correct version fails in the other configuration. In the model, formatting is not a real formatter but a fixed rule for two languages: in Turkish the decimal separator is a comma, in English a dot. In a real app this job is left to the platform's formatter. #### The Bounding Measurement The measurement below applies the text form of the `discount-result` check in two configurations. The expected text is written for the Turkish configuration as `180,00`. Both versions are formatted separately in each configuration. ```swift // main.swift — same expectation, two configurations func formattedTotal(_ total: Int, language: String) -> String { language == "tr" ? "\(total),00" : "\(total).00" } let correctTotal = discountedTotal(correctPrice) let faultyTotal = discountedTotal(faultyPrice) let turkish = contrast("tr-configuration", correct: formattedTotal(correctTotal, language: "tr"), faulty: formattedTotal(faultyTotal, language: "tr"), check: { $0 == "180,00" }).describe() let english = contrast("en-configuration", correct: formattedTotal(correctTotal, language: "en"), faulty: formattedTotal(faultyTotal, language: "en"), check: { $0 == "180,00" }).describe() countSeparations([turkish, english]) ``` ``` tr-configuration | correct=180,00/pass faulty=198,00/fail -> separating en-configuration | correct=180.00/fail faulty=198.00/fail -> noisy separating: 1 blind: 0 noisy: 1 inverted: 0 ``` In the Turkish configuration the check separates. In the English configuration the same check becomes noisy: the correct version also produces `180.00`, so it fails. When the plan runs both configurations together, the report is red, and the cause of the red is not a fault. After a while the team gets used to this red and, when a real fault appears, passes over it with the same habit. The solution is not to write the expectation tied to a language: either the expected text is produced according to the configuration's language, or the check looks at the numeric value rather than the text. This measurement compares only two configurations; both run with the two versions that carry the same fault. **TP3.** We did not measure the percentage of a real coverage report anywhere in this lesson. The model counts the markers that ran itself; a percentage is a product output and is not this lesson's measure. **TP4.** The formatting in the model is a fixed rule for two languages. A real formatter carries more region and currency detail. #### Common Mistakes - Counting the coverage percentage as a quality indicator. The percentage does not say whether the result is correct. - Writing tests that do not check the result in order to raise coverage. - Writing the expected text for one language and running the plan in two languages. The second configuration stays red all the time. - Treating the red in a configuration as normal and not looking at it. - Adding every test and every configuration to the plan. If the plan does not stay fast, the team does not run it. #### Summary - A test plan gathers test targets and configurations in one file; the same test runs separately in each configuration. - Coverage counts which line ran; it does not ask whether the result is correct. - Of two checks, one came out blind and one separating: the coverage check passed on both versions, the outcome check passed only on the correct version. - When the expectation was written for the Turkish format, the same check became noisy in the second configuration. - Coverage is a floor; it shows a missing test, it does not show a wrong test. - This lesson's separation: **blind**. #### Next Step Tests check the correctness of code; they do not check its speed. In the next lesson, Profiling Tools will count the work a list cell does on every draw and measure when a check that looks at the amount of work separates. ### Profiling Tools Source: https://academia.sh/en/courses/architecture-testing-and-release/profiling-tools How a profiler shows where an app spends time through timelines and signposts; measuring whether a work unit budget and a check on the drawn text tell the correct version from the faulty one. In the previous lesson (Test Plans and Coverage) we saw that a test entering a line does not say that line is correct. In this lesson we look at code that is correct but slow. If a list cell builds the same formatter from scratch every time it is drawn, the text shown on the screen is correct and the test stays green; but while scrolling, the app does more work than it needs to. The measurement axis is **separation** again: we apply the same check to the correct version and to the faulty version. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. We give the core again, because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on each; the `separation` label derives from these two verdicts. `contrast` applies the same closure to both outputs, and `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the two versions apart. #### What Profiling Is **Profiling** is measuring a running app's time, memory and rendering work. A test says yes or no to a question; a profiler answers the question "where". You record a single request, a single scroll or a single screen, and the tool shows the timeline of the recording. Profiling has two rules. The first is that the measurement is taken with the release configuration; the debug configuration turns off optimizations and distorts the numbers. The second is that the measurement is taken on a physical device where possible; the simulator runs on the desktop processor and does not carry the memory and rendering limits of a physical device. A profiler offers several timelines. The **CPU timeline** shows which functions spend time as a call tree; the heaviest call is often the place being sought. The **memory timeline** tracks allocated objects and their lifetimes. The **rendering timeline** shows how long each frame takes to prepare. Each timeline answers a different question: the CPU "what is running", memory "what is accumulating", rendering "is the screen keeping up". #### Measuring with Signposts The tool shows the whole app, but the section you are interested in may be small. A **signpost** is a start and end pair you put in the code; the tool shows this pair on the timeline as a named interval. ```swift // cell drawing signpost — SOURCE import os let signposter = OSSignposter(subsystem: "com.example.shop", category: "list") func configure(cell: PriceCell, price: Decimal) { let state = signposter.beginInterval("configure") cell.show(price) signposter.endInterval("configure", state) } ``` This fragment is not run; it has meaning inside the profiler. The interval's name and category are used as filters on the timeline. The cost of placing a signpost is low, and signposts can be left in the release configuration. The older, function-based form is written with `os_signpost`; the new form uses the `OSSignposter` object. Both do the same thing. Now consider the faulty cell. Building a new formatter on every call is one of the most common slowdowns in a price cell, because a formatter is an expensive object. ```swift // cell configuration — SOURCE import UIKit final class PriceCell: UITableViewCell { func show(_ price: Decimal) { let formatter = NumberFormatter() formatter.numberStyle = .currency textLabel?.text = formatter.string(from: price as NSDecimalNumber) } } ``` Every time `show(_:)` is called it builds a `NumberFormatter`. While a list is scrolled, hundreds of cells are drawn, so hundreds of formatters are built. The fix is to build the formatter once and share it in every cell. ```swift // cell configuration, fixed — SOURCE import UIKit final class PriceCell: UITableViewCell { private static let formatter: NumberFormatter = { let formatter = NumberFormatter() formatter.numberStyle = .currency return formatter }() func show(_ price: Decimal) { textLabel?.text = Self.formatter.string(from: price as NSDecimalNumber) } } ``` The two versions show the same text on the screen. The difference is only in how many times an object is built. #### The Work Unit Model The **model** below sets up what a profiler measures; it is not the tool itself. We will not measure time: milliseconds vary from machine to machine, and a number printed here would not be repeatable. Instead we count each piece of work as a **work unit**. Building a formatter is one unit, formatting a price is one unit. With the number of cells the same, the number of objects built determines the work units. The correct version builds the formatter once and reuses it on every row. The faulty version rebuilds it on every row. Twenty rows are drawn. ```swift // Cells.swift — correct and faulty versions of the price cell, with a work unit counter struct Rendered { let firstRow: String let units: Int } func formatPrice(_ cents: Int) -> String { let rest = cents % 100 return "\(cents / 100).\(rest < 10 ? "0" : "")\(rest)" } func renderRows(count: Int, rebuildsFormatter: Bool) -> Rendered { var units = 0 var first = "" if !rebuildsFormatter { units += 1 } for row in 1...count { if rebuildsFormatter { units += 1 } units += 1 if row == 1 { first = formatPrice(1000 + row * 25) } } return Rendered(firstRow: first, units: units) } ``` `renderRows` draws twenty rows. In the correct version the formatter is built once (one unit), and then each row spends one unit. In the faulty version each row spends two units. `firstRow` is the text of the first row; the same price comes out in both versions. Two checks are applied. The first looks at the **drawn text**: is the first row `10.25`. The second looks at the **work unit budget**: does the total number of units exceed twenty-five. Twenty-five is this lesson's rule; it is not a real budget. ```swift // main.swift — drawn text and work unit budget let correct = renderRows(count: 20, rebuildsFormatter: false) let faulty = renderRows(count: 20, rebuildsFormatter: true) let text = contrast("rendered-text", correct: correct.firstRow, faulty: faulty.firstRow, check: { $0 == "10.25" }).describe() let budget = contrast("work-unit-budget", correct: correct.units, faulty: faulty.units, check: { $0 <= 25 }).describe() countSeparations([text, budget]) ``` ``` rendered-text | correct=10.25/pass faulty=10.25/pass -> blind work-unit-budget | correct=21/pass faulty=40/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` `rendered-text` is blind: both versions show the same price. `work-unit-budget` is separating: the correct version spends twenty-one units and fits the budget, the faulty version spends forty units and does not. Forty is almost twice twenty-one, and the difference comes exactly from the formatter rebuilt on every row. There is an important result here: no test that checks the visible behavior can find this fault. A unit test, a UI test and a coverage threshold all approve this faulty cell. The fault is seen only by a check that looks at the amount of work, or by the profiler's timeline. **TP1.** The model counts building a formatter and formatting a price at the same cost. The real cost ratio was not measured and is not the subject of this lesson; only how many times it is built is counted. **TP2.** The model draws twenty rows. As the number of rows grows, the difference between the two versions grows; this lesson's measure covers only twenty rows. #### Reading the Timeline When a profile recording is opened, there is an order to reading it. First find the interval on the timeline where the slowdown shows; signposts make this easier. Then, within that interval, look at the call tree in the CPU timeline and start from the call that spends the most time. Inverting the call tree, that is, seeing the bottom-most calls at the top, quickly shows the function doing the spending. In the cell above, the call tree could have shown the `NumberFormatter` setup among the heavy calls. The tree itself is not a diagnosis of a fault; it says "time is going here". Understanding why the time goes and deciding whether it is necessary is still the reader's job. Another trap in profiling is making the measurement depend on the measuring. The tool slows the app down while recording, and placing signposts too often changes what is measured. So the recording is kept short and a single scenario is measured. #### The Bounding Measurement The work unit budget was a separating check. But the limit of the budget determines the category of the check. Below, the same two versions are checked with three different budgets: twenty-five, fifty and ten. ```swift // main.swift — same counter, three budgets let correct = renderRows(count: 20, rebuildsFormatter: false) let faulty = renderRows(count: 20, rebuildsFormatter: true) let tight = contrast("tight-budget", correct: correct.units, faulty: faulty.units, check: { $0 <= 10 }).describe() let fitting = contrast("fitting-budget", correct: correct.units, faulty: faulty.units, check: { $0 <= 25 }).describe() let loose = contrast("loose-budget", correct: correct.units, faulty: faulty.units, check: { $0 <= 50 }).describe() countSeparations([tight, fitting, loose]) ``` ``` tight-budget | correct=21/fail faulty=40/fail -> noisy fitting-budget | correct=21/pass faulty=40/fail -> separating loose-budget | correct=21/pass faulty=40/pass -> blind separating: 1 blind: 1 noisy: 1 inverted: 0 ``` Ten units is too tight: even the correct version exceeds it and the check becomes noisy. Fifty units is too loose: the faulty version fits as well and the check becomes blind. Only twenty-five stands above the correct version and below the faulty one. The same counter, the same kind of check; only the threshold changed, and the category changed three times. A budget is itself such a check. The person who sets the budget must measure what the correct version spends; a budget set without measuring is either red from the first day or never red. And as the correct version slows down and approaches the faulty one, the budget's margin changes too. **TP3.** The model's thresholds are the lesson's rule. The number for a real budget is found only by measurement, and it is different in every app. **TP4.** All three measurements use the same twenty-row drawing. With another number of rows the most suitable budget would be another number. #### Common Mistakes - Profiling in the debug configuration. Because optimizations are off, the numbers mislead. - Measuring only on the simulator. A physical device has different limits. - Putting signposts on everything. Signposts fill the timeline and change the work being measured. - Counting green tests that check the visible output as a performance guarantee. - Setting the budget by guess, without measuring. #### Summary - Profiling answers the question "where"; a test says yes or no. - The CPU, memory and rendering timelines correspond to different questions; signposts name the interval you are interested in on the timeline. - Of two checks, one came out blind and one separating: the drawn text was the same in both versions, the work unit budget passed only on the correct version. - The same budget check became noisy, separating or blind depending on the threshold; a budget must be set by measuring. - Tests that look at the visible output cannot find a performance fault. - This lesson's separation: **separating**. #### Next Step Lost time is a performance problem; an object that stays in memory is a separate fault. In the next lesson, Memory Leak Detection will measure, through a retain cycle, an object that is still alive after a screen has closed. ### Memory Leak Detection Source: https://academia.sh/en/courses/architecture-testing-and-release/memory-leak-detection How a retain cycle forms, how a weak reference breaks it, and testing whether an object is really released; measuring whether two leak checks tell the correct version from the faulty one. In the previous lesson (Profiling Tools) we measured a cell whose visible output was correct but which did more work than it needed to. In this lesson the fault is not in time but in memory. A screen closes, the user never returns to it, but the screen's model stays in memory. The screen showed the right data and no test complains; only the app's memory use grows a little with every screen opening. The measurement axis is **separation** again: we apply the same check to the correct version and to the faulty version. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. The core is the same as in the previous lessons; we give it again because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on each; the `separation` label derives from these two verdicts. `contrast` applies the same closure to both outputs, and `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the two versions apart. #### The Retain Cycle In Swift, **reference counting** determines the lifetime of objects. Each object counts how many places hold it strongly, and when the count drops to zero its memory is freed. If one object holds another strongly and the second also holds the first strongly, neither count ever drops to zero. This is called a **retain cycle**. Nobody from the outside can reach the objects, but the objects keep holding each other: this is a **memory leak**. By default, a closure captures the objects it uses strongly. If a screen model stores a closure in its own property and the closure uses `self`, the model holds the closure and the closure holds the model. ```swift // screen model — SOURCE final class OrderDetailViewModel { var onUpdate: (() -> Void)? func bind(service: OrderService) { service.observe { order in self.apply(order) } onUpdate = { self.refresh() } } } ``` Because the `onUpdate` closure calls `self.refresh()`, it captures `self` strongly; and `self` holds `onUpdate` strongly. Even when the screen closes, the count does not drop to zero. The fix is to give the closure a **capture list** and capture `self` weakly. ```swift // screen model, fixed — SOURCE final class OrderDetailViewModel { var onUpdate: (() -> Void)? func bind(service: OrderService) { service.observe { [weak self] order in self?.apply(order) } onUpdate = { [weak self] in self?.refresh() } } } ``` `[weak self]` gives the closure a **weak reference** to `self`. A weak reference does not increase the object's count; when the object is freed, the reference becomes `nil` on its own. This is why `self` is an optional value inside the closure. There is also `unowned`: it does not increase the count either, but using it crashes if the object is gone. It is used when it is certain that the object will outlive the closure; when you are not sure, `weak` is chosen. Not every closure creates a cycle. A closure passed as a function parameter that runs immediately and finishes does not store the object anywhere. A cycle forms when the closure is stored and the one storing it is the captured object. #### Finding the Leak There are two ways to find a leak. The first is the **memory graph** view in the debug area of the development environment: it pauses the running app and shows the live objects and the links between them as arrows. The second is done inside a test. The test builds an object, attaches a weak observer to it, drops the object's strong reference and checks that the observer is `nil`. ```swift // leak test — SOURCE import XCTest final class LeakTests: XCTestCase { func testViewModelIsReleased() { var model: OrderDetailViewModel? = OrderDetailViewModel() weak var observer = model model?.bind(service: StubOrderService()) model = nil XCTAssertNil(observer) } } ``` This fragment is not run; the types above and `StubOrderService` are not defined here. The test builds a model, gives it a second reference with `weak var observer`, and drops the only strong reference with `model = nil`. If there is no cycle, the object is freed at that moment and `observer` becomes `nil`. If there is a cycle, the object lives and `XCTAssertNil` fails. To repeat the same check in several tests, `addTeardownBlock(_:)` can be used: a closure that checks whether the object was released runs at the end of every test when the test finishes. #### The Leak Model The **model** below sets up what release says about the two versions; it is not the memory graph view itself. The model uses real objects and real reference counting: the retain cycle really forms in this measurement and is really broken. No bytes, addresses or amounts of memory are printed; only whether an object is released and how many objects are alive are counted. The screen model stores a closure with `bind`. The correct version uses `[weak self]` in the closure. The faulty version captures `self` strongly. ```swift // Models.swift — correct and faulty versions of the screen model, with a live object counter final class LiveCount { var screens = 0 var loaders = 0 } final class LoaderModel { let counts: LiveCount var completion: (() -> Void)? init(counts: LiveCount) { self.counts = counts counts.loaders += 1 } deinit { counts.loaders -= 1 } func arm(leaking: Bool) { if leaking { completion = { _ = self.counts } } else { completion = { [weak self] in _ = self?.counts } } } } final class DetailViewModel { let title: String let counts: LiveCount var onUpdate: (() -> Void)? init(title: String, counts: LiveCount) { self.title = title self.counts = counts counts.screens += 1 } deinit { counts.screens -= 1 } func bind(captureStrongly: Bool) { if captureStrongly { onUpdate = { _ = self.title } } else { onUpdate = { [weak self] in _ = self?.title } } } func startLoading(leaking: Bool) { let loader = LoaderModel(counts: counts) loader.arm(leaking: leaking) } } func shownTitle(captureStrongly: Bool) -> String { let model = DetailViewModel(title: "order", counts: LiveCount()) model.bind(captureStrongly: captureStrongly) return model.title } func closeScreen(captureStrongly: Bool) -> Bool { weak var observer: DetailViewModel? do { let model = DetailViewModel(title: "order", counts: LiveCount()) model.bind(captureStrongly: captureStrongly) observer = model } return observer == nil } func closeAndCount(leaking: Bool) -> LiveCount { let counts = LiveCount() do { let model = DetailViewModel(title: "order", counts: counts) model.startLoading(leaking: leaking) } return counts } ``` `closeScreen` is the test's model: `weak var observer` is the weak observer, the end of the `do` block is the closing of the screen, and the returned value is whether the observer is `nil`. `shownTitle` returns the title the screen shows. The first looks at **release**, the second at **visible data**. `LiveCount` and `startLoading` are for this lesson's second measurement. ```swift // main.swift — visible data and release let title = contrast("screen-data", correct: shownTitle(captureStrongly: false), faulty: shownTitle(captureStrongly: true), check: { $0 == "order" }).describe() let released = contrast("screen-released", correct: closeScreen(captureStrongly: false), faulty: closeScreen(captureStrongly: true), check: { $0 }).describe() countSeparations([title, released]) ``` ``` screen-data | correct=order/pass faulty=order/pass -> blind screen-released | correct=true/pass faulty=false/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` `screen-data` is blind: both versions show the same title. The screen's visible behavior is exactly the same in the two versions. `screen-released` is separating: in the correct version the weak observer becomes `nil`, in the faulty version the object stays alive and the observer still points to an object. This is why a memory leak is invisible to ordinary behavior tests. No check that looks at what the screen shows says where the screen is after it closes. The cycle is real in the model too: in the faulty version the object stays until the process ends, because nobody is left to release it. **TP1.** The model measures a single closure capture. The same cycle can also form through strong properties two objects give each other, or through a timer; these should be measured separately. **TP2.** In the faulty version `closeScreen` really leaves a leak. Since the measurement runs in a single short-lived process it does no harm; in a real app every screen opening adds another leak. #### What Is Counted A second form of finding a leak in a test is to look at the **live object count**: it should be zero before the screen opens and zero again after it closes. The counter looks simple and strong. But what the counter counts determines the result. Closures do not capture only the screen itself. The screen can start a loader object, and the loader's completion closure may capture the loader. In that case the screen is freed but the loader leaks. The measurement below sets up this situation. The screen starts a loader with `startLoading`. In the faulty version the loader's closure captures the loader strongly; in the correct version it captures it weakly. The screen is freed on its own in both versions. Two values are read from the same counter: the number of live screens and the number of live loaders. ```swift // main.swift — same counter, two counted objects let correct = closeAndCount(leaking: false) let faulty = closeAndCount(leaking: true) let screens = contrast("counted-screen", correct: correct.screens, faulty: faulty.screens, check: { $0 == 0 }).describe() let loaders = contrast("counted-loader", correct: correct.loaders, faulty: faulty.loaders, check: { $0 == 0 }).describe() countSeparations([screens, loaders]) ``` ``` counted-screen | correct=0/pass faulty=0/pass -> blind counted-loader | correct=0/pass faulty=1/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` The live screen count is zero in both versions, because the screen was freed. This check is blind: it cannot see the leak. The live loader count is zero in the correct version and one in the faulty version, and the check separates. The counter is the same, the fault is the same; only the counted object changed. The lesson here is this: a leak test that does not count the leaking object cannot find the leak. It is hard to know in advance which object may leak; this is why the memory graph view, by looking at all live objects, shows the places the test skipped. A test answers the question it is asked; a view can also show the question that was not asked. **TP3.** The model sets the loader up as a helper of the screen. In a real app the leak is mostly born in a layer the screen knows nothing about. **TP4.** The counter counts only two types of objects. A leak of another type is not seen in this measurement. #### Common Mistakes - Using `self` in a closure without a capture list and storing the closure in a property. - Writing `[weak self]` in every closure. It is unnecessary in a closure that runs immediately and finishes. - Using `unowned` when the object's lifetime is uncertain. If the object is gone, the app crashes. - Checking only that the screen was released and skipping the helper objects the screen started. - Looking for leaks only with behavior tests. Memory grows while the behavior stays correct. #### Summary - A retain cycle is two objects holding each other strongly; the reference count never drops to zero and memory leaks. - A closure captures `self` strongly; `[weak self]` breaks the cycle, while `unowned` crashes if the object is gone. - Of two checks, one came out blind and one separating: the visible data was the same in both versions, and release happened only in the correct version. - The same live object counter was blind when it counted the screen and separating when it counted the leaking loader. - A test that finds a leak must count the leaking object; the memory graph view shows the objects that were skipped. - This lesson's separation: **separating**. #### Next Step The leak was one fault; unnecessary work done at startup is another. In the next lesson, Launch and Scrolling Performance will measure how a performance check compared with a recorded baseline can turn inverted depending on which version the baseline was taken from. ### Launch and Scrolling Performance Source: https://academia.sh/en/courses/architecture-testing-and-release/launch-and-scrolling-performance Deferring the work done at launch and reusing row objects; measuring how a performance check that compares against a recorded baseline can turn inverted or blind depending on the version the baseline was taken from. In the previous lesson (Memory Leak Detection) the fault was an object that stayed alive after a screen had closed. In this lesson the fault is in the first two places the user sees: when the app launches and when a list is scrolled. In both the output is correct; the screen opens and the rows are drawn. The difference is in the amount of work done to reach that result. Performance checks mostly compare against a **baseline**: a value measured in an earlier run is recorded, and the new run's value must not exceed it. This lesson's question is what the check does depending on which version the baseline was taken from. The measurement axis is **separation** again: we apply the same check to the correct version and to the faulty version. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. The core is the same as in the previous lessons; we give it again because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on each; the `separation` label derives from these two verdicts. `contrast` applies the same closure to both outputs, and `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the two versions apart. #### Work Done at Launch The time from the user tapping the icon until the first frame is drawn is the app's first impression. Every piece of work that runs in this time keeps the user waiting. Not all the work done at launch is needed for the first frame: starting an analytics library, warming a cache or preparing the data of a second screen can be left until after the first frame. ```swift // app entry — SOURCE @main struct ShopApp: App { init() { configureTheme() restoreSession() // Not needed for the first frame; left for later. // warmImageCache() // startAnalytics() } var body: some Scene { WindowGroup { RootView() } .task { await deferredStartup() } } } ``` This fragment is not run; `configureTheme`, `restoreSession` and `deferredStartup` are not defined here. The view's `task` modifier starts an asynchronous task before the view appears; the task is suspended at its first `await` point and the rest of the deferred work does not hold up the first frame. The rule is simple: the work needed to draw the first frame goes in the entry, the rest comes later. In real apps, launch time and scrolling smoothness are also collected from users' phones. The platform's `MetricKit` framework delivers this data to the app as daily summaries. ```swift // collected measurements — SOURCE import MetricKit final class MetricsReceiver: NSObject, MXMetricManagerSubscriber { func didReceive(_ payloads: [MXMetricPayload]) { for payload in payloads { if let launch = payload.applicationLaunchMetrics { record(launch.histogrammedTimeToFirstDraw) } } } } ``` The summaries give the distribution under real users' real conditions; a single measurement taken during development cannot give this. This lesson's measurements, in contrast, rest on **counted work** that does not depend on conditions: how many steps ran, how many objects were created. Time is neither measured nor printed. #### Objects Created While Scrolling In a long list only a few rows are visible on the screen at the same time. When a row leaves the screen its object is not discarded; it is put in a pool and reused for a newly visible row. This removes the work of creating an object and allocating memory for every row. Without reuse a new object is created for every row, and the work grows as the scroll gets longer. ```swift // row creation — SOURCE func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { let cell = tableView.dequeueReusableCell(withIdentifier: "Row", for: indexPath) cell.textLabel?.text = items[indexPath.row].title return cell } ``` `dequeueReusableCell(withIdentifier:for:)` takes an idle row from the pool and creates a new one if there is none. The faulty version builds a new row object every time. The **model** below is the pure counting counterpart of this; it is not a list view. It scrolls a hundred-row list through a window where ten rows are visible at a time and counts how many row objects are created. The correct version puts a row leaving the screen into the pool and uses it for the next row. The faulty version uses no pool. ```swift // Scroll.swift — scrolling and launch model, measurement with a baseline struct Measured: CustomStringConvertible { let value: Int let baseline: Int var description: String { "\(value) baseline=\(baseline)" } } func cellsCreated(rows: Int, visible: Int, reuse: Bool) -> Int { var pool = 0 var created = 0 var onScreen = 0 for _ in 0.. 0 { pool -= 1 } else { created += 1 } onScreen += 1 } return created } func drawnRows(rows: Int, reuse: Bool) -> Int { rows } func launchWork(deferHeavy: Bool) -> Int { let light = 3 let heavy = 5 return deferHeavy ? light : light + heavy } ``` `cellsCreated` runs the scroll with a loop: when the window is full, the oldest row leaves the screen, and if reuse is on it goes into the pool. `drawnRows` is the number of rows drawn in both versions. `launchWork` counts the steps that run before the first frame at launch: three light steps always, five heavy steps only when they are not deferred. `Measured` holds a value together with its recorded baseline. ##### The Source of the Baseline We apply three checks to the same scroll. The first looks at the number of rows drawn. The second says "the value must equal the baseline" and takes the baseline from the **faulty** version. The third says "the value must not exceed the baseline" and takes the baseline from the **correct** version. ```swift // main.swift — three checks, the source of the baseline let drawn = contrast("drawn-rows", correct: drawnRows(rows: 100, reuse: true), faulty: drawnRows(rows: 100, reuse: false), check: { $0 == 100 }).describe() let faultyBaseline = cellsCreated(rows: 100, visible: 10, reuse: false) let equalToFaulty = contrast("equal-baseline", correct: Measured(value: cellsCreated(rows: 100, visible: 10, reuse: true), baseline: faultyBaseline), faulty: Measured(value: cellsCreated(rows: 100, visible: 10, reuse: false), baseline: faultyBaseline), check: { $0.value == $0.baseline }).describe() let correctBaseline = cellsCreated(rows: 100, visible: 10, reuse: true) let notAboveCorrect = contrast("within-baseline", correct: Measured(value: cellsCreated(rows: 100, visible: 10, reuse: true), baseline: correctBaseline), faulty: Measured(value: cellsCreated(rows: 100, visible: 10, reuse: false), baseline: correctBaseline), check: { $0.value <= $0.baseline }).describe() countSeparations([drawn, equalToFaulty, notAboveCorrect]) ``` ``` drawn-rows | correct=100/pass faulty=100/pass -> blind equal-baseline | correct=10 baseline=100/fail faulty=100 baseline=100/pass -> inverted within-baseline | correct=10 baseline=10/pass faulty=100 baseline=10/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 1 ``` `drawn-rows` is blind: both versions draw a hundred rows, and the list the user sees is the same. `equal-baseline` is inverted: the baseline was taken from the faulty version, that is, a hundred objects; the correct version, with ten objects, does not equal this value, so the check fails, while the faulty version equals its own value and passes. The improvement looks red and the fault looks green. `within-baseline` is separating: the baseline was taken from the correct version, the correct version passes and the faulty version exceeds it. An inverted check is worse than being blind, because it reverses the relation between what is measured and the baseline. A blind check says nothing; an inverted check says the wrong thing and pushes the team to roll back the correct version. This is why, when a baseline is recorded, the version it was taken from is recorded too, and "does not exceed" is said instead of "equal": a performance improvement must not be counted as an error. **TP1.** Taking the baseline from the faulty version is a design mistake, but it happens often in practice: a team records a value without knowing about the fault and fixes it later. **TP2.** The model measures only the number of objects created. In a real list the row's content, size and image also affect scrolling; these are not counted. #### The Bounding Measurement The baseline has one more problem: refreshing it. To keep the check from turning red, some teams pull the baseline to the current value on every run. The check "the value must not exceed the baseline" then always passes. The measurement below puts three things side by side: the "does not exceed" check with a fixed baseline, the same check with a baseline refreshed on every run, and the check with a fixed baseline applied to the work that runs before the first frame at launch. ```swift // main.swift — fixed and refreshed baseline, launch let correctCells = cellsCreated(rows: 100, visible: 10, reuse: true) let faultyCells = cellsCreated(rows: 100, visible: 10, reuse: false) let fixed = contrast("fixed-baseline", correct: Measured(value: correctCells, baseline: correctCells), faulty: Measured(value: faultyCells, baseline: correctCells), check: { $0.value <= $0.baseline }).describe() let refreshed = contrast("refreshed-baseline", correct: Measured(value: correctCells, baseline: correctCells), faulty: Measured(value: faultyCells, baseline: faultyCells), check: { $0.value <= $0.baseline }).describe() let correctLaunch = launchWork(deferHeavy: true) let launch = contrast("pre-first-frame-work", correct: Measured(value: correctLaunch, baseline: correctLaunch), faulty: Measured(value: launchWork(deferHeavy: false), baseline: correctLaunch), check: { $0.value <= $0.baseline }).describe() countSeparations([fixed, refreshed, launch]) ``` ``` fixed-baseline | correct=10 baseline=10/pass faulty=100 baseline=10/fail -> separating refreshed-baseline | correct=10 baseline=10/pass faulty=100 baseline=100/pass -> blind pre-first-frame-work | correct=3 baseline=3/pass faulty=8 baseline=3/fail -> separating separating: 2 blind: 1 noisy: 0 inverted: 0 ``` `fixed-baseline` is the separating check of the previous measurement. `refreshed-baseline` is blind: since the faulty version's baseline is also pulled to its own value, a hundred objects do not exceed a hundred objects. Because the check never breaks, believing that it protects something is misleading. `pre-first-frame-work` is separating: the undeferred launch runs eight steps and exceeds the fixed three-step baseline. Refreshing the baseline is not wrong in itself; after a deliberate improvement a new baseline must be recorded. What is wrong is doing it automatically on every run. A baseline changes by a decision, not by a run. **TP3.** The model counts "steps" and "objects". In a real launch the steps do not take equal time; a single heavy step can take longer than all eight. **TP4.** The launch measurement counts only the work before the first frame. The work deferred until after the first frame also has a cost; deferring does not remove it. #### Common Mistakes - Recording the baseline from a faulty or unknown version and saying "equal". - Pulling the baseline to the current value on every run. The check never breaks. - Doing at launch the work that is not needed for the first frame. - Creating a new object for every row; skipping the reuse pool. - Treating a single local measurement as representative of users' conditions. #### Summary - The work before the first frame keeps the user waiting; only what is needed for the first frame is done in the entry, the rest is deferred. - Reusing row objects keeps the number of objects created while scrolling at the number of visible rows. - If the baseline is taken from the faulty version, the "equal" check becomes inverted: the improvement looks red and the fault looks green. - If the baseline is taken from the correct version, the "does not exceed" check separates; a baseline refreshed on every run leaves the check blind. - A baseline changes by a decision, not by a run. - This lesson's separation: **inverted**. #### Next Step A performance check can be written as a rule; so can code quality checks. In the next lesson, Code Quality Tools will measure whether a formatting check and a rule that scans the source text can tell the same fault apart. ### Code Quality Tools Source: https://academia.sh/en/courses/architecture-testing-and-release/code-quality-tools The duties and limits of formatters and static analysis rules; measuring whether a format check and a rule that scans the source text can tell the same fault apart, and how a false positive spoils a check. In the previous lesson (Launch and Scrolling Performance) the fault was in the amount of a measured piece of work. In this lesson the fault is in the source text itself: the code compiles, the tests pass, but under the right conditions one line crashes the app. Code quality tools check the code by looking at its text, without running it. There are two kinds of tool. A **formatter** arranges spaces, indentation and line endings; it does not look at what the code does. A **static analysis** tool reads the source text and finds structures that do not follow the defined rules: an unused variable, a function that is too long, a force-unwrapped optional value. Both make the team write in a common style; but the faults they catch are different. The measurement axis is **separation** again: we apply the same check to the correct version and to the faulty version. A **separating** check passes on the correct version and fails on the faulty one. A **blind** check passes on both. A **noisy** check fails on both. An **inverted** check fails on the correct version and passes on the faulty one. The core is the same as in the previous lessons; we give it again because each lesson runs on its own. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` `Contrast` holds the output of the two versions and the check's verdict on each; the `separation` label derives from these two verdicts. `contrast` applies the same closure to both outputs, and `countSeparations` counts the four categories. **The reason for the rule:** a check's value lies not in passing but in telling the two versions apart. #### Force Unwrapping In Swift an optional value can be `nil`. Force unwrapping with an exclamation mark assumes that the value is not `nil`; if the assumption is wrong, the app crashes on that line. The faulty version converts a text to a number and force unwraps the result. The correct version handles the `nil` case with `guard let`. ```swift // faulty and fixed — SOURCE func total(of text: String) -> Int { let value = Int(text)! return value } func total(of text: String) -> Int { guard let value = Int(text) else { return 0 } return value } ``` The text "12" the user enters is converted to twelve in both versions. When the input is "twelve", the faulty version crashes and the correct version returns zero. If the test never tried this input, both versions pass; the fault shows up in production, on a user's unexpected input. #### Format Check and Rule Check The **model** below sets up the duties of the two tools on source text; it is not a real tool itself. The source is kept as an array of lines. The model does not run the code; it only scans the text, just like a static analysis tool. `formatViolations` is the format check: it counts tabs, trailing spaces at the end of a line and indentation that is not a multiple of four spaces. `forceUnwraps` is the rule check: it counts exclamation marks that come right after a name, `)` or `]` and are not followed by `=`. The not-equal comparison and a leading mark that negates a value are not force unwraps and are not counted. When `allowLiterals` is on, force unwrapping a call whose only argument is a literal text is allowed. ```swift // Lint.swift — format check, force unwrap rule, source lines func formatViolations(in lines: [String]) -> Int { var count = 0 for line in lines { let indent = line.prefix(while: { $0 == " " }).count if line.contains("\t") || line.last == " " || indent % 4 != 0 { count += 1 } } return count } func isLiteralCall(_ chars: [Character], closeIndex: Int) -> Bool { var depth = 0 var open = closeIndex while open >= 0 { if chars[open] == ")" { depth += 1 } if chars[open] == "(" { depth -= 1 } if depth == 0 { break } open -= 1 } guard open >= 0 else { return false } let inner = chars[(open + 1).. Int { var count = 0 for line in lines { let chars = Array(line) for index in chars.indices where chars[index] == "!" && index > 0 { let previous = chars[index - 1] let next: Character = index + 1 < chars.count ? chars[index + 1] : " " let postfix = previous.isLetter || previous.isNumber || previous == "_" || previous == ")" || previous == "]" if !postfix || next == "=" { continue } if allowLiterals && previous == ")" && isLiteralCall(chars, closeIndex: index - 1) { continue } count += 1 } } return count } let correctLines = [ "func total(of text: String) -> Int {", " guard let value = Int(text) else { return 0 }", " return value", "}" ] let faultyLines = [ "func total(of text: String) -> Int {", " let value = Int(text)!", " return value", "}" ] let correctLiteralLines = [ "func home() -> URL {", " return URL(string: \"https://example.com\")!", "}" ] let faultyLiteralLines = [ "func link(_ input: String) -> URL {", " return URL(string: input)!", "}" ] ``` There are four sources. `correctLines` and `faultyLines` are written in the same format; only the second line of the second one force unwraps. `correctLiteralLines` force unwraps a constant address: this address is known and valid at compile time, so there is no chance of a crash. `faultyLiteralLines` force unwraps a text that comes from the input and crashes on an invalid address. ##### The Format Is Blind, the Rule Separates The first measurement applies the two checks to the two versions of the `total` function. ```swift // main.swift — format check and force unwrap rule let format = contrast("format-check", correct: formatViolations(in: correctLines), faulty: formatViolations(in: faultyLines), check: { $0 == 0 }).describe() let rule = contrast("force-unwrap-rule", correct: forceUnwraps(in: correctLines, allowLiterals: false), faulty: forceUnwraps(in: faultyLines, allowLiterals: false), check: { $0 == 0 }).describe() countSeparations([format, rule]) ``` ``` format-check | correct=0/pass faulty=0/pass -> blind force-unwrap-rule | correct=0/pass faulty=1/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 0 ``` `format-check` is blind: both versions are indented properly and have no trailing spaces. The format does not say that the code may crash. `force-unwrap-rule` is separating: the correct version has no exclamation mark, the faulty version has one. This does not mean that the formatter is unnecessary; it answers a different question. The format ensures that the code is readable and that no unnecessary differences appear during review. Finding the fault is the rule check's job. One does not replace the other; the sentence "the format check is clean" says nothing about the code being correct. **TP1.** The model scans the lines as text and does not tell an exclamation mark inside a string apart. A real tool works on a syntax tree and eliminates these false matches. **TP2.** The model counts only a single rule. A real tool runs dozens of rules, and each rule's separation must be measured separately. #### The Bounding Measurement A rule can find the fault and still be a bad check: if it counts safe use as a fault too. Force unwrapping a constant address does not crash; if the rule flags this as well, the team learns to ignore the warnings. The measurement below applies the strict rule and the rule with a literal text exemption to the correct version that force unwraps a constant address and to the faulty version that force unwraps a text from the input. ```swift // main.swift — false positive and exemption let strict = contrast("strict-rule", correct: forceUnwraps(in: correctLiteralLines, allowLiterals: false), faulty: forceUnwraps(in: faultyLiteralLines, allowLiterals: false), check: { $0 == 0 }).describe() let exempt = contrast("exempt-rule", correct: forceUnwraps(in: correctLiteralLines, allowLiterals: true), faulty: forceUnwraps(in: faultyLiteralLines, allowLiterals: true), check: { $0 == 0 }).describe() countSeparations([strict, exempt]) ``` ``` strict-rule | correct=1/fail faulty=1/fail -> noisy exempt-rule | correct=0/pass faulty=1/fail -> separating separating: 1 blind: 0 noisy: 1 inverted: 0 ``` `strict-rule` is noisy: the correct version is flagged too, because the rule does not tell a constant address from a text that comes from the input. Both versions fail and the check says nothing. `exempt-rule` is separating: a call whose argument is a literal text is allowed, and a text from the input is flagged. Defining an exemption is a decision too. An exemption that is too wide hides the fault again; one that is too narrow keeps the false positive. Here the exemption depends on a single condition, that the argument is a literal text, and this condition is written in the rule itself; comments added above a line saying "skip this line" are each a separate loss of checking. **TP3.** The exemption covers only a single call form. That a literal text is really valid must be verified separately, outside the rule. **TP4.** The model does not run the crash. That the faulty version crashes on input is known from the structure of the source text, not measured. #### Common Mistakes - Reading a clean format check as the code being correct. - Turning rules on and not examining their results; silently turning off a rule that gives false positives. - Flagging all force unwraps and not separating the safe ones; the team gets used to ignoring the warning. - Silencing every warning with a line comment. This disables the rule on that line. - Putting static analysis in place of a test. A rule finds only the pattern it describes; it does not test behavior. #### Summary - A formatter arranges spaces and indentation; a static analysis rule looks for a specific structure. They answer different questions. - Force unwrapping compiles and runs in tests, but crashes the app on unexpected input. - The format check passed on both versions and stayed blind; the rule check passed the correct version and failed the faulty one, and separated. - A strict rule that also flags safe use failed on both versions and became noisy; the literal text exemption brought the separation back. - A rule's value lies in finding the fault and in not interfering with safe use. - This lesson's separation: **blind**. #### Next Step After the test and performance checks, the code is packaged to reach users. The next topic, Distribution, explains with its first lesson, Signing and Provisioning Profiles, how the app's identity is proven. ## Distribution Signing and provisioning profiles with identity and entitlement files, continuous integration with automated builds and tests, release automation with version numbering and upload steps, beta distribution to a limited audience, the store review process and listing, and crash reporting and analytics. ### Signing and Provisioning Profiles Source: https://academia.sh/en/courses/architecture-testing-and-release/signing-and-provisioning-profiles A measurement showing that comparing a package's entitlements with a recorded list passes the faulty version and rejects the correct one, while certificate and profile checks stay blind. Until the previous lesson, the checks read code: tests, linters, formatters. From this lesson on, the checks move to the package after it has been built. Before a package reaches a device or the store, it is matched against three things. The **certificate** says who signed the package. The **provisioning profile** is a signed file that lists which bundle identifier may run with which entitlements and which certificates. The **entitlements file** states which entitlements the package itself asks for. Signing looks at whether these three agree. The question of this course is not what it means when a check passes or fails. The question is **whether the two answers differ when the same check runs against the correct version and against the faulty version.** The measure is called **separation**, and it has four categories: **separating**, where the correct version passes and the faulty version fails; **blind**, where both pass; **noisy**, where both fail; and **inverted**, where the correct version fails and the faulty version passes. In the output they appear as `separating`, `blind`, `noisy`, and `inverted`. This lesson measures with the core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is the following: the app asks for the remote notification entitlement, but the package does not carry it. One entry is missing from the entitlements file, and the package still builds, gets signed, and opens. The gap shows up when the app tries to register for remote notifications. A real entitlements file looks like the one below; the model keeps entitlements under its own names, which are not the key names of the real file. ```xml aps-environment production ``` The entitlements a package actually carries are read with the `codesign -d --entitlements :- Shop.app` command. This lesson does not measure the output of that command; what it measures is the signing decision, built as a model in plain Swift. The model builds what the check says about the two versions; it is not the signing tool itself. #### The Signing Model The model holds three values: the package's identifier and entitlements, the profile's allowances and expiry counter, and the certificate's expiry counter. Time is modeled not by a clock but by a **version counter**: an expiry value counts as expired when the counter reaches or passes it. `effective` returns the entitlements a signed package can actually use: the intersection of what the package asks for and what the profile allows. `sign` makes the decision and **rejects** any entitlement the profile does not match: if the package carries an entitlement that is not in the profile, the signature is not accepted. ```swift // Signing.swift — models a package's entitlements, the certificate, and the provisioning profile struct Package { let identifier: String let entitlements: Set } struct Profile { let identifier: String let allowed: Set let expiresAt: Int } struct Certificate { let expiresAt: Int } func effective(_ package: Package, profile: Profile) -> [String] { package.entitlements.intersection(profile.allowed).sorted() } func sign(_ package: Package, profile: Profile, certificate: Certificate, counter: Int) -> String { if certificate.expiresAt <= counter { return "reject:certificate" } if profile.expiresAt <= counter { return "reject:profile" } if package.identifier != profile.identifier { return "reject:identifier" } if !package.entitlements.isSubset(of: profile.allowed) { return "reject:entitlement" } return "signed" } ``` The identifier check looks small but is necessary: a profile with an explicit identifier is bound to a single bundle identifier, and an app packaged under another identifier cannot be signed with that profile, even if it carries the same entitlements. In the model this stands as `reject:identifier`. No input in this lesson triggers it; identifier matching stays outside the measurement, and the fault does not touch it. The lesson's two packages carry the same bundle identifier. The correct package carries two entitlements, and the faulty package carries only one of them. The profile allows both, and the app asks for both. ```swift // Shop.swift — the lesson's two packages, the profile, and the certificate let requested: Set = ["keychain", "remote-notification"] let shopProfile = Profile(identifier: "com.example.shop", allowed: ["keychain", "remote-notification"], expiresAt: 8) let shopCertificate = Certificate(expiresAt: 5) let correctPackage = Package(identifier: "com.example.shop", entitlements: ["keychain", "remote-notification"]) let faultyPackage = Package(identifier: "com.example.shop", entitlements: ["keychain"]) ``` **DS1.** This measurement gives the expiry of the certificate and the profile as a version counter. How a real expiry date maps to the counter was not measured. #### Four Checks Four checks are applied to the same two packages. The first compares the entitlements the package carries with a **recorded list**. The recorded list is an entitlement list that the team recorded once and then treated as the yardstick for correctness; in this lesson, that list was copied from the faulty build. The second looks at whether the effective entitlements of the signed package cover the entitlements the app asks for. The third looks only at whether the profile allows the entitlements the app asks for, and the fourth only at whether the certificate is valid for the counter. ```swift // main.swift — four checks, three separations let counter = 3 let registered = ["keychain"] let listed = contrast("registered-list-equals", correct: correctPackage.entitlements.sorted(), faulty: faultyPackage.entitlements.sorted()) { $0 == registered }.describe() let covered = contrast("effective-entitlements-cover", correct: effective(correctPackage, profile: shopProfile), faulty: effective(faultyPackage, profile: shopProfile)) { requested.isSubset(of: $0) }.describe() let profileOnly = contrast("profile-covers", correct: shopProfile.allowed.sorted(), faulty: shopProfile.allowed.sorted()) { requested.isSubset(of: $0) }.describe() let certificateOk = contrast("certificate-valid", correct: shopCertificate.expiresAt, faulty: shopCertificate.expiresAt) { $0 > counter }.describe() countSeparations([listed, covered, profileOnly, certificateOk]) ``` ```text registered-list-equals | correct=["keychain", "remote-notification"]/fail faulty=["keychain"]/pass -> inverted effective-entitlements-cover | correct=["keychain", "remote-notification"]/pass faulty=["keychain"]/fail -> separating profile-covers | correct=["keychain", "remote-notification"]/pass faulty=["keychain", "remote-notification"]/pass -> blind certificate-valid | correct=5/pass faulty=5/pass -> blind separating: 1 blind: 2 noisy: 0 inverted: 1 ``` Four lines, three categories. `effective-entitlements-cover` looks at the value the fault changes: in the faulty package, remote notification is missing from the effective entitlements and the check fails; in the correct package it is present and the check passes. This is the **separating** check. `profile-covers` and `certificate-valid`, on the other hand, see the same value in both versions, because the fault touches neither the profile nor the certificate. On both lines the `faulty` side says `pass`; these two **blind** checks give the confidence common in a green signing pipeline and say nothing about the fault. #### The Inversion of the Equality Check The first line is the interesting one. `registered-list-equals` rejects the correct package and accepts the faulty one: `correct=…/fail`, `faulty=…/pass`, result **inverted**. The check is not faulty; the problem is the source of the expected value. Because the recorded list was copied from the faulty build, it counted the missing entitlement as "correct". The team that later adds the entitlement sees the check turn red and, more often than not, updates the list together with the package. From that moment on, the fault has settled into the check itself. This is a trap specific to equality checks: if a check takes its expected value **from a version**, it also takes that version's fault. An equality check protects an expectation, but it does not say whether the expectation is correct. A separating check solves this another way. `effective-entitlements-cover` takes its expected value from no build; it reads the entitlements the app asks for from their own source, the request list kept together with the app's code. It would also be wrong to say the equality check against the recorded list is useless. It is a good way to catch entitlements that **quietly grow**: if the record was taken correctly, a package with an extra entitlement fails. The fault measured in this lesson is a different one, a missing entitlement. Separation is measured against one fault; the same check can fall into another category against another fault. **DS2.** This measurement assumes that the signature is accepted when the profile's entitlements cover the package's entitlements. How the real signing tool is affected by extra entitlements in the profile was not measured. #### The Comfort of a Blind Check Why are certificate and profile checks so common? Because their failures are loud and warn the team immediately. A certificate that has expired does not let the package be signed, and a package without a profile is not installed on a device. That is why these checks sit at the start of the pipeline and pass every day, giving the team a habit. But a check that has been green hundreds of times does not show that it saw a difference between the correct version and the faulty one. Both versions are signed with the same certificate and bound to the same profile; `blind` is therefore unavoidable. The takeaway is this: a blind check does not have to be deleted, because it catches another fault. What is needed is **knowing whether it counts as a check that catches the fault.** If the green output of a pipeline consists of three blind checks and one separating check, the fault stays invisible unless it gets past the separating one. In practice this means writing into the pipeline report not only the number of checks that passed but also which fault each check could see; the number can be high while the share of separating checks stays close to zero. #### Testing a New Check with Mutation Testing When a new check is added to the signing pipeline, the first question should not be "does this check pass today". Passing today gives a green pipeline, and that proves nothing. The right question is this: **if I break the package on purpose, will this check fail?** This method is called **mutation testing**. Here, breaking means deliberately removing an entitlement from the package; if the check does not reject that package, it is either blind or inverted. The lesson's measurement is a small form of mutation testing: the faulty package was made by removing a single entitlement from the correct package, and four checks were applied to it. The difference is this: mutation testing only asks "was it caught". This course also looks at the answer of the correct version, because a check that rejects the correct package keeps the team busy with a wrong red every time, whatever its success at catching. Noisy and inverted separate only when the answers of the two versions are put side by side. | Check | What it looks at | Sees the fault | Separation | |---|---|---|---| | `registered-list-equals` | the package's entitlements, the recorded list | only as long as the list is correct | inverted | | `effective-entitlements-cover` | the effective entitlements, the app's requests | yes | separating | | `profile-covers` | the profile, the app's requests | no | blind | | `certificate-valid` | the certificate, the counter | no | blind | The third column in the table states not a result but a **condition**. In the first row, seeing the fault requires that the recorded list was not taken from the faulty build. That condition does not exist inside the check itself; where the list came from stays outside the check, and only the person who wrote it knows. #### Correcting the Recorded List When the inversion is found, there are two ways. The first is to record the list again from the correct package. That makes the check correct until the next fault, but it again takes the list from a build; if an entitlement is quietly lost in a new build, the inversion returns. The second is to derive the expected value from outside the package, from the app's request list: this is what `effective-entitlements-cover` does. The second way removes the work of refreshing the record, because the expected value comes from the app itself every time. The two ways can also be set up together. The check derived from the request list catches the missing entitlement, and the equality check against the recorded list catches the extra one. The two look at different faults, and the lesson treats them as complementary, not as rivals. #### The Bounding Measurement Separation is not a property of a check alone; it changes with the fault and the input. In the measurement below, a single check, the `sign` decision being `signed`, is applied to three inputs. At counter 3, both the certificate and the profile are valid, and both packages get signed. At counter 9, the certificate has expired and both packages are rejected. In the third input, the profile is kept narrow and allows only the `keychain` entitlement: an entitlement without a match is rejected in the model too, so the correct package that carries remote notification is rejected. ```swift // main.swift — the same signing check gives three separations on three inputs let narrowProfile = Profile(identifier: "com.example.shop", allowed: ["keychain"], expiresAt: 8) let early = contrast("counter-3", correct: sign(correctPackage, profile: shopProfile, certificate: shopCertificate, counter: 3), faulty: sign(faultyPackage, profile: shopProfile, certificate: shopCertificate, counter: 3)) { $0 == "signed" }.describe() let late = contrast("counter-9", correct: sign(correctPackage, profile: shopProfile, certificate: shopCertificate, counter: 9), faulty: sign(faultyPackage, profile: shopProfile, certificate: shopCertificate, counter: 9)) { $0 == "signed" }.describe() let narrow = contrast("narrow-profile", correct: sign(correctPackage, profile: narrowProfile, certificate: shopCertificate, counter: 3), faulty: sign(faultyPackage, profile: narrowProfile, certificate: shopCertificate, counter: 3)) { $0 == "signed" }.describe() countSeparations([early, late, narrow]) ``` ```text counter-3 | correct=signed/pass faulty=signed/pass -> blind counter-9 | correct=reject:certificate/fail faulty=reject:certificate/fail -> noisy narrow-profile | correct=reject:entitlement/fail faulty=signed/pass -> inverted separating: 0 blind: 1 noisy: 1 inverted: 1 ``` The same check, three inputs, three categories. At `counter-3`, the faulty package is signed too; the signing pipeline is green and the fault passes, **blind**. At `counter-9`, the time has run out, both packages get `reject:certificate`, and the check fails on both versions, **noisy**: the reason for failing is the certificate, not the fault. At `narrow-profile`, the correct package is rejected with `reject:entitlement`, because remote notification is not in the profile; the faulty package carries exactly the one entitlement the profile allows, so it gets `signed`, **inverted**. In real life too, if a profile does not contain the entitlement, the correct package cannot be signed; the faulty package, which works with a missing entitlement, never meets this obstacle. This measurement also shows a limit. On none of the three inputs did the `signed` check come out separating: this check alone cannot separate this fault, because the signing decision only says whether the package **fits the profile**, not what the app **needs**. The app's need is in the `requested` set, and the only check that reads it is `effective-entitlements-cover`. **DS3.** This measurement gives the order of the signing decision as the model's order: first the certificate, then the profile, the identifier, and the entitlement. Which mismatch the real signing tool checks first was not measured. #### Summary - The certificate says who signed, the provisioning profile says which bundle identifier works with which entitlements, and the entitlements file says what the package asks for; signing looks at whether these three agree. - A check that compares the package's entitlements with a recorded list rejects the correct package and passes the faulty one if the list was taken from the faulty build: inverted. - A check that looks at whether the effective entitlements cover what the app asks for separates the faulty package: separating. - Checks that look only at the certificate or only at the profile pass on both versions: blind. - The same signing check can be blind, noisy, or inverted depending on the counter and the profile; separation is a property not of the check but of the check and the input together. - This lesson's separation: **inverted**. #### Next Step A package that can be signed can still compute the wrong amount; the signing decision does not verify the computation. In the next lesson, Continuous Integration, we will measure these checks running one after another in a pipeline, with the pipeline stopping at the first step that fails. ### Continuous Integration Source: https://academia.sh/en/courses/architecture-testing-and-release/continuous-integration A measurement showing that an automated pipeline on a clean machine fails both versions over a missing build setting, that the test step separates the fault once the setting is fixed, and that steps that never ran fall outside the categories. The idea that the same steps run in the same order every time a change is made, tied to nobody's machine, is called **continuous integration**. The steps are written as a pipeline: build first, then test, then package. If the pipeline fails on a change, the change stops before it reaches the team. It is a frame that also holds the previous lesson's signing checks; the signing step can sit at the end of the pipeline, and in this lesson it stays outside the pipeline model. The measure of this course is called **separation**: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: **separating**, where the correct version passes and the faulty version fails; **blind**, where both pass; **noisy**, where both fail; and **inverted**, where the correct version fails and the faulty version passes. In the output they appear as `separating`, `blind`, `noisy`, and `inverted`. This lesson measures with the core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is in the basket total. When the total of a three-item basket is computed, the last item is silently skipped: the correct version adds up all the items, and the faulty version leaves out the last one. The computation raises no error and does not crash; it only returns a number that is too small. What finds a fault of this kind is usually a test, and a test finds it only if it runs. The question is whether the pipeline actually runs the test. #### The Pipeline Model The model runs a pipeline step by step. Each step gets one of three results: `pass`, `fail`, or `skipped`. The pipeline tries the steps in order; it **stops** at the first step that gets `fail`, and all the remaining steps are marked `skipped`. This matches the behavior of real pipelines: if the build has failed, there is no point in running tests, and the pipeline does not spend time. There are three steps. The `build` step looks at the target's required setting; if the setting is missing, the build fails. The `test` step compares the basket total with the expected value. The `package` step checks that the total is greater than zero and stands here only as the last link of the pipeline. The settings are of the `Settings` type; its single field says whether the target setting is present. ```swift // Pipeline.swift — a pipeline model that runs steps in order and stops at the first failing step struct Settings { let hasTargetSetting: Bool } let pipelineSteps = ["build", "test", "package"] func runStep(_ name: String, total: Int, expected: Int, settings: Settings) -> Bool { switch name { case "build": return settings.hasTargetSetting case "test": return total == expected case "package": return total > 0 default: return false } } func runPipeline(total: Int, expected: Int, settings: Settings, steps: [String] = pipelineSteps) -> [String] { var outcomes: [String] = [] var stopped = false for name in steps { if stopped { outcomes.append("\(name):skipped") } else if runStep(name, total: total, expected: expected, settings: settings) { outcomes.append("\(name):pass") } else { outcomes.append("\(name):fail") stopped = true } } return outcomes } func outcome(of step: String, in outcomes: [String]) -> String { for entry in outcomes where entry.hasPrefix(step + ":") { return String(entry.dropFirst(step.count + 1)) } return "none" } func trace(_ outcomes: [String]) -> String { outcomes.joined(separator: " ") } func stepContrast(_ label: String, step: String, correct: [String], faulty: [String]) -> String { let onCorrect = outcome(of: step, in: correct) let onFaulty = outcome(of: step, in: faulty) if onCorrect == "skipped" || onFaulty == "skipped" { print("\(label) | correct=\(onCorrect) faulty=\(onFaulty) -> outside-category") return "outside-category" } return contrast(label, correct: onCorrect, faulty: onFaulty) { $0 == "pass" }.describe() } func countOutside(_ results: [String]) { print("outside-category: \(results.filter { $0 == "outside-category" }.count)") } ``` `stepContrast` compares the result of a single step for the two versions, but it does not classify when the step **did not run** in one of the two versions. Why it is done this way is explained in a separate section; for now it is enough to know that the output says `outside-category`. This is the lesson's own rule: the four categories of separation stay in the core, and `outside-category` is written only in this lesson, when a step did not run. The basket and the expected value live in a separate file. The correct function adds up all the items, and the faulty function skips the last one. The expected total is the correct version's value, and the test step takes it from there. ```swift // Basket.swift — the lesson's basket and its two total functions let basketPrices = [40, 25, 15] let expectedTotal = 80 func basketTotal(_ prices: [Int]) -> Int { prices.reduce(0, +) } func faultyBasketTotal(_ prices: [Int]) -> Int { prices.dropLast().reduce(0, +) } ``` **DS4.** This measurement models the pipeline as a single job that runs three steps in sequence. The behavior of parallel steps and of the dependency graph in real pipelines was not measured. #### The Pipeline on a Clean Machine The pipeline runs for the first time on a clean machine. A clean machine is an environment that carries none of the settings and local cache a developer's machine has accumulated. On the developer's machine, the build setting was made once and forgotten; it was never written into the pipeline definition. On the clean machine, that setting does not exist. **DS5.** This measurement assumes that the clean environment carries no local cache or developer settings. Which settings a real pipeline environment brings along on its own was not measured. The pipeline is applied to the two versions, and three checks run: that the pipeline as a whole is green, that the build step passes, and that the test step passes. ```swift // main.swift — three checks on a clean machine let cleanMachine = Settings(hasTargetSetting: false) let correctRun = runPipeline(total: basketTotal(basketPrices), expected: expectedTotal, settings: cleanMachine) let faultyRun = runPipeline(total: faultyBasketTotal(basketPrices), expected: expectedTotal, settings: cleanMachine) let allGreen = contrast("pipeline-green", correct: trace(correctRun), faulty: trace(faultyRun)) { !$0.contains("fail") && !$0.contains("skipped") }.describe() let buildStep = stepContrast("build-step-passed", step: "build", correct: correctRun, faulty: faultyRun) let testStep = stepContrast("test-step-passed", step: "test", correct: correctRun, faulty: faultyRun) countSeparations([allGreen, buildStep, testStep]) countOutside([allGreen, buildStep, testStep]) ``` ```text pipeline-green | correct=build:fail test:skipped package:skipped/fail faulty=build:fail test:skipped package:skipped/fail -> noisy build-step-passed | correct=fail/fail faulty=fail/fail -> noisy test-step-passed | correct=skipped faulty=skipped -> outside-category separating: 0 blind: 0 noisy: 2 inverted: 0 outside-category: 1 ``` On both versions the pipeline stops in the same place: `build:fail`. The `pipeline-green` check, which looks at the pipeline as a whole, fails on both versions, so it is **noisy**. The check that looks at the build step itself is the same: `fail` on both versions. The pipeline is red, and the fault has no share in that; a missing setting stopped the pipeline before the basket total was ever computed. The practical consequence is harmful. When the pipeline turns red, the team first repairs the red, because no change can move forward while the pipeline is red. The cause of the red is the setting, and the repair is adding the setting. But during the time until the setting is added, nobody knows whether the pipeline would have seen the fault; the pipeline never reached the fault. A red pipeline is very easy to read as "the fault was caught", while the measurement says the opposite: whether the fault was caught does not come out of this run. The third line, `test-step-passed`, was not classified. The test step did not run on either version, because the pipeline stopped before it. The output says `correct=skipped faulty=skipped` and the result is `outside-category`. This line is not separating, not blind, not noisy, and not inverted; the check ran on neither version, and a check that did not run has no answer for the two versions. #### Once the Setting Is Fixed The setting is added to the pipeline definition, and the clean machine now passes the build. The same three checks, plus a fourth that looks at the package step, are applied to the same two versions. ```swift // main.swift — four checks after the setting is fixed let fixedMachine = Settings(hasTargetSetting: true) let correctRun = runPipeline(total: basketTotal(basketPrices), expected: expectedTotal, settings: fixedMachine) let faultyRun = runPipeline(total: faultyBasketTotal(basketPrices), expected: expectedTotal, settings: fixedMachine) let allGreen = contrast("pipeline-green", correct: trace(correctRun), faulty: trace(faultyRun)) { !$0.contains("fail") && !$0.contains("skipped") }.describe() let buildStep = stepContrast("build-step-passed", step: "build", correct: correctRun, faulty: faultyRun) let testStep = stepContrast("test-step-passed", step: "test", correct: correctRun, faulty: faultyRun) let packageStep = stepContrast("package-step-passed", step: "package", correct: correctRun, faulty: faultyRun) countSeparations([allGreen, buildStep, testStep, packageStep]) countOutside([allGreen, buildStep, testStep, packageStep]) ``` ```text pipeline-green | correct=build:pass test:pass package:pass/pass faulty=build:pass test:fail package:skipped/fail -> separating build-step-passed | correct=pass/pass faulty=pass/pass -> blind test-step-passed | correct=pass/pass faulty=fail/fail -> separating package-step-passed | correct=pass faulty=skipped -> outside-category separating: 2 blind: 1 noisy: 0 inverted: 0 outside-category: 1 ``` Once the setting is fixed, the table changes. `test-step-passed` passes on the correct version and fails on the faulty version: **separating**. `pipeline-green`, which looks at the pipeline as a whole, is also separating, because the pipeline is red when the test fails. `build-step-passed` passes on both versions, **blind**: the fault does not touch the build, and the build finishes clean on both versions. The fourth line is again `outside-category`. The difference between the first measurement and the second is a single setting. The code is the same, the basket is the same, the test is the same; but the `pipeline-green` check moved from noisy to separating. This shows that separation is a function not only of the check or the code but also of the **environment**. The same pipeline, as long as the setting is missing, says nothing about the fault. #### Why a Step That Did Not Run Is Not a Category Returning to the fourth line: `package-step-passed`. On the correct version the packaging passed; on the faulty version it did not run. If the core classified this pair, `pass` and `skipped`, directly, it would count the check as **separating**: the correct version passed, the faulty version "could not pass". But the packaging step never saw the fault. In the faulty version the basket total is too small, and the `total > 0` check of the `package` step would have passed with the too-small total as well; the step did not run only because the pipeline stopped at the test. This difference matters, because otherwise every step at the end of the pipeline would look separating. Packaging, signing, upload: each sits after the pipeline's first failing step, so each would count as separating whenever the fault is caught earlier in the pipeline. But to separate is for a check to give a different answer to the faulty version **on its own**. A step that did not run has no answer; that answer is determined by the pipeline's first failing step. The distinction the lesson makes is this. If the step ran on both versions, the result falls into one of the four categories. If it did not run on one of them, `outside-category` is written and counted separately. The last line of the output gives this number; so "how many checks were classified" and "how many checks never ran" are not mixed up. If thirty of the forty steps of a green pipeline did not run, that pipeline is as silent as thirty checks. #### The Pipeline Definition A pipeline definition is a configuration file and usually lives in the repository like code. The notation below is a generic form; these are not the real key names of any service, and this lesson does not measure the pipeline definition. What is measured is the pipeline model. ```yaml steps: - name: build run: xcodebuild build -scheme Shop - name: test run: xcodebuild test -scheme Shop - name: package run: xcodebuild archive -scheme Shop ``` The definition looks like a three-line list and matches the model's `pipelineSteps` array one to one. What matters is the **order** of the list and whether each step stops the pipeline on failure. Most pipeline definitions do this by default; some can be configured to continue even when a step fails. In a pipeline that continues there is no `skipped`, and what every step says about the two versions can be seen. This is a trade-off: the pipeline takes longer and produces more output, but the `outside-category` lines turn into lines that can be classified. If a setting is missing from the pipeline definition, the gap becomes visible only on the first clean machine. That is why it is a good habit to run the pipeline's **first** run with a deliberately faulty version as well: can the pipeline actually reach the test step, and if it does, does it fail on the faulty version. This single trial catches, while the pipeline is being built, a check problem that would otherwise stay inconclusive while the pipeline is green. #### The Bounding Measurement Separation also depends on the order of the steps. In the measurement below, the same clean machine, that is, the same missing setting, runs in two different orders. In the first order, the build comes first; in the second, the test. When the order changes, the test step is reached; but the missing setting still breaks the build step. The fifth check looks at the build step once the setting is fixed. ```swift // main.swift — same machine, different step order, same check let cleanMachine = Settings(hasTargetSetting: false) let fixedMachine = Settings(hasTargetSetting: true) let testFirst = ["test", "build", "package"] let correctTotalNow = basketTotal(basketPrices) let faultyTotalNow = faultyBasketTotal(basketPrices) let buildFirstClean = ( runPipeline(total: correctTotalNow, expected: expectedTotal, settings: cleanMachine), runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: cleanMachine)) let testFirstClean = ( runPipeline(total: correctTotalNow, expected: expectedTotal, settings: cleanMachine, steps: testFirst), runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: cleanMachine, steps: testFirst)) let buildFirstFixed = ( runPipeline(total: correctTotalNow, expected: expectedTotal, settings: fixedMachine), runPipeline(total: faultyTotalNow, expected: expectedTotal, settings: fixedMachine)) let first = stepContrast("build-first-test", step: "test", correct: buildFirstClean.0, faulty: buildFirstClean.1) let second = stepContrast("test-first-test", step: "test", correct: testFirstClean.0, faulty: testFirstClean.1) let third = stepContrast("setting-fixed-test", step: "test", correct: buildFirstFixed.0, faulty: buildFirstFixed.1) let fourth = contrast("test-first-pipeline", correct: trace(testFirstClean.0), faulty: trace(testFirstClean.1)) { !$0.contains("fail") }.describe() let fifth = stepContrast("setting-fixed-build", step: "build", correct: buildFirstFixed.0, faulty: buildFirstFixed.1) countSeparations([first, second, third, fourth, fifth]) countOutside([first, second, third, fourth, fifth]) ``` ```text build-first-test | correct=skipped faulty=skipped -> outside-category test-first-test | correct=pass/pass faulty=fail/fail -> separating setting-fixed-test | correct=pass/pass faulty=fail/fail -> separating test-first-pipeline | correct=test:pass build:fail package:skipped/fail faulty=test:fail build:skipped package:skipped/fail -> noisy setting-fixed-build | correct=pass/pass faulty=pass/pass -> blind separating: 2 blind: 1 noisy: 1 inverted: 0 outside-category: 1 ``` The same test-step check, the same fault, three different answers: `build-first-test` is not classified, while `test-first-test` and `setting-fixed-test` separate. Whether the test step is seen while the missing setting stands depends entirely on the step order. A pipeline that runs the test first catches the fault even when the setting is missing. But `test-first-pipeline`, which looks at the pipeline as a whole, still stays **noisy**: the build breaks in the next step, and the pipeline is red on both versions. To see that the test caught the fault, one has to look not at the pipeline result but at the step's own result. These two observations lean on each other. The order determines separation; but choosing the order for the sake of separation is a bad idea, because the pipeline cannot test without a build. In a real application, a test needs the built output, and the order change in this measurement is possible only in the model. The lesson's takeaway is that when writing a pipeline definition, besides asking whether the step is "in a position to see the fault", one must also ask whether the fault is **reachable**. **DS6.** This measurement treats the order change as free in the model; the test step runs without depending on the build. The test step's dependence on the build output in a real pipeline was not measured. #### Summary - Continuous integration runs the same steps in order in a clean environment for each change and stops at the first failing step. - On a clean machine, a missing build setting breaks the pipeline on both versions. The checks that look at the pipeline as a whole and at the build step become noisy. - Once the setting is fixed, the test step passes on the correct version and fails on the faulty version: separating. The build step passes on both versions: blind. - Steps that did not run after the pipeline stopped are not classified; in this lesson they are counted separately and written as `outside-category`. - Separation depends not only on the check but also on the environment and the order of the steps; a red pipeline does not mean the fault was caught. - This lesson's separation: **noisy**. #### Next Step A pipeline whose test step catches the fault also reaches the steps that upload a version to the store. In the next lesson, Release Automation, we will measure which checks separate the upload script that raises the version and build numbers. ### Release Automation Source: https://academia.sh/en/courses/architecture-testing-and-release/release-automation A measurement showing that a check on whether the build number is greater than the last number in the store separates an upload script that does not raise the build number, while a format check stays blind. Every step done by hand after a version is ready is a chance for a mistake: raising the number, producing the archive, signing, uploading. **Release automation** gathers these steps into a single script and runs them in the same order every time. If the script is written wrongly, the mistake is repeated every time as well; that is why the script itself is checked too. The measure of this course is called **separation**: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: **separating**, where the correct version passes and the faulty version fails; **blind**, where both pass; **noisy**, where both fail; and **inverted**, where the correct version fails and the faulty version passes. In the output they appear as `separating`, `blind`, `noisy`, and `inverted`. This lesson measures with the core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is a small oversight: the upload script **does not raise the build number**. The version number stays the same, which is normal, but the build number stays the same too. The script runs, the archive is produced, the signing passes, and the upload command runs. No step turns red. The problem surfaces only when the store sees the same number a second time, and by then the upload has already been made. #### The Version Number and the Build Number An application has two numbers. The **version number** (such as `2.4.0`) is visible to the user and says which form of the product it carries. The **build number** tells apart the successive uploads of the same version: if three uploads were made while the version was `2.4.0`, there are three different build numbers. Because the two numbers have different roles, the mistake shows up in different places. Someone who writes the version number wrongly notices at once, because it stands on the screen. Someone who forgets to raise the build number sees nothing. The script below is the skeleton of an upload step. The tool names are kept generic: the first command is a small helper that raises the number, and the third is the uploader. The script is not measured; the lesson only shows what it does. ```zsh ./bump-build-number.sh xcodebuild archive -scheme Shop -archivePath Shop.xcarchive ./upload-to-store.sh Shop.xcarchive ``` The faulty script lacks the first line. The other two lines are identical. The eye reading the script finds the fault only by searching for the line that slipped away, and to search for the line that slipped away, one must first know that something slipped away. The job of a check is exactly this. #### Who Raises the Number The job of raising the build number can be placed in three spots, and each spot has its own mistake. If it is raised by hand, it is forgotten; this is a special case of the faulty script. If the script raises it, one must be sure the script raises it on every run: if the script runs again after a failed upload, it raises the number a second time and the numbers skip. If the number is derived from the pipeline's counter, every pipeline run raises the number automatically, but this time the link between the number and the version history loosens: two uploads carry different numbers even if they came from the same change. No option is free of mistakes, and the choice is not this lesson's subject. The lesson's subject is whether the chosen path is checked for whether it **actually raised** the number. A team that raises it by hand keeps the check as a checklist; a team that raises it with a script puts the check on the script's output. In both cases the check asks the same question: is the upload number greater than the previous one. #### Building the Model The model holds an upload as two fields: the version number and the build number. `prepareUpload` imitates the work of the script: it takes the last number in the store, raises it by one in the correct script, and leaves it as it is in the faulty script. `hasVersionShape` checks that the version number has the `x.y.z` form, that is, three groups of digits separated by dots. `Reading` carries an upload together with **the last number it is compared against**; this will be needed in the last section of the lesson. ```swift // Release.swift — a model of the upload script's version and build numbers struct Upload: CustomStringConvertible { let version: String let buildNumber: Int var description: String { "\(version)(\(buildNumber))" } } struct Reading: CustomStringConvertible { let upload: Upload let recorded: Int var description: String { "\(upload)/recorded=\(recorded)" } } func hasVersionShape(_ version: String) -> Bool { let parts = version.split(separator: ".", omittingEmptySubsequences: false) return parts.count == 3 && parts.allSatisfy { !$0.isEmpty && $0.allSatisfy { $0.isNumber } } } func prepareUpload(version: String, lastUploaded: Int, bumpsNumber: Bool) -> Upload { Upload(version: version, buildNumber: bumpsNumber ? lastUploaded + 1 : lastUploaded) } let releaseVersion = "2.4.0" let storeLast = 12 ``` The correct script produces version `2.4.0` with build number `13`. The faulty script produces `12` with the same version, which repeats the last number in the store. **DS7.** This measurement takes as the lesson's assumption that the store expects the build number for the same version to be greater than the previous one. Which number the store actually accepts or rejects was not measured. #### Three Checks The upload the script produces is tested with three checks. The first looks at whether the version number has the `x.y.z` form: it is the check most often written in upload scripts, because it is easy to write. The second looks at whether the build number is greater than the last number in the store. The third looks at whether the build number is positive. ```swift // main.swift — three checks, two categories let withBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: true) let withoutBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: false) let shape = contrast("shape-x-y-z", correct: withBump, faulty: withoutBump) { hasVersionShape($0.version) }.describe() let greater = contrast("number-greater-than-last", correct: withBump, faulty: withoutBump) { $0.buildNumber > storeLast }.describe() let positive = contrast("number-positive", correct: withBump, faulty: withoutBump) { $0.buildNumber > 0 }.describe() countSeparations([shape, greater, positive]) ``` ```text shape-x-y-z | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind number-greater-than-last | correct=2.4.0(13)/pass faulty=2.4.0(12)/fail -> separating number-positive | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind separating: 1 blind: 2 noisy: 0 inverted: 0 ``` Three lines, two categories. `shape-x-y-z` and `number-positive` give the same answer on both uploads: the version number is `2.4.0` in both uploads, and the build number is positive in both. The fault does not touch these two values, because the fault is a **comparison**: the number is not wrong, it is equal to the last number in the store. Only `number-greater-than-last` makes this comparison. On the correct upload `13 > 12` passes, and on the faulty upload `12 > 12` fails: **separating**. #### Why the Format Check Is Common and Why It Is Blind The format check is common because it is easy to write and it catches many faults: a letter in the version number, a missing group, an extra dot. The faults it catches are real. But what the check looks at is **the consistency of a single upload within itself**; it does not compare the upload with another upload. A number that does not increase becomes visible only when read together with the previous upload. Looked at alone, `2.4.0(12)` is a valid upload, and so is `2.4.0(13)`. This is a trait common to blind checks: the check does not look at the value the fault changes. Here the fault changes the **relation of the build number to the previous value**, while the `x.y.z` format looks only at the version number. A green format check does not carry the sentence "the numbers are correct"; it carries only "the version number is well formed". Writing "numbers are valid" in the script's report without separating these two sentences puts the first in the place of the second. The `number-positive` check is weaker still. It asks for the build number to be greater than zero; the faulty number is `12` as well. A weak check does not mean a bad check: against another fault that produces a zero or a negative number, this check would be separating. The distinction belongs to the pair of check and fault; when the fault changes, the table changes too. #### The Data the Greater-Than Check Depends On `number-greater-than-last` is the separating one, but in this measurement the `storeLast` value was taken as if it came directly from the store. In a real script, where the last number is read from is a separate question, and the check is as reliable as that source is accurate. There are three common sources. One is the store itself: the script asks for the last number before uploading. One is a local file: the team keeps the last uploaded number in a file and updates it with each upload. One is the version control history: the number is read from tags. Each ages in a different way. The local file is the easiest and the one that causes the most trouble. The file is written in the step that raises the number. In the faulty script that step does not exist, so the file is not written either. The file holds a number left over from an earlier upload, **older** even than the one in the store. If the check compares against this old number, the faulty upload looks greater than the old number and passes. The correct upload, on the other hand, has updated the file itself; the new number is equal to itself, and `>` fails. This is the flipping of a separating check into its opposite. **DS8.** This measurement assumes that the local record is updated only in the step that raises the number. When and how a real script writes the record was not measured. #### The Bounding Measurement The same check, the build number being greater than the last number, runs with three different sources of the last number. The first source is the store: the last number is `12` on both uploads. The second source is a file written after the upload: both scripts write the number they uploaded to the file, so the file carries its own number on each upload. The third source is a file updated only in the step that raises the number: the correct script writes `13`, the faulty script does not write, and the file stays at `11`. ```swift // main.swift — the same greater-than check, three sources of the last number let withBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: true) let withoutBump = prepareUpload(version: releaseVersion, lastUploaded: storeLast, bumpsNumber: false) let fromStore = contrast("last-number-store", correct: Reading(upload: withBump, recorded: storeLast), faulty: Reading(upload: withoutBump, recorded: storeLast)) { $0.upload.buildNumber > $0.recorded }.describe() let afterUpload = contrast("last-number-file-after-upload", correct: Reading(upload: withBump, recorded: withBump.buildNumber), faulty: Reading(upload: withoutBump, recorded: withoutBump.buildNumber)) { $0.upload.buildNumber > $0.recorded }.describe() let staleFile = contrast("last-number-file-in-bump-step", correct: Reading(upload: withBump, recorded: withBump.buildNumber), faulty: Reading(upload: withoutBump, recorded: 11)) { $0.upload.buildNumber > $0.recorded }.describe() let shapeAgain = contrast("shape-x-y-z", correct: withBump, faulty: withoutBump) { hasVersionShape($0.version) }.describe() countSeparations([fromStore, afterUpload, staleFile, shapeAgain]) ``` ```text last-number-store | correct=2.4.0(13)/recorded=12/pass faulty=2.4.0(12)/recorded=12/fail -> separating last-number-file-after-upload | correct=2.4.0(13)/recorded=13/fail faulty=2.4.0(12)/recorded=12/fail -> noisy last-number-file-in-bump-step | correct=2.4.0(13)/recorded=13/fail faulty=2.4.0(12)/recorded=11/pass -> inverted shape-x-y-z | correct=2.4.0(13)/pass faulty=2.4.0(12)/pass -> blind separating: 1 blind: 1 noisy: 1 inverted: 1 ``` The same comparison, `buildNumber > recorded`, fell into three different categories with three sources, and with the format check on the fourth line all four categories appear. In the first line the source tells the truth: **separating**. In the second line the file was written after the upload; since each upload equals its own number, both versions fail: **noisy**. A red check gives the impression that the script broke the upload, yet what broke it is the comparison itself. In the third line the correct upload fails and the faulty upload passes: **inverted**. This third line explains why a team sometimes switches off a check it once trusted. The check is red on the correct upload, the team counts it as a false alarm, and the threshold is loosened: `>=` is written instead of `>`. That way both the faulty and the correct upload pass, and the check goes blind. Instead of loosening a check that seems to raise false alarms, one must examine its **source**; `>` was written correctly in this lesson, and the fault was in the source. The limit of this measurement must be stated openly. The upload and the store's rejection were not measured; the script was shown only as source, and the model compared not the result of an upload but the two numbers that lead to it. Whether the store really rejects a second identical number, with what message, and how quickly, remains outside this lesson. **DS9.** In this measurement, the file's `11` value is a previous record that the faulty script did not write, and it is older than the last number in the store. How old a stale record can be in a real script was not measured. #### What the Script Output Says The summary printed at the end of the script is the only place where the team reads about the upload. A script that writes "version 2.4.0, build 12, uploaded" in the summary shows the faulty upload as clean as the correct one. For the summary to carry separating information, it is enough to write the number **together with the previous number**: "build 12, previous 12". A team that sees these two numbers side by side notices the number that does not increase even if the script ran no check at all. The same principle holds elsewhere. A number alone has little meaning; if the number it is compared against is written down, the check becomes readable. For that reason the upload record should carry not only the latest state but also the previous one. A record that does not carry the previous state silently produces a wrong comparison when it is stale or unwritten, as in the last measurement of this lesson. Finally, recall the question to ask when writing a new check: what does this check say about the upload the faulty script produces? To answer it, it is enough to break the script on purpose; deleting the first line is the first mutation test. If a check does not fail on the first attempt, it is blind or inverted, and, as the lesson showed, the cause usually lies in the data the check looks at. #### Summary - The version number is visible to the user, and the build number tells apart successive uploads of the same version; a build number that does not increase turns no step red. - Checks that look only at the `x.y.z` format or at whether the number is positive pass on both uploads: blind. - The check on whether the build number is greater than the last number in the store separates the faulty upload: separating. - The same greater-than check becomes noisy if the last number is read from a file written after the upload, and inverted if it is read from a stale file updated only in the step that raises the number. - Before loosening a red greater-than check, the source of the last number is examined. - This lesson's separation: **separating**. #### Next Step An upload whose number is raised correctly still does not have to be opened to everyone. In the next lesson, Beta Distribution, we will measure opening a new feature to a limited audience, and a flag that stays open outside that audience as well. ### Beta Distribution Source: https://academia.sh/en/courses/architecture-testing-and-release/beta-distribution A measurement showing that a check saying the feature is on for beta testers cannot see a flag that is also open outside the beta audience, while a check saying it is off for non-beta users can. A build can be given to a small audience before it is opened to everyone. This audience is the team itself, invited outside testers, or both. Through the platform's beta distribution service, the build reaches the devices of the invited users, and they try it in real use. The goal is to find the problem before it reaches everyone. A build reaching an audience is not the same thing as a feature being open to that audience. The build is the same for everyone; a new feature, however, can arrive switched off behind a **feature flag** and be turned on only for certain users. A flag opened beyond the beta audience removes the very reason beta distribution exists. The measure of this course is called **separation**: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: **separating**, where the correct version passes and the faulty version fails; **blind**, where both pass; **noisy**, where both fail; and **inverted**, where the correct version fails and the faulty version passes. In the output they appear as `separating`, `blind`, `noisy`, and `inverted`. This lesson measures with the core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is in the flag's configuration. In the correct version, the new feature is on only for the users on an allowlist. In the faulty version, the configuration was left as "open to everyone" and the allowlist is never read. Nothing changes for beta testers: the feature is on for them and it works. The difference shows only for users outside the beta. #### The Flag Model The flag consists of two values: whether it is open to everyone, and an allowlist. `isOn` says whether the flag is on for a user: on if the flag is open to everyone, otherwise on if the user is in the allowlist. `exposure` writes the state of each user in a list into a single string, so the answers of the two versions can be read side by side. The configuration usually sits in a file that looks like the one below. The key names are not those of a real service; the lesson only shows the form. ```yaml flags: new-checkout: enabled_for: beta allowlist: [u01, u02, u03] ``` In the faulty version the `enabled_for` value was left as `everyone`; the allowlist is in place but nobody looks at it. An eye that reads the configuration notices this, because it is a single value. For a check that does not read it and only looks at behavior, the difference can stay invisible. ```swift // Beta.swift — the feature flag's allowlist and user states struct Rollout { let everyone: Bool let allowlist: Set } func isOn(_ user: String, rollout: Rollout) -> Bool { rollout.everyone || rollout.allowlist.contains(user) } func exposure(of users: [String], rollout: Rollout) -> String { users.map { "\($0):\(isOn($0, rollout: rollout) ? "on" : "off")" }.joined(separator: " ") } let allowlist: Set = ["u01", "u02", "u03"] let betaTesters = ["u01", "u02", "u03"] let newTester = ["u04"] let outsiders = ["u10", "u11"] let enabledForBeta = Rollout(everyone: false, allowlist: allowlist) let enabledForAll = Rollout(everyone: true, allowlist: allowlist) ``` The lesson's users are written with short identifiers: three beta testers, one newly invited beta tester, and two non-beta users. The newly invited user exists in the beta service but has not yet been added to the allowlist; why this matters will show in the bounding measurement. **DS10.** This measurement represents users with a small list of identifiers. The size of the list says nothing about the size of the audience, and how a real beta audience is chosen was not measured. #### Three Checks Three checks are applied to the flag. The first is the most natural: is the feature on for beta testers. The second looks at whether the feature is off for non-beta users. The third looks not at behavior but at configuration: is the flag not open to everyone. ```swift // main.swift — three checks, two categories let onBeta = contrast("on-for-beta-testers", correct: exposure(of: betaTesters, rollout: enabledForBeta), faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("off") }.describe() let onOutsiders = contrast("off-for-non-beta-users", correct: exposure(of: outsiders, rollout: enabledForBeta), faulty: exposure(of: outsiders, rollout: enabledForAll)) { !$0.contains("on") }.describe() let configured = contrast("configuration-not-open-to-everyone", correct: enabledForBeta.everyone, faulty: enabledForAll.everyone) { !$0 }.describe() countSeparations([onBeta, onOutsiders, configured]) ``` ```text on-for-beta-testers | correct=u01:on u02:on u03:on/pass faulty=u01:on u02:on u03:on/pass -> blind off-for-non-beta-users | correct=u10:off u11:off/pass faulty=u10:on u11:on/fail -> separating configuration-not-open-to-everyone | correct=false/pass faulty=true/fail -> separating separating: 2 blind: 1 noisy: 0 inverted: 0 ``` All three checks pass on the correct version. On the faulty version the first one passes too: for beta testers the feature **really is** on, in the faulty version as well. This check correctly measures whether the feature works, and it says nothing about whom the flag is open to. It gives the same answer on both versions: **blind**. The second check looks at the audience the fault changed. For a non-beta user the correct version shows off and the faulty version shows on, and the check fails only on the faulty version: **separating**. The third check is also separating, but for another reason: the fault sits directly in the configuration value, and the check looks at that value. The difference between the two separating checks is that the first measures behavior and the second reads the file. The check that measures behavior is also robust against a fault where the configuration changes by another route; the check that reads the configuration does not show whether the configuration is actually read in the application. #### Invitation, Build, and Flag Who sees what in beta distribution is determined by three separate layers, and each layer has its own list. The first is the invitation list: it says which users can download the build. The second is the build itself: it determines which code is in the downloaded package. The third is the flag's allowlist: it says whom the feature inside the package is opened to. The three lists live in different places and are updated by different people. This separation explains why the lesson's fault stays silent. The invitation list can be right, the build can be packaged right, and the flag can still have been left open to everyone. Each layer is consistent within itself; the inconsistency lies between the layers. A test that checks one layer cannot see the fault of the others. The newly invited user example is a small form of this: the user is in the invitation list, is not in the allowlist, and the two lists do not look at each other. #### Why the "On" Check Gets Written The check for the feature being on for beta testers is common, because the purpose of beta distribution is to try the feature, and the team naturally asks "does the feature work". The question is valid and must stand somewhere in the pipeline. But it is the question **does the feature work**, not the question **does the feature work only for the right audience**. The two questions catch different faults. The first finds a flag that arrives closed; the second finds a flag that is opened too much. A flag opened too much is a less known risk, because nobody complains. Beta testers see the feature on and are satisfied, and non-beta users also see the feature on and do not complain, since the feature works. The mistake comes out only when the feature does something unexpected, or when the release owner looks at whom the feature is visible to. The blind check is a silent danger because it reports this silence as green output. #### Completing the Blind Check A blind check is not deleted; it is completed. Here the complementary check is the one that says off for non-beta users. Together they cover the two faces of the flag: on where it should be on, off where it should be off. Writing only one of them is acceptable provided one knows which face is being tested; a first check written without knowing gives the impression that writing the second is unnecessary. The mutation test applies here as well. Deliberately setting `enabled_for` to `everyone` in the configuration and running the checks shows which check turns red. The lesson's first measurement is the result of that attempt: of the three checks, one never turned red. #### What Beta Feedback Does Not Say At the end of the beta period, the team usually looks at feedback: how many bug reports came in, what the users wrote. Few reports give the impression that the build is sound. But feedback reflects only **seen** behavior and is silent about behavior that is not seen. A flag left open to non-beta users is a typical example of this silence: if the feature works, nobody writes. For this reason, beta feedback does not take the place of a check. Feedback answers the question "did the users notice anything"; a check answers the question "is the flag open to the right audience". The two run together in the same beta period, and neither closes the gap of the other. If the summary the team writes at the end of the beta carries two separate lines, feedback and flag state, the reader knows which one proves what. The cost of a blind check is visible here too. The check saying on for beta testers is green in every beta period, and this green gives the team confidence that the beta period went without problems. If underneath the green lies a fault where the flag is open to everyone, the confidence is in the wrong place. The solution is not to delete the green but to put the check on the audience outside the beta next to it. #### When the Audience Grows The beta audience grows: another team is added, outside testers are invited. With every expansion the allowlist changes, and the definition of "non-beta" changes with it. A list of non-beta users written by hand can contain a beta tester after the expansion. A check like that fails on the correct version, because the feature is on for a user who is now a beta tester. The check raises a false alarm and the team switches it off; but switching off a check because the audience grew leaves the risk of the flag opening beyond the audience uncovered. The solution is to **derive** the non-beta users from the invitation list instead of choosing them by hand: every user not in the invitation list is outside the beta. Because the definition stands in one place, expansion does not break the check. Unifying the data source the check reads in this way frees separation from depending on user selection. #### When the Flag Is Rolled Back If a problem appears during the beta, the flag is turned off; no new build needs to be uploaded. This is one of the main values of flags, and rollback is also a subject for checks. If, after the flag is turned off, it is checked that the feature is **off** for beta testers, that check shows that the turning off was actually applied. Here too the same question applies: is the check written to catch a flag that was not turned off? Writing the opposite of the check saying on for beta testers is not enough; one has to look at what answer it gives on which version. #### The Bounding Measurement The same checks fall into different categories with different user lists. The first list is three beta testers. The second list is a single beta tester who was newly invited and has not yet been added to the allowlist. The third and fourth lines apply the off-for-non-beta check to the list of beta testers and to a single non-beta user. ```swift // main.swift — the same checks, different user lists let open = contrast("on-three-beta-testers", correct: exposure(of: betaTesters, rollout: enabledForBeta), faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("off") }.describe() let invited = contrast("on-newly-invited", correct: exposure(of: newTester, rollout: enabledForBeta), faulty: exposure(of: newTester, rollout: enabledForAll)) { !$0.contains("off") }.describe() let closedOnBeta = contrast("off-on-beta-testers", correct: exposure(of: betaTesters, rollout: enabledForBeta), faulty: exposure(of: betaTesters, rollout: enabledForAll)) { !$0.contains("on") }.describe() let closedOnOutsider = contrast("off-one-non-beta-user", correct: exposure(of: ["u10"], rollout: enabledForBeta), faulty: exposure(of: ["u10"], rollout: enabledForAll)) { !$0.contains("on") }.describe() countSeparations([open, invited, closedOnBeta, closedOnOutsider]) ``` ```text on-three-beta-testers | correct=u01:on u02:on u03:on/pass faulty=u01:on u02:on u03:on/pass -> blind on-newly-invited | correct=u04:off/fail faulty=u04:on/pass -> inverted off-on-beta-testers | correct=u01:on u02:on u03:on/fail faulty=u01:on u02:on u03:on/fail -> noisy off-one-non-beta-user | correct=u10:off/pass faulty=u10:on/fail -> separating separating: 1 blind: 1 noisy: 1 inverted: 1 ``` Four lines, four categories. `on-three-beta-testers` is the same as in the first measurement: **blind**. `on-newly-invited` is **inverted**: since the newly invited user is not in the allowlist, the feature is off for them in the correct version and the check fails; in the faulty version the flag is open to everyone, the feature is on for them as well, and the check passes. So a team that forgets to update the allowlist gets an error on the correct version and sees no error at all on the faulty version. The red one is the correct version. In the third line the check looks for "off" among beta testers and fails on both versions, **noisy**: the feature is on for beta testers in both versions. This line shows the design of a bad check: the check's expectation and the users it looks at do not match. The fourth line, with a single non-beta user, is again **separating**. So finding the separating check does not require a large user list; it is enough that the right user is in the list. This measurement shows the importance of user selection, but it also has a limit. Each list is small, and in a real audience the allowlist also has gaps, misspelled identifiers, and old accounts. The audience size and proportions were not measured. **DS11.** This measurement assumes that the allowlist is kept in a place separate from the beta invitations. How the invitation and the list are kept in sync in a real pipeline was not measured. **DS12.** This measurement does not model the time the beta service takes to deliver the build to the user. When the invitation arrived or when the build reached the device was not measured. #### Summary - Beta distribution delivers the build to a limited audience; the feature flag determines whom the feature is open to. - A check that the feature is on for beta testers passes on both versions when the flag is also open outside the audience: blind. - A check that the feature is off for non-beta users separates the fault: separating. - The same "on" check becomes inverted for a newly invited user who has not yet been added to the allowlist. - The separation of a check depends on the user list it looks at; a blind check is not deleted but completed with a counterpart that checks what should be off. - This lesson's separation: **blind**. #### Next Step The build tried in the beta audience finally goes to the store's review. In the next lesson, The Store Review Process, we will measure which items of a readiness checklist separate the fault and which fail on both versions. ### The Store Review Process Source: https://academia.sh/en/courses/architecture-testing-and-release/store-review-process A measurement showing that a readiness-list item checking whether the permission description is filled separates the fault, while an outdated item left on the list fails on both versions and makes the list noisy. Before a build is sent to the store, teams mostly go through a readiness list: are the screenshots in place, does the privacy link work, have the permission descriptions been written. The list is drawn from problems experienced in past submissions and grows with every new problem. The store's review decision is made outside this list; the list is the team's own way of protecting itself. The measure of this course is called **separation**: when the same check is applied to the correct version and to the faulty version, do the two answers differ. There are four categories: **separating**, where the correct version passes and the faulty version fails; **blind**, where both pass; **noisy**, where both fail; and **inverted**, where the correct version fails and the faulty version passes. In the output they appear as `separating`, `blind`, `noisy`, and `inverted`. This lesson measures with the core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is in a field of the metadata: the **description text** to be shown to the user for the location permission is empty. The application may ask for the location, but the permission screen does not tell the user why it is being asked. The build passes, the signing passes, the tests pass; this field is not something any build step checks. In the store's review, such a gap may cause the submission to be turned back; the lesson encodes this possibility only as an item of the readiness list. #### The Readiness List Model The model holds a submission with three fields: the location permission description, the screenshot sizes, and the presence of the privacy link. The readiness list consists of three items. `purpose-filled` checks that the description is not empty, and counts a text made only of spaces as empty. `privacy-link-present` checks that the link is defined. `old-size-screenshot-present` asks for a size named `old` to be present in the screenshot set; this item is left over from an old submission of the team. `readiness` applies each item to the submission and writes `item:pass` or `item:fail`. `Submission` summarizes itself in one line, so the output shows which field carries what. ```swift // Review.swift — the submission's fields and the team's readiness list struct Submission: CustomStringConvertible { let locationPurpose: String let screenshotSizes: [String] let privacyLink: Bool var purposeState: String { if locationPurpose.isEmpty { return "empty" } if locationPurpose.allSatisfy({ $0.isWhitespace }) { return "blank" } return "filled" } var description: String { "purpose=\(purposeState) sizes=\(screenshotSizes.joined(separator: ",")) privacy=\(privacyLink ? "present" : "missing")" } } let readinessItems = ["privacy-link-present", "old-size-screenshot-present", "purpose-filled"] func itemPasses(_ item: String, _ submission: Submission) -> Bool { switch item { case "purpose-filled": return submission.purposeState == "filled" case "purpose-not-empty": return !submission.locationPurpose.isEmpty case "privacy-link-present": return submission.privacyLink case "old-size-screenshot-present": return submission.screenshotSizes.contains("old") default: return false } } func readiness(_ submission: Submission, items: [String]) -> [String] { items.map { "\($0):\(itemPasses($0, submission) ? "pass" : "fail")" } } func listPasses(_ submission: Submission, items: [String]) -> Bool { readiness(submission, items: items).allSatisfy { $0.hasSuffix(":pass") } } let correctSubmission = Submission(locationPurpose: "Your location is used to show nearby stores", screenshotSizes: ["large", "medium"], privacyLink: true) let emptySubmission = Submission(locationPurpose: "", screenshotSizes: ["large", "medium"], privacyLink: true) let spaceSubmission = Submission(locationPurpose: " ", screenshotSizes: ["large", "medium"], privacyLink: true) ``` The two submissions differ only in the description field. The screenshots are in the `large` and `medium` sizes; the `old` size was removed from the team's visual layout and is absent from both submissions. The item, however, stayed on the list. **DS13.** The readiness list in this measurement is the team's own list. The items are not a rule published by the store, and the grounds on which the store turns a submission back were not measured. #### Four Checks Four checks are applied to the same two submissions: the description item, the old-size item, the privacy item, and the whole list. ```swift // main.swift — three items and the whole list let purpose = contrast("purpose-filled", correct: correctSubmission, faulty: emptySubmission) { itemPasses("purpose-filled", $0) }.describe() let oldSize = contrast("old-size-screenshot-present", correct: correctSubmission, faulty: emptySubmission) { itemPasses("old-size-screenshot-present", $0) }.describe() let privacy = contrast("privacy-link-present", correct: correctSubmission, faulty: emptySubmission) { itemPasses("privacy-link-present", $0) }.describe() let wholeList = contrast("whole-list-passed", correct: correctSubmission, faulty: emptySubmission) { listPasses($0, items: readinessItems) }.describe() countSeparations([purpose, oldSize, privacy, wholeList]) ``` ```text purpose-filled | correct=purpose=filled sizes=large,medium privacy=present/pass faulty=purpose=empty sizes=large,medium privacy=present/fail -> separating old-size-screenshot-present | correct=purpose=filled sizes=large,medium privacy=present/fail faulty=purpose=empty sizes=large,medium privacy=present/fail -> noisy privacy-link-present | correct=purpose=filled sizes=large,medium privacy=present/pass faulty=purpose=empty sizes=large,medium privacy=present/pass -> blind whole-list-passed | correct=purpose=filled sizes=large,medium privacy=present/fail faulty=purpose=empty sizes=large,medium privacy=present/fail -> noisy separating: 1 blind: 1 noisy: 2 inverted: 0 ``` Four lines show three different behaviors of the list. The description item passes the correct submission and fails the submission with the empty description: **separating**. The privacy link item passes on both submissions; since the link exists in both versions, it does not see the fault and is therefore **blind**. Being blind is not a defect; the item was written for another problem and has nothing to say about this fault. The real lesson is in the old-size item. The item fails on both submissions, because there is no `old` size in the screenshot set. Even the correct submission cannot pass the list. The faulty submission fails too, but the reason for failing has nothing to do with the fault. Since the two answers are the same, the item counts as **noisy**: it looks red and carries no separation. #### The Cost of Noise This cost shows in the whole-list line. `whole-list-passed` fails even on the correct submission, and the only reason is the old-size item. The person reading the list meets two situations. In the first, they see the red, look at the cause, understand that they got stuck on an old item, and move on. In the second, they no longer look at all, because they already know about the red. Teams most often fall into the second path, because a red that separates nothing becomes invisible after a while. This habit of ignoring continues when a real fault turns red on the list. The description item may catch the fault correctly; but since the list is red anyway, the caught fault is lost in the crowd. An item being separating on its own does not mean the list works. What carries the separation is the behavior of the whole list, and with the noisy item it is noisy. **DS14.** This measurement does not know why the item stayed outdated. That the screenshot layout changed and the `old` size is no longer produced is the lesson's assumption. In a real team's list there may also be a case where such an item is still needed. #### How an Old Item Is Cleaned Up Two things can be done with a noisy item: delete it or update it. Deleting is right if the concern the item covered no longer exists. If the screenshot layout changed and the old size is not asked for in any submission, the item does not belong on the list. Updating is done when the concern continues but its wording has aged: the item counts the sizes that are asked for today. The question to ask when deciding is the measurement's question: when I put a faulty submission and a correct submission together, does this item separate the two answers? If it does not and gives the same answer on every submission, then either the answer is always pass and the item is blind, or the answer is always fail and the item is noisy. Each may have its place. A blind item may stand like a guard watching for a fault experienced only once; a noisy item does not even do that job. A measurement is also taken before deleting. Taking the item off the list and checking the same two submissions again shows that the whole list now passes the correct submission and fails the faulty one. This is the proof that the item really consisted of noise. Without the proof, deletion is a dry guess. #### A Red Item and the Decision to Submit Between the list coming out red and the submission being stopped there is a decision, and the list does not make that decision. The team reads the red item and stops the submission, or decides the item is old and submits. The first path is safe but slow; the second is fast but breaks the list's own word. A noisy item recreates exactly this dilemma with every submission. A separating item has no such dilemma. Red means something in the submission is wrong, and green means that thing is in place. The decision becomes easier accordingly. This is the practical return of measuring the list: looking at the red and fixing it instead of looking at the red and arguing. There is also the question of who reads. The readiness list is mostly read by the person who makes the submission; they do not remember when the items were written. Someone who does not know why an item is on the list takes the old-size item seriously and may start preparing a visual for nothing. A noisy item thus costs not only time but also effort. #### Empty Versus Only Spaces The description item can be written in two forms. `purpose-filled` also checks that the text is not made only of spaces. `purpose-not-empty` checks only that the text is not of zero length. The two seem written for the same purpose, but they behave differently against a description made of three spaces. A description that shows three spaces on the user's screen is no different from an empty description; but the second item cannot see this. The next measurement applies the same two checks to the submission with three spaces. It also puts the old-size item next to them, so that it can be seen that the noisy item stays the same when the input changes. #### The Bounding Measurement The same items are applied to the submission that carries a description made only of spaces instead of an empty description. On the screen the two show the same blank. ```swift // main.swift — the same items, empty and blank descriptions let emptyByLength = contrast("not-empty-empty-description", correct: correctSubmission, faulty: emptySubmission) { itemPasses("purpose-not-empty", $0) }.describe() let blankByLength = contrast("not-empty-blank-description", correct: correctSubmission, faulty: spaceSubmission) { itemPasses("purpose-not-empty", $0) }.describe() let blankByContent = contrast("filled-blank-description", correct: correctSubmission, faulty: spaceSubmission) { itemPasses("purpose-filled", $0) }.describe() let oldSizeOnBlank = contrast("old-size-blank-description", correct: correctSubmission, faulty: spaceSubmission) { itemPasses("old-size-screenshot-present", $0) }.describe() countSeparations([emptyByLength, blankByLength, blankByContent, oldSizeOnBlank]) ``` ```text not-empty-empty-description | correct=purpose=filled sizes=large,medium privacy=present/pass faulty=purpose=empty sizes=large,medium privacy=present/fail -> separating not-empty-blank-description | correct=purpose=filled sizes=large,medium privacy=present/pass faulty=purpose=blank sizes=large,medium privacy=present/pass -> blind filled-blank-description | correct=purpose=filled sizes=large,medium privacy=present/pass faulty=purpose=blank sizes=large,medium privacy=present/fail -> separating old-size-blank-description | correct=purpose=filled sizes=large,medium privacy=present/fail faulty=purpose=blank sizes=large,medium privacy=present/fail -> noisy separating: 2 blind: 1 noisy: 1 inverted: 0 ``` The item that looks at length separates on the empty description and is blind on the description of spaces. Two appearances of the same fault make the same item give different answers. The item that looks at content separates on both. The old-size item stays red whatever the input is, and in this measurement too it is noisy. The limits are clear. This lesson did not measure the store's review decision or its duration; and that is not what should be measured, because that decision is an output rather than an input in the team's hands. The lesson also did not measure on which grounds the store turns submissions back. The reasons for rejection are countless: permission descriptions, privacy texts, the content of screenshots, the application behaving differently from what it declares, and many more. Out of this countlessness, what enters a list is what the team has experienced in the past. **DS15.** Every item in the readiness list is the team's list, not the store's rule. Passing the list does not mean the review will pass; failing the list does not say the submission will be turned back. The lesson measures only this: does an item on the list separate a fault the team knows from a submission without that fault. #### Keeping the List Honest A readiness list decays in two directions. The first is an item staying on the list after the source of the fault is gone; the old-size item is this. The second is a new fault passing without entering the list. When a submission is turned back, the work to do is to turn the reason for the rejection into an item and to test that item with this lesson's measurement: apply the item to the faulty submission, apply it to the correct submission, and see that the two answers differ. This test is done before the item is added, because every added item brings a maintenance load. An item that is not separating brings load but no separation. A blind item will not catch the next fault; a noisy item accustoms the reader of the list to red. An inverted item is the worst: it fails the correct submission and passes the faulty one, and pushes the team in the wrong direction. Maintenance of the list works if it is tied to one person or one step. After every submission the list is reviewed with a single question: which item stayed red and only red in this submission? That item is either updated or deleted. This habit turns the list into a tool that watches for today's faults and not the past's. #### Summary - The readiness list before the store's review comes from the team's own history of faults; it is not a rule published by the store. - The item checking whether the permission description is filled is separating, because it tells the submission with an empty description from the correct submission. - An outdated item stays red on both versions and makes the whole list noisy. - The two appearances of the same fault, empty and made only of spaces, give a different answer to an item that looks only at length; an item that looks at content separates both. - Before adding an item to the list or removing it, it is measured on a faulty and a correct submission. - This lesson's separation: **noisy**. #### Next Step The readiness list checks the content of the submission; the page the user sees in the store is a separate job. In the next lesson, The Store Listing, we will measure whether the screenshots and the subtitle separate the fault when compared with a recorded reference, and a case where an equality check can work backwards. ### The Store Listing Source: https://academia.sh/en/courses/architecture-testing-and-release/store-listing A measurement showing that an equality check against a recorded screenshot fails the correct version and passes the faulty version when the reference was taken from the faulty build, and that a subtitle limit check does not see the fault. The application page in the store is the only thing a user sees before downloading the application: a few screenshots, a name, a subtitle, and a description. These parts are kept apart from the application's code, are mostly uploaded by hand from the store console, and for that reason do not pass through the code's checks. When a mismatch appears between the page and the application, the first to notice it is often a user. The measure of this course is called **separation**. We apply the same check to the correct version and to the faulty version; if the check tells the two versions apart it is **separating**, if it passes both it is **blind**, if it fails both it is **noisy**, and if it fails the correct one and passes the faulty one it is **inverted**. In the output these four names are written as `separating`, `blind`, `noisy`, and `inverted`. This lesson also measures with the same core file. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault is in the page's screenshot: the screenshot shows the old interface, not the new one. The application has been renewed, the page has not. The user sees an old interface, downloads, and meets a completely different application. The subtitle, on the other hand, is the same in both versions and stays within the limit the lesson sets. #### The Listing Model The model extracts two things from a build: which interface the screenshot shows, and the text of the subtitle. `listing(of:)` collects the two in a `Listing` value; `Listing` summarizes itself in one line, so the output shows what the screen displays and how many characters the subtitle has. There are three checks. `matchesReference` compares the screenshot with a recorded reference; it is the simplified form of the equality check teams perform with image comparison. `subtitleFits` looks at whether the character count of the subtitle stays within the limit. `showsNewInterface` tests directly that the screenshot shows the new interface. ```swift // Listing.swift — a listing from a build, three checks struct Build { let interface: String let subtitle: String } struct Listing: CustomStringConvertible { let screenshot: String let subtitle: String var description: String { "screen=\(screenshot) subtitle=\(subtitle.count)" } } func listing(of build: Build) -> Listing { Listing(screenshot: "screen-\(build.interface)", subtitle: build.subtitle) } func matchesReference(_ listing: Listing, reference: String) -> Bool { listing.screenshot == reference } func subtitleFits(_ listing: Listing, limit: Int) -> Bool { listing.subtitle.count <= limit } func showsNewInterface(_ listing: Listing) -> Bool { listing.screenshot == "screen-new" } let subtitleLimit = 30 let correctBuild = Build(interface: "new", subtitle: "Track all orders in one place") let faultyBuild = Build(interface: "old", subtitle: "Track all orders in one place") ``` **DS16.** The subtitle limit is thirty characters. This number is the lesson's rule; the store's real limit was not measured, and nothing measured depends on it. The lesson asks only this: does a check that looks at the subtitle not exceeding the limit see this fault. #### Three Checks The reference is recorded the first time a screenshot is uploaded to the page. The person who takes the record has taken the image from the build in their hands that day. In this measurement that build is faulty: the screenshot was recorded as `screen-old`. The equality check works against this record. ```swift // main.swift — equality with the recorded reference, subtitle limit, direct test let correctListing = listing(of: correctBuild) let faultyListing = listing(of: faultyBuild) let recordedFromFaulty = faultyListing.screenshot let equality = contrast("screen-equals-reference", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: recordedFromFaulty) }.describe() let subtitle = contrast("subtitle-within-limit", correct: correctListing, faulty: faultyListing) { subtitleFits($0, limit: subtitleLimit) }.describe() let direct = contrast("screen-shows-new-interface", correct: correctListing, faulty: faultyListing) { showsNewInterface($0) }.describe() countSeparations([equality, subtitle, direct]) ``` ```text screen-equals-reference | correct=screen=screen-new subtitle=29/fail faulty=screen=screen-old subtitle=29/pass -> inverted subtitle-within-limit | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/pass -> blind screen-shows-new-interface | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating separating: 1 blind: 1 noisy: 0 inverted: 1 ``` The equality check came out **inverted**. The correct version differs from the recorded image, and the check fails. The faulty version is identical to the record, and the check passes. The check works correctly in itself: it looks at whether two things are equal. What is wrong is what it should be equal to. Because the reference was recorded from a faulty build, the check confirms that the fault is preserved. This is the most insidious form of being inverted. A noisy item turns red and draws attention; a blind item passes silently. An inverted item is green on the faulty version, and the team reads this green less as a warning than as a confirmation: we checked the page, it is identical to the record. When the correct version arrives, the check turns red, and the team may deem the new, correct image wrong and try to protect the old record. The subtitle check stayed **blind**. The subtitle is the same in both versions and within the limit; the check measures something correct, but this fault has nothing to do with the subtitle. The direct test, on the other hand, came out separating: the claim that the screenshot shows the new interface is tied to the content, not to a reference. #### Where the Reference Comes From This is the common weakness of equality checks. The check assumes the reference is correct, and nothing verifies the reference. If the build was faulty when the reference was taken, the equality check locks in the fault. If the record was taken from the correct build, the same check becomes separating; this will be seen in the second part of the measurement. Four situations are told apart. If the reference was taken from the correct build, the check separates. If the reference was taken from the faulty build, the check works backwards. If no reference was ever taken, the check fails on both versions and is noisy. If the reference is not updated between versions, even a correct new interface stays against the old record; this too is a kind of inverted behavior. Teams usually do not notice this weakness, because the person who takes the reference and the person who writes the check are different. The first uploads the image and moves on; the second finds a file to compare the image with and writes the equality. Nobody writes down where the reference came from. **DS17.** This lesson did not measure the search visibility of the store page, the download rate, or what the user reads on the page. How much the screenshot and the subtitle influence the user is not the subject of this lesson; the lesson measures only the separation of the checks that look at whether the page and the application agree. #### When Screenshots Are Updated Once the reference problem is solved, a second question appears: whenever the interface changes, the reference must change too. These two changes often happen in different hands and on different days. The person who changes the interface ships the application's code; the person who will update the page goes on with the old images without hearing about it. An equality check can catch this separation, but only if the reference has been updated as well. When a new interface is shipped without updating the reference, the check fails the new interface; the team should read this inverted behavior not as the interface being broken but as a reminder to renew the reference. This reading needs a separate discipline: a red check can have one of two causes, either the image is wrong or the reference is old. To say which, a direct test must stand next to the check. The direct test thus provides two-way evidence. If the equality check is red and the direct test is green, the problem is in the reference. If the equality check is green and the direct test is red, the problem is in the record the equality check rests on. If both are red, the problem is in the image itself. A single check cannot tell these three situations apart. #### The Place of the Subtitle The subtitle check came out blind in this lesson, and coming out blind was natural. Still, it does not need to be deleted. The subtitle limit looks at a real fault: a subtitle that exceeds the limit breaks the store's character rule. This fault did not exist in this lesson, so the check could not separate. Whether to keep a blind check depends on whether the fault it was written to guard against is still possible. It is possible in a team that writes the subtitle by hand; it is even more possible for a subtitle machine-translated from another language, because a translation often comes out longer than the original text. A blind check turns into noise only when the fault it guards against can no longer occur at all. This distinction also explains the difference between blind and noisy. A blind check is useless today and may be useful tomorrow. A noisy check gives the same answer today and tomorrow and has no tie left to any fault. The subtitle check is blind in this measurement; it is not noisy like the old size. #### The Store Console The listing information lives not in the application's repository but in the store's console. In the console there is a name, a subtitle, a description, and sets of screenshots for each language. Teams often keep a copy of this information in the repository as a configuration file; this way a change passes a review and is carried to the console in one step. ```yaml # listing.yaml — listing configuration carried to the console (SOURCE, not measured) language: en subtitle: Track all orders in one place screenshots: - file: screens/home.png - file: screens/orders.png reference: recorded_from: "" ``` This configuration was not measured; it is a yaml example, and nothing the lesson runs depends on it. What matters is the `recorded_from` field. Writing which build the reference was taken from makes the weakness of the equality check visible. When the field is left empty, the source of the reference is not on record and nobody can check it. **DS18.** The store console was described in this lesson with prose and a configuration example. The console's real fields, limits, and upload steps were not measured; neither was how everything in the console affects the page. #### The Bounding Measurement This time the reference is taken from the correct build. The separation returns with the same equality check and the same two versions. Next to it are put the case where the reference was never recorded and the subtitle check that stays the same on both versions. ```swift // main.swift — the same checks when the source of the reference changes let correctListing = listing(of: correctBuild) let faultyListing = listing(of: faultyBuild) let recordedFromCorrect = correctListing.screenshot let fromCorrect = contrast("reference-from-correct-build", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: recordedFromCorrect) }.describe() let missing = contrast("reference-never-recorded", correct: correctListing, faulty: faultyListing) { matchesReference($0, reference: "") }.describe() let withinLimit = contrast("subtitle-within-thirty", correct: correctListing, faulty: faultyListing) { subtitleFits($0, limit: subtitleLimit) }.describe() let seesNew = contrast("direct-new-interface-check", correct: correctListing, faulty: faultyListing) { showsNewInterface($0) }.describe() countSeparations([fromCorrect, missing, withinLimit, seesNew]) ``` ```text reference-from-correct-build | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating reference-never-recorded | correct=screen=screen-new subtitle=29/fail faulty=screen=screen-old subtitle=29/fail -> noisy subtitle-within-thirty | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/pass -> blind direct-new-interface-check | correct=screen=screen-new subtitle=29/pass faulty=screen=screen-old subtitle=29/fail -> separating separating: 2 blind: 1 noisy: 1 inverted: 0 ``` When the reference is taken from the correct build, the equality check becomes separating; the check's code did not change at all, only the source of the record changed. When there is no reference, the check fails on both versions and is noisy. The subtitle check is blind again; the page's subtitle is not the carrier of this fault. Since the direct test needs no reference, it is separating in both measurements. The limits are clear. This lesson did not measure how visible the page is in search results. The character limit of the subtitle is the lesson's rule, not the store's. The store console was described only with a configuration example and prose. What was measured is whether the code that checks the listing separates the fault; what the user sees on the page or understands from it is outside this measurement. #### Verifying the Reference An equality check against a reference becomes separating only if the reference is verified. Three habits provide this. The first is to write, when recording the reference, which build the record was taken from; the `recorded_from` field in the configuration example is for this. The second is for the person who takes the record to compare the screen with the application with their own eyes; the human approval covers only that record and is repeated every time the reference is renewed. The third is to put next to the equality check a direct test that needs no reference. If the direct test and the equality check diverge, one of the two rests on a wrong reference. This third habit is the cheapest inference of the lesson. When `showsNewInterface` and `matchesReference` were applied to the same two versions, they gave different answers: one separated, the other worked backwards. The divergence between the two checks is a warning even without a measurement. #### A Closing Thought At the start of the lesson it was said that the listing does not pass through the code's checks. The measurement showed that this has a price: the person who writes the check chooses what the check works against, and this choice may be wrong. In code checks, the correct version is written in the test input. In a listing, the correct version is an image, and once an image has been recorded nobody goes back and looks at it. #### Summary - The store page stands apart from the application's code; for this reason the code's checks do not cover the page. - An equality check against a recorded reference fails the correct version and passes the faulty version when the reference was taken from a faulty build; that is, it works backwards. - The check on whether the subtitle stays within the limit gives the same answer on both versions; it is blind on this fault. - When the reference is taken from the correct build, the same equality check becomes separating; when there is no reference, it becomes noisy. - Writing down the source of the reference and putting a direct test next to the equality check bring the separation back. - This lesson's separation: **inverted**. #### Next Step When the page and the submission are ready, one question remains: when the released version crashes for a user, how do you find out? In the next lesson, Crash Reporting and Analytics, we will measure whether a fault that crashes only in one language configuration is separated by a total counter and by a per-configuration counter. ### Crash Reporting and Analytics Source: https://academia.sh/en/courses/architecture-testing-and-release/crash-reporting-and-analytics A measurement showing that a total crash-free session check stays blind to a fault that crashes in only one language configuration, that a per-configuration check separates it, and the closing table of the course's nineteen lessons. When a released version crashes for a user, the way to find out is a crash report. The report carries the trace of the crashed session back to the team. Analytics, on the other hand, counts what the user does in sessions that do not crash. Both are counters, and counters mislead when they are read without knowing where a fault lives. The measuring question of the course is always the same: when the same check is applied to the correct version and to the faulty version, do the two answers differ. A check where the correct version passes and the faulty one fails is **separating**, one where both pass is **blind**, one where both fail is **noisy**, and one where the correct version fails and the faulty one passes is **inverted**. In the output they are written as `separating`, `blind`, `noisy`, and `inverted`. The core comes in unchanged, as in the earlier lessons. ```swift // Separation.swift — classifies whether a check tells the correct version from the faulty one struct Contrast { let label: String let correct: Output let faulty: Output let passesCorrect: Bool let passesFaulty: Bool var separation: String { switch (passesCorrect, passesFaulty) { case (true, false): return "separating" case (true, true): return "blind" case (false, false): return "noisy" case (false, true): return "inverted" } } func describe() -> String { let onCorrect = passesCorrect ? "pass" : "fail" let onFaulty = passesFaulty ? "pass" : "fail" print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)") return separation } } func contrast(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast { Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty)) } func countSeparations(_ results: [String]) { for name in ["separating", "blind", "noisy", "inverted"] { print("\(name): \(results.filter { $0 == name }.count)") } } ``` This lesson's fault lives in only one language configuration: the application crashes for users whose date format cannot be resolved, and works in the other configurations. The crash itself does not happen in the model; the model keeps the session and crash counts for each configuration and reads the numbers like a report. #### The Report Model `ConfigTally` is the session and crash count of one configuration. `Telemetry` carries three configurations and a single analytics event name; it summarizes itself as `sessions`, `crashes`, and `event`. `overallPasses` looks over all sessions at whether the crash-free ratio stays at the threshold, and `configPasses` looks at the same thing over a single configuration. `dashboardCount` returns the record the dashboard counts with a given event name. ```swift // Crash.swift — session and crash counts per configuration, a single event name struct ConfigTally { let config: String let sessions: Int let crashes: Int } struct Telemetry: CustomStringConvertible { let tallies: [ConfigTally] let eventName: String var sessions: Int { tallies.reduce(0) { $0 + $1.sessions } } var crashes: Int { tallies.reduce(0) { $0 + $1.crashes } } var description: String { "sessions=\(sessions) crashes=\(crashes) event=\(eventName)" } } func crashFree(sessions: Int, crashes: Int, threshold: Int) -> Bool { (sessions - crashes) * 100 >= sessions * threshold } func overallPasses(_ telemetry: Telemetry, threshold: Int) -> Bool { crashFree(sessions: telemetry.sessions, crashes: telemetry.crashes, threshold: threshold) } func configPasses(_ telemetry: Telemetry, config: String, threshold: Int) -> Bool { telemetry.tallies.filter { $0.config == config }.allSatisfy { crashFree(sessions: $0.sessions, crashes: $0.crashes, threshold: threshold) } } func dashboardCount(_ telemetry: Telemetry, name: String) -> Int { telemetry.eventName == name ? telemetry.sessions : 0 } let crashFreeThreshold = 90 let correctTelemetry = Telemetry( tallies: [ConfigTally(config: "en", sessions: 40, crashes: 0), ConfigTally(config: "tr", sessions: 30, crashes: 0), ConfigTally(config: "ja", sessions: 10, crashes: 0)], eventName: "order-completed") let faultyTelemetry = Telemetry( tallies: [ConfigTally(config: "en", sessions: 40, crashes: 0), ConfigTally(config: "tr", sessions: 30, crashes: 0), ConfigTally(config: "ja", sessions: 10, crashes: 4)], eventName: "order-completed") ``` **DS21.** The session and crash counts are the model's own numbers; they do not show the crash rate of a real application. The ninety percent threshold is also the lesson's rule, not the threshold of a platform or a service. #### Three Checks The first check looks at all sessions. The second looks only at the counter of the `ja` configuration. The third looks for the record the dashboard still counts under the old name, because the event was renamed from `order-placed` to `order-completed` in the version in question. The event is sent under the new name in both versions. ```swift // main.swift — total counter, configuration counter, old event name let overall = contrast("total-crash-free-sessions-at-threshold", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: crashFreeThreshold) }.describe() let japanese = contrast("ja-config-crash-free-at-threshold", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "ja", threshold: crashFreeThreshold) }.describe() let oldName = contrast("dashboard-has-old-event-name", correct: correctTelemetry, faulty: faultyTelemetry) { dashboardCount($0, name: "order-placed") > 0 }.describe() countSeparations([overall, japanese, oldName]) ``` ```text total-crash-free-sessions-at-threshold | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind ja-config-crash-free-at-threshold | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating dashboard-has-old-event-name | correct=sessions=80 crashes=0 event=order-completed/fail faulty=sessions=80 crashes=4 event=order-completed/fail -> noisy separating: 1 blind: 1 noisy: 1 inverted: 0 ``` The total check came out **blind**. In the faulty version four sessions crash, but within eighty sessions these four do not have to fall below the threshold. The check looks at something correct, only it does not look at the place where the fault lives. The per-configuration check separates the fault from the same data, because the four crashed sessions are concentrated in a small configuration. Within an average, the crash of a small group of users is lost. The dashboard check came out **noisy**. Because the event was renamed, the dashboard that counts under the old name sees zero in both versions and fails. A red dashboard does not show the crash; it shows that the dashboard does not know the new name. #### Reporting Configuration The crash reporting service and symbolication are not measured in this lesson. Both remain SOURCE: the configuration says that reporting is on and that symbols are uploaded. ```yaml # telemetry.yaml — reporting configuration (SOURCE, not measured) crash_reporting: enabled: true upload_symbols: true analytics: events: - order-completed ``` ```zsh # symbol upload step — SOURCE # 1. symbol files are extracted from the archive # 2. symbol files are uploaded to the reporting service # 3. the service makes the incoming crash trace readable with these files ``` A crash trace that arrives without symbols consists of address lines and does not say in which function the crash happened. For this reason the symbol upload step is a step that turns no step of the pipeline red, but that makes the report useless if it is forgotten. **DS22.** With what delay and at what rate a crash report reaches the team was not measured. The reporting service, symbolication, and the real behavior of the dashboard were not measured either. #### The Bounding Measurement The blindness of the total check depends on the threshold. Below, the same two versions are checked with changing values of the threshold, with the `tr` configuration that does not crash, and with the new event name. ```swift // main.swift — when the threshold and the configuration change let strictOverall = contrast("total-threshold-96", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: 96) }.describe() let turkish = contrast("tr-config", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "tr", threshold: crashFreeThreshold) }.describe() let newName = contrast("dashboard-has-new-event-name", correct: correctTelemetry, faulty: faultyTelemetry) { dashboardCount($0, name: "order-completed") > 0 }.describe() let strictJapanese = contrast("ja-config-threshold-100", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "ja", threshold: 100) }.describe() countSeparations([strictOverall, turkish, newName, strictJapanese]) ``` ```text total-threshold-96 | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating tr-config | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind dashboard-has-new-event-name | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind ja-config-threshold-100 | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating separating: 2 blind: 2 noisy: 0 inverted: 0 ``` When the threshold goes up from ninety to ninety-six, the total check becomes separating. The check that looks at the `tr` configuration, which does not crash, is blind again, because the fault does not live there. The dashboard that counts under the new name passes on both versions and is blind: the event the dashboard counts correctly knows nothing about the crash. The limits are clear. Separation is not the whole selection criterion of an architecture, a test suite, or a release process. Cost, the size of the team, and the sustainability of a structure over time were not measured in this course and were not reduced to a number; they are discussed in prose. **DS23.** The thresholds are the lesson's rule. Which threshold is right depends on how much crashing the team tolerates for its users, and this lesson does not say it. #### Course Wrap-Up The nineteen lessons of the course stand side by side in the table below, each with its own fault. The separation column was taken from the decision in each lesson's own `## Summary` section. | Lesson | Fault | Separation | Bounding measurement | |---|---|---|---| | Architectural Patterns | the discount is applied twice | all four (2/2/1/1) | when the percentage changes, the same check is blind, then separating | | The Coordinator Pattern | wrong screen after payment | blind | when the fault shifts to an intermediate step, only the last screen is blind | | Unidirectional Data Flow | the reducer produces a wrong state | separating | inverted when the recorded sample is taken from the faulty one | | Dependency Injection | the list is deleted on an empty response | noisy | blind when the stub always returns full | | Modularization | a feature module is tied to another feature | blind | rebuild budget: one separating, one blind | | Unit Tests | the fee is still charged at the threshold | separating | inverted when the expected value is copied from the faulty one | | UI Tests | the button does not send the order | noisy | noisy when looking before the screen settles | | Test Plans and Coverage | the discount is calculated wrongly | blind | an expectation written in a single language is noisy | | Profiling Tools | a cell does more work than needed | separating | by budget: noisy, separating, blind | | Memory Leak Detection | the closure holds the object strongly | separating | blind when the counter counts the screen | | Launch and Scrolling Performance | extra work at launch and in scrolling | inverted | a renewed baseline is blind | | Code Quality Tools | force unwrapping crashes on empty input | blind | a strict rule is noisy | | Signing and Provisioning Profiles | the package does not carry the notification entitlement | inverted | the same check gives three separations on three inputs | | Continuous Integration | the cart total skips the last item | noisy | when the step order changes, the separation changes | | Release Automation | the build number does not increase | separating | noisy or inverted depending on the source of the last number | | Beta Distribution | the flag is open outside the audience | blind | inverted for a newly invited tester | | The Store Review Process | the permission description is empty | noisy | the length item is blind on a description of spaces | | The Store Listing | the screenshot shows the old interface | inverted | separating with the correct reference | | Crash Reporting and Analytics | crashes in only one language | blind | separating when the threshold changes | The separation column of the table writes the opening lesson as all four and the other eighteen lessons with a single separation. The measurement below counts these eighteen decisions; the last decision comes from this lesson's own total check. Under it, the six checks of the core are put separately. ```swift // main.swift — the distribution of the eighteen lessons other than the opening one let ownLesson = contrast("wrap-up", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: crashFreeThreshold) }.separation let remaining = ["blind", "separating", "noisy", "blind", "separating", "noisy", "blind", "separating", "separating", "inverted", "blind", "inverted", "noisy", "separating", "blind", "noisy", "inverted", ownLesson] print("opening-excluded-eighteen") countSeparations(remaining) print("core-six-checks") countSeparations(["separating", "separating", "blind", "blind", "noisy", "inverted"]) ``` ```text opening-excluded-eighteen separating: 5 blind: 6 noisy: 4 inverted: 3 core-six-checks separating: 2 blind: 2 noisy: 1 inverted: 1 ``` The distribution of the eighteen lessons other than the opening one is five separating, six blind, four noisy, and three inverted. The 2/2/1/1 distribution of the core is a different thing: the core counts the six checks in the first lesson, while this distribution counts the separation of eighteen lessons. What should be read from the table is not the distribution itself, but that the same kind of check can fall into all four separations on different faults. Blind and inverted together come out in nine lessons; separating stays at five. M30 closes here: six courses were completed, namely the Swift language, the platform and tools, declarative interfaces, imperative interfaces and bridging, data and networking, and architecture and release. #### Summary - If the crash counter is read over the total of all sessions, a fault that crashes in only one language configuration is lost in the total. - The check that splits the crash counter per configuration separates the fault from the same data. - A dashboard that counts a renamed event under its old name sees zero in both versions and becomes noisy. - When the threshold of the total check is tightened, the same check becomes separating; being blind is a shared property of the check and the threshold, not of the check alone. - Across the course's nineteen lessons the separation spreads over four categories; no kind of check stays in a single category. - This lesson's separation: **blind**. #### Next Step This is the last lesson of the course. The question to ask when writing a new check does not change: apply the check to the correct version and to the faulty version, and read the two answers side by side.