Cloud Sync

The first five lessons of this topic dealt with the data on a single device. If the user has a phone, a tablet, and a computer, the same record stands in several places. Cloud sync brings these copies closer to each other through a server: each device sends the changes it makes, the server merges them, and the other devices pull the result. No new question arrives; the existing question is asked in a new place: if this operation is repeated or reordered, does the result change? The difference is this: the two devices do not wait for each other, so the order in which the writes reach the server may not be in the app’s hands.

// Replay.swift — classifies an operation's sensitivity to repetition and order
struct Trial<State: Equatable & CustomStringConvertible> {
    let label: String
    let once: State
    let twice: State
    let forward: State
    let backward: State

    var sensitivity: String {
        switch (once != twice, forward != backward) {
        case (false, false): return "robust"
        case (true, false): return "repeat-sensitive"
        case (false, true): return "order-sensitive"
        case (true, true): return "doubly-sensitive"
        }
    }

    func report() -> String {
        print("\(label) | once=\(once) twice=\(twice) ab=\(forward) ba=\(backward) -> \(sensitivity)")
        return sensitivity
    }
}

func replay<State>(_ label: String, from start: State, _ a: (State) -> State, then b: (State) -> State) -> Trial<State> {
    Trial(label: label, once: a(start), twice: a(a(start)), forward: b(a(start)), backward: a(b(start)))
}

func countSensitivities(_ results: [String]) {
    for name in ["robust", "repeat-sensitive", "order-sensitive", "doubly-sensitive"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

once is the operation applied to the starting state one time, twice is two times; ab is a first and then b, and ba is the reverse order. The model is the pure Swift counterpart of a single record on the server. It does not build the network, the delay, or the real delivery order; it only builds the result a write leaves in the record.

Conflict and Rule

If two devices write different values to the same field, a conflict arises on the server. A rule is needed: which value stays? The most common rule is that the last writer wins; the value that reaches the server later takes the place of the earlier one. Another rule is to put a stamp on every write and keep the one with the highest stamp. A third way is to merge the value itself: if two devices added different tags to a set of tags, the result is the union of the two.

import CloudKit

let recordID = CKRecord.ID(recordName: "k1")
let record = CKRecord(recordType: "Note", recordID: recordID)
record["theme"] = "dark"

let database = CKContainer.default().privateCloudDatabase
database.save(record) { saved, error in
    if let error = error as? CKError, error.code == .serverRecordChanged {
        // the record on the server has changed in the meantime; the app applies its own rule
    }
}

This fragment is SOURCE; it needs a cloud account and a network, so it does not run here. The serverRecordChanged error reports that the server’s version of the record that is being saved has changed in the meantime. Resolving the conflict is left to the app: the version on the server returns with the error, and the app decides which value stays. The measurement of this lesson compares three forms of this decision. The error itself and the real delivery order are not measured.

The Model of the Record

The model keeps the scalar fields and the set fields in a single record. Each scalar value carries a stamp. The server’s rule is a setting: an incoming write either always wins, or wins only if its stamp is not smaller than the current stamp.

// SyncModel.swift — a pure Swift model of a record on the server
struct SyncBox: Equatable, CustomStringConvertible {
    enum Rule {
        case lastArrival
        case highestStamp
    }

    struct Stamped {
        let value: String
        let stamp: Int
    }

    private(set) var scalars: [String: Stamped] = [:]
    private(set) var sets: [String: Set<String>] = [:]
    let rule: Rule

    init(rule: Rule = .lastArrival) {
        self.rule = rule
    }

    var description: String {
        let scalarText = scalars.keys.sorted().map { "\($0)=\(scalars[$0]?.value ?? "-")@\(scalars[$0]?.stamp ?? 0)" }
        let setText = sets.keys.sorted().map { "\($0)={\((sets[$0] ?? []).sorted().joined(separator: ","))}" }
        let all = scalarText + setText
        return all.isEmpty ? "-" : all.joined(separator: " ")
    }

    static func == (lhs: SyncBox, rhs: SyncBox) -> Bool {
        lhs.description == rhs.description
    }

    mutating func write(_ value: String, stamp: Int, field: String) {
        if rule == .highestStamp, let current = scalars[field], current.stamp > stamp {
            return
        }
        scalars[field] = Stamped(value: value, stamp: stamp)
    }

    mutating func merge(_ items: [String], field: String) {
        sets[field, default: []].formUnion(items)
    }
}

func writing(_ value: String, stamp: Int, field: String) -> (SyncBox) -> SyncBox {
    { record in
        var next = record
        next.write(value, stamp: stamp, field: field)
        return next
    }
}

func merging(_ items: [String], field: String) -> (SyncBox) -> SyncBox {
    { record in
        var next = record
        next.merge(items, field: field)
        return next
    }
}

The model builds the two devices as two separate operations; ab is the case where the first device’s write reaches the server first, and ba is the case where the second one’s does.

PS14. The model assumes that the server applies the rule consistently: the rule is a setting and works on every write exactly. The real delivery order, the delay, and serverRecordChanged were not measured.

PS15. The stamps are numbers given to the model; no clock is read or stamped. The drift of real device clocks was not built.

PS16. The model assumes that every write reaches the server. Lost, delayed, and repeated delivery were not built.

Measuring Two Operations

Two operations are built. The first is two devices writing different values to the same field. The second is two devices adding different tags to the same set of tags.

// main.swift — two devices write the same record
let twoDevices = replay("two-devices-same-field", from: SyncBox(), writing("dark", stamp: 1, field: "theme"), then: writing("light", stamp: 2, field: "theme")).report()
let tags = replay("set-union", from: SyncBox(), merging(["a1"], field: "tags"), then: merging(["b2"], field: "tags")).report()

countSensitivities([twoDevices, tags])

two-devices-same-field | once=theme=dark@1 twice=theme=dark@1 ab=theme=light@2 ba=theme=dark@1 -> order-sensitive
set-union | once=tags={a1} twice=tags={a1} ab=tags={a1,b2} ba=tags={a1,b2} -> robust
robust: 1
repeat-sensitive: 0
order-sensitive: 1
doubly-sensitive: 0

In the first line, repeating the same write changes nothing: once and twice are the same. But when the values of the two devices reach the server in a different order, the result changes: ab is light, ba is dark. The last writer wins, and the arrival order decides the winner, that is, the network and not the app. The class is order-sensitive. This is the end of the chain that has been building up since the beginning of the topic: the preference of two screens, the repeated keychain item, the append to a file, the row with a primary key, the object created twice, and now the value written by two devices. At every step a local problem showed up again in another place in the next one.

In the second line, two devices add different tags to the same set. Repetition does not change the set, and the order does not change the result either: both orders give {a1,b2}. A set union makes the order irrelevant, because no value takes the place of the other. The class is robust. This is why a per-field merge is the most durable way of resolving conflicts; its cost is that every field must have a structure that can be merged. A theme has no union.

The count is one robust, one order-sensitive.

What to Merge

The measurement returns to the app’s data model as a question: can this field be merged? Tags, read marks, and the list of attached files merge. A theme, a name, a date do not merge; a winner has to be chosen for these. Using a stamp when choosing the winner reduces the dependence on order but does not remove it. The bounding measurement below shows this.

Common Mistakes

• Assuming that the last writer will win and not noticing the data loss. The losing value is silently deleted.
• Writing a field that can be merged as a scalar. One of the tags added by the two devices is lost.
• Trusting the device clock and taking the stamp from there. Clocks drift.
• Ignoring the conflict error. The record stays unsaved and the user does not notice.
• Running sync immediately on every change. The calls are repeated and the order becomes even less controlled.
• Binding a delete made on one device to the same rule as a write on another. A delete is a separate operation and is thought about separately.

The Bounding Measurement

The order sensitivity of the first line rested on the server’s rule that lets the latest arrival win. We measure the same two writes with a rule that keeps the one with the larger stamp, and we also try once more with equal stamps.

// main.swift — server rule
let arrival = replay("rule-arrival-order", from: SyncBox(rule: .lastArrival), writing("dark", stamp: 1, field: "theme"), then: writing("light", stamp: 2, field: "theme")).report()
let stamped = replay("rule-stamp", from: SyncBox(rule: .highestStamp), writing("dark", stamp: 1, field: "theme"), then: writing("light", stamp: 2, field: "theme")).report()
let tie = replay("rule-stamp-tie", from: SyncBox(rule: .highestStamp), writing("dark", stamp: 1, field: "theme"), then: writing("light", stamp: 1, field: "theme")).report()

countSensitivities([arrival, stamped, tie])

rule-arrival-order | once=theme=dark@1 twice=theme=dark@1 ab=theme=light@2 ba=theme=dark@1 -> order-sensitive
rule-stamp | once=theme=dark@1 twice=theme=dark@1 ab=theme=light@2 ba=theme=light@2 -> robust
rule-stamp-tie | once=theme=dark@1 twice=theme=dark@1 ab=theme=light@1 ba=theme=dark@1 -> order-sensitive
robust: 1
repeat-sensitive: 0
order-sensitive: 2
doubly-sensitive: 0

The same two writes fell into two separate classes depending on the rule. It is order-sensitive under the arrival order rule; robust under the highest stamp rule, because the write with the larger stamp wins whichever order it arrives in. When the stamps were equal, the order came back: ab is light, ba is dark. The stamp did not remove the order; it only narrowed the case in which the order comes into play.

The limit is PS14. The model assumes that the server applies the rule correctly and consistently. In a real service, where and how the rule is applied, when the error is returned, and in what order the writes arrive are outside this model. What is measured is the result of the rule; not that the rule really works.

Where It Shows Up in a Real App

Think of a notes app. The user changes the title of a note on the phone, and at the same time turns the title of the same note into something else on the tablet. Both devices are offline and connect a few minutes later. Whichever reaches the server first, the second to arrive takes the place of the first; the title that remains is decided by the delay of the network, not by the user. The two-devices-same-field line in the measurement is the model of this situation.

Tags are different. The phone adds a tag, the tablet adds another tag. The field is kept as a set, and the server records the union of the two sets. Which device reached first no longer matters: in the set-union line both orders gave the same result. This difference is the first question to ask when building the data model of an app: can this field be merged, or must a winner be chosen?

Deletion brings a separate problem. One device deletes the record, the other updates the same record. Will the server count the record as deleted or as updated? The model of this lesson does not build deletion; but in a real sync design this decision is as important as the scalar conflict rule and must be made explicitly.

A Wrong Setup and Its Fix

The most common wrong setup is for every device to stamp all the records with its local clock and let the newest stamp win. Device clocks drift; the user can change the clock by hand, and when the clock of one device is wrongly ahead, all its writes crush the correct writes made later. As we showed in the bounding measurement, the stamp rule reduces the dependence on order but the order returns when the stamps are equal. When real clocks are involved, this equality is not rare.

The fix has two layers. The first is to take the stamp not from the device clock but from an increasing version number given by the server; that way the stamps are comparable and the drift has no effect. The second is to move every field that can be, to a structure that can be merged. A set of tags, a set of read marks, and a list of attached files are of this class. No conflict rule is needed for merged fields.

The second mistake is ignoring the conflict error. If the app stays silent when the server reports that the record has changed in the meantime, the user’s change has never been saved, and the user does not notice. The correct setup is to take the server version that returns with the error, apply the app’s own rule, and send the result again. PS15 records that the model takes the stamps as numbers given from outside; real clock drift is not in the model.

Reading the SOURCE Fragment

The unnamed fragment at the start of the lesson creates a record, writes a value to a field of it, and saves it to a private database. The completion of the save call reports the result of the operation as an error or a saved record. One of the error types says that the server’s version of the record that is being saved has changed in the meantime. This error is not an exception but a normal outcome: in an app with several devices, a conflict happens sooner or later.

Our model does not build this error or the real delivery order; PS14 assumes that the server applies the rule consistently, and PS16 that every write reaches the server. On a real network, writes are lost, delayed, and repeated. A repeated write being applied a second time on the server brings the repeat sensitivity question back onto the agenda, and it will be taken up in the next lesson, in the context of notifications arriving twice.

From Measurement to Design

What the measurement says in a sync design is that a decision has to be made field by field. For each field one of three options is chosen: turning it into a structure that can be merged, choosing a winner with a stamp, or asking the user about the conflict. Tags, marks, and lists fall into the first group; they came out robust in the measurement. Fields such as a name, a theme, and a date are in the second group and at best narrow the dependence on order. The third group is rare, but for some data it is the only honest option; the user choosing between two different texts is better than silently deleting one.

The rule must always be written down. “The last writer wins” is a rule, and a correct one, as long as it was chosen consciously. What is dangerous is that the rule is never thought about and the server’s default becomes the product’s rule.

The chain of this topic closes here. In the simple preference, the writing of two screens depended on the order; in the keychain, the insertion on repetition; in the file, on both repetition and order; in the table, on the key; in the object framework, on the identity; and in sync, on the nature of the field. In every layer the same two questions were asked: what happens if it is repeated and if its order changes. The answers changed, the question did not. Each layer of persistence does not bring a new flaw; it houses the same two flaws in a new place.

One last note: the question of how a sync success is shown to the user is also a part of the design. The change shows up locally at once and goes to the server later. If the user does not know that the value has not been sent yet and closes the device, the change stays in a waiting queue. The pending operations have to be stored and sent again when the connection returns; sending again is also repetition. So the pending writes of sync must be robust against repetition.

Summary

• Cloud sync lets several devices write the same record; a rule decides which value stays in a conflict.
• The different values written by two devices to the same field depend on the arrival order: ab is light, ba is dark; the class is order-sensitive.
• A per-field set union is robust: both orders give {a1,b2}.
• The highest stamp rule makes the same pair of writes robust, but the order returns when the stamps are equal.
• The server rule, the real delivery order, and the conflict error were not measured; they stayed as PS14.
• This lesson’s sensitivity: order-sensitive.

Next Step

Sync described how a value travels between devices. In the next lesson, Push Notifications, we move on to the path on which the server informs the app on its own: next to a device token, a permission, and a badge, we will measure whether the same notification arriving twice changes the result.