Mock Objects

The previous lesson measured that a fixture dropped distinct verdict count from two to one by rebuilding shared state fresh for every test. It could do this because what it built was in its own hands: an in-memory object, a counter. Rebuilding was as cheap as an assignment, and it decided the outcome completely.

Not everything a test touches is like this. A resolver asks a package registry to learn accepted versions; the registry sits at the other end of a network, its content is not under our control, and it does not give the same answer to the same question every time. This lesson’s question: how many distinct outcomes does a test reaching such a source give, and what does that number drop to once a mock object takes its place?

The External Source’s Three Modes

The external source in the measurement is a registry client, modeled within the lesson; no product name appears, no real request gets made. The client has a single method: it returns a package’s published versions. The same call does three separate things across three separate modes.

Full. The registry returns every known version. Resolution gives the shared setup’s result.

Stale mirror. A copy of the registry has not been updated yet and does not carry the latest version. The request succeeds, the response is valid, but incomplete — and the incompleteness is not visible inside the response.

Outage. The registry cannot be reached, and the call raises an exception. Resolution produces no result at all.

All three modes get seen in a real setup, and none is a defect; each is part of the registry’s normal operation. From the test’s point of view, though, the three are one thing: the same test, three distinct outcomes.

The number three is the measurement’s choice, not the source’s limit. In a real registry, mode count is larger: a new version gets published and the list grows, a version gets pulled and the list shrinks, a request gets rate-limited and the response gets delayed. What the number means is not “the source does exactly three things,” it means whatever the source’s mode count is, the test’s outcome count equals it. If a test depends on an external source, every distinct behavior of the source is a distinct outcome of the test.

Where a Mock Can Be Put

A mock object being able to substitute for the real thing depends on a design condition: the real thing being supplied from outside. The resolve function in the measurement does not produce its version list internally; it asks the client it is given. For this reason, putting a different object in the client’s place amounts to nothing more than changing an argument.

Had the same function asked the registry directly — built and used the client internally — there would be no place left for a mock to go. The test would then have two options: ask the real registry and stay exposed to the three modes, or rebind the name at runtime and change the object the function sees. The second works, but ties the test to the function’s internal implementation: the moment the function starts building the client under a different name, the test fails without the behavior ever changing.

Supplying a dependency from outside was built as “testable design” in the Unit Testing and Test-Driven Development course; not repeated here. The only thing worth noting: a mock object’s ability to drop distinct outcome count depends on the code being written to accept it.

What a Mock Object Changes

A mock object (mock / stub) is an object put in place of a real dependency that answers the same calls. The test itself decides the answer; there is no network, no delay, no outage. This has a direct effect on the measurement: whatever the source’s mode count is, the mock object gives a single answer, and the test produces 1 distinct outcome.

The types of mock objects and which one fits which situation were built in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum; not repeated here, only referenced. The only thing measured in this lesson is how many distinct outcomes isolation makes singular.

The mock object in the measurement does two jobs. First, it fixes the answer: it returns the version dictionary it was given. Second, it records the calls it receives — how many times, and for which package it was called. This second job lets the test check not just the returned value, but the call made too; the measurement’s last line writes this record.

What Isolation Costs

A mock object frees the test from the source’s modes, but at a cost: the test no longer touches the real source. Nothing is left to say the fixed answer matches what the real registry actually gives. The mock object gets written once, while the registry keeps changing.

This is called contract drift, and the measurement’s second part counts it. When a version not present in the real registry gets added to the mock’s version list, the test still passes — because the test’s expectation is written alongside the mock, looking at the mock. The expectation and the mock verify each other, and the two drift away from reality together.

Drift has two directions, and both are silent. The mock can keep giving an answer the real thing no longer gives; or the real thing can start giving an answer the mock never anticipated. In the first case, the test stays green and the setup breaks; in the second, the test stays green too, and the setup follows a path the mock never tried.

The mechanism that makes this drift visible is called a contract test: the same calls get made to both the mock and the real source, and the answers get compared. A test like this breaks isolation — it touches the real source, and so is exposed to the source’s three modes again. For this reason, it does not run alongside the unit tests, it gets kept as a separate, sparsely run set.

The measurement’s assumptions:

The Measurement’s Assumptions

• TE19 — The registry client is modeled within the lesson; no real network request is made, no real package is downloaded, no product name is written.
• TE20 — The oracle is the shared setup’s registry: we know which package carries which versions because we wrote it. REGISTRY and MANIFEST are not changed.
• TE21 — The client has three modes: full, stale mirror, and outage. Stale mirror drops every package’s latest version; outage raises an exception.
• TE22 — Resolution runs with the “newest” strategy and gets the version list only from the client; it looks at no other source.
• TE23 — The outage result gets written as “no result” and counts as a distinct outcome; an error is an outcome too.
• TE24 — The mock object fixes the answer and records calls. The concept of mode does not exist in the mock; it gives the same answer in all three runs.
• TE25 — The drifted mock adds to the metrics package a version not present in the real registry. The other two packages stay untouched.
• TE26 — The test’s expectation is written alongside the mock in use, looking at the mock. This is the ordinary writing style in a suite that works with a mock object.
• TE27 — The contract test asks the same three calls of both sources and compares the answers package by package; it gives the count of packages where drift is found.
• TE28 — Duration is never measured. What gets counted is distinct outcomes, packages where drift is found, and calls made to the mock.

The Measurement

"""Mock object: an external source's three answers versus a single answer.
The client is modeled within the lesson.

Part 1 - the real client in three modes, the mock object in a single mode.
Part 2 - what happens when the mock drifts from the registry.
"""

REGISTRY = {"metrics": [(1, 0), (1, 1), (1, 2), (2, 0)],
            "report": [(0, 9), (1, 0), (1, 1)],
            "common": [(3, 0), (3, 1), (3, 2), (4, 0)]}
MANIFEST = {"metrics": ((1, 0), (2, 0)), "report": ((1, 0), (2, 0)),
            "common": ((3, 0), (4, 0))}


class Outage(Exception):
    """Signals that the external source cannot be reached."""


class RealClient:
    """External source: the same call gives a separate answer per mode."""

    def __init__(self, mode):
        self.mode = mode
        self.calls = 0

    def versions(self, package):
        self.calls += 1
        if self.mode == "outage":
            raise Outage(package)
        if self.mode == "stale mirror":
            return REGISTRY[package][:-1]
        return list(REGISTRY[package])


class MockClient:
    """Mock object that fixes the answer; also counts calls."""

    def __init__(self, source=None):
        self.source = source if source is not None else REGISTRY
        self.calls = 0
        self.requested = []

    def versions(self, package):
        self.calls += 1
        self.requested.append(package)
        return list(self.source[package])


def resolve(client, strategy="newest"):
    """Resolves the same manifest; gets the version list from the client."""
    choice = {}
    for package in sorted(MANIFEST):
        low, high = MANIFEST[package]
        candidates = [v for v in client.versions(package) if low <= v < high]
        choice[package] = candidates[-1] if strategy == "newest" else candidates[0]
    return choice


def render(choice):
    return " ".join(".".join(map(str, choice[p])) for p in ("metrics", "common", "report"))


def attempt(client):
    try:
        return render(resolve(client))
    except Outage:
        return "no result"


MODES = ("full", "stale mirror", "outage")

print(f"{'run':<22s} {'real client':>16s} {'mock object':>16s}")
real, mock = [], []
for mode in MODES:
    r, m = attempt(RealClient(mode)), attempt(MockClient())
    real.append(r)
    mock.append(m)
    print(f"{'mode: ' + mode:<22s} {r:>16s} {m:>16s}")
print(f"same test, real source gives {len(set(real))} distinct outcomes; "
      f"mock object gives {len(set(mock))}")

print()
DRIFTED = dict(REGISTRY, metrics=REGISTRY["metrics"] + [(1, 3)])
drifted_mock = MockClient(DRIFTED)
drifted = attempt(drifted_mock)          # the expectation is written from the mock
print(f"{'run':<31s} {'result':>12s} {'expected':>12s} {'verdict':>7s}")
for label, result, expected in (("mock, matches registry", attempt(MockClient()),
                                  attempt(MockClient())),
                                 ("mock, drifted from registry", drifted, drifted),
                                 ("real source, drifted expectation",
                                  attempt(RealClient("full")), drifted)):
    print(f"{label:<31s} {result:>12s} {expected:>12s} "
          f"{('passes' if result == expected else 'fails'):>7s}")
print(f"between the drifted mock and the real source there are "
      f"{len({drifted, attempt(RealClient('full'))})} distinct outcomes")

contract_drift = [p for p in sorted(MANIFEST)
                   if MockClient(DRIFTED).versions(p) != RealClient("full").versions(p)]
print(f"the contract test asks the same three calls of both sources and finds "
      f"drift in {len(contract_drift)} package(s): {', '.join(contract_drift)}")
print(f"the mock object received {drifted_mock.calls} calls, requested packages "
      f"{', '.join(drifted_mock.requested)}")

run                         real client      mock object
mode: full                  1.2 3.2 1.1      1.2 3.2 1.1
mode: stale mirror          1.2 3.2 1.0      1.2 3.2 1.1
mode: outage                  no result      1.2 3.2 1.1
same test, real source gives 3 distinct outcomes; mock object gives 1

run                                   result     expected verdict
mock, matches registry           1.2 3.2 1.1  1.2 3.2 1.1  passes
mock, drifted from registry      1.3 3.2 1.1  1.3 3.2 1.1  passes
real source, drifted expectation  1.2 3.2 1.1  1.3 3.2 1.1   fails
between the drifted mock and the real source there are 2 distinct outcomes
the contract test asks the same three calls of both sources and finds drift in 1 package(s): metrics
the mock object received 3 calls, requested packages common, metrics, report

From Three to One

The upper table gives what isolation pays off. With the real client, the same test produces 3 distinct outcomes: 1.2 3.2 1.1 in full mode, 1.2 3.2 1.0 in stale mirror, no result in outage. With the mock object, all three runs give 1.2 3.2 1.1 — 1 distinct outcome.

The middle row is the most dangerous of the three and deserves a separate look. Outage announces itself: it raises an exception, the run goes red, and the reason is plain. Stale mirror announces nothing. The request succeeds, the response is a valid list, resolution completes without a hitch, and only the report package’s version changes. The only thing that catches a difference like this is the expectation being written exactly — the test saying “a result came back” is not enough.

What the number dropping from three to one means: the mock object freed the test from the source’s modes. The test is now insensitive to the registry’s delay, its outages, and changes to its content — not a small thing; like order dependency, source mode is a factor that changes the verdict from outside the code.

The Correctness of a Singular Outcome

The lower table gives what the mock object cannot free the test from, and repeats the previous lesson’s warning on a different surface.

The first two rows give the same verdict: passes. Both mocks return a single answer, both produce 1 distinct outcome. But one carries the registry’s actual content, the other makes up a version that does not exist in the registry. The reason the test passes is the same in both: the expectation was written looking at the mock. Whatever the mock says, the expectation writes down, and the comparison holds on its own.

The third row exists only if something calls the real source. The drifted expectation fails once it gets asked of the real source, because 2 distinct outcomes stand between them. These two outcomes never meet in any run of the suite; the unit tests run with the mock, and the real source only enters during setup.

A contract test closes exactly this gap: it asks the same three calls of both sources and finds drift in 1 package — metrics. The drift being in a single package gives the measurement’s scale too: in two of the three packages, the mock is still correct; in one, it has drifted. Without a contract test, this ratio is never known, because knowing it requires asking the real source.

The last line shows the mock object’s second job. The mock received 3 calls and asked about all three packages. This record allows checking something beyond the returned value: did the resolver genuinely ask its registry for every package, or did it get some of them from a cache? The returned value does not carry the answer to this question, the call record does.

The call record has its own cost too. A test that checks call count ties itself to how the resolver works: a cache gets added one day, call count drops from three to one, and the test fails — while resolution keeps giving the same result. A test on the returned value is unaffected by a change like this. The record, for this reason, only gets checked when the call itself is part of the contract — when asking each package separately is a requirement, say.

Reading the three sections together: a mock object removes the source’s variability and drops distinct outcome count from three to one; against that, it says nothing about agreement with the source itself, since it never asks the real source. What isolation makes singular is the outcome, not correctness — and the only mechanism tying the singular outcome back to reality is a test that measures that tie separately.

Summary

• The same test reaching an external source gives 3 distinct outcomes across the source’s three modes: full answer, incomplete answer, and error. All three are part of the source’s normal operation.
• Stale mirror is the quietest of the three modes: the request succeeds, the response is valid, and only one package’s selected version changes.
• A mock object drops the count to 1 distinct outcome by fixing the answer and frees the test from the source’s modes; it also lets a test check beyond the returned value by recording the calls it receives.
• A mock drifted from the registry also gives 1 distinct outcome and passes the test, because the expectation is written looking at the mock; 2 distinct outcomes stand between the mock and the real source.
• A contract test asks the same calls of both sources and finds the drift in 1 package; because it breaks isolation, it does not run alongside the unit tests, it gets kept as a separate set.

Next Step

Everything tested up to here was written in a test file: the assertion, the fixture, the mock object. But a project’s correctness claims do not sit only there — they sit in documentation too. A function’s docstring saying “this call returns that” is a claim, and no run tests it; it falsifies silently when the code changes. The next lesson makes these claims testable, and counts exactly one thing: after the same change, how many stale claims does the same document turn into sound, and how many does it leave silent?