Testing Frameworks

Being correctly bound to a library doesn’t show that the code using it works correctly. Testing this itself carries a visibility problem: if a test sees the state left behind by another test, you can no longer tell which test actually failed. A small test framework solves this problem by running each test in its own copy.

The Test Table and Isolated Running

The core of a test framework is a table of function pointers, a single assertion macro, and a pass/fail count. When an assertion fails, the test function returns 0; if it finishes with nothing left to assert, it returns 1.

// framework.h — declaration of the test table, assertion macro, and shared state
#ifndef FRAMEWORK_H
#define FRAMEWORK_H

typedef int (*TestFunction)(void);

struct TestEntry {
    const char *name;
    TestFunction run;
};

#define ASSERT(condition) do { if (!(condition)) return 0; } while (0)

extern int shared_state;            /* to observe leakage between tests */
extern int mock_call_count;         /* how many times the mock dependency was called */
extern int fixture_setup_count;     /* how many times the fixture was set up before a test */
extern int fixture_teardown_count;  /* how many times the fixture was torn down after a test */

int test_set_state(void);
int test_state_is_zero(void);
int test_see_fixture_value(void);
int test_write_to_file(void);
int test_read_from_file(void);
int mock_dependency(int input);
int stub_dependency(int input);
int do_real_work(int input, int (*dependency)(int));
int fixture_setup(void);
int fixture_teardown(void);

#endif

// framework.c — test bodies, mock dependency, and two file-based tests
#include <stdio.h>
#include "framework.h"

int shared_state = 0;
int mock_call_count = 0;

int test_set_state(void) {
    shared_state = 9;
    return 1;
}

int test_state_is_zero(void) {
    ASSERT(shared_state == 0);
    return 1;
}

int test_see_fixture_value(void) {
    ASSERT(shared_state == 5);   /* fixture_setup must have brought this to 5 */
    return 1;
}

int test_write_to_file(void) {
    FILE *f = fopen("trace.dat", "w");
    if (!f) return 0;
    fputc('1', f);
    fclose(f);
    return 1;
}

int test_read_from_file(void) {
    FILE *f = fopen("trace.dat", "r");
    if (!f) return 0;
    int value = fgetc(f);
    fclose(f);
    return value == '1';
}

int mock_dependency(int input) {
    mock_call_count++;
    return input * 2;
}

int fixture_setup_count = 0;
int fixture_teardown_count = 0;

int stub_dependency(int input) {
    (void)input;              /* the stub's answer is fixed, it never reads the input at all */
    return 42;                 /* a predetermined answer that never changes */
}

int do_real_work(int input, int (*dependency)(int)) {
    return dependency(input) + 1;
}

int fixture_setup(void) {
    fixture_setup_count++;
    shared_state = 5;    /* the known starting state the fixture sets up */
    return 1;
}

int fixture_teardown(void) {
    fixture_teardown_count++;
    shared_state = 0;    /* leave the state clean for the next test */
    return 1;
}

PI5. The test table is shown with only two entries; the mechanism generalizes to any number of entries, because running them in order is nothing more than a loop over the table.

The helper that runs each test in its own child uses the forking contract set up in the previous module: the buffer is flushed, the child runs the test and reports the result through its exit status, and the parent only reads that status.

PI6. The fixture is shown here with a single pair of functions — one setup, one teardown; in a real framework the fixture can vary from test to test, but the requirement of being a pair (one teardown for every setup) doesn’t change.

If there is common preparation that has to run before and after the test itself, pulling that preparation out into a separate fixture (a setup/teardown pair) instead of writing it into the body of every test one by one keeps the same preparation from being written over and over, and, more importantly, guarantees that the teardown always runs regardless of the test’s result. A fixture’s contract is this: the setup runs exactly once before the test, the teardown runs exactly once after the test — whether the test passes or fails, even if the test returns early at an assertion. The fixture_setup/fixture_teardown pair below models this contract: fixture_setup brings shared_state to a known starting value (5) and increments the setup count; fixture_teardown resets the state and increments the teardown count. These two counts are the way to verify that the fixture is really set up once and torn down once for every test.

A stub serves a different purpose than a mock, and this lesson keeps the two apart. A stub (stub_dependency) returns a fixed, predetermined answer no matter what input it’s given, and never records that it was called at all — its only purpose is to stand in for the real dependency so the rest of the test can run. A mock (mock_dependency), on top of that, makes its own calling pattern observable; in this lesson it does that with a call count. The distinction comes down to this: a stub prepares the test’s input, a mock becomes part of the test’s verification. If a test only wants to see that do_real_work produces the right number, a stub is enough and simpler; if the test also wants to verify how many times the dependency was called, a mock is needed. Mixing the two up and using a mock everywhere piles an unnecessary observation burden onto tests that never ask about call counts; using a stub everywhere leaves blind the tests that need to verify calling behavior.

// measurement.c — isolated running, unisolated leakage, and the filesystem boundary in one run
#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include "framework.h"

static int run_isolated(TestFunction func) {
    fflush(stdout);   /* otherwise the child would also carry the parent's unflushed buffer */
    pid_t pid = fork();
    if (pid == 0) {
        _exit(func() ? 0 : 1);
    }
    int status = 0;
    waitpid(pid, &status, 0);
    return WIFEXITED(status) && WEXITSTATUS(status) == 0;
}

/* run_with_fixture: setup and teardown in the parent, the test itself in an isolated child */
static int run_with_fixture(TestFunction func) {
    fixture_setup();
    int result = run_isolated(func);
    fixture_teardown();
    return result;
}

int main(void) {
    printf("isolated set state: %s\n",
           run_isolated(test_set_state) ? "passed" : "failed");
    printf("isolated state is zero: %s\n",
           run_isolated(test_state_is_zero) ? "passed" : "failed");

    /* the same two tests, this time called directly in the same process, unisolated */
    test_set_state();
    printf("unisolated state is zero: %s\n",
           test_state_is_zero() ? "passed" : "failed");

    int result = do_real_work(3, mock_dependency);
    printf("mock dependency call count: %d\n", mock_call_count);
    printf("do_real_work result: %d\n", result);

    int stub_result = do_real_work(3, stub_dependency);
    printf("do_real_work result with stub: %d\n", stub_result);

    printf("fixtured see-value test: %s\n",
           run_with_fixture(test_see_fixture_value) ? "passed" : "failed");
    printf("fixture setup count: %d\n", fixture_setup_count);
    printf("fixture teardown count: %d\n", fixture_teardown_count);
    printf("shared_state in parent after fixture: %d\n", shared_state);

    printf("isolated write to file: %s\n",
           run_isolated(test_write_to_file) ? "passed" : "failed");
    printf("isolated read from file: %s\n",
           run_isolated(test_read_from_file) ? "passed" : "failed");
    return 0;
}

isolated set state: passed
isolated state is zero: passed
unisolated state is zero: failed
mock dependency call count: 1
do_real_work result: 7
do_real_work result with stub: 43
fixtured see-value test: passed
fixture setup count: 1
fixture teardown count: 1
shared_state in parent after fixture: 0
isolated write to file: passed
isolated read from file: passed

The first two lines show two tests that use the same shared_state variable running in separate children: test_set_state sets shared_state to 9 in its own child, but when that child exits, the change disappears along with it. When test_state_is_zero runs in its own child, it inherits a copy of the parent’s shared_state, still 0, and the assertion passes. The third line runs the same two tests unisolated: both run in a single process, on the real shared_state, and the 9 left behind by the first directly affects the second, so the assertion fails — this is exactly the leakage that process isolation prevents.

The dangerous side of this leak is that the result depends on the order in the test file. If the same two tests had been written with test_state_is_zero running first, the leak would never show up, and both tests would pass even in their unisolated form — because shared_state wouldn’t have been pulled to 9 yet. This is the most deceptive property of an unisolated test suite: the suite looks completely green in one order, and turns red in the same suite once the tests are reordered or a new test steps in between, even though no test’s own logic has changed. The source of the fault is not in any single test but in the hidden dependency between them — and that dependency stays invisible to code review, because the two tests never call each other, they only share the same variable. The real payoff of process isolation shows up here: because every test runs in its own child, the result stays the same no matter what order the tests are written in — order stops being a variable that affects the correctness of a test.

mock_dependency is a function pointer that stands in for the real dependency; do_real_work calls it without changing its own logic, and the call count is measured through mock_call_count. This count, just like shared_state, lives in a single process’s memory — if a test read it inside a forked child, the count that child sees would also be its own copy and wouldn’t make it back to the parent; the only thing that returns to the parent is whether the test passed.

What the mock doesn’t change here is do_real_work‘s own logic: the function still calls the pointer it’s given, only the body that pointer points to has been swapped from the real dependency for a body that keeps a count. This splits what the test measures in two — the correctness of do_real_work‘s own arithmetic on one side, and how many times and in what order the dependency it calls was invoked on the other. The second question can’t be tested with the real dependency, because there’s no way to observe the real dependency’s own call count from outside; the mock is the only tool that makes this observation possible.

The result obtained with stub_dependency is 43 — the 1 that do_real_work adds on top of the stub, which returns a fixed 42 no matter the input. This line says nothing about the call count, because the stub doesn’t count anything to begin with; it only shows that do_real_work’s own arithmetic correctly processes the value the dependency returns. Compared against the result of 7 obtained with the mock, the two tests test the same function for two different questions: one, “does it produce the right answer”, the other, “does it call the dependency the right number of times”.

PI7. The fixture_setup/fixture_teardown pair here runs in the parent process, immediately before and immediately after the call to run_isolated; the test itself still runs in its own child, the fixture is not moved into the child.

The run_with_fixture line confirms the fixture’s contract with three numbers. The setup count is 1, the teardown count is 1 — that is, both ran exactly once, no more and no less. The test itself passed the shared_state == 5 assertion, because fixture_setup set this value in the parent before the test’s child, and the child inherited this value through the fork. The last line shows that shared_state in the main process is back to 0 after fixture_teardown runs — the fixture’s teardown happens in the main process after the test’s child has already exited, because the setup and teardown never fork, they run directly in the parent. This shows why the fixture has to stay outside the isolated run: the fixture is common preparation repeated before and after every test, it doesn’t need to be moved into a separate child each time the way the test itself does — if it were moved, the effect of the setup (shared_state being 5) could never reach the test’s child at all, because every child inherits only the state at the moment of its own fork.

The Bounding Measurement

The last two lines show where process isolation stops. test_write_to_file writes a value to the trace.dat file in its own child, and the child exits; the memory state disappears along with it. But test_read_from_file, even though it runs in an entirely separate child, can open the same file and read the value the previous test wrote, and the assertion passes. Forking only copies memory; the filesystem is an object that lives outside the process and isn’t part of the area fork isolates. If two tests touch the same file, they can affect each other even though there is isolation between them at the memory level.

This boundary isn’t limited to files; every shared resource that lives outside the process carries the same problem. A network socket, a lock file, an environment variable kept by the operating system itself, or a shared-memory region — none of these are inside the memory area fork copies, so none of them get automatically isolated by forking. Knowing this distinction while writing a test suite has a practical consequence: running tests in separate processes doesn’t guarantee that the tests are independent of each other — it only guarantees independence at the memory level. Tests that touch a resource outside the process need an extra precaution: either the resource itself is reset before every test, or the tests are designed to never share the resource at all — for instance, by having every test generate its own temporary file name.

Another side of the filesystem boundary is whether the test gets cleaned up. The trace.dat file stays on disk after this run; on the next run, test_write_to_file writes over a file that already exists, which doesn’t change the result in this example because the function reopens and rewrites the file every time, but if a different test used the same name for another purpose, the content left over from the old run would be the first value the new test reads. Memory doesn’t have this problem, because every run starts its own process from scratch and the previous run’s memory is reclaimed by the operating system; the filesystem, on the other hand, keeps sitting there unchanged even after the process has ended. This is why the contract for tests that use files is one step broader than for tests that use memory: a clean starting point has to be guaranteed not only between tests but also between runs — the fixture can step in here too, by adding deleting the file to the teardown step.

This extended fixture could have been built by having fixture_teardown not only reset shared_state but also call remove("trace.dat"); this step wasn’t added here, because the goal was to show that the filesystem falls outside fork, not how file cleanup should be done. The general rule is this: a fixture only tears down the resource it itself set up — if fixture_setup touches shared_state, then fixture_teardown only reverts that; if there were another fixture that created a file, its teardown would likewise be limited to deleting its own file. Changing, at the teardown stage, a resource the setup never touched widens the fixture’s contract and makes it harder to track which test is responsible for which resource.

Summary

• A test framework consists of a table of function pointers, an assertion macro, and a pass/fail count; each entry is a row in the table.
• Running each test by forking it into a separate child keeps the change the test leaves in memory from reaching the next test — the class this lesson measures is copy.
• When the same tests run unisolated, in a single process, the first one’s change leaks into the second, and the run order can change the result.
• Turning a dependency into a mock through a function pointer measures the call count without changing the real dependency; a stub, without counting calls, prepares the test’s input by returning a fixed answer — the two serve different purposes.
• A fixture separates setup and teardown from the test itself and guarantees the teardown always runs regardless of the test’s result; it runs in the parent process, it isn’t moved into the test’s child.
• Process isolation cuts off memory, not the filesystem; two tests touching the same file aren’t separated from each other by forking. This boundary isn’t limited to files — every resource outside the process, such as a socket, a lock, or an environment variable, is equally outside isolation.

Next Step

When a test fails, you need to see what deviated from expectation, and at which step. The next lesson looks at how stopping a running program and stepping through it makes the intermediate values in its memory visible.