Testing Frameworks

The previous lesson showed that the format of the code does not change the run, and that style checking therefore stands outside the run. Does a test stand outside the run or inside it? The answer depends on what the test relies on. The smallest checking tool in PHP is the assert() function, and its behavior depends on one of the seven differences between the two configuration files: zend.assertions.

Three kinds of tests are separated by the size of the thing under test. A unit test checks a single function or class in isolation. An integration test checks several parts working together, for example the behavior of a function together with a file or a database. A behavior test checks the system from the outside, through the result the user sees. A test framework provides a common skeleton for writing and running these tests: it finds the tests, runs each one separately, collects the results, and reports which passed and which failed. This lesson names no test framework; instead it measures the mechanism test frameworks rely on: the assertion.

The fifth course of the Rust curriculum (M10/K05) searched for where assurance ends. Here the question is not the limit of assurance but the profile in which an assertion is compiled.

Testing with assert()

The measurement core is the same as in the previous lessons. side.php does not change at all.

<?php
// side.php — does this behavior reach production?
declare(strict_types=1);

// The two configuration files PHP ships (php.ini-development, php.ini-production) differ
// in seven directives; all seven are copied here verbatim. The last two production lines
// are comments in the shipped file: code cache on for the command line too, file checks off.
const PROFILES = [
    'development' => [
        'display_errors' => '1',
        'display_startup_errors' => '1',
        'error_reporting' => '32767',
        'mysqlnd.collect_memory_statistics' => '1',
        'zend.assertions' => '1',
        'zend.exception_ignore_args' => '0',
        'zend.exception_string_param_max_len' => '15',
    ],
    'production' => [
        'display_errors' => '0',
        'display_startup_errors' => '0',
        'error_reporting' => '24575',
        'mysqlnd.collect_memory_statistics' => '0',
        'zend.assertions' => '-1',
        'zend.exception_ignore_args' => '1',
        'zend.exception_string_param_max_len' => '0',
        'opcache.enable_cli' => '1',
        'opcache.validate_timestamps' => '0',
    ],
];

/** Runs the script in a separate process with no configuration file, only the profile's directives. */
function run(string $script, array $directives): bool {
    $command = ['php', '-n'];
    foreach ($directives as $name => $value) {
        array_push($command, '-d', "$name=$value");
    }
    $command[] = $script;
    $process = proc_open($command, [1 => ['pipe', 'w'], 2 => ['file', '/dev/null', 'w']], $pipes);
    $output = stream_get_contents($pipes[1]);
    fclose($pipes[1]);
    proc_close($process);
    $lines = array_values(array_filter(array_map('trim', explode("\n", $output))));
    return end($lines) === 'yes';
}

/** Derives the side from whether the behavior was observed in the two runs. */
function side(bool $inDevelopment, bool $inProduction): string {
    return match (true) {
        $inDevelopment && $inProduction => 'both sides',
        $inDevelopment => 'development only',
        $inProduction => 'production only',
        default => 'not in the run',
    };
}

function sideReport(string $name, string $script): string {
    $inDevelopment = run($script, PROFILES['development']);
    $inProduction = run($script, PROFILES['production']);
    $finalSide = side($inDevelopment, $inProduction);
    echo "$name → $finalSide\n    development ", $inDevelopment ? 'yes' : 'no',
        ' · production ', $inProduction ? 'yes' : 'no', "\n";
    return $finalSide;
}

The code under test is an addition function that is written wrong on purpose: add() subtracts instead of adding two numbers. The assertionCaught() function runs the assertion assert(add(2, 3) === 5) in a try block and returns true if an AssertionError is thrown. The tally() function increments a counter every time it is called: it exists to measure how a call that changes state behaves inside an assertion. The third helper, mustEqual(), does not use assert(); it compares two values with strict equality and throws a RuntimeException if they differ.

<?php
// testing.php — a deliberately wrong function and three checking helpers
declare(strict_types=1);

function add(int $a, int $b): int {
    return $a - $b;
}

function assertionCaught(): bool {
    try {
        assert(add(2, 3) === 5);
        return false;
    } catch (AssertionError) {
        return true;
    }
}

$counter = 0;

function tally(): bool {
    global $counter;
    $counter++;
    return true;
}

function mustEqual(mixed $expected, mixed $actual): void {
    if ($expected !== $actual) {
        throw new RuntimeException('expected and actual differ');
    }
}

The driver runs three probes in the two profiles. The first asks whether the failing assertion is caught, the second asks whether the call inside the assertion runs, and the third asks whether the same check written with mustEqual() catches the mistake.

<?php
// tests.php — three checks, two profiles
declare(strict_types=1);
require 'side.php';

$probes = [
    'a failing assertion is caught' => <<<'PHP'
        <?php
        require 'testing.php';
        echo assertionCaught() ? "yes\n" : "no\n";
        PHP,
    'a call inside the assertion runs' => <<<'PHP'
        <?php
        require 'testing.php';
        assert(tally());
        echo $counter === 1 ? "yes\n" : "no\n";
        PHP,
    'own comparison helper catches the mistake' => <<<'PHP'
        <?php
        require 'testing.php';
        try {
            mustEqual(5, add(2, 3));
            echo "no\n";
        } catch (RuntimeException) {
            echo "yes\n";
        }
        PHP,
];

foreach ($probes as $name => $code) {
    file_put_contents('probe.php', $code);
    sideReport($name, 'probe.php');
}

a failing assertion is caught → development only
    development yes · production no
a call inside the assertion runs → development only
    development yes · production no
own comparison helper catches the mistake → both sides
    development yes · production yes

The first two probes appear only in development. In the development profile (zend.assertions=1) a failing assertion throws an AssertionError and the check fails. In the production profile (zend.assertions=-1) PHP does not compile the assert() call at all; the assertion neither produces a value nor throws an exception. The same wrong function passes the check in the production profile. A test passing does not say that the code is correct; it also says in which profile the test ran.

The second probe shows the detail of this. When the assertion is not compiled, the expression inside it is not evaluated either. The call inside assert(tally()) increments the counter in development and does not in production. The script below prints the two values directly.

# counter.sh — in which profile does the call inside the assertion run
cat > counter.php <<'PHP'
<?php
require 'testing.php';
assert(tally());
echo "counter: $counter\n";
PHP
echo "zend.assertions=1:"
php -n -d zend.assertions=1 counter.php
echo "zend.assertions=-1:"
php -n -d zend.assertions=-1 counter.php

zend.assertions=1:
counter: 1
zend.assertions=-1:
counter: 0

This is the known trap of using an assertion as a checking tool: if a call that makes something happen is put inside the assertion, the work is done in only one profile. The expression inside an assertion should only read a condition, not change state.

Your Own Comparison Helper

The third probe gives a different result: mustEqual() catches the mistake in both profiles. The helper does not call assert() and does not rely on AssertionError; it evaluates its own condition and throws its own exception. For this reason the zend.assertions setting does not affect it. The comparison methods of test frameworks are not built on assert() for the same reason: a test that passes silently in the production profile means that the test checks nothing.

The cost of this is small: a two-line helper does the same job as assert() and is independent of the profile. One distinction remains. assert() is an assertion written inside the code that questions an assumption while the program runs; it gives extra assurance in development and its cost is reduced to zero in production. A test, by contrast, is a separate program that stands outside the run and calls the code, and it should produce the same result in every profile.

What a Test Framework Does

The driver of this lesson is a small test framework. It gathers the probes in a list, writes each one to a separate file, runs each file in a separate process, and reports the result in a single line. Three jobs are also the core of real test frameworks. Finding the tests builds the list. Isolation prevents the state one test leaves behind from affecting another: the counter in the second probe is independent of the counter in the first, because every probe is a new process. Reporting collects the passed and the failed in one place.

Isolation has a cost. A separate process also makes the settings the test sees separate. In this lesson every probe runs with no configuration file, only the profile’s directives, because the setting a test runs with determines its result. The same test suite can pass with the development settings and fail with the production settings, or the other way around. The first probe shows exactly this: a check that relies on an assertion finds the mistake only when zend.assertions=1.

Two consequences follow in practice. Tests should also be run in a profile close to the configuration that will go to production; otherwise a behavior that changes in production reaches the live system without ever being checked. On the other hand, the assertions themselves should keep running in development, because their job is to check an assumption inside the code. Separating the two jobs, leaving the check to a comparison helper that is independent of the profile and the assumption to assert(), is therefore a consistent distinction.

Unit and Integration Testing

Going back to the difference between a unit test and an integration test, we measure it with a small example. A pure function, withTax(), depends only on the value it receives: the same input gives the same output in every run. A function that writes to a file, addNote(), depends on state outside it: a file left over from the previous run changes the result.

The driver runs four probes. The first checks the pure function. The second checks the function that writes to the file, without cleaning the file. The third writes the same check, deleting the file at the start. The fourth asks whether the failure message is the same in the two profiles. The driver deletes the file before every probe; the two runs inside a probe (development, then production) share the same file.

<?php
// unit.php — a pure function and a function that writes to a file
declare(strict_types=1);
require 'side.php';

$probes = [
    'unit: pure function' => <<<'PHP'
        <?php
        require 'testing.php';
        function withTax(int $amount): int {
            return intdiv($amount * 118, 100);
        }
        try {
            mustEqual(118, withTax(100));
            echo "yes\n";
        } catch (RuntimeException) {
            echo "no\n";
        }
        PHP,
    'integration: test that does not clean the file' => <<<'PHP'
        <?php
        function addNote(string $file, string $line): void {
            file_put_contents($file, $line . "\n", FILE_APPEND);
        }
        addNote('note.txt', 'first');
        echo count(file('note.txt')) === 1 ? "yes\n" : "no\n";
        PHP,
    'integration: test that cleans first' => <<<'PHP'
        <?php
        function addNote(string $file, string $line): void {
            file_put_contents($file, $line . "\n", FILE_APPEND);
        }
        @unlink('note.txt');
        addNote('note.txt', 'first');
        echo count(file('note.txt')) === 1 ? "yes\n" : "no\n";
        PHP,
    'the failure message is the same' => <<<'PHP'
        <?php
        require 'testing.php';
        try {
            mustEqual(5, add(2, 3));
            echo "no\n";
        } catch (RuntimeException $e) {
            echo $e->getMessage() === 'expected and actual differ' ? "yes\n" : "no\n";
        }
        PHP,
];

foreach ($probes as $name => $code) {
    @unlink('note.txt');
    file_put_contents('probe.php', $code);
    sideReport($name, 'probe.php');
}

unit: pure function → both sides
    development yes · production yes
integration: test that does not clean the file → development only
    development yes · production no
integration: test that cleans first → both sides
    development yes · production yes
the failure message is the same → both sides
    development yes · production yes

The check of the pure function passes on both sides: it depends on nothing but its input and output. The integration check that does not clean up, however, passes only in development. This is not a profile difference; it is a trace of the driver’s run order. The development run creates the file for the first time and sees one line; the production run appends a second line to the same file and sees two lines. If the run order were reversed, the result would be reversed too. The check that cleans at the start therefore passes on both sides.

This also gives the definition of an integration test: a test that uses state outside itself must set up that state and clean it up by itself. Otherwise the test result measures not the correctness of the code but the order in which the tests ran. The result shows why isolation is a basic job of a test framework. The fourth probe looks at another distinction: mustEqual() produces the failure message independently of the settings, so the test report carries the same text in the two profiles, and the cause of a failure can be read without looking at the profile.

The Bounding Measurement

zend.assertions takes three values. 1 compiles and runs the assertion; -1 does not compile it; 0 compiles it but does not run it. Throughout the lesson the production profile used -1. The question to ask is this: does the value 0 silence the assertion as well, and is there a behavioral difference between the two values? The core profile is not changed; the driver calls the run() function in side.php with a copy of the production profile and changes only the zend.assertions value.

<?php
// limit.php — the difference between 0 and -1 for zend.assertions
declare(strict_types=1);
require 'side.php';

$probes = [
    'assertion caught' => <<<'PHP'
        <?php
        require 'testing.php';
        echo assertionCaught() ? "yes\n" : "no\n";
        PHP,
    'assertion caught after enabling at run time' => <<<'PHP'
        <?php
        require 'testing.php';
        ini_set('zend.assertions', '1');
        echo assertionCaught() ? "yes\n" : "no\n";
        PHP,
];

foreach (['0', '-1'] as $value) {
    $profile = ['zend.assertions' => $value] + PROFILES['production'];
    foreach ($probes as $name => $code) {
        file_put_contents('probe.php', $code);
        echo "zend.assertions=$value · $name: ", run('probe.php', $profile) ? 'yes' : 'no', "\n";
    }
}

zend.assertions=0 · assertion caught: no
zend.assertions=0 · assertion caught after enabling at run time: yes
zend.assertions=-1 · assertion caught: no
zend.assertions=-1 · assertion caught after enabling at run time: no

With the value 0 as well, the failing assertion is not caught: even if the assertion is compiled, it is not run. The difference between the two values appears at run time. With 0 the setting can be turned to 1 with ini_set() and the assertion catches from that point on; with -1 the same call changes nothing, because the assertion was never compiled. This shows that the boundary that turns assertions on and off lies not only in the value of the setting but also in the setting in force at the moment the file is compiled. The production profile choosing -1 is therefore a hardening decision: the way to turn assertions back on within a request is closed.

Summary

• A test framework is a skeleton that finds, runs, and reports tests; unit, integration, and behavior tests are separated by the size of the thing under test.
• An assert()-based check catches a failing assertion only in development; in the production profile (zend.assertions=-1) the assertion is not compiled and wrong code passes the check.
• A call that changes state inside an assertion does not run in production: the counter stays at 1 in development and at 0 in production.
• A comparison helper of your own that does not rely on AssertionError works on both sides; for this reason a test relies on such a helper, not on assert().
• zend.assertions=0 compiles the assertion but does not run it, and it can be turned to 1 at run time. With -1 this way is closed.

Next Step

In this lesson the setting that makes an assertion visible in development silently removed it in production. The same direction applies to the information we see while debugging: the argument values in a trace, the warning printed to the screen, and the variables on the line where a debugger stops. The next lesson asks whether this information seen in development reaches production, and measures where a value is shown or hidden in the exception trace.