Testing and Property-Based Testing

The Dependencies and Feature Flags lesson measured a body the compiler never sees at all — there, the guarantee never even started. But the previous lesson’s closing question pointed in a different direction: a body like running_body enters compilation, passes type checking, has its names resolved correctly, and can still produce a wrong result. The compiler checks that a body exists and that its type is consistent; it never checks that the body is correct — correctness is a property of behavior, not of type, and type checking knows nothing about behavior.

One way to close this gap is to test by running: run a body against specific inputs and check whether the result carries an expected property. This is different from writing an example test with a single fixed input — an example test only shows that something is correct for that one input, and says nothing about any input that was not chosen. This lesson does this not with a single fixed input but with a large number of inputs produced by a fixed-seed generator — that is property-based testing’s core idea: instead of a single example, test the same property across many generated examples. That the inputs come from a deterministic generator is not a side condition, it is one of the necessary requirements for the observation this lesson establishes: if a generator relying on the system clock or the operating system’s source of randomness were used, the same test would run with different inputs on every machine and every run, and the number of inputs it held for would change from run to run — this lesson rules out that variability from the start. The test itself is not without checking: what decides which property is tested, which range the inputs come from, and how many inputs are enough is not the compiler, it is the party who writes the test. The course’s shared question — where does the compiler’s guarantee end here — finds its answer in this lesson: where type checking ends, behavior checking begins, but the party who writes it also draws this checking’s scope.

Testing a Property by Running

A property-based test is built from three parts: a source that produces random inputs, a condition to test on each of those inputs, and a loop that counts how many inputs the condition holds for. Rust’s standard library has no random-number generator, and this curriculum forbids third-party crates, so the generator is hand-written. PL5. The only thing that needs to be random is the distribution of inputs — the test itself must stay deterministic, or the same run would give a different result on different machines. The generator below is a linear congruential generator that starts from a fixed seed and produces the same sequence on every call: given the same seed, the output sequence is bit-for-bit identical on every run.

The generator’s state is a single u64 number, updated on every call by multiplying and adding two fixed numbers; wrapping_mul and wrapping_add silently wrap any overflow in this operation, so it never halts with a panic. length narrows this raw number to an upper bound (deciding how many elements an array carries), and element produces an integer between -500 and 499. vector combines the two: it first draws a length, then produces that many elements. The constants used come from a known family of linear congruential generators, and the generator’s good distribution is not this lesson’s subject — the only property sought here is that it produces the same sequence on every run with the same seed; this property gave bit-for-bit identical numbers across every run above.

// linear_generator.rs — fixed-seed, deterministic pseudo-random generator
pub struct LinearGenerator {
    state: u64,
}

impl LinearGenerator {
    pub fn new(seed: u64) -> Self {
        LinearGenerator { state: seed }
    }

    pub fn next(&mut self) -> u64 {
        self.state = self
            .state
            .wrapping_mul(6364136223846793005)
            .wrapping_add(1442695040888963407);
        self.state
    }

    pub fn length(&mut self, upper_bound: usize) -> usize {
        (self.next() % (upper_bound as u64 + 1)) as usize
    }

    pub fn element(&mut self) -> i32 {
        (self.next() % 1000) as i32 - 500
    }

    pub fn vector(&mut self, max_len: usize) -> Vec<i32> {
        let n = self.length(max_len);
        (0..n).map(|_| self.element()).collect()
    }
}

The property to be tested is a classic example: reversing an array twice should return the original array. This property is built using Vec’s reverse method and is tested against every generated input. Two hundred inputs are produced, each carrying at most twenty elements; the count of inputs the property holds for is tallied.

// property_test.rs — does applying reversal twice give identity
mod linear_generator;
use linear_generator::LinearGenerator;

fn reverse_reverse_identity(v: &[i32]) -> bool {
    let mut copy = v.to_vec();
    copy.reverse();
    copy.reverse();
    copy == v
}

fn main() {
    let mut generator = LinearGenerator::new(42);
    let total_inputs = 200;
    let mut held = 0;

    for _ in 0..total_inputs {
        let input = generator.vector(20);
        if reverse_reverse_identity(&input) {
            held += 1;
        }
    }

    println!("inputs tested: {}", total_inputs);
    println!("inputs the property held for: {}", held);
}

inputs tested: 200
inputs the property held for: 200

When the generator starts with seed 42, the first array it produces carries fourteen elements, the second eleven, the third eighteen — lengths, just like elements, change deterministically with the fixed seed. Only the total and held counts are printed in this run, but each of the two hundred arrays the generator produces arrives this way, with different lengths and different elements; the upper bound given to vector (20) states that an array can carry at most twenty elements, and at least zero (an empty array). The smallest length was left at zero because an empty array is also a valid input — applying reversal twice should also give identity on an empty array, and the generator does produce this edge case from time to time.

Applying reversal twice is a small example chosen to make explaining property-based testing easier; the same approach can be applied to larger bodies. Converting a structure a parser produced back to a string and parsing it again should give the same result as the original structure; a sorting function’s output should carry the same elements as the input and have its neighboring elements in order. The property chosen in this lesson is one of the simplest — an operation combined with its own inverse gives identity — but the way of measuring (generator, property, counter) stays the same: whatever body is tested, closing the place the compiler does not check by running always requires the same three parts.

All two hundred inputs confirmed the property. PL6. This count is deterministic together with the generator: as long as the seed is 42, each of the two hundred arrays produced arrives in the same order, the same length, and with the same elements; no matter how many times the run is repeated, the result does not change. This also clarifies what the word “random” means in this lesson: the produced inputs are varied (not written out one by one by hand), but the production process is not random — it produces the same sequence on every run.

Distinguishing this from a related idea: the Production Go: Testing, Tooling and Performance course’s Coverage Reporting lesson measured which lines a test ran — the coverage ratio showed how much of the source passed through the run. This lesson does not count lines; it counts how many inputs a property holds for. A line running does not mean it runs correctly — it only means it ran; what this lesson measures is the strength of the guarantee that running gives about correctness.

The guarantee’s class here is handed to the writer: the compiler checked reverse_reverse_identity’s type and stopped there; it never asked whether the function gives the correct result for every input. The party who wrote the test was the one who asked and answered this question — the writer decided which property to test, which range the inputs would come from (-500 to 499, at most twenty elements), and how many inputs (two hundred) would be enough. The compiler took no part in any of these decisions; it only checked that the written test code itself was type-consistent.

The handed-to-the-writer class carries the same word as the transfer measured in the unsafe Rust lesson, but it sits in a different place: there, what was handed off was the contract that a memory access stays within bounds; here, what is handed off is the claim that a behavior is correct. What the two share is that the compiler stops at some point and leaves the remaining responsibility to whoever wrote the source. One consequence of this transfer: if the reverse_reverse_identity function itself had been written wrong — say, the second reverse call had been forgotten — the compiler would not have noticed that either, because a wrongly written test passes type checking exactly like a correctly written one. A test’s correctness, just like the correctness of the body it tests, stays the writer’s responsibility.

How many inputs there are is also part of the same transfer. Two hundred was chosen in this lesson; had fifty been chosen, the test would have seen fewer inputs, had two thousand been chosen, more. As the number grows, the covered set of inputs widens, but no number covers all inputs. The compiler expresses no opinion about this number at all — no matter how small or how large the number is, the test code passes type checking and compiles the same way. Deciding whether the number is sufficient is part of the same responsibility as deciding the generator’s range: both rest in the writer’s hands, entirely outside the compiler’s sphere of checking.

A Bounding Measurement

The run above showed that the chosen property holds for every input — but that outcome comes from the chosen property being a true mathematical fact: reversing an array twice always gives back the original array, regardless of order. Not every property is like this. The run below tests, with the same generator, a property that holds for some inputs but not others: an array being equal to its own reverse, that is, a palindrome condition.

// bounding_test.rs — a property that holds for some inputs, not for others
mod linear_generator;
use linear_generator::LinearGenerator;

fn equals_own_reverse(v: &[i32]) -> bool {
    let mut reversed = v.to_vec();
    reversed.reverse();
    reversed == v
}

fn main() {
    let mut generator = LinearGenerator::new(7);
    let total_inputs = 200;
    let mut held = 0;
    let mut first_non_holding: Option<Vec<i32>> = None;

    for _ in 0..total_inputs {
        let input = generator.vector(6);
        if equals_own_reverse(&input) {
            held += 1;
        } else if first_non_holding.is_none() {
            first_non_holding = Some(input);
        }
    }

    println!("inputs tested: {}", total_inputs);
    println!("inputs the property held for: {}", held);
    println!("first non-holding input length: {}", first_non_holding.map(|v| v.len()).unwrap_or(0));
}

inputs tested: 200
inputs the property held for: 62
first non-holding input length: 3

PL7. The property held for sixty-two of the two hundred inputs, and failed to hold for the remaining hundred thirty-eight — because the generator can also produce short arrays (at most six elements, including zero), some inputs turned out palindromic by chance: an empty array and a one-element array are always equal to their own reverse, and this raises the holding count. The first non-holding input’s length is three — a three-element array being a palindrome depends on the elements themselves, not on its length.

This measurement limits what a passing property test means. In the previous run, the property held for all two hundred of two hundred inputs, because the tested property was always a true mathematical fact; in this run, the property held for only sixty-two inputs, because the tested property is sometimes true. The test itself does not show the difference between the two — both look like a “passing test,” except that the count of passing inputs is not entirely equal. A property test passing proves that the property holds for the tested set of inputs; it says nothing about an input that was not tested. Two hundred inputs may be a wide sample, but the generator draws from a bounded range (-500 to 499, at most twenty elements) — an input outside this range, or beyond this length, was never tested at all. PL8. The guarantee a passing test gives is bounded by the number and range of inputs it tested; when a claim is written without being tied to that number, the guarantee’s real scope is hidden.

The measurement carries one more limit: the first non-holding input was reported only by its length, not by its elements — this is not a choice made because of the environment, it is a deliberate one. Real property-based testing tools, once they find a non-holding input, reduce it to a smaller, more readable counterexample through a process called shrinking; this lesson does not write such a shrinking step, it only reports the first non-holding input’s length. This is another limit of the measurement: showing that a property does not hold is not the same as finding the most understandable counterexample — what is counted in this lesson is only how many inputs it holds for, not which input is the most illustrative counterexample.

The number sixty-two itself also carries a limit: it is the count these particular two hundred inputs, produced with seed 7, gave. Had seed 11 or 100 been chosen, the count of holding inputs would have come out different even with the same property tested across the same range — because the set of produced inputs changes depending on the seed. This lesson does not measure that variability; with a single seed, it makes a single run deterministic and reports that run’s result. Determinism here guarantees repeatability (the same seed always gives the same sixty-two), but it does not guarantee the tested range’s representativeness — that strength depends on how the range and the generator are designed, not on a single run’s result.

Summary

• Rust’s standard library has no random-number generator; this lesson used a fixed-seed, hand-written linear congruential generator — the same seed produces the same input sequence on every run.
• The property that applying reversal twice gives identity held for all two hundred of the two hundred generated inputs.
• The guarantee’s class is handed to the writer: the compiler checked the test code’s type and stopped; the writer decided which property to test, over which input range, and with how many inputs.
• A property that is sometimes true (being equal to its own reverse), tested with the same generator, held for only sixty-two of the two hundred inputs — a passing test does not mean correct for every input.
• This measurement’s limit: the guarantee a property test gives extends only as far as the number and range of tested inputs; an input outside the range was never tested at all.
• The number of inputs, their range, and the tested property itself — including how much of each is enough — are decisions made by the party who writes the test, not the compiler; the same three parts (generator, property, counter) apply to bodies other than reversal, too.

Next Step

This lesson measured that we can show, by running, that a property runs correctly, but that this demonstration stays bounded by a limited number of inputs and the writer’s decisions. The next lesson applies the same “guarantee established by running” idea to a completely different question: which of two implementations does less work can also only be learned by running, a question the compiler never asks, just like correctness. But this time what is tested is not correctness, it is operation count; and instead of a number of inputs, this time it is how many comparisons and how many copies happen on a single input that gets counted. The Benchmarking and Profiling lesson examines what this number does and does not say — specifically, why not a duration.