Benchmarking

The previous four lessons each asked one question at a time about a program: did a test pass, how many subtests did a case turn into, which status code did a request trigger, did a dependency’s real and fake versions give the same verdict. Each question carried its own source/tool/run class, but none of them asked how these classes are distributed in total. This lesson steps back and asks the same question all at once: if we gather eleven separate things that can be known about a program, how many are written in the source, how many does a tool state without a run, how many only emerge by running it?

This lesson is also where the measurement the course’s name points to — the benchmark — first appears. This measure produces two numbers on its own: how much memory a piece of work allocates and how long it takes. Only the first of these is used in this course; why the second is left out is the subject of the second half of this lesson.

Eleven Pieces of Information, Three Tellers

The measurement below writes a small source — a struct, two functions, a three-case test table, an extra file kept by a build tag — into a temporary module, then asks eleven separate questions about this one source. Each question is passed to the observe routine together with who states the answer (source, tool, run); observe counts these three classes. This gathers everything the previous four lessons asked separately under one roof; each question is answered with a mechanism already seen before (reading the source as a string and counting, calling a tool from inside the program and reading only its result).

// tools.go — the shared routines of this "who states this" frame, and two benchmark writings
package main

import (
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"testing"
)

var source, tool, run int

func observe(fact, statedBy, evidence string) {
	switch statedBy {
	case "source":
		source++
	case "tool":
		tool++
	default:
		run++
	}
	fmt.Printf("%-35s %-7s %s\n", fact, statedBy, evidence)
}

func buildModule(files map[string]string) string {
	dir, _ := os.MkdirTemp("", "measurement")
	os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module measurement\n\ngo 1.24\n"), 0o644)
	for name, content := range files {
		os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644)
	}
	return dir
}

func runTool(dir string, env []string, args ...string) (string, bool) {
	cmd := exec.Command("go", args...)
	cmd.Dir = dir
	cmd.Env = append(append(os.Environ(), "GOTOOLCHAIN=local"), env...)
	output, err := cmd.CombinedOutput()
	return string(output), err == nil
}

func escapeCount(output string) int { return strings.Count(output, "escapes to heap") }

func coverageRatio(output string) string {
	for _, s := range strings.Split(output, "\n") {
		if i := strings.Index(s, "coverage: "); i >= 0 {
			return strings.TrimSpace(s[i+len("coverage: "):])
		}
	}
	return "none"
}

// growSlice appends eight times without preallocating.
func growSlice(b *testing.B) {
	for i := 0; i < b.N; i++ {
		var s []int
		for j := 0; j < 8; j++ {
			s = append(s, j)
		}
		_ = s
	}
}

// preallocSlice does the same work, appending into a space preallocated for eight elements.
func preallocSlice(b *testing.B) {
	for i := 0; i < b.N; i++ {
		s := make([]int, 0, 8)
		for j := 0; j < 8; j++ {
			s = append(s, j)
		}
		_ = s
	}
}

// runFiveTimes runs the same benchmark five times; it returns the first value and how
// many distinct values were seen.
func runFiveTimes(k func(*testing.B)) (first int64, distinctValues int) {
	seen := map[int64]bool{}
	for i := 0; i < 5; i++ {
		r := testing.Benchmark(k)
		seen[r.AllocsPerOp()] = true
		if i == 0 {
			first = r.AllocsPerOp()
		}
	}
	return first, len(seen)
}

TM13. Two of the eleven questions asked are answered from the source: the case count in the test table and the module’s language version — both are written in the text itself, readable without running anything.

// measurement.go — eleven pieces of information, split among three tellers
package main

import (
	"fmt"
	"os"
	"strings"
)

const sourceText = `package main

import "fmt"

type Record struct{ Name string }

func staysLocal() int {
	r := Record{Name: "bolt"}
	return len(r.Name)
}

func escapes() *Record {
	r := Record{Name: "nut"}
	return &r
}

var version = "unknown"

func main() {
	fmt.Println(staysLocal(), escapes().Name, version)
}
`

const testText = `package main

import "testing"

var cases = []struct {
	name     string
	input    string
	length   int
}{
	{"short", "ab", 2},
	{"medium", "abcd", 4},
	{"empty", "", 0},
}

func TestLength(t *testing.T) {
	for _, c := range cases {
		if len(c.input) != c.length {
			t.Errorf("%s: expected %d", c.name, c.length)
		}
	}
	if staysLocal() != 4 {
		t.Error("staysLocal")
	}
	if escapes().Name != "nut" {
		t.Error("escapes")
	}
}
`

const taggedText = `//go:build extra

package main

func extra() int { return 1 }
`

func main() {
	fmt.Println("-- who states this --")

	dir := buildModule(map[string]string{"main.go": sourceText, "main_test.go": testText, "extra.go": taggedText})
	defer os.RemoveAll(dir)

	observe("table case count", "source", fmt.Sprint(strings.Count(testText, "\t{\"")))
	observe("module language version", "source", "written in the go.mod line")

	escapeOutput, _ := runTool(dir, nil, "build", "-gcflags=-m", "-o", os.DevNull, ".")
	observe("heap-escaping value count", "tool", fmt.Sprint(escapeCount(escapeOutput)))

	_, vetClean := runTool(dir, nil, "vet", "./...")
	observe("static analysis finding", "tool", fmt.Sprint("clean=", vetClean))

	_, crossBuilt := runTool(dir, []string{"GOOS=linux", "GOARCH=arm64"}, "build", "-o", os.DevNull, ".")
	observe("buildability for another platform", "tool", fmt.Sprint("built=", crossBuilt))

	untagged, _ := runTool(dir, nil, "list", "-f", "{{len .GoFiles}}", ".")
	tagged, _ := runTool(dir, nil, "list", "-tags", "extra", "-f", "{{len .GoFiles}}", ".")
	observe("file count changed by build tag", "tool",
		fmt.Sprint("untagged=", strings.TrimSpace(untagged), " tagged=", strings.TrimSpace(tagged)))

	_, testPassed := runTool(dir, nil, "test", "./...")
	observe("test verdict", "run", fmt.Sprint("passed=", testPassed))

	coverageOutput, _ := runTool(dir, nil, "test", "-cover", "./...")
	observe("line coverage ratio", "run", coverageRatio(coverageOutput))

	linkedOutput, _ := runTool(dir, nil, "run", "-ldflags=-X main.version=1", ".")
	fields := strings.Fields(strings.TrimSpace(linkedOutput))
	observe("version embedded in the binary", "tool", fields[len(fields)-1])

	firstAllocs, distinctValues := runFiveTimes(growSlice)
	observe("allocation count", "run", fmt.Sprint(firstAllocs, " distinct values across five runs=", distinctValues))

	filterOutput, _ := runTool(dir, nil, "test", "-run", "TestLength", "-v", "./...")
	observe("filter-selected test count", "run", fmt.Sprint(strings.Count(filterOutput, "--- PASS")))

	fmt.Printf("information: %d source, %d tool, %d run\n", source, tool, run)
}

-- who states this --
table case count                    source  3
module language version             source  written in the go.mod line
heap-escaping value count           tool    3
static analysis finding             tool    clean=true
buildability for another platform   tool    built=true
file count changed by build tag     tool    untagged=1 tagged=2
test verdict                        run     passed=true
line coverage ratio                 run     80.0% of statements
version embedded in the binary      tool    1
allocation count                    run     4 distinct values across five runs=1
filter-selected test count          run     1
information: 2 source, 5 tool, 4 run

TM14. Five pieces of information come from a tool: heap-escaping value count, the static analyzer’s finding, buildability for another platform, the file count changed by the tag, the version embedded in the binary. What these five share is that they require no run — the compiler, the checker, the analyzer, and the linker each give the same answer to the same source every time they are called.

TM15. The remaining four pieces of information can only be learned by running: the test verdict, the covered line ratio, the allocation count, the filter-selected test count. The sum of the eleven pieces of information accumulates in a single run for the first time in this lesson: 2 source, 5 tool, 4 run. This is the topic’s binding distribution — this number is referred to with a single sentence in every lesson after this one, it is never reproduced again.

Each of these eleven lines is asked in exactly the same way the previous four lessons’ questions were asked — only this time all of them are gathered in a single run, against a single source. The table case count line repeats Table-Driven Tests’ question, the test verdict line repeats Test Package Basics’ question exactly; the only difference is that here all these questions are gathered in the same run and turned into a distribution. Nine of the eleven pieces of information (the tool and run sum) require a run or at least one tool call; reading the source alone gives only two of these eleven — the great majority of what can be known about the program itself lies outside the text.

Only One of the Benchmark’s Two Numbers Is Reproducible

The tenth of the eleven lines above — the allocation count — is this lesson’s real subject, and it is measured differently from the remaining ten: testing.Benchmark runs the same work five times, AllocsPerOp returns one number from each run, and how many distinct values these five numbers fall into is counted separately. In the run above, this number is 1 — five runs, a single value.

growSlice was already used once in the full run above — it was the evidence for the “allocation count” line. The measurement below calls it again, but this time not alone, together with a second writing it will compare against.

TM16. The measurement below compares two writings of the same work: growSlice appends eight elements without preallocating, preallocSlice appends the same eight elements into a space preallocated beforehand. Both are run five times each.

// comparison.go — the same work's two writings compared by allocation count
package main

import "fmt"

func main() {
	fmt.Println("-- benchmark: two writings compared by allocation count --")

	noPreallocFirst, noPreallocDistinct := runFiveTimes(growSlice)
	fmt.Println("without preallocation, allocations per operation:", noPreallocFirst,
		"distinct values across five runs:", noPreallocDistinct)

	preallocFirst, preallocDistinct := runFiveTimes(preallocSlice)
	fmt.Println("with preallocation, allocations per operation:", preallocFirst,
		"distinct values across five runs:", preallocDistinct)
}

-- benchmark: two writings compared by allocation count --
without preallocation, allocations per operation: 4 distinct values across five runs: 1
with preallocation, allocations per operation: 0 distinct values across five runs: 1

The benchmark tool itself, here just as in the previous run, returns two separate numbers — allocations per operation and elapsed time. This lesson never prints, measures, or compares the time anywhere: only AllocsPerOp is read from what testing.Benchmark returns; the field carrying elapsed time is never used at all. The reason for this is not a preference but an observation: even measured five times in a row on the same machine, on the same source, elapsed time changes from run to run — it depends on what else the operating system happens to be running at that moment, on the cache’s state at that moment, even on how much the measurement itself warms the machine. Allocation count is not like this: growSlice allocates exactly four times on every run, because the slice’s underlying array goes through the same growth decisions every time until it reaches eight elements, and this chain of decisions does not change from run to run.

preallocSlice allocates zero times — the space for eight elements is allocated all at once with make([]int, 0, 8), and the compiler’s escape analysis (this separation itself is measured in the topic’s next topic) can keep this slice on the stack without ever moving it to the heap. The gap between four and zero is the concrete counterpart of preallocating: the same eight elements are added in the same order, but one is left to the slice’s own growth mechanism (doubling capacity as it fills), while the other asks for enough space up front.

The Limiting Measurement: Even Allocation Count Cannot Be Compared Without Saying What It Counts

TM17. The two writings’ allocation counts — 4 and 0 — do not on their own say “one is better”; without saying what each number counts, these two values cannot be compared. What AllocsPerOp returns is allocations per operation: testing.Benchmark decides b.N on its own, runs the inner loop that many times, and produces this number by dividing the total allocations by b.N. The total allocation count — how many times memory was allocated across the whole run — is a separate number, and it grows as b.N grows; the per-operation number stays fixed while the total number never stays fixed at all.

When this distinction is missed, a misleading comparison follows: the sentence “the first writing allocates four times more memory than the second” is correct, but “the first writing allocated four times more memory in total” is a different claim — the second sentence assumes the two runs’ b.N values were the same, and that assumption was never checked anywhere. The number this lesson measures is always allocations per operation; total allocations were never computed in this lesson at all, because the topic’s question is “which writing allocates less memory per operation” — not “how much memory did this run allocate in total.” If a benchmark result is shared without saying which of these two numbers is meant, the same two numbers (4 and 0) can be offered as evidence for an entirely different claim.

The runFiveTimes routine itself carries this distinction implicitly too: the first value it returns is each run’s AllocsPerOp — already a per-operation number, already divided by b.N by testing.Benchmark. This routine never sees the total allocation count at all, because testing.Benchmark itself never exposes that number — it returns only allocations per operation and time per operation. This is a consequence of the measurement tool’s own design — benchmark tools generally prefer the per-operation number, because it is exactly this normalized form that makes two measurements run with different b.N values comparable at all.

Summary

• Eleven pieces of information were gathered in one run and split into three classes: 2 source, 5 tool, 4 run — this distribution is the topic’s binding number, it is not reproduced in later lessons.
• Five pieces of tool information require no run, giving the same answer to the same source on every call; four pieces of run information can only be learned by actually running it.
• The benchmark tool returns two numbers, but this lesson read only one of them — allocations per operation; time was never printed anywhere, never compared anywhere.
• Allocation count landed on the same value across five runs (1 distinct value); this is proof that the number is reproducible.
• The limiting measurement: allocations per operation and total allocations are separate numbers; a comparison cannot be read without saying which one is meant.

Next Step

This lesson measured how eleven pieces of information about a program split into three classes, and that only one of the benchmark’s two numbers is reproducible. Five pieces of tool information and four pieces of run information stayed within this lesson’s own example; both reappear in their own corner in later lessons, but the full eleven-line table is never printed again. One of these eleven pieces of information — covered line ratio — passed in this lesson only as a single number: 80.0% of statements. The next lesson measures exactly what this number counts, which lines are left out of coverage, and how three separate test sets change this ratio — the topic’s closing lesson.