--- title: Benchmarking source: 'https://academia.sh/en/courses/go-production/comparative-measurement' course: 'Production Go: Testing, Tooling and Performance' language: en updated: '2026-09-14T10:59:33+00:00' license: 'CC BY-SA 4.0' objectives: - 'Splits eleven pieces of information into source, tool, and run classes, producing the 2/5/4 split by running it' - 'Compares the allocation count of two writings of the same work across five runs and reads which one is reproducible' - 'Limits that allocations per operation and total allocation count are separate measures, by reading the same comparison two ways' --- # Benchmarking Eleven pieces of information are gathered about one program and split among three tellers — 2 by the source, 5 by a tool with no run needed, 4 only by running it. The benchmark tool produces one of these four run pieces of information: the allocation count of two writings of the same work gives the same value across five runs; the limiting measurement shows that even this number cannot be compared without saying what it counts. The previous four lessons each asked one question at a time about a program: did a test pass, how many subtests did a case turn into, which status code did a request trigger, did a dependency's real and fake versions give the same verdict. Each question carried its own source/tool/run class, but none of them asked how these classes are distributed **in total**. This lesson steps back and asks the same question all at once: if we gather eleven separate things that can be known about a program, how many are written in the source, how many does a tool state without a run, how many only emerge by running it? This lesson is also where the measurement the course's name points to — the benchmark — first appears. This measure produces two numbers on its own: how much memory a piece of work allocates and how long it takes. Only the first of these is used in this course; why the second is left out is the subject of the second half of this lesson. ## Eleven Pieces of Information, Three Tellers The measurement below writes a small source — a struct, two functions, a three-case test table, an extra file kept by a build tag — into a temporary module, then asks eleven separate questions about this one source. Each question is passed to the `observe` routine together with who states the answer (source, tool, run); `observe` counts these three classes. This gathers everything the previous four lessons asked separately under one roof; each question is answered with a mechanism already seen before (reading the source as a string and counting, calling a tool from inside the program and reading only its result). ```go // tools.go — the shared routines of this "who states this" frame, and two benchmark writings package main import ( "fmt" "os" "os/exec" "path/filepath" "strings" "testing" ) var source, tool, run int func observe(fact, statedBy, evidence string) { switch statedBy { case "source": source++ case "tool": tool++ default: run++ } fmt.Printf("%-35s %-7s %s\n", fact, statedBy, evidence) } func buildModule(files map[string]string) string { dir, _ := os.MkdirTemp("", "measurement") os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module measurement\n\ngo 1.24\n"), 0o644) for name, content := range files { os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644) } return dir } func runTool(dir string, env []string, args ...string) (string, bool) { cmd := exec.Command("go", args...) cmd.Dir = dir cmd.Env = append(append(os.Environ(), "GOTOOLCHAIN=local"), env...) output, err := cmd.CombinedOutput() return string(output), err == nil } func escapeCount(output string) int { return strings.Count(output, "escapes to heap") } func coverageRatio(output string) string { for _, s := range strings.Split(output, "\n") { if i := strings.Index(s, "coverage: "); i >= 0 { return strings.TrimSpace(s[i+len("coverage: "):]) } } return "none" } // growSlice appends eight times without preallocating. func growSlice(b *testing.B) { for i := 0; i < b.N; i++ { var s []int for j := 0; j < 8; j++ { s = append(s, j) } _ = s } } // preallocSlice does the same work, appending into a space preallocated for eight elements. func preallocSlice(b *testing.B) { for i := 0; i < b.N; i++ { s := make([]int, 0, 8) for j := 0; j < 8; j++ { s = append(s, j) } _ = s } } // runFiveTimes runs the same benchmark five times; it returns the first value and how // many distinct values were seen. func runFiveTimes(k func(*testing.B)) (first int64, distinctValues int) { seen := map[int64]bool{} for i := 0; i < 5; i++ { r := testing.Benchmark(k) seen[r.AllocsPerOp()] = true if i == 0 { first = r.AllocsPerOp() } } return first, len(seen) } ``` **TM13.** Two of the eleven questions asked are answered from the source: the case count in the test table and the module's language version — both are written in the text itself, readable without running anything. ```go // measurement.go — eleven pieces of information, split among three tellers package main import ( "fmt" "os" "strings" ) const sourceText = `package main import "fmt" type Record struct{ Name string } func staysLocal() int { r := Record{Name: "bolt"} return len(r.Name) } func escapes() *Record { r := Record{Name: "nut"} return &r } var version = "unknown" func main() { fmt.Println(staysLocal(), escapes().Name, version) } ` const testText = `package main import "testing" var cases = []struct { name string input string length int }{ {"short", "ab", 2}, {"medium", "abcd", 4}, {"empty", "", 0}, } func TestLength(t *testing.T) { for _, c := range cases { if len(c.input) != c.length { t.Errorf("%s: expected %d", c.name, c.length) } } if staysLocal() != 4 { t.Error("staysLocal") } if escapes().Name != "nut" { t.Error("escapes") } } ` const taggedText = `//go:build extra package main func extra() int { return 1 } ` func main() { fmt.Println("-- who states this --") dir := buildModule(map[string]string{"main.go": sourceText, "main_test.go": testText, "extra.go": taggedText}) defer os.RemoveAll(dir) observe("table case count", "source", fmt.Sprint(strings.Count(testText, "\t{\""))) observe("module language version", "source", "written in the go.mod line") escapeOutput, _ := runTool(dir, nil, "build", "-gcflags=-m", "-o", os.DevNull, ".") observe("heap-escaping value count", "tool", fmt.Sprint(escapeCount(escapeOutput))) _, vetClean := runTool(dir, nil, "vet", "./...") observe("static analysis finding", "tool", fmt.Sprint("clean=", vetClean)) _, crossBuilt := runTool(dir, []string{"GOOS=linux", "GOARCH=arm64"}, "build", "-o", os.DevNull, ".") observe("buildability for another platform", "tool", fmt.Sprint("built=", crossBuilt)) untagged, _ := runTool(dir, nil, "list", "-f", "{{len .GoFiles}}", ".") tagged, _ := runTool(dir, nil, "list", "-tags", "extra", "-f", "{{len .GoFiles}}", ".") observe("file count changed by build tag", "tool", fmt.Sprint("untagged=", strings.TrimSpace(untagged), " tagged=", strings.TrimSpace(tagged))) _, testPassed := runTool(dir, nil, "test", "./...") observe("test verdict", "run", fmt.Sprint("passed=", testPassed)) coverageOutput, _ := runTool(dir, nil, "test", "-cover", "./...") observe("line coverage ratio", "run", coverageRatio(coverageOutput)) linkedOutput, _ := runTool(dir, nil, "run", "-ldflags=-X main.version=1", ".") fields := strings.Fields(strings.TrimSpace(linkedOutput)) observe("version embedded in the binary", "tool", fields[len(fields)-1]) firstAllocs, distinctValues := runFiveTimes(growSlice) observe("allocation count", "run", fmt.Sprint(firstAllocs, " distinct values across five runs=", distinctValues)) filterOutput, _ := runTool(dir, nil, "test", "-run", "TestLength", "-v", "./...") observe("filter-selected test count", "run", fmt.Sprint(strings.Count(filterOutput, "--- PASS"))) fmt.Printf("information: %d source, %d tool, %d run\n", source, tool, run) } ``` ``` -- who states this -- table case count source 3 module language version source written in the go.mod line heap-escaping value count tool 3 static analysis finding tool clean=true buildability for another platform tool built=true file count changed by build tag tool untagged=1 tagged=2 test verdict run passed=true line coverage ratio run 80.0% of statements version embedded in the binary tool 1 allocation count run 4 distinct values across five runs=1 filter-selected test count run 1 information: 2 source, 5 tool, 4 run ``` **TM14.** Five pieces of information come from a tool: heap-escaping value count, the static analyzer's finding, buildability for another platform, the file count changed by the tag, the version embedded in the binary. What these five share is that they **require no run** — the compiler, the checker, the analyzer, and the linker each give the same answer to the same source every time they are called. **TM15.** The remaining four pieces of information can only be learned by running: the test verdict, the covered line ratio, the allocation count, the filter-selected test count. The sum of the eleven pieces of information accumulates in a single run for the first time in this lesson: **2 source, 5 tool, 4 run**. This is the topic's binding distribution — this number is referred to with a single sentence in every lesson after this one, it is never reproduced again. Each of these eleven lines is asked in exactly the same way the previous four lessons' questions were asked — only this time all of them are gathered in a single run, against a single source. The `table case count` line repeats Table-Driven Tests' question, the `test verdict` line repeats Test Package Basics' question exactly; the only difference is that here all these questions are gathered in the same run and turned into a distribution. **Nine** of the eleven pieces of information (the tool and run sum) require a run or at least one tool call; reading the source alone gives only two of these eleven — the great majority of what can be known about the program itself lies outside the text. ## Only One of the Benchmark's Two Numbers Is Reproducible The tenth of the eleven lines above — the allocation count — is this lesson's real subject, and it is measured differently from the remaining ten: `testing.Benchmark` runs the same work five times, `AllocsPerOp` returns one number from each run, and how many **distinct** values these five numbers fall into is counted separately. In the run above, this number is 1 — five runs, a single value. `growSlice` was already used once in the full run above — it was the evidence for the "allocation count" line. The measurement below calls it again, but this time not alone, together with a second writing it will compare against. **TM16.** The measurement below compares two writings of the same work: `growSlice` appends eight elements without preallocating, `preallocSlice` appends the same eight elements into a space preallocated beforehand. Both are run five times each. ```go // comparison.go — the same work's two writings compared by allocation count package main import "fmt" func main() { fmt.Println("-- benchmark: two writings compared by allocation count --") noPreallocFirst, noPreallocDistinct := runFiveTimes(growSlice) fmt.Println("without preallocation, allocations per operation:", noPreallocFirst, "distinct values across five runs:", noPreallocDistinct) preallocFirst, preallocDistinct := runFiveTimes(preallocSlice) fmt.Println("with preallocation, allocations per operation:", preallocFirst, "distinct values across five runs:", preallocDistinct) } ``` ``` -- benchmark: two writings compared by allocation count -- without preallocation, allocations per operation: 4 distinct values across five runs: 1 with preallocation, allocations per operation: 0 distinct values across five runs: 1 ``` The benchmark tool itself, here just as in the previous run, returns two separate numbers — allocations per operation and elapsed time. This lesson **never prints, measures, or compares the time anywhere**: only `AllocsPerOp` is read from what `testing.Benchmark` returns; the field carrying elapsed time is never used at all. The reason for this is not a preference but an observation: even measured five times in a row on the same machine, on the same source, elapsed time changes from run to run — it depends on what else the operating system happens to be running at that moment, on the cache's state at that moment, even on how much the measurement itself warms the machine. Allocation count is not like this: `growSlice` allocates exactly four times on every run, because the slice's underlying array goes through the same growth decisions every time until it reaches eight elements, and this chain of decisions does not change from run to run. `preallocSlice` allocates zero times — the space for eight elements is allocated all at once with `make([]int, 0, 8)`, and the compiler's escape analysis (this separation itself is measured in the topic's next topic) can keep this slice on the stack without ever moving it to the heap. The gap between four and zero is the concrete counterpart of preallocating: the same eight elements are added in the same order, but one is left to the slice's own growth mechanism (doubling capacity as it fills), while the other asks for enough space up front. ## The Limiting Measurement: Even Allocation Count Cannot Be Compared Without Saying What It Counts **TM17.** The two writings' allocation counts — 4 and 0 — do not on their own say "one is better"; without saying **what** each number counts, these two values cannot be compared. What `AllocsPerOp` returns is allocations **per operation**: `testing.Benchmark` decides `b.N` on its own, runs the inner loop that many times, and produces this number by dividing the total allocations by `b.N`. The total allocation count — how many times memory was allocated across the whole run — is a separate number, and it grows as `b.N` grows; the per-operation number stays fixed while the total number never stays fixed at all. When this distinction is missed, a misleading comparison follows: the sentence "the first writing allocates four times more memory than the second" is correct, but "the first writing allocated four times more memory in total" is a different claim — the second sentence assumes the two runs' `b.N` values were the same, and that assumption was never checked anywhere. The number this lesson measures is always allocations per operation; total allocations were never computed in this lesson at all, because the topic's question is "which writing allocates less memory per operation" — not "how much memory did this run allocate in total." If a benchmark result is shared without saying which of these two numbers is meant, the same two numbers (4 and 0) can be offered as evidence for an entirely different claim. The `runFiveTimes` routine itself carries this distinction implicitly too: the `first` value it returns is each run's `AllocsPerOp` — already a per-operation number, already divided by `b.N` by `testing.Benchmark`. This routine never sees the total allocation count at all, because `testing.Benchmark` itself never exposes that number — it returns only allocations per operation and time per operation. This is a consequence of the measurement tool's own design — benchmark tools generally prefer the per-operation number, because it is exactly this normalized form that makes two measurements run with different `b.N` values comparable at all. ## Summary - Eleven pieces of information were gathered in one run and split into three classes: **2 source, 5 tool, 4 run** — this distribution is the topic's binding number, it is not reproduced in later lessons. - Five pieces of tool information require no run, giving the same answer to the same source on every call; four pieces of run information can only be learned by actually running it. - The benchmark tool returns two numbers, but this lesson read only one of them — allocations per operation; time was never printed anywhere, never compared anywhere. - Allocation count landed on the same value across five runs (1 distinct value); this is proof that the number is reproducible. - The limiting measurement: allocations per operation and total allocations are separate numbers; a comparison cannot be read without saying which one is meant. ## Next Step This lesson measured how eleven pieces of information about a program split into three classes, and that only one of the benchmark's two numbers is reproducible. Five pieces of tool information and four pieces of run information stayed within this lesson's own example; both reappear in their own corner in later lessons, but the full eleven-line table is never printed again. One of these eleven pieces of information — covered line ratio — passed in this lesson only as a single number: `80.0% of statements`. The next lesson measures exactly what this number counts, which lines are left out of coverage, and how three separate test sets change this ratio — the topic's closing lesson.