Coverage Reporting

The previous lesson split eleven pieces of information about a program into source, tool, and run classes; one of these eleven — covered line ratio — passed in that lesson only as a single number (80.0% of statements), with no detail given. This closing lesson is the topic’s wrap-up, and it opens that detail: what exactly does coverage ratio count, and what actually changes when this number rises?

Coverage ratio is a commonly used indicator in production codebases: a review, a threshold (“new code must be covered at least this much”), or a single percentage shown on a dashboard. What this indicator says and does not say is the final exam of the “who states this” question the topic’s five lessons have built — because coverage, looked at on its own, proves not that a program was tested, but only that it was run. To answer this question, the lesson returns one last time to the inventory package used throughout the topic; this time the subject is not the package itself but the three different test sets that test the same package.

Same Function, Three Test Sets, Three Ratios

One last function is added to the inventory package: Label looks at an item’s quantity and returns one of four states — a negative quantity "invalid", zero "out of stock", under three "low stock", the rest "sufficient". Each of the four states has its own return line, and the four states are mutually exclusive: a single call falls into exactly one of them.

This function was built deliberately as an example with an adjustable branch count — four distinct branches, four distinct test cases, and exactly how many separate blocks each case adds to coverage can be traced one by one. Valid (Table-Driven Tests) and ReduceStock (Faking and Interface Separation) also carried similarly branched shapes; this lesson reuses the same shape to see how the coverage tool counts this branching.

All three test sets run against the same inventoryText source — Label itself never changes, only which states get called changes. This shows that coverage ratio is not a property of a source but a property of a source-test pair: the same source’s coverage ratio can take any value between 57.1% and 100.0%, depending on which test set asks the question.

The Setup and Run routines have not changed since the topic’s first lesson; this lesson too builds a completely empty temporary module for every test set, carrying no trace of a previous run.

TM18. The same Label function is tested with three different test sets: T1 tests only the “sufficient” state, T2 tests three states (“sufficient”, “out of stock”, “low stock”), T3 calls all four.

// tools.go — setup, run, and coverage-reading routines, and the Label used throughout the topic
package main

import (
	"os"
	"os/exec"
	"path/filepath"
	"strings"
)

func Setup(files map[string]string) string {
	dir, _ := os.MkdirTemp("", "trial")
	os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module trial\n\ngo 1.24\n"), 0o644)
	for name, content := range files {
		os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644)
	}
	return dir
}

func Run(dir string, extraEnv []string, args ...string) (string, bool) {
	cmd := exec.Command("go", args...)
	cmd.Dir = dir
	cmd.Env = append(append(os.Environ(), "GOTOOLCHAIN=local"), extraEnv...)
	output, err := cmd.CombinedOutput()
	return string(output), err == nil
}

func coverageRatio(output string) string {
	for _, s := range strings.Split(output, "\n") {
		if i := strings.Index(s, "coverage: "); i >= 0 {
			return strings.TrimSpace(s[i+len("coverage: "):])
		}
	}
	return "none"
}

// uncoveredBlockCount counts the blocks that never ran at all in the coverage profile.
func uncoveredBlockCount(profile []byte) int {
	n := 0
	for _, line := range strings.Split(string(profile), "\n")[1:] {
		fields := strings.Fields(line)
		if len(fields) == 3 && fields[2] == "0" {
			n++
		}
	}
	return n
}

const inventoryText = `package trial

type Item struct {
	Name     string
	Quantity int
	Price    int
}

func Label(it Item) string {
	if it.Quantity < 0 {
		return "invalid"
	}
	if it.Quantity == 0 {
		return "out of stock"
	}
	if it.Quantity < 3 {
		return "low stock"
	}
	return "sufficient"
}
`

const testTextT3 = `package trial

import "testing"

func TestLabel(t *testing.T) {
	if got := Label(Item{"pen", 5, 250}); got != "sufficient" {
		t.Errorf("got %s", got)
	}
	if got := Label(Item{"pen", 0, 250}); got != "out of stock" {
		t.Errorf("got %s", got)
	}
	if got := Label(Item{"pen", 2, 250}); got != "low stock" {
		t.Errorf("got %s", got)
	}
	Label(Item{"pen", -1, 250})
}
`

// measurement.go — three test sets, three separate coverage ratios and uncovered block counts
package main

import (
	"fmt"
	"os"
	"path/filepath"
)

const testTextT1 = `package trial

import "testing"

func TestLabelSufficient(t *testing.T) {
	if got := Label(Item{"pen", 5, 250}); got != "sufficient" {
		t.Errorf("got %s", got)
	}
}
`

const testTextT2 = `package trial

import "testing"

func TestLabel(t *testing.T) {
	if got := Label(Item{"pen", 5, 250}); got != "sufficient" {
		t.Errorf("got %s", got)
	}
	if got := Label(Item{"pen", 0, 250}); got != "out of stock" {
		t.Errorf("got %s", got)
	}
	if got := Label(Item{"pen", 2, 250}); got != "low stock" {
		t.Errorf("got %s", got)
	}
}
`

func main() {
	fmt.Println("-- run: three separate ratios with three test sets --")
	sets := []struct{ name, text string }{
		{"T1", testTextT1},
		{"T2", testTextT2},
		{"T3", testTextT3},
	}
	for _, set := range sets {
		dir := Setup(map[string]string{"inventory.go": inventoryText, "inventory_test.go": set.text})
		output, _ := Run(dir, nil, "test", "-coverprofile=coverage.out", "./...")
		profile, _ := os.ReadFile(filepath.Join(dir, "coverage.out"))
		fmt.Println(set.name, "coverage ratio:", coverageRatio(output),
			"uncovered block count:", uncoveredBlockCount(profile))
	}
}

-- run: three separate ratios with three test sets --
T1 coverage ratio: 57.1% of statements uncovered block count: 3
T2 coverage ratio: 85.7% of statements uncovered block count: 1
T3 coverage ratio: 100.0% of statements uncovered block count: 0

TM19. The case count could also be counted from the source in the previous two lessons, and it can be counted here too — T1 is one call, T2 three, T3 four. But the coverage ratio itself cannot be read from the source at all: while the same Label function stays unchanged, the ratio climbs from 57.1% to 100.0% only because which test set calls it changes. The profile file produced with the -coverprofile flag also gives which blocks this ratio comes from: uncoveredBlockCount reads each line of the profile and counts the ones whose last column is zero — this column is how many times that block ran during the run. In T1, three blocks never run (three return lines), in T3, none.

The coverage tool itself produces run information, because which block ran and which did not can only be known once the tests are actually run — anyone reading the source can see that four states exist, but cannot derive from the source alone which of these four states a given test set triggers. This is the last appearance, in this topic, of the distinction measured in Table-Driven Tests: one number (case count) sits in the source, another number (coverage ratio) only emerges by running it.

It is worth noting that all three sets are run with the same routines — Setup, Run; the only thing that changes is the content of the inventory_test.go file. Run is called this time with the -coverprofile flag, and this flag tells go test to write not just a ratio but the raw data behind that ratio to a file as well. -cover gave only the summary ratio (the form used in the previous lesson); -coverprofile records the same information block by block, and this lesson reads that detail with the uncoveredBlockCount routine. The profile file itself sits temporarily in this program’s working directory too, and its path is never printed anywhere — only the number derived from its content is read.

The Limiting Measurement: A Hundred Percent Coverage Does Not Show Correctness

T3 calls all four states, and its last call — Label(Item{"pen", -1, 250}) — never takes or checks the return value at all. In the coverage tool’s eyes, this line is a full test: Label was called, the relevant block ran, the ratio climbed to 100%. But whether this call produced a correct result was never checked anywhere.

It is worth stressing that this is not a typo — anyone reading T3’s source sees the fourth call, confirming that Label‘s fourth state was called. What is missing is that an if got := ...; got != "expected" pattern was never added to that call; this pattern was present through the third call, and was dropped for the fourth. The coverage tool does not notice this drop, because the question it asks is “did this line run,” not “was this line’s result bound to an assertion.“

This connects to the limit measured in Faking and Interface Separation: there it was shown that a fake does not carry all the real one’s assumptions; here it is shown that a test set’s own assertions do not cover all of its own branches — both from the same root: a test’s “passing” carries no guarantee about anything outside the scope of what it tested.

TM20. The measurement below makes this concrete: a broken version of Label — one that also labels a negative quantity as "out of stock", never producing the "invalid" state at all — is run with the same T3 test set.

// boundary.go — what T3, with full coverage, says about a broken source
package main

import (
	"fmt"
	"strings"
)

const brokenInventoryText = `package trial

type Item struct {
	Name     string
	Quantity int
	Price    int
}

func Label(it Item) string {
	if it.Quantity < 0 {
		return "out of stock"
	}
	if it.Quantity == 0 {
		return "out of stock"
	}
	if it.Quantity < 3 {
		return "low stock"
	}
	return "sufficient"
}
`

func main() {
	fmt.Println("-- limiting measurement: what T3, with full coverage, says about a broken source --")
	dir := Setup(map[string]string{"inventory.go": brokenInventoryText, "inventory_test.go": testTextT3})
	output, _ := Run(dir, nil, "test", "-cover", "-v", "./...")
	passed := strings.Contains(output, "--- PASS: TestLabel")
	fmt.Println("test passed:", passed, "coverage ratio:", coverageRatio(output))
}

-- limiting measurement: what T3, with full coverage, says about a broken source --
test passed: true coverage ratio: 100.0% of statements

TM21. In the broken version, a negative quantity now returns "out of stock" instead of "invalid" — a real defect, because these two states can carry different meanings elsewhere in the inventory package (an item that ran out of stock and an item whose stock count is broken are not the same thing). But T3 never catches this defect at all: the test still passes, coverage is still 100.0%. The reason is simple — T3’s last line calls Label but does nothing at all with the value it returns; the coverage tool sees that this line ran, it never asks whether the line that ran produced a correct value.

The coverage ratio of both runs (correct source, broken source) is also byte-for-byte the same: 100.0% of statements in both. The coverage tool sees no difference at all between the two sources, because the difference is not in whether a line ran, but in which value a line that did run produced. This is the same limit that applies to every piece of tool information in this topic, applied to coverage: a tool answers only the question it itself asks, and the question the coverage tool asks is always the same — did this line run — it never asks which value it produced.

This is the last form of the same limit repeated across the topic’s six lessons: in Test Package Basics a test that verified nothing still passed; here a single call that verifies nothing carries the coverage ratio of a test set whose other three assertions are sound all the way to 100%. The coverage tool counts blocks that ran, not blocks that were correctly tested — and these two numbers, as in the T3 example, can look identical while saying something different. A fully covering test set is a guarantee that every line of the source ran at least once; it is not a guarantee that every line of the source ran correctly.

This has a concrete production-side counterpart: as a codebase’s coverage ratio approaches 100%, the people adding to it tend to read that ratio as a sign of completeness. The measurement above shows this reading is incomplete — the ratio says the lines ran, it does not say the values those lines produced were verified. A codebase can misbehave even with a hundred percent coverage, because the coverage tool itself never asks whether an assertion exists at all — it only counts whether a line ran or not. This is the carryover, into coverage ratio, of the claim set up in Test Package Basics that a passing test does not say it is correct: a high coverage ratio does not state correctness either, it only states what was run.

This closing measurement shows that coverage itself is also an indicator, not a target: reaching 100% is easy, seeing which questions are still left unanswered while reaching it is hard. The coverage tool does not solve this difficulty, it only shows which lines were never run at all — the rest, whether an assertion actually checks the value that line produced, is left to the person who wrote it.

Summary

• When the same Label function was tested with three test sets, coverage ratio took three different values (57.1%, 85.7%, 100.0%); case count could be counted from the source, coverage ratio could only be learned by running it.
• The coverage profile also gives which block never ran at all; the uncovered block count dropped from 3 to 0 as the test set widened.
• The limiting measurement: a test set that calls all four states but never checks its last call’s result still passed and still gave 100% coverage when run against a broken source.
• The coverage tool counts lines that ran, not lines that were correctly tested — this distinction is the last form of the same limit repeated across the topic’s six lessons.
• The fake dependency family, the table-driven test pattern, and coverage measurement were built in the Unit Testing and Test-Driven Development course; this lesson only measured Go’s own coverage tool.

Next Step

The same question was asked six times across this topic: a test’s verdict, a case list, an HTTP handler’s response, a fake dependency’s limit, which number of a benchmark is reproducible, and now what coverage ratio counts — all were classified with a source, tool, or run label, and each of the six lessons showed its own limiting measurement by running it: a passing test does not say it is correct, adding a case does not raise coverage, a test helper tests the handler and not the network, a fake implementation does not carry all the real one’s assumptions, allocation count cannot be compared without saying what it counts, and now coverage counts lines that ran, it does not say they were correctly tested. The next topic asks the same question of memory: a profile, too, just like coverage, sees the path that ran and states where memory was allocated — but it does not say why it was slow.