--- title: 'Review Metrics' source: 'https://academia.sh/en/courses/code-review/review-metrics' course: 'Code Review and Team Process' language: en updated: '2026-08-17T18:10:46+00:00' license: 'CC BY-SA 4.0' --- # Review Metrics Three reviewers on the same axis produce 6 review rounds, 36 units of wait, and 3 remaining defects; one reviewer on five axes gives 2 rounds, 12 units, and 0 remaining, and the same gap turns into 2 versus 15 rounds of staying open in the review queue. Three lessons arrived at the same two numbers. Change size set the chunks read and what was found; note style set how fast the found closed; the resolution rule set whether it closed at all. In all three, the result was a **review round** count and an amount of **wait**. This lesson takes those two numbers directly as **metrics** and adds a third: what review does not to a single change but to the **review queue**. What is asked is this — what does each of these three numbers measure, which decision does it support, and what does it hide when read alone. ## Three Metrics **Wait time** is the span from the moment a change enters review to the moment it closes. In the shared fixture, every review round adds 6 units to wait; wait is a direct function of review round count. **Review round count** is how many times a change goes back and forth before it closes. It has one source, and the previous three lessons showed it three separate ways: **work that does not close in the first round**. A defect not seen in the first round, a closing criterion not written, and an argument not resolved all land in the same place. **Review queue health** is a metric not of a single change but of all changes open at the same time. A team's review capacity is limited: only a certain number of changes can be reviewed at once. As every change takes more review rounds, capacity stays tied up longer and changes coming in behind it wait. Queue health shows whether this backlog is forming. ## Why Metrics Are Not Reported Per Person These three metrics are the process's metrics, and they are not reported per person, because what produces every row of the table is not people's effort but the **axis assignment** — three reviewers producing 6 review rounds happens even though all three do their job flawlessly. The measurement's second row proves this. There are three reviewers, all three find every defect on the axis they are responsible for, and the result is **6** of 24 defects. If a per-person number were produced, all three reviewers would look weak; yet the only thing that needs to change is that all three are looking at the same axis. When a metric is tied to the wrong subject, the fix goes to the wrong place too. A second justification comes from the previous lesson. When note style changed, the number of defects found never changed — it stayed fixed at **22**. That is, note count and defects-found count are independent of each other. In a system that reports note count per person, what grows is note count, not defects found. The measurement's assumptions: - **RC20** — The shared fixture does not change: 600 lines, 12 chunks, 24 defects, fixed class distribution. The only thing that changes is the **axis assignment**. - **RC21** — Four assignments are measured: one reviewer on all axes; three reviewers on the same axis; three reviewers on separate axes; five reviewers on five axes. Every reviewer's attention is **12** chunks. - **RC22** — Found defects are fixed in every review round; the remaining defect carries to the next round, and each round clears half of what was found. A change can stay open for at most **6** review rounds. - **RC23** — One review round adds **6 units** to wait, independent of assignment. - **RC24** — In the review queue, one change enters per round; **12** changes enter in total. The team's capacity is **3** simultaneous reviews. - **RC25** — All changes in the queue are reviewed with the same assignment; this is why every one of them takes the same review round count. - **RC26** — The queue measurement is tracked across **60** rounds; this span is enough for all changes to close in all four of the four assignments. - **RC27** — The set's resolution is **1/24 = 0.042** at 24 defects; wait and review round are counted per unit. ## Measurement ```python """Review metrics: found, review round, wait, and queue health. Part 1 - what four axis assignments find and miss. Part 2 - the same assignments' review round, wait, and remaining defects. Part 3 - what the same rounds do to the review queue. """ SEED = 20260815 AXES = ("interface", "implementation", "test", "documentation", "style") UNWRITTEN = "unwritten requirement" CLASSES = AXES + (UNWRITTEN,) ATTENTION = 12 CHUNK = 50 CAPACITY = 3 # changes reviewable at the same time INCOMING = 12 # changes entering the queue during the measurement def rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def change(lines, defect_count=24, seed=SEED): r, defects = rng(seed), [] chunk_count = max(1, lines // CHUNK) for i in range(defect_count): defects.append({"no": i + 1, "class": CLASSES[r(6)], "chunk": r(chunk_count)}) return {"lines": lines, "chunks": chunk_count, "defects": defects} def review(d, axes, attention=ATTENTION): read = set(range(min(attention, d["chunks"]))) return {k["no"] for k in d["defects"] if k["class"] in axes and k["chunk"] in read} def panel(d, assignments, attention=ATTENTION): found = set() for axes in assignments: found |= review(d, axes, attention) return found def round_cycle(found, defect_count, wait_per_round=6): """Each round fixes what is found; the remainder carries to the next round.""" remaining, rnd, wait = defect_count - len(found), 1, wait_per_round while remaining > 0 and rnd < 6: rnd += 1 wait += wait_per_round remaining -= max(1, len(found) // 2) return rnd, wait, max(0, remaining) def queue(rnd, incoming=INCOMING, capacity=CAPACITY, duration=60): """One change enters per round; at most `capacity` are reviewed at once.""" pending, working, done, snapshot = [], {}, [], 0 for t in range(1, duration + 1): if t <= incoming: pending.append(t) for no in [n for n, b in working.items() if b == t]: done.append((no, t)) del working[no] while pending and len(working) < capacity: working[pending.pop(0)] = t + rnd if t == incoming: snapshot = len(pending) + len(working) return snapshot, max(b - g for g, b in done), max(b for _, b in done) FULL = set(AXES) ASSIGNMENTS = { "one reviewer, all axes": [FULL], "three reviewers, same axis": [{"implementation"}] * 3, "three reviewers, separate axes": [{"interface"}, {"implementation"}, {"test", "documentation"}], "five reviewers, five axes": [{a} for a in AXES], } d = change(600) print(f"{'assignment':<32s} {'reviewers':>9s} {'found':>7s} {'missed':>6s} " f"{'unwritten missed':>17s}") for name, assignment in ASSIGNMENTS.items(): b = panel(d, assignment) missed = [k for k in d["defects"] if k["no"] not in b] unw = sum(1 for k in missed if k["class"] == UNWRITTEN) print(f"{name:<32s} {len(assignment):9d} {len(b):7d} {len(missed):6d} {unw:17d}") print() print(f"{'assignment':<32s} {'round':>5s} {'wait':>6s} {'remaining':>10s}") for name, assignment in ASSIGNMENTS.items(): rnd, wait, remaining = round_cycle(panel(d, assignment), len(d["defects"])) print(f"{name:<32s} {rnd:5d} {wait:6d} {remaining:10d}") print() print(f"review queue: one change enters per round, {INCOMING} changes, " f"capacity {CAPACITY}") print(f"{'assignment':<32s} {'round':>5s} {'open at round 12':>17s} " f"{'longest open':>13s} {'last close':>11s}") for name, assignment in ASSIGNMENTS.items(): rnd = round_cycle(panel(d, assignment), len(d["defects"]))[0] snapshot, longest, last = queue(rnd) print(f"{name:<32s} {rnd:5d} {snapshot:17d} {longest:13d} {last:11d}") ``` ``` assignment reviewers found missed unwritten missed one reviewer, all axes 1 22 2 2 three reviewers, same axis 3 6 18 2 three reviewers, separate axes 3 15 9 2 five reviewers, five axes 5 22 2 2 assignment round wait remaining one reviewer, all axes 2 12 0 three reviewers, same axis 6 36 3 three reviewers, separate axes 3 18 0 five reviewers, five axes 2 12 0 review queue: one change enters per round, 12 changes, capacity 3 assignment round open at round 12 longest open last close one reviewer, all axes 2 2 2 14 three reviewers, same axis 6 9 15 27 three reviewers, separate axes 3 3 3 15 five reviewers, five axes 2 2 2 14 ``` ## What Lengthens the Round Reading the first two tables side by side gives the topic's conclusion. When three reviewers work the same axis, **6** defects are found, the process costs **6** review rounds and **36** units of wait, and **3** defects still remain in the end. When one reviewer looks at five axes, **22** defects are found, the process costs **2** review rounds and **12** units, remaining is **0**. The comparison matters because the second row has **three times** as many reviewers. Three people produce six times the wait and end up unable to deliver three defects; one person does the same job with a third of the wait, and flawlessly. **What lengthens the review queue is not reviewer count, it is what goes unseen in the first round.** The link is straightforward arithmetic. If little is found in the first round, a lot of defects remain; the remaining defects clear in later rounds at only half the speed of what was found; so finding little means both more review rounds and defects left at the limit. When three reviewers spread across separate axes, **15** are found and the process closes at **3** rounds / **18** units: same three people, only the axis spread changed. The rightmost missed column is **2** in every row. All four of the four configurations miss the two unwritten requirement defects. This number does not fall by adding reviewers or by adding axes — no axis searches for what was never written. In reading the metrics, this row serves as a **floor**: **2** of 24 defects sit where review can never reach, and a process's target is **22**. A reading that does not know the floor calls every review incomplete and sets an unreachable number as the goal; a metric's ceiling is the ceiling of the process it measures. ## The Health of the Queue The third table shows what the same gap does outside a single change. The team is the same team, capacity is **3** simultaneous reviews, one change enters per round. When review round count is **2**, the queue is healthy: at round twelve only **2** changes are open, the longest open stays **2** rounds, and the last change closes at round **14**. When review round count is **3**, the system sits exactly at the limit — capacity 3, service time 3 — and the open count settles at **3**. When review round count is **6**, the queue breaks down. At round twelve, **9** changes are open; the longest open climbs from **2** rounds to **15**; the last change closes at round **27**. On a single change the gap was 2 versus 6; in the queue it is **2 versus 15**. Once capacity runs out, delay does not just add — it **compounds**. A simple rule of queue health follows from this: **the queue grows once review round count exceeds capacity.** This has nothing to do with how many people are assigned to review, and everything to do with how many rounds a review takes to close — and the previous three lessons each showed, one by one, where that number comes from. The arithmetic of the backlog is simple too. With capacity 3 and service time 6, the system can close at most **0.5** changes per round, while **1** change arrives per round. On top of that, the first closing cannot happen before round **7**; of the 12 changes arrived by round twelve, only **3** have closed and **9** are open. Stopping the inflow does not erase this backlog right away: the last-arriving change reaches its turn at round **21** and closes at round **27**. **The queue does not recover at the speed it broke down.** ## What the Three Metrics Hide Alone No metric is read alone, because each one can take the same value for two separate reasons. **Short wait** can be a good sign or the sign of a change that was never read. In the change size lesson, a 2400-line review also got approved in a single round; defects found was **2**. Read alone, wait makes this review look exemplary. **Few review rounds** comes either from finding a lot in the first round or from finding nothing at all. What separates the two is the second table's last column: remaining defects. In the disagreement lesson, the deferral rule gave **2** rounds / **12** units — the same as the table's best row — but left **8** defects in the code. **A healthy queue** comes either from reviews that close fast or from changes that never enter the queue at all. The remedy for all three is the same: metrics are read **together with found and missed defects**. This is why every table in this topic has carried both sides. ## Which Metric Supports Which Decision A metric's value is only as good as the decision it supports. The three metrics look at three separate decisions and do not substitute for one another. **Review round count** supports the axis-assignment decision. The second table gives this directly: the same three people produce **6** rounds on the same axis, **3** rounds on separate axes. When round count rises, the question to ask is not "who is slow," it is **which axis is going unlooked-at**. **Wait time** supports the change-size decision. Wait is six times the review round count, and what sets review round count is what gets found in the first round; what most reduces the first round's find is the **unread chunk**. Lengthening wait is, most often, the sign of a change sent to review that does not fit inside attention. **Queue health** supports the capacity decision. The third table shows the backlog starting exactly when service time exceeds capacity. The decision here runs two ways: either review round count is lowered, or simultaneous review capacity is raised. The two do not cost the same — the first is paid with axis spread and change size, the second with time taken from other work. Read together, the table reduces to a single sentence: **every fix that raises what is found also fixes wait and the queue.** The measurement has no row that contradicts this; the two configurations that find the most are also the ones with the least wait and the healthiest queue. ## Summary - The three metrics are the process's metrics: wait time, review round count, and review queue health. They are not reported per person, because what produces their values is the axis assignment — three reviewers producing 6 review rounds happens even though all three do their job flawlessly. - Three reviewers on the same axis find **6** defects, cost **6** rounds / **36** units, and leave **3** remaining; one reviewer on five axes finds **22**, costs **2** rounds / **12** units, leaves **0**. Three times the reviewers, three times the wait. - What lengthens the review queue is not reviewer count but **what goes unseen in the first round**: finding little produces both more review rounds and defects left at the limit. - In a queue with capacity 3, the longest open is **2** rounds when review round count is 2, and **15** rounds when it is 6; open changes at round twelve rise from **2** to **9**. Once capacity is exceeded, delay compounds. - All four of the four configurations miss the **2** unwritten requirement defects; this is the floor metrics are read against — the most review can ever reach is **22** of 24. - No metric is read alone: short wait, few review rounds, and a healthy queue can each take the same value in both the best and the worst case; what separates them is defects found and missed. ## Next Step In this topic, review always stayed around a single change: a change arrives, is read, notes are written, it closes. The third table took the first step outside that circle and showed changes waiting on one another. But what if the change itself is not a starting point but an **element inside a longer chain**? The next topic treats review as one stop on that chain: where the element stands before it reaches review and after it closes, and what share of the total time review holds.