Static Analysis

The previous lesson made the test stable. A test verifies by running code; some code, however, can be said to be wrong without ever being run. An unused resource, a stream that was not closed, and an API called on the wrong version are examples. The linter does this job automatically. The unit of work measured in this lesson is a rule violation, and the measurement is taken per module. The course’s question is the same: in how many samples does this measurement exceed its budget? The answers are within, minority, majority, and all; this time the sample is a module.

The core is the same as in the previous lessons.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

Reading Code Without Running It

Static analysis reads the source code without running it and checks it against a list of rules. Everything it finds is called a violation. A violation can be a defect, it can be a warning, or it can be a false positive of the linter: the code is correct but the rule reads it wrongly. Telling these three apart is the job of the team, not of the linter.

The behavior of the linter comes from a configuration file: which rule is on and which is counted as an error is written there. The violations accepted as a baseline are kept in a separate file.

<!-- Platform code: lint.xml -->
<lint>
    <issue id="UnusedResources" severity="error" />
    <issue id="HardcodedText" severity="warning" />
    <issue id="OldTargetApi" severity="error">
        <ignore path="legacy/**" />
    </issue>
</lint>

This piece belongs to the platform: which rule is counted as an error, which stays a warning, and which path is exempted from a rule. The model does not read the configuration; it takes the number of violations per module from a fixed table and compares it with the budget.

TE1. In the model, the violation counts of eight modules are a fixed table; in a real project the numbers vary with the linter’s version and rule set. The table represents a single check made with the same rule set.

Violations per Module

The helper file defines the modules and the number of violations. Each module has two numbers: the violations that existed before the baseline and the newly added violations. In addition, the violations for which the linter gave a false positive are counted separately in each module.

// Lint.kt — the violation table of the modules
class Module(val name: String, val existing: Int, val added: Int, val falseAlarms: Int)

val modules: List<Module> = listOf(
    Module("app", 6, 2, 1),
    Module("core", 3, 0, 0),
    Module("data", 7, 1, 2),
    Module("ui", 9, 3, 1),
    Module("network", 2, 0, 0),
    Module("auth", 5, 1, 0),
    Module("feature-cart", 8, 0, 2),
    Module("feature-search", 4, 2, 1),
)

fun lintFindings(m: Module): Int = m.existing + m.added

fun cleanedFindings(m: Module): Int = lintFindings(m) - m.falseAlarms

fun newFindings(m: Module): Int = m.added

lintFindings is the raw number of violations of the module. cleanedFindings is the number after the false positives have been filtered out. newFindings is what remains after the baseline: the newly added violations. The team allows each module to carry at most five violations.

// Main.kt
fun main() {
    val samples = modules.map { lintFindings(it) }
    println("findings: $samples")

    val overrun = Overrun("raw-findings", Budget("finding", 5), samples)
    overrun.line()
    countOverruns(listOf(overrun))
}

findings: [8, 3, 8, 12, 2, 6, 8, 6]
raw-findings: budget=finding<=5 | over=6/8 -> majority
within: 0
minority: 0
majority: 1
all: 0

Six of the eight modules are above five violations. Only core and network are within the budget. The measurement is majority. The rule does not see the magnitude this time either: ui with twelve and auth with six count as having exceeded in the same way. It does not see the cause either: one module may have piled up its violations over years, and another may have been written within a week.

Filtering Out False Positives

The first reaction is that some violations are false positives. That is true: there are seven false positives across the eight modules. If they are filtered out, the count changes. The question is in how many modules this rescues the budget.

// Main.kt — after the false positives are filtered out
fun main() {
    val samples = modules.map { cleanedFindings(it) }
    println("filtered findings: $samples")

    val overrun = Overrun("filtered-findings", Budget("finding", 5), samples)
    overrun.line()
    countOverruns(listOf(overrun))
}

filtered findings: [7, 3, 6, 11, 2, 6, 6, 5]
filtered-findings: budget=finding<=5 | over=5/8 -> majority
within: 0
minority: 0
majority: 1
all: 0

Five modules are still above the budget, and the measurement stayed majority. feature-search dropped to five violations and stayed at the limit of the budget; because only strictly exceeding counts, it has no longer exceeded. Filtering out the false positives lowered the number but did not change the class. The false positives were not the main cause of the overrun.

The Kinds of Violations

A linter does not give the same weight to everything it finds. Each rule has a severity, and the configuration file sets it. A violation at the error level stops the continuous integration pipeline; a violation at the warning level appears in the report but does not stop the pipeline; a violation at the information level is only a suggestion. The same number of violations carries very different meanings depending on the distribution of severity. In this lesson the measurement does not separate severity: all violations count as one unit. This is a simplification, and in a real project the team can track those at the error level with a separate budget.

Violations also have two separate origins. Some rules catch genuinely wrong code: a stream that is not closed, a missing permission in the manifest, an unused resource. Others enforce a style preference: hardcoded text or naming. The first group aims at defects, the second at conformity. The team must know which of them it counts toward the budget; otherwise a budget overrun can be read as neither a defect nor a style issue.

Reading the Configuration Piece Line by Line

The root element <lint> of the lint.xml file defines the scope of the configuration. Each <issue> element names a rule, and the id field is the identity of the rule. The first element raises the UnusedResources rule to the error level: an unused resource now stops the pipeline. The second element leaves the HardcodedText rule at the warning level: hardcoded text appears in the report but does not stop the pipeline.

The third element makes the OldTargetApi rule an error and carries an <ignore path="legacy/**" /> line inside. This line says that the files under the legacy directory will not be checked for this rule. This is not a baseline; it is not applying the rule to a certain path. The difference between the two is important. An exemption is a decision and stays visible in the configuration. A baseline is a record of the past and is kept separately as a file.

The Causes of False Positives

A false positive means the linter misunderstood the code. A rule looks only at the structure of the code and does not know the runtime. A resource used only through reflection makes the rule think that the resource is unused. If a function is called dynamically from another module, the rule may count it as dead code. In these cases the right remedy is not to fix the violation in the code but to exempt the line or the path deliberately.

Filtering out false positives has two risks. The first is that real violations are filtered out as false positives; the team loosens the limit to lower the number. The second is that the filtering is not documented; a year later nobody knows why it was filtered out. In this lesson we kept the number of false positives explicitly separate in the table, because making the filtering visible keeps the budget honest.

The Risks of a Baseline

A baseline file is a ledger of debt, and like every ledger of debt it can grow. Three risks stand out.

The first is invisible debt. The 44 violations that are hidden are not discussed in team meetings, because they are not in the report. Counting only new violations makes people forget that accumulated debt exists. For this reason the hidden number is reported together with the visible number.

The second is fragility of matching. A baseline ties a violation to a file and a rule id. If the file is reorganized or the code is moved, the same violation does not match the baseline and shows up again as a new violation. It looks like a false positive, but its cause is the change in the code.

The third is the renewal of the baseline. If the team one day produces the baseline from scratch, all the violations added up to that day turn into old debt. This operation should be recorded as a decision and its reason written; otherwise the baseline turns into a silent way of resetting the budget.

Where the Check Sits in the Pipeline

Static analysis can run the same rules in three places, and each has a different trade-off. If it runs on the developer’s local machine, while the code is written or saved, the feedback is fastest, but there is no guarantee that everyone on the team runs with the same version and settings. If it runs in a pull request before a change is merged, it is known that everyone is checked with the same settings, and a new violation is caught before merging. In a full check that runs at night, the raw violation counts of all modules are collected.

The two numbers measured in this lesson correspond to two different places. The new-violation budget suits the pull request check: a change may bring at most one new violation. The raw budget of five violations per module suits the full check that runs at night: the total debt of the module is tracked.

Tightening the Budget over Time

While violations have accumulated, it is not possible to pull the budget to zero all at once. Teams usually lower the budget gradually: five violations per module today, four after three months. With each drop the number of exceeding modules may rise, because the limit has tightened. In this lesson the effect of moving from 5 to 4 can be seen: in the filtered count feature-search stayed at the limit with five violations, and if the budget goes down to four, the number of exceeding modules rises to six.

Tightening and improvement are not the same thing. When the number of exceeding modules goes up, the team may think the code has gotten worse; yet the limit has changed. For this reason a change of budget is stated as a separate note in the measurement report. Improvement is the number of exceeding modules going down with the same budget.

Rule Selection

As the rule set grows, the number of violations grows, but the value does not grow at the same rate. If the team turns on all the rules, every module exceeds the budget and the measurement comes out all; all no longer distinguishes anything. If all the rules are turned off, no module exceeds and the measurement says nothing. A useful rule set consists of the rules the team is really willing to fix. Keeping a rule on that will not be fixed only produces noise.

One way to add a rule is a trial period. A new rule is first turned on at the warning level, the number of violations is tracked for a few weeks, and the false positive rate is observed. If the rule finds useful violations, it is raised to the error level; if it produces noise, it is turned off or narrowed with an exemption specific to a path. This order keeps the rule from appearing to the team as a sudden obstacle, and it preserves the meaning of the budget: only the rules the team has adopted count toward the budget.

A violation of a rule that has not been adopted inflates the budget and makes it harder for real problems to show. For this reason reviewing the rule set is as regular a job as reviewing the budget; every quarter the team asks which rules are working and removes the ones that are not from the set.

Common Misconceptions

The first misconception is the thought that “if the number of violations is zero, the code is correct”. The linter knows only the rules that are defined. A logic error, a wrong calculation, or a security hole often falls outside the rule set. Zero violations means there are no violations that the rules can find.

The second misconception is the thought that “a baseline fixes the violations”. We saw this in the measurement: 44 violations stayed in the baseline, the count changed, the code did not. A baseline is a measuring tool, not a repair tool.

The third misconception is the thought that “the linter is useless because there are many false positives”. Seven false positives are a small part of the forty-four violations. In the measurement, filtering out the false positives brought the number of exceeding modules from six to five; that is, the false positives were real but they were not the main cause of the overrun. Giving up the linter means giving up the real violations as well.

The fourth misconception is the thought that “the budget must be the same in all modules”. A core module and an experimental module may not be measured with the same expectation. But separating the budget per module muddles the measurement: if each module has its own limit, the number of exceeding modules no longer answers the same question. In this lesson we set a single budget and we state openly that this is a simplification.

The Bounding Measurement

The bounding measurement here is the baseline file. The reasoning is this: the raw count mixes debt and new violations. Taking the old violations out of the count re-measures the same table of modules in a way that answers another question. The budget fits this question as well: the new-violation budget is 1, because each change should be able to bring at most one new violation.

It is not possible to fix the accumulated violations overnight. The common remedy is the baseline file: the linter records that day’s violations and in later checks reports only the new violations. The old violations drop out of the count. The budget is also set according to the new violations: at most one per module.

// Main.kt — only new violations with the baseline file
fun main() {
    val samples = modules.map { newFindings(it) }
    println("new findings: $samples")

    val hidden = modules.sumOf { it.existing }
    val visible = modules.sumOf { newFindings(it) }
    println("hidden in baseline: $hidden")
    println("visible: $visible")

    val overrun = Overrun("baseline", Budget("new-finding", 1), samples)
    overrun.line()
    countOverruns(listOf(overrun))
}

new findings: [2, 0, 1, 3, 0, 1, 0, 2]
hidden in baseline: 44
visible: 9
baseline: budget=new-finding<=1 | over=3/8 -> minority
within: 0
minority: 1
majority: 0
all: 0

The measurement came from majority to minority: only app, ui, and feature-search exceed the new-violation budget. But the reason for this is not that the violations were fixed; it is that the count changed. The 44 hidden violations stand where they stood. A baseline file does not fix a violation; it changes what the linter looks at. These two measurements answer different questions: the first asks “how much debt is in this module”, the second asks “what was newly added to this module”. The two do not stand in for each other.

TE2. In the model, the baseline is a single number: the earlier number of violations. A real baseline file records the violations one by one; when the code changes, a violation may not match its record and may show up again as a new violation.

Summary

• This lesson’s overrun: majority. Six of the eight modules exceeded the budget of five violations per module.
• Filtering out false positives brought the number of exceeding modules from six to five but left the measurement at majority; the main cause was not the false positives.
• A baseline file does not fix violations, it changes the count: 44 violations were hidden, 9 new violations became visible, and the measurement came to minority.
• The raw count answers “how much debt is there”, and the count with a baseline answers “what was added”; the two are read together.
• The rule does not see the magnitude or the cause of a violation; it only counts in how many modules the budget is exceeded.

Next Step

Testing and checking examined the correctness and the order of the code. How the app first appears to the user is a separate measurement. The next topic begins with Startup Time, a lesson that measures in how many samples the work done while the app opens exceeds its startup budget.