Static Application Security Testing

Every measurement up to this point assumed the system’s own legitimate user. The load generator sent valid requests, the scenario imitated a realistic reader, and bottleneck analysis asked where a correctly working path slowed down. No measurement asked what a client that deliberately misuses the system would find.

The cheapest place to ask this question is the source code. Static application security testing looks at a program’s text without running it: which calls it makes, how it builds its queries, which constant values sit inside the code. This lesson’s subject is not the vulnerabilities themselves — their client-side counterparts were built in the Frontend Quality course, and the risk of a concatenated query in the Advanced SQL course — it is measuring the scan itself: how many defects it finds, how many warnings turn out empty, and which defect never shows up at any threshold.

The Codebase and Known Defects

Six files of the lending system will be scanned. The location of each defect has been manually verified; the scanner’s result will be compared against this list.

NF7 (assumption): the manually verified defect list is complete. A line absent from the list is counted as clean. In a real codebase this assumption does not hold; the measured ratios are ratios relative to this list.

// source.mjs — the codebase to scan for the lending system and the manually verified defect list
export const CODE = {
  "catalog.mjs": [
    `const SEARCH = "SELECT isbn, title FROM book WHERE title LIKE ?";`,
    `export const search = (db, q) => db.prepare(SEARCH).all("%" + q + "%");`,
    `export const sort = (db, field) =>`,
    `  db.prepare("SELECT isbn FROM book ORDER BY " + field).all();`,
  ],
  "loan.mjs": [
    `const DAYS = 21;`,
    `export function lend(db, memberId, isbn) {`,
    `  const s = "INSERT INTO loan(memberId, isbn, days) VALUES(?, ?, " + DAYS + ")";`,
    `  return db.prepare(s).run(memberId, isbn);`,
    `}`,
    `export const history = (db, memberId) =>`,
    "  db.prepare(`SELECT * FROM loan WHERE memberId = '${memberId}'`).all();",
  ],
  "member.mjs": [
    `const token_key = "k9TdQ2vRm7XzP4bH";`,
    `export const sign = (memberId) => memberId + ":" + token_key.length;`,
    `export const password_field = "user-password";`,
  ],
  "notification.mjs": [
    `export function format(template, data) {`,
    "  return new Function(\"v\", \"return `\" + template + \"`\")(data);",
    `}`,
    `// eval( ) call removed from this file`,
  ],
  "report.mjs": [
    `const rule = { late: "days > 21", penalty: "days * 2" };`,
    `export const apply = (name, v) => eval(rule[name]);`,
    `export const headers = ["memberId", "isbn", "days"];`,
  ],
  "config.mjs": [
    `export const secret_path = process.env.LOAN_KEY;`,
    `export const channel_password = "password123";`,
    `export const key_name = "LOAN_KEY";`,
    `export const server = "127.0.0.1";`,
  ],
};

// Manually verified defects: file, line (from 1), class.
export const ACTUAL = [
  { file: "catalog.mjs", line: 4, class: "concatenated-query" },
  { file: "loan.mjs", line: 7, class: "concatenated-query" },
  { file: "member.mjs", line: 1, class: "hardcoded-secret" },
  { file: "notification.mjs", line: 2, class: "forbidden-call" },
  { file: "report.mjs", line: 2, class: "forbidden-call" },
  { file: "config.mjs", line: 2, class: "hardcoded-secret" },
];

export const lineCount = Object.values(CODE).reduce((t, s) => t + s.length, 0);

The scanner consists of three rules. Each rule carries a pattern and a score; the score is the scanner’s own estimate of the probability that a finding is a real defect.

// scanner.mjs — three-rule static scanner; every finding carries a score
const secret = /\b(\w*(?:key|password|token|secret)\w*)\s*=\s*["']([^"']{8,})["']/i;
const query = /(SELECT|INSERT|UPDATE|DELETE)[^"'`]*["'`]\s*\+\s*(\w+)/i;

export const RULE = [
  {
    name: "forbidden-call", pattern: /\beval\s*\(|\bnew\s+Function\s*\(/,
    score: (s) => (s.trimStart().startsWith("//") ? 20 : 90),
  },
  {
    name: "concatenated-query", pattern: query,
    // An all-uppercase name is a constant; a lowercase name could be an outside value.
    score: (s) => (/^[A-Z0-9_]+$/.test(s.match(query)[2]) ? 30 : 80),
  },
  {
    name: "hardcoded-secret", pattern: secret,
    // A long, mixed value is likely a generated secret; a short or uniform one is usually a name.
    score: (s) => {
      const d = s.match(secret)[2];
      return d.length >= 16 && /[a-z]/.test(d) && /[A-Z]/.test(d) && /\d/.test(d) ? 85 : 40;
    },
  },
];

export function scan(code) {
  const findings = [];
  for (const [file, lines] of Object.entries(code)) {
    lines.forEach((s, i) => {
      for (const r of RULE) if (r.pattern.test(s)) findings.push({ file, line: i + 1, class: r.name, score: r.score(s) });
    });
  }
  return findings;
}

export const comparisons = (code) =>
  Object.values(code).reduce((t, s) => t + s.length, 0) * RULE.length;

Findings

// findings.mjs — the scanner's findings and each one's match against the actual defect list
import { CODE, ACTUAL, lineCount } from "./source.mjs";
import { scan, comparisons, RULE } from "./scanner.mjs";

const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));
const findings = scan(CODE).sort((a, b) => b.score - a.score);

console.log(`${Object.keys(CODE).length} files, ${lineCount} lines, ${RULE.length} rules -> ${comparisons(CODE)} comparisons`);
console.log(`${findings.length} findings, ${ACTUAL.length} manually verified defects\n`);
console.log(`${"location".padEnd(21)}${"class".padEnd(23)}score  actual`);
for (const b of findings) {
  console.log(`${key(b).padEnd(21)}${b.class.padEnd(23)}${String(b.score).padStart(5)}  ${actualSet.has(key(b)) ? "yes" : "no"}`);
}

const found = new Set(findings.map(key));
const missed = ACTUAL.filter((g) => !found.has(key(g)));
console.log(`\ndefect that appears at no score: ${missed.map((g) => `${key(g)} ${g.class}`).join(", ")}`);

6 files, 25 lines, 3 rules -> 75 comparisons
9 findings, 6 manually verified defects

location             class                  score  actual
notification.mjs:2   forbidden-call            90  yes
report.mjs:2         forbidden-call            90  yes
member.mjs:1         hardcoded-secret          85  yes
catalog.mjs:4        concatenated-query        80  yes
member.mjs:3         hardcoded-secret          40  no
config.mjs:2         hardcoded-secret          40  yes
config.mjs:3         hardcoded-secret          40  no
loan.mjs:3           concatenated-query        30  no
notification.mjs:4   forbidden-call            20  no

defect that appears at no score: loan.mjs:7 concatenated-query

Five of the nine findings are real, four are false positives. What the false positives share is that the pattern matched but the context was innocent: a call name inside a comment line, query text concatenated with a constant, and two variables carrying a field name rather than a secret value. The scanner sees the line; it does not see what the line means.

The sixth defect never appeared at any score. The query inside loan.mjs does not use a concatenation operator; it places the value inside a template literal. The pattern looks for " +, and no such sequence exists there. This is the missed class: code that writes the same risk in a different syntax. Extending the rule is possible, but every extension brings a new false positive; the next section counts that trade-off.

Threshold Scanning

Not every score is carried to the gate. A threshold is chosen, and findings below it are listed but do not stop the release. Where does the threshold come from?

NF8 (assumption): the cost of a missed defect is five times the cost of reviewing a false positive. This figure comes not from a measurement but from an acceptance, and throughout the lesson the weight is called w. The threshold is derived directly from this weight.

// threshold.mjs — threshold scan: false positives, false negatives and cost under two weights at every threshold
import { CODE, ACTUAL } from "./source.mjs";
import { scan } from "./scanner.mjs";

const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));
const findings = scan(CODE);

// NF8: the cost of a missed defect is WEIGHT times the cost of reviewing a false positive.
const measure = (threshold) => {
  const remaining = findings.filter((b) => b.score >= threshold);
  const falsePositive = remaining.filter((b) => !actualSet.has(key(b))).length;
  const seen = new Set(remaining.filter((b) => actualSet.has(key(b))).map(key)).size;
  return { threshold, remaining: remaining.length, falsePositive, falseNegative: ACTUAL.length - seen };
};

const THRESHOLDS = [20, 30, 40, 50, 60, 70, 80, 90];
const rows = THRESHOLDS.map(measure);
console.log(`${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=1".padStart(10)}${"cost w=5".padStart(10)}`);
for (const r of rows) {
  console.log(`${String(r.threshold).padStart(9)}${String(r.remaining).padStart(11)}${String(r.falsePositive).padStart(16)}${String(r.falseNegative).padStart(16)}${String(r.falsePositive + r.falseNegative).padStart(10)}${String(r.falsePositive + 5 * r.falseNegative).padStart(10)}`);
}

for (const w of [1, 5]) {
  const best = rows.reduce((a, b) => (b.falsePositive + w * b.falseNegative < a.falsePositive + w * a.falseNegative ? b : a));
  console.log(`w=${w} -> lowest-cost threshold ${best.threshold} (false positive ${best.falsePositive}, false negative ${best.falseNegative})`);
}

threshold  remaining  false positive  false negative  cost w=1  cost w=5
       20          9               4               1         5         9
       30          8               3               1         4         8
       40          7               2               1         3         7
       50          4               0               2         2        10
       60          4               0               2         2        10
       70          4               0               2         2        10
       80          4               0               2         2        10
       90          2               0               4         4        20
w=1 -> lowest-cost threshold 50 (false positive 0, false negative 2)
w=5 -> lowest-cost threshold 40 (false positive 2, false negative 1)

In the table, false fail is the false-positive column: a clean line turns the gate red. False pass is the false-negative column: a real defect passes through the gate. The two columns move in opposite directions, and there is no outright winner between them.

The weight decides. A team that prefers two reviews over one miss keeps the threshold at 40; a team for which review is expensive raises it to 50 and knowingly lets two defects pass through the gate. Same scanner, same code, same finding set — a different threshold. The threshold’s source is NF8, not a measurement; what gets measured is the threshold’s result.

Gate and Remediation

Threshold 40 is chosen, and the scan runs as the quality gate defined in the Quality and Testing Fundamentals course. The gate turns red, four defects are fixed, and the code is scanned again.

// gate.mjs — rescanning after the fix and the quality gate running at threshold 40
import { CODE, ACTUAL } from "./source.mjs";
import { scan } from "./scanner.mjs";

const THRESHOLD = 40;
const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));

// The fixed form of four defects; loan.mjs:7 was deliberately left untouched.
const FIX = {
  "catalog.mjs": { 4: `  db.prepare("SELECT isbn FROM book " + ORDER[field]).all();` },
  "member.mjs": { 1: `const token_key = process.env.LOAN_KEY;` },
  "notification.mjs": { 2: "  return template.replace(/{(\\w+)}/g, (_, a) => data[a]);" },
  "report.mjs": { 2: `export const apply = (name, v) => RULES[name](v);` },
  "config.mjs": { 2: `export const channel_password = process.env.CHANNEL_PASSWORD;` },
};

const fixed = Object.fromEntries(Object.entries(CODE).map(([d, s]) =>
  [d, s.map((line, i) => FIX[d]?.[i + 1] ?? line)]));

const gate = (code, suppressed = []) => {
  const remaining = scan(code).filter((b) => b.score >= THRESHOLD && !suppressed.includes(key(b)));
  return { status: remaining.length === 0 ? "green" : "red", remaining };
};

for (const [label, code] of [["before the fix", CODE], ["after the fix", fixed]]) {
  const { status, remaining } = gate(code);
  console.log(`${label.padEnd(17)} threshold ${THRESHOLD} -> ${status}, ${remaining.length} findings` +
    `, ${remaining.filter((b) => !actualSet.has(key(b))).length} of them false positive`);
  for (const b of remaining) console.log(`  ${key(b).padEnd(21)}${b.class.padEnd(23)}${b.score}`);
}

const SUPPRESSED = ["member.mjs:3", "config.mjs:3"];   // reviewed, a field name; a ticket was opened
console.log(`\nsuppression list ${SUPPRESSED.length} lines -> gate ${gate(fixed, SUPPRESSED).status}`);

const remainingActual = ACTUAL.filter((g) =>
  !Object.keys(FIX[g.file] ?? {}).includes(String(g.line)));
console.log(`actual defect still in the codebase while the gate is green: ${remainingActual.length}` +
  ` (${remainingActual.map((g) => `${key(g)} ${g.class}`).join(", ")})`);

before the fix    threshold 40 -> red, 7 findings, 2 of them false positive
  catalog.mjs:4        concatenated-query     80
  member.mjs:1         hardcoded-secret       85
  member.mjs:3         hardcoded-secret       40
  notification.mjs:2   forbidden-call         90
  report.mjs:2         forbidden-call         90
  config.mjs:2         hardcoded-secret       40
  config.mjs:3         hardcoded-secret       40
after the fix     threshold 40 -> red, 2 findings, 2 of them false positive
  member.mjs:3         hardcoded-secret       40
  config.mjs:3         hardcoded-secret       40

suppression list 2 lines -> gate green
actual defect still in the codebase while the gate is green: 1 (loan.mjs:7 concatenated-query)

The catalog fix reads the sort field from a fixed table; the rule still matches, but because its operand is now an uppercase constant the score drops from 80 to 30 and falls below the threshold. After the fix the gate is still red, and both of the two remaining findings are false positives. This is where static scanning spends the most time: to turn the gate green, either the rule is narrowed or each finding is reviewed one by one and written to the suppression list. Once the list grows by two lines the gate turns green — and a real defect still sits in the codebase.

Who owns the decision: a red gate stops the release, and every suppressed line opens a review record. The suppression list grows over time, and when it is not revisited it silently narrows the area the scan actually watches.

The Cost of Scanning

The run-independent cost lives in three numbers. The first is the scan itself: 25 lines, 3 rules, 75 comparisons; zero processes start, zero requests go out, zero test data is prepared. This is why the scan can run on every change.

The second is review: at threshold 40, 7 findings need a human eye, and 2 of them turn out empty. As the codebase grows this number grows with the line count, not with the defect count.

The third is maintenance: the three rules and the two-line suppression list were written by hand. When a new syntax comes into use, the rule stays silent — loan.mjs:7 is the proof of this — and a rule that stays silent is indistinguishable from a rule that does not exist.

Summary

• Static application security testing scans the source without running it; in this lesson a three-rule scanner produced 9 findings over 6 files and 25 lines, 5 of them real defects.
• The false positives share a common cause, context blindness: a comment line, a query concatenated with a constant, and variables carrying a field name rather than a secret value all tripped the pattern.
• Threshold scanning yields two columns together: at threshold 40, 2 false fails and 1 false pass; at threshold 50, 0 false fails and 2 false passes. The choice derives from NF8’s weight.
• The caught class is a flawed line that matches a pattern; the missed class is code that writes the same risk in a different syntax — the query built with a template literal never appeared at any threshold.
• The cost: 75 comparisons, zero processes, zero requests; in exchange, 7 findings manually reviewed at threshold 40 and the upkeep of a growing suppression list.
• When the gate turned green, a real defect still sat in the codebase; a green gate does not mean flawless code, only that the rules have gone quiet.

Next Step

Everything this scan saw was read from text: which call was written, how the query was built, which value was embedded in the code. Text, however, states what the program writes, not what it does. The call inside report.mjs could sit on a branch that is never reached; the value inside config.mjs could be overridden in production by an environment variable. The reverse also holds: in a system where two files are each clean on their own, the combination of their endpoints can meet a request with a completely unexpected response. The next lesson actually brings the lending service up, sends requests to its endpoints, and measures a single question: how much of the endpoints can a test that exercises the running application reach, and where do the defect classes the two methods see diverge.