Security Requirements

The previous lesson counted that a quarter of the 250 hours prevented comes from items born in the requirement stage, but left the check that catches those items in a single table row: eighteen requirement reviews, five alerts, three real items. How a review catches an item was never opened up. It has exactly one condition for catching one: the requirement has to be written in a form where whether it is met is testable. The line “the system must be secure” produces no item in any review, because there is no observation that would show it has been violated.

This lesson takes testability out of being an adjective and splits it into three parts, then runs it on a list of twenty requirements. In the second half it asks where the list comes from: the requirements written by reading a feature list and the requirements written by considering an actor pursuing a target the system must deny do not produce the same set.

• DA12. The requirement list is fictional and is modeled with node. Each requirement is written with four fields: subject, predicate type, threshold, scope.
• DA13. Testability is not given, it is derived: if the predicate is observable and both the threshold and the scope are written, the requirement is automatically testable. If one of the parts is missing it is checked by eye; if the predicate is qualitative it can never be tested.
• DA14. The predicate-type and mode fields are a record of how the requirement is written, not a judgment of its content’s value. A requirement written untestably is not unimportant; the only thing it lacks is that it cannot be checked.
• DA15. An abuse case is modeled with an actor and a target; no step is given in an executable form. Trying out the scenario outside an organization’s own system requires written authorization and a defined scope; otherwise it is both illegal and unprofessional.
• DA16. The run counts and the manual-check minutes are model values; they are consistent with the change and deployment counts from the previous lessons.

The Three Parts of Testability

For a requirement to be automatically testable, three things are needed: an observable predicate where the violation would be seen, a threshold that says it has been met, and a scope that says where it applies. If one of the three is missing, testing is handed to a human.

// requirement.mjs — security requirements of the measurement network software (model)
// row: name | subject | predicate type | threshold | scope | mode
// predicate: event = observable event, state = auditable state, qualitative = qualitative adjective
// mode: provides = the work the system does, denies = the work the system refuses
const parse = (s) => s.trim().split('\n').map((r) => {
  const [name, subject, predicate, threshold, scope, mode] = r.trim().split(/\s+/);
  return { name, subject, predicate, threshold, scope, mode };
});
export const REQUIREMENT = parse(`
  session-duration-limit        session               state        30-min            subscriber-portal denies
  failed-login-limit            login-endpoint        event        5-attempts        subscriber-portal denies
  transport-encryption          external-connection   state        required          all-endpoints     provides
  measurement-data-retention    measurement-record    state        24-months         measurement-store provides
  authorization-check-coverage  read-endpoint         event        every-request     billing           provides
  secret-in-source              source-file           state        0-count           all-stores        denies
  personal-data-in-logs         log-line              state        0-fields          all-services      denies
  dependency-age                dependency            state        180-days          all-builds        provides
  admin-endpoint-access         admin-endpoint        event        internal-network  meter-collection  denies
  backup-encryption             backup-file           state        required          billing-db        provides
  request-rate-limit            general-endpoint      event        60-requests-min   subscriber-portal denies
  session-termination           logout-action         event        1-request         subscriber-portal provides
  file-upload-size              upload-endpoint       state        10-mb             field-app         denies
  error-message-content         error-response        state        -                 all-endpoints     denies
  audit-log-retention           authorization-change  event        every-change      -                 provides
  password-storage-format       subscriber-record     state        -                 subscriber-portal provides
  authentication-strength       subscriber-login      qualitative  -                 subscriber-portal provides
  interface-security            field-app             qualitative  -                 -                 provides
  data-integrity                measurement-record    qualitative  -                 -                 provides
  separation-of-duties          billing-approval      qualitative  -                 billing           denies`);

// Testability is derived: with an observable predicate + threshold + scope, it is tested automatically.
export function classify(g) {
  if (g.predicate === 'qualitative') return 'untestable';
  const missing = ['threshold', 'scope'].filter((a) => g[a] === '-');
  return missing.length === 0 ? 'automatic' : 'manual';
}
export const missingPart = (g) => ['threshold', 'scope'].filter((a) => g[a] === '-');

// Abuse case: an actor pursues a target the system must deny.
// Written as a model; no step is given in an executable form.
export const SCENARIO = [
  ['subscriber-account-takeover', 'external-actor',
    ['failed-login-limit', 'session-duration-limit', 'session-termination', 'login-notification']],
  ['reading-someone-elses-bill', 'authorized-subscriber',
    ['authorization-check-coverage', 'record-id-guessability']],
  ['altering-a-measurement-record', 'field-staff',
    ['audit-log-retention', 'measurement-write-authorization', 'measurement-change-approval']],
  ['bulk-subscriber-data-pull', 'authorized-subscriber',
    ['request-rate-limit', 'bulk-download-limit', 'outbound-volume-threshold']],
  ['device-registration-spoofing', 'external-actor',
    ['device-registration-verification', 'registration-rate-limit']],
];
// Requirements that come out of the scenarios and are not on the list, written the same way:
export const NEW = parse(`
  login-notification                successful-login        event  every-login            subscriber-portal provides
  record-id-guessability            billing-record          state  not-guessable          billing           denies
  measurement-write-authorization   measurement-record      event  role-matched           measurement-store denies
  measurement-change-approval       measurement-correction  event  two-signatures         measurement-store denies
  bulk-download-limit               external-export         state  -                      subscriber-portal denies
  outbound-volume-threshold         external-export         state  50-mb-day              all-endpoints     denies
  device-registration-verification  device-registration     event  field-approval         field-app         denies
  registration-rate-limit           device-registration     event  10-registrations-hour  -                 denies`);

// run1.mjs — requirements are split by testability; missing parts are counted
import { REQUIREMENT, classify, missingPart } from './requirement.mjs';

const count = (l, s) => l.filter((g) => classify(g) === s).length;
console.log(`${REQUIREMENT.length} requirements; automatic ${count(REQUIREMENT, 'automatic')}, ` +
  `manual ${count(REQUIREMENT, 'manual')}, untestable ${count(REQUIREMENT, 'untestable')}`);

console.log('\nclass       requirements  has threshold  has scope  observable predicate');
for (const s of ['automatic', 'manual', 'untestable']) {
  const l = REQUIREMENT.filter((g) => classify(g) === s);
  console.log(s.padEnd(12) + String(l.length).padEnd(14) +
    String(l.filter((g) => g.threshold !== '-').length).padEnd(15) +
    String(l.filter((g) => g.scope !== '-').length).padEnd(11) +
    l.filter((g) => g.predicate !== 'qualitative').length);
}

console.log('\nrequirements that do not reach automatic, and their missing parts');
for (const g of REQUIREMENT.filter((x) => classify(x) !== 'automatic'))
  console.log('  ' + g.name.padEnd(25) + classify(g).padEnd(12) +
    (g.predicate === 'qualitative' ? 'qualitative predicate: cannot change class without a rewrite'
      : `missing: ${missingPart(g).join(', ')}`));

const singleMissing = REQUIREMENT.filter((g) => classify(g) === 'manual' && missingPart(g).length === 1);
console.log(`\n${singleMissing.length} requirements move to automatic by adding a single part; ` +
  `${count(REQUIREMENT, 'untestable')} requirements need a rewrite`);
console.log(`after the fix, automatic ${count(REQUIREMENT, 'automatic') + singleMissing.length}/` +
  `${REQUIREMENT.length}, remaining ${REQUIREMENT.length - count(REQUIREMENT, 'automatic') - singleMissing.length}`);

20 requirements; automatic 13, manual 3, untestable 4

class       requirements  has threshold  has scope  observable predicate
automatic   13            13             13         13
manual      3             1              2          3
untestable  4             0              2          0

requirements that do not reach automatic, and their missing parts
  error-message-content    manual      missing: threshold
  audit-log-retention      manual      missing: scope
  password-storage-format  manual      missing: threshold
  authentication-strength  untestable  qualitative predicate: cannot change class without a rewrite
  interface-security       untestable  qualitative predicate: cannot change class without a rewrite
  data-integrity           untestable  qualitative predicate: cannot change class without a rewrite
  separation-of-duties     untestable  qualitative predicate: cannot change class without a rewrite

3 requirements move to automatic by adding a single part; 4 requirements need a rewrite
after the fix, automatic 16/20, remaining 4

Thirteen of the twenty requirements are automatically testable, three are checked by eye, four can never be tested. The real information is in what separates the second row from the third. The three requirements checked by eye have observable predicates; what is missing is a single field. error-message-content says what an error response must not contain but writes no threshold; audit-log-retention asks for a log entry of every authorization change but does not say which services it applies to. Both move to automatic by adding one line.

The four untestable ones are a different case. data-integrity or interface-security are not requirements with a missing field; their predicates do not correspond to an observation. Writing a threshold for them would not help either, because it is not clear what would be measured. What they need to change class is not completion but a rewrite: each one has to be translated into an event or a state whose violation can be seen, and this translation usually splits a single requirement into several.

The decision counterpart of the distinction is clear. For three requirements the job is filling in a field; for four requirements the job is a design discussion. After the fix the list splits into 16 automatic and 4 untestable requirements — and these four requirements produce no alert in the code stage no matter how much checking is set up.

A warning is needed here: testability is not correctness. Having written thirty minutes as the session duration limit makes it automatically testable, but it does not show that thirty minutes is the right number; a wrong threshold is tested just as flawlessly. This classification measures whether a requirement is fit for checking, not whether it is well-aimed. Mixing the two up produces a common flaw: the threshold gets written so that it is testable, and where the number came from is never discussed.

What an Abuse Case Adds

The twenty requirements above are written by reading the feature list: they are derived from the work the system must do. An abuse case starts from the opposite direction — an actor pursues a target the system must deny, and the requirements needed to deny that target are written down.

// run2.mjs — how much and in what form abuse cases grow the list
import { REQUIREMENT, NEW, SCENARIO, classify } from './requirement.mjs';

const existing = new Set(REQUIREMENT.map((g) => g.name));
console.log('scenario                       actor                  requirements  on list  new');
let total = 0, inList = 0;
for (const [name, actor, produced] of SCENARIO) {
  const v = produced.filter((x) => existing.has(x)).length;
  total += produced.length; inList += v;
  console.log(name.padEnd(31) + actor.padEnd(23) + String(produced.length).padEnd(14) +
    String(v).padEnd(9) + (produced.length - v));
}
console.log(`${SCENARIO.length} scenarios, ${total} requirement references, ${inList} on the list, ` +
  `${total - inList} new; the list grows ${(((total - inList) / REQUIREMENT.length) * 100).toFixed(0)}%`);

const count = (l, s) => l.filter((g) => classify(g) === s).length;
const modeCount = (l, k) => l.filter((g) => g.mode === k).length;
console.log('\nset                requirements  automatic  manual  untestable  denies  denies %');
for (const [name, l] of [['from feature list', REQUIREMENT], ['from scenarios', NEW]])
  console.log(name.padEnd(19) + String(l.length).padEnd(14) + String(count(l, 'automatic')).padEnd(11) +
    String(count(l, 'manual')).padEnd(8) + String(count(l, 'untestable')).padEnd(12) +
    String(modeCount(l, 'denies')).padEnd(8) + ((modeCount(l, 'denies') / l.length) * 100).toFixed(0));

// Testability class determines which stage a requirement is tested in, and its cost.
const DEPLOY_RUNS = 60, CHANGE_RUNS = 420, REVIEW_MIN = 8;
console.log('\nclass       tested in stage  run count     human minutes');
const all = [...REQUIREMENT, ...NEW];
for (const [s, stage, runs, min] of [['automatic', 'code', CHANGE_RUNS, 0],
  ['manual', 'deployment', DEPLOY_RUNS, REVIEW_MIN], ['untestable', 'runtime', 0, 0]])
  console.log(s.padEnd(12) + stage.padEnd(17) + String(count(all, s) * runs).padEnd(14) +
    count(all, s) * runs * min);
const manualMinutes = count(all, 'manual') * DEPLOY_RUNS * REVIEW_MIN;
console.log(`\n${all.length} requirements: ${count(all, 'automatic')} tested in the code stage, ` +
  `${count(all, 'manual')} in the deployment stage, ${count(all, 'untestable')} never tested`);
console.log(`manual checking's yearly total is ${manualMinutes} minutes (${(manualMinutes / 60).toFixed(1)} hours); ` +
  `if the missing parts are written, all ${(manualMinutes / 60).toFixed(1)} hours drop`);

scenario                       actor                  requirements  on list  new
subscriber-account-takeover    external-actor         4             3        1
reading-someone-elses-bill     authorized-subscriber  2             1        1
altering-a-measurement-record  field-staff            3             1        2
bulk-subscriber-data-pull      authorized-subscriber  3             1        2
device-registration-spoofing   external-actor         2             0        2
5 scenarios, 14 requirement references, 6 on the list, 8 new; the list grows 40%

set                requirements  automatic  manual  untestable  denies  denies %
from feature list  20            13         3       4           9       45
from scenarios     8             6          2       0           7       88

class       tested in stage  run count     human minutes
automatic   code             7980          0
manual      deployment       300           2400
untestable  runtime          0             0

28 requirements: 19 tested in the code stage, 5 in the deployment stage, 4 never tested
manual checking's yearly total is 2400 minutes (40.0 hours); if the missing parts are written, all 40.0 hours drop

Five abuse cases touch fourteen requirements; six of them are already on the list, eight are new. The list grows by forty percent, and where the growth concentrates matters: both of the two requirements the last scenario produces are new. Device registration is a feature on the feature list, and that feature’s requirements are written; being able to register a device under someone else’s name is not a feature, so it cannot be derived from any feature-list line.

A terminology distinction is needed here. A use case writes out, step by step, a use the system supports and wants that use to produce the right outcome. An abuse case uses the same form, but its owner sits on the system’s opposing side, and the outcome it wants is the outcome the system must not give. The second table turns the difference into a number: 45% of the twenty requirements from the feature list are in denying mode, versus 88% of the eight requirements from the scenarios. If the share of denying mode in a requirement list is low, what is missing is not security attention, it is source.

Six of the eight requirements coming from the scenarios are written directly as automatically testable. This is not because the writer was more careful, it is the shape of the problem: a target that must be denied already demands an observation, a threshold, and a scope. The sentence “device registration requires field approval” carries both the event where the violation would be seen and the threshold; “the interface must be secure” carries neither.

There is something the table does not measure, and that is this lesson’s limit: where the five scenarios came from. The scenarios are on the list because someone thought of them. The table has no answer to how many more requirements a sixth scenario would add, because the scenario set itself was not produced by a systematic method. What is measured is the gain per scenario, not the share of targets covered; when the two get conflated, a team that has written five scenarios considers its job done.

The Stage Counterpart of Testability

The last table ties back into this course’s measurement axis. Nineteen of the twenty-eight requirements are automatically testable, so they are tested in the code stage and produce 7,980 runs a year; the human cost of these runs is zero. Five requirements can only be checked by eye, so the point of testing shifts to the deployment stage: 300 checks a year, 2,400 minutes, that is 40 hours.

The two numbers should be read side by side. An automatically tested requirement is tested at the first opportunity after the stage it is born in, and on every change; its violation is caught in the code stage. A requirement checked by eye is only looked at while a deployment is being prepared, and its violation shows up in the previous lesson’s table as an item caught three stages late. Testability looks like a property of how something is written, but its consequence is directly the detection stage.

The four untestable requirements sit in the third row and produce no runs at all. They are not ignored by the system; their violations are only reported by an incident, not a check. In the previous lesson’s language: these four requirements are the requirement-stage source of the item class no check ever sees. A requirement list’s security value is measured not by how many lines it contains, but by how many of those lines correspond to a run count; in this list, that ratio is nineteen out of twenty-eight.

Summary

• Automatic testability is derived from three parts: an observable predicate, a threshold, a scope. If one of the three is missing the requirement is checked by eye; if the predicate is qualitative it can never be tested.
• Of the 20 requirements, 13 are automatic, 3 are checked by eye, 4 are untestable. The 3 checked by eye move to automatic by adding a single field; the 4 untestable ones cannot change class without a rewrite.
• 5 abuse cases touch 14 requirements, 6 of which are already on the list, 8 of which are new; the list grows by 40 percent, and the scenario that is entirely new is the one with no counterpart feature on the feature list.
• 45% of the requirements from the feature list are in denying mode, versus 88% of those from the scenarios; a low share of denying mode reflects the list’s source, not the writer’s attention.
• 19 of the 28 requirements are tested in the code stage 7,980 times a year at zero human cost; 5 are tested in the deployment stage, consuming 40 hours across 300 checks, and 4 are never tested.
• When the missing parts are written, the entire 40-hour load of manual checking drops; the four untestable requirements, meanwhile, keep producing no alert in any check.

Next Step

This lesson measured how a requirement should be written and produced two jobs: writing the missing field into three requirements, rethinking four requirements. Both are jobs someone has to do, and who that someone is was never asked. The person who will write the missing threshold, the person who will run the abuse case, and the person who will notice an unobservable predicate in a design session are not the same person, and they are not present on all five teams. The previous two lessons spread the checks across stages, this lesson turned requirements into a testable form — but both assumed that the competence to do the work is ready and waiting on every team. The next lesson measures that assumption: how many teams there are, how many of them have someone with a security eye, how many teams a single person’s capacity can cover, and how many reviews go through without a security eye when capacity falls short.