--- title: 'Configuration Benchmarks' source: 'https://academia.sh/en/courses/endpoint-security/configuration-benchmarks' course: 'Endpoint and System Hardening' language: en updated: '2026-09-26T03:02:51+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts noncompliant items separately from items inapplicable in this environment, without merging the two into one ratio' - 'Compares the raw compliance ratio against the ratio computed over applicable items only' - 'Counts how many guide items touch a foothold point and the legitimate-work cost of applying them' --- # Configuration Benchmarks The benchmark guide's 26 items are applied to four machine classes' baseline: 11 of 104 decisions are inapplicable in this environment, 42 are noncompliant, the compliance ratio gives two different numbers with two different denominators, and applying every item narrows the write authority of 588 points while dropping 48 legitimate tasks to a manual step. In the previous two lessons, someone decided which item was necessary and which calendar was appropriate; the reasoning behind those decisions was never written down anywhere. **Security benchmark guides** exist to fill that gap: they collect the same decisions item by item and propose a baseline. The guide itself is an external source, it does not know the environment, and its items are not written based on what a specific endpoint does. This lesson's measure is not how many items the guide contains, but what those items do in this environment. When three distinct outcomes are collapsed into a single ratio, the decision-carrying information is lost. ## Three Distinct Outcomes When a guide item is applied to a machine class, one of three outcomes results. **Compliant**: the baseline already satisfies the item. **Noncompliant**: the baseline does not satisfy the item and could. **Inapplicable**: the item does not make sense in this environment or makes the machine's job impossible. The third outcome differs from the first two; a screen-lock timeout is not a deficiency on a machine with no screen — it is that the guide was not written for that machine. A fourth axis layers on top of these: whether **legitimate work stops** when the item is applied. An item being applicable does not mean it is free. This lesson's point column must also be read carefully. Guide items do not remove a foothold point — that work was done in the first lesson, and the remaining points stay because the job requires them. What a guide item does is narrow the **set of subjects who can write** that point's definition. The three lessons' measures line up this way: the first closed the point itself, the second shortened the **duration** for which the point is reachable by a known path, the third narrows the **set of subjects** who can write to the point. The number of closed points is zero here too. - **SH13.** The 26-item guide, four machine classes, and baseline are a **model**. The guide's name is not written; items are written at a type level, and no real audit is run. - **SH14.** An item is either applicable or inapplicable in a class; partial application is not modeled. - **SH15.** Compliance status is treated as read from the baseline and is fixed in the model. - **SH16.** The point column is the number of points per machine whose write authority narrows to the administrator when the item is applied. - **SH17.** Cost is the number of legitimate tasks that fall to a manual step when the item is applied; written per class. - **SH18.** Machine counts come from the previous lesson: 60, 24, 12, and 6. ```js // benchmark/benchmark.mjs — applying a security benchmark to the baseline IS A MODEL. // The guide's name is not written; items are written at a type level. No real audit is run. export const CLASS = [ { name: 'field-laptop', machines: 60 }, { name: 'branch-endpoint', machines: 24 }, { name: 'collector', machines: 12 }, { name: 'processing-server', machines: 6 }, ]; // row: item | write-authority-narrowing points (per machine) | applicability | compliance | cost // applicability: u = applicable, x = inapplicable in this environment // compliance: v = baseline already compliant, a = noncompliant, - = none, because inapplicable // cost: number of legitimate tasks falling to a manual step when the item is applied (digit per class) export const CONTROL = ` unnecessary-service-disabling 2 uuuu vvvv 0000 autorun-restriction 1 uuuu avvv 2000 scheduled-task-write-restriction 1 uuuu aavv 1100 logon-script-write-restriction 1 uuxu av-v 2001 driver-signature-requirement 1 uuuu vvaa 0011 boot-order-lock 1 uuuu vvaa 0000 local-administrator-membership-restriction 1 uuuu aavv 3200 remote-management-restriction 1 uuuu vvva 0001 software-source-restriction 1 uuuu aaav 4210 script-runtime-restriction 1 uuxx aa-- 3200 plugin-directory-lock 1 uuxx av-- 1000 management-tool-access-restriction 1 xuuu -vaa 0011 default-account-disablement 0 uuuu vvvv 0000 password-length 0 uuuu vvvv 0000 account-lockout-threshold 0 uuuu vvva 0001 session-timeout 0 uuuu avvv 1000 screen-lock-timeout 0 uuxx aa-- 2100 host-firewall-default 0 uuuu aavv 1100 inbound-connection-restriction 0 uuuu vvaa 0022 share-disabling 0 uuuu avvv 1000 log-retention-period 0 uuuu aaav 0000 process-creation-logging 0 uuuu aaaa 0000 log-centralization 0 uxuu a-vv 0000 disk-encryption-requirement 0 uuuu avva 0000 removable-media-restriction 0 uuxu aa-v 3200 automatic-update-requirement 0 uxuu v-aa 0011` .trim().split('\n').map((r) => { const [name, points, applicability, compliance, cost] = r.trim().split(/\s+/); return { name, points: Number(points), applicability, compliance, cost }; }); export const applicable = (m, i) => m.applicability[i] === 'u'; export const compliant = (m, i) => m.compliance[i] === 'v'; export const noncompliant = (m, i) => m.compliance[i] === 'a'; export const costOf = (m, i) => Number(m.cost[i]); // A scenario: which noncompliant items are applied. 'none' | 'free' | 'all' export const applied = (m, i, scenario) => noncompliant(m, i) && (scenario === 'all' || (scenario === 'free' && costOf(m, i) === 0)); export function scan(scenario) { let cell = 0, inapplicable = 0, alreadyCompliant = 0, narrowed = 0, stopped = 0, exception = 0; for (const m of CONTROL) for (const [i, s] of CLASS.entries()) { cell += 1; if (!applicable(m, i)) { inapplicable += 1; continue; } if (compliant(m, i)) { alreadyCompliant += 1; continue; } if (applied(m, i, scenario)) { narrowed += m.points * s.machines; stopped += costOf(m, i); } else exception += 1; } const compliantCount = alreadyCompliant + (cell - inapplicable - alreadyCompliant - exception); return { cell, inapplicable, alreadyCompliant, narrowed, stopped, exception, rawPercent: (100 * compliantCount) / cell, applicablePercent: (100 * compliantCount) / (cell - inapplicable) }; } ``` ## Applying Items to the Baseline ```js // benchmark/measure.mjs — noncompliant items are separated from inapplicable items; compliance percent and narrowed points import { CLASS, CONTROL, applicable, compliant, noncompliant, costOf, scan } from './benchmark.mjs'; const row = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); const f1 = (x) => x.toFixed(1); const pointItems = CONTROL.filter((m) => m.points > 0); console.log(`${CONTROL.length} guide items x ${CLASS.length} machine classes = ` + `${CONTROL.length * CLASS.length} decisions; ${pointItems.length} items touch a foothold point`); const A = [-18, 7, 14, 11, 14, 12, 17, 8]; console.log('\n1. item status per class'); row(A, 'class', 'items', 'inapplicable', 'compliant', 'noncompliant', 'raw %', 'applicable %', 'cost'); for (const [i, s] of CLASS.entries()) { const ux = CONTROL.filter((m) => !applicable(m, i)).length; const uv = CONTROL.filter((m) => compliant(m, i)).length; const ua = CONTROL.filter((m) => noncompliant(m, i)).length; row(A, s.name, CONTROL.length, ux, uv, ua, f1((100 * uv) / CONTROL.length), f1((100 * uv) / (CONTROL.length - ux)), CONTROL.filter((m) => noncompliant(m, i)).reduce((a, m) => a + costOf(m, i), 0)); } const B = [-46, 8, 14, 14, 17, 8]; console.log('\n2. items that touch a foothold point'); row(B, 'item', 'points', 'inapplicable', 'noncompliant', 'narrowed points', 'cost'); for (const m of pointItems) row(B, m.name, m.points, CLASS.filter((_, i) => !applicable(m, i)).length, CLASS.filter((_, i) => noncompliant(m, i)).length, CLASS.reduce((a, s, i) => a + (noncompliant(m, i) ? m.points * s.machines : 0), 0), CLASS.reduce((a, _, i) => a + (noncompliant(m, i) ? costOf(m, i) : 0), 0)); const C = [-24, 12, 18, 17, 14, 12]; console.log('\n3. three scenarios'); row(C, 'scenario', 'raw %', 'applicable %', 'narrowed points', 'stopped work', 'exceptions'); for (const [name, scenario] of [['none applied', 'none'], ['free ones applied', 'free'], ['all applied', 'all']]) { const t = scan(scenario); row(C, name, f1(t.rawPercent), f1(t.applicablePercent), t.narrowed, t.stopped, t.exception); } const t0 = scan('none'), t1 = scan('free'), t2 = scan('all'); console.log(`\ninapplicable decisions ${t0.inapplicable} / ${t0.cell}`); console.log(`items that close no point ${CONTROL.length - pointItems.length} / ${CONTROL.length}`); console.log(`compliance ${f1(t0.rawPercent)}% -> ${f1(t2.rawPercent)}% gain ` + `${t2.narrowed} points, ${t2.stopped} manual steps`); console.log(`share of free steps ${t1.narrowed}/${t2.narrowed} points, ` + `${t1.stopped}/${t2.stopped} manual steps`); ``` ``` 26 guide items x 4 machine classes = 104 decisions; 12 items touch a foothold point 1. item status per class class items inapplicable compliant noncompliant raw % applicable % cost field-laptop 26 1 9 16 34.6 36.0 24 branch-endpoint 26 2 15 9 57.7 62.5 11 collector 26 5 13 8 50.0 61.9 6 processing-server 26 3 14 9 53.8 60.9 7 2. items that touch a foothold point item points inapplicable noncompliant narrowed points cost unnecessary-service-disabling 2 0 0 0 0 autorun-restriction 1 0 1 60 2 scheduled-task-write-restriction 1 0 2 84 2 logon-script-write-restriction 1 1 1 60 2 driver-signature-requirement 1 0 2 18 2 boot-order-lock 1 0 2 18 0 local-administrator-membership-restriction 1 0 2 84 5 remote-management-restriction 1 0 1 6 1 software-source-restriction 1 0 3 96 7 script-runtime-restriction 1 2 2 84 5 plugin-directory-lock 1 2 1 60 1 management-tool-access-restriction 1 1 2 18 2 3. three scenarios scenario raw % applicable % narrowed points stopped work exceptions none applied 49.0 54.8 0 0 42 free ones applied 60.6 67.7 18 0 30 all applied 89.4 100.0 588 48 0 inapplicable decisions 11 / 104 items that close no point 14 / 26 compliance 49.0% -> 89.4% gain 588 points, 48 manual steps share of free steps 18/588 points, 0/48 manual steps ``` ## Two Compliance Percentages The first table's last two percentage columns produce two different numbers from the same baseline. **Raw compliance** divides the compliant items by the guide's total item count. **Applicable compliance** divides the same numerator by a denominator reduced after removing the inapplicable items. The second number is always larger than the first, and it is usually the one written in reports. The collector-machine row shows where the difference comes from. Raw compliance is 50.0 percent; applicable compliance is 61.9 percent. The entire 11.9-point gap comes from five items dropping out of the denominator — screen-lock timeout on a machine with no screen, script-runtime restriction on a machine whose job is done via script, removable-media restriction because meter data is collected via removable media. Marking these five items inapplicable is correct. What is wrong is that a correct decision looks like an **improvement**: no point narrowed, no configuration changed, and the percentage rose 11.9 points. The third table gives the limit of this. When **all** noncompliant items are applied, applicable compliance rises to 100 percent, while raw compliance stays at 89.4 percent. Eleven decisions can never be compliant, because the item is not meaningful in that environment. "100 percent compliance" is therefore an unmeasurable goal; the measurable goal is **naming the inapplicable items explicitly**. The field laptop is the class with the lowest of both numbers (34.6 and 36.0) and the highest cost (24 manual steps). Reading the two numbers together shows why it is low: most of the sixteen noncompliant items restrict items the user installed themselves, and applying them stops legitimate work. Noncompliance here is not negligence — it is an unpaid cost. ## Items That Do not Close a Point Fourteen of the twenty-six items touch no foothold point at all. Password length, session timeout, log retention period, process-creation logging, and inbound-connection restriction are real controls, valuable for other things; they add no number to this course's measure. Yet they carry the same weight as the other items in the compliance ratio's denominator. The result: there is no fixed relationship between the compliance ratio and the number of narrowed points. Going from 49.0 percent to 89.4 percent narrows 588 points' write authority, but much of that same 40-point gain comes from items that never touch the point column. The reverse is also possible: a team that applies only non-point-closing items raises the ratio while making zero progress in the point column. The `unnecessary-service-disabling` row in the second table is the other end of the same relationship. It narrows 2 points per machine, the strongest item, and it is already **compliant** in all four classes — because that work was exactly what got done in the first lesson. The guide adds nothing here; it confirms. The guide's contribution can only be measured in the noncompliant column. ## Exception List and Maintenance The last two rows give the tradeoff facing whoever makes the decision. Applying the zero-cost noncompliant items raises compliance to 60.6 percent with no legitimate work stopped — and narrows only 18 of 588 points, 3 percent. Nearly all the narrowing comes from items with a cost. Software-source restriction alone narrows 96 points and brings 7 manual steps; local-administrator-membership restriction narrows 84 points and 5 steps. What is cheap does very little; nothing that does real work is cheap. Every unapplied noncompliant decision is an **exception item**, and an exception is not a one-time task. 30 exception items remain in the free-of-cost scenario, 42 when nothing is applied. Each item needs a reason, an owner, and a re-evaluation date; every new version of the guide remaps items and re-tests the inapplicable markers. The 11 inapplicable decisions sit on the same list, because when a machine's job changes, what was inapplicable can become applicable. This is the true measure of maintenance: 53 of 104 decisions — 42 noncompliant plus 11 inapplicable — form a continuously maintained list. This is the number that shows applying the guide is far more expensive than reading it. The boundary between the two markers must therefore be protected. "Inapplicable" states an environmental fact and shrinks the denominator; "not applied" states a decision and stays in the denominator. The only difference between them is one letter, and the same person who writes that letter also reports the ratio. Marking a high-cost item as inapplicable finishes the work and raises the ratio at the same time; the result is a table with an empty noncompliant column, a high raw-compliance ratio, and zero narrowed points. What distinguishes this from a lie is that every inapplicable marking states, alongside it, **which environmental condition** made the item meaningless. A machine with no screen is a testable justification; "the restriction makes our job harder" is a cost statement and belongs in the noncompliant column. ## Summary - When a guide item is applied to a class, one of three outcomes results: compliant, noncompliant, inapplicable. 51 of 104 decisions are compliant, 42 noncompliant, 11 inapplicable in this environment. - Two compliance ratios give two different numbers from the same baseline: raw 50.0 on the collector machine, applicable 61.9. The 11.9-point gap is earned with zero configuration change, purely by shrinking the denominator. - Even if every item is applied, raw compliance stays at 89.4 percent; 11 decisions can never be compliant. The measurable goal is not 100 percent compliance — it is naming the inapplicable items explicitly. - A guide item does not close a foothold point; it narrows the set of subjects who can write its definition: zero points closed, 588 points narrowed. The three lessons' measures are, in sequence, the point itself, the duration it stays open, and the set of subjects who can write to it. - Fourteen of the twenty-six items touch no point at all but carry the same weight as the others in the compliance ratio; the strongest item is already compliant in all four classes, because that work was done in the first lesson. - Zero-cost noncompliant items narrow only 18 of 588 points; 97 percent of the narrowing comes at the cost of 48 manual steps. Every unapplied decision is an exception item, and 53 items require ongoing maintenance. ## Next Step This lesson's most expensive item was the software-source restriction: it narrows 96 points, drops 7 legitimate tasks to a manual step, and is already compliant in one class. Behind the item is a decision mentioned in a single line — deciding in advance what software can run on the endpoint. That is the next lesson's topic. An **application allowlist** is not a configuration item but a decision made at every execution attempt, and its accuracy can only be measured against a real execution set: how many legitimate items the list blocks, how many items it drops to manual approval, and what a signature-based rule misses differently from a path-based rule.