--- title: 'Resource Limits' source: 'https://academia.sh/en/courses/system-administration/resource-limits' course: 'System Administration' language: en updated: '2026-08-17T18:10:04+00:00' license: 'CC BY-SA 4.0' --- # Resource Limits Eight distinct failure causes collapse into three error messages and one silence: 13 of 60 events produce no message at all, a diagnosis based on the message alone is wrong 21 times, one based on the limit reading alone is wrong 29 times, and the one combining both is wrong 13 times. Priority affects **how fast** a process runs; it does not limit **how much it can consume.** The process pulled back in the previous lesson's table could still open as many files as it liked, spawn as many children as it liked, request as much memory as it liked. Limiting consumption has its own mechanism. This lesson measures that mechanism and the failure that appears when a limit is exceeded. The question is this: does the error message say which limit it was — or do distinct causes show up under the same sentence. ## The Per-Process Constraint Every process has a set of resource limits, and every limit has **two** values. The **soft limit** is the value in effect; a process can raise it itself, up to the hard limit. The **hard limit** is the ceiling and can only be raised by a privileged user. Limits are read and written with the shell's `ulimit` builtin: ```text $ ulimit -a # the limits of the server in the fiction, sample transcript core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 1024 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited $ ulimit -Hn # hard limit 4096 $ ulimit -Sn 2048 # raise the soft limit up to the hard limit $ cat /proc/1014/limits Limit Soft Limit Hard Limit Units Max open files 2048 4096 files Max processes 4096 8192 processes ``` This transcript is **illustrative and has not been run.** Two reading paths appear here: `ulimit` gives the limits of **the calling shell**, while the file under `/proc` gives the limits of **a specific process**. The two are not the same thing, and the difference is the source of part of this lesson's diagnostic errors. The critical rule is **inheritance (at fork)**: limits are inherited from the parent process at fork time and cannot be changed from outside afterward. Three consequences follow. First, a limit raised with `ulimit` in a shell covers only **that shell's children from that point on.** Second, a process started by the service manager inherits nothing from the operator's shell; its limits are written in the unit definition. Third, raising a running process's limit later cannot be done with a shell builtin — the process must be restarted. This is the measure behind "I raised the limit but the service still gives the same error." The per-process constraint is not the only kind of constraint. Constraints placed on the **total** of a group of processes are a separate mechanism; namespaces and control groups are not covered in this course and are left to the **Kernel Interfaces and Isolation** course. Every number here is the limit of **a single process.** - **PM30.** **60 failure events** are seen in the observation window. - **PM31.** The failures have **eight distinct causes**, and the causes are not equally frequent; their relative frequencies are fixed in the model. - **PM32.** Eight causes produce **three error messages** and **one silence**. Each message corresponds to **two distinct causes**. - **PM33.** The file size and CPU time limits produce no message at all: the process is killed by signal and only an exit status remains. - **PM34.** Six of the causes are **per-process** limits; two (the system-wide descriptor limit and free memory) are not per-process and do not appear in the process's own limit reading. - **PM35.** The first diagnostic path is **the message alone**. The best diagnosis that can be drawn from the message is the most frequent of the causes that produce it. - **PM36.** The second diagnostic path is **the limit reading**: whether the counter is still sitting at its limit is checked. - **PM37.** Part of the events are **transient**: by the time it is looked at, the counter has dropped back and the reading shows nothing. The evidence has vanished on its own. - **PM38.** The second seed is **20260219**. ```python # --- common definition, in the form used in the first lesson SEED, SECOND_SEED = 20260218, 20260219 USER = ("root", "app", "backup", "monitor") COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector", "queue-worker", "cache-cleaner") def generator(seed): d = seed def next_value(n): nonlocal d d = (d * 1103515245 + 12345) % 2147483648 return d % n return next_value def processes(seed=SEED, count=24): r = generator(seed) result = [] for i in range(count): kind = r(10) pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy") result.append({"pid": 1000 + i * 7 + r(5), "user": USER[r(len(USER))], "command": COMMAND[r(len(COMMAND))], "pattern": pattern, "priority": r(11) - 5, "memory": 20 + r(400), "ignores_signal": r(10) < 2}) return result # --- this lesson's layer: eight causes, three messages, one silence # (name, message, is it a per-process limit) CAUSE = ( ("process count limit", "resource temporarily unavailable", True), ("thread count limit", "resource temporarily unavailable", True), ("per-process descriptor limit", "too many open files", True), ("system-wide descriptor limit", "too many open files", False), ("address space limit", "cannot allocate memory", True), ("no free memory in the system", "cannot allocate memory", False), ("file size limit", "", True), ("CPU time limit", "", True), ) FREQUENCY = (5, 2, 8, 1, 4, 1, 2, 3) # relative frequency of causes POOL = [i for i, w in enumerate(FREQUENCY) for _ in range(w)] def failures(processes_, seed=SEED, count=60): """Failure events seen over the window. Oracle: the REAL cause of each event.""" r = generator(seed) event = [] for _ in range(count): s = processes_[r(len(processes_))] name, message, per_process = CAUSE[POOL[r(len(POOL))]] event.append({"pid": s["pid"], "cause": name, "message": message, "per_process": per_process, "transient": r(10) < 4}) # counter has dropped back by the time it is checked return event S = processes() EVENT = failures(S) print("failure events:", len(EVENT), "| distinct causes:", len({o["cause"] for o in EVENT})) counts = {} for o in EVENT: counts.setdefault(o["message"] or "(no message, killed by signal)", []).append(o["cause"]) print("message events distinct causes") for i, n in sorted(counts.items()): print(f" {i:38s} {len(n):4d} {len(set(n)):16d}") ``` ``` failure events: 60 | distinct causes: 8 message events distinct causes (no message, killed by signal) 13 2 cannot allocate memory 12 2 resource temporarily unavailable 16 2 too many open files 19 2 ``` The table gives this lesson's structure in one glance. Eight causes collapse onto four rows; **each row carries two distinct causes.** The top row has no message at all: **13 events** produced no text whatsoever. The process that exceeds the file size limit gets `SIGXFSZ`, the one that exceeds the CPU time limit gets `SIGXCPU`; if no handler is defined the process dies silently and all that remains is an exit code in the form of 128 plus the signal number. **The failure has no text, only a number.** ## The Cause the Message Hides The message looks as if it shows a cause, but what it actually shows is the **pair** the cause belongs to. The "resource temporarily unavailable" message can come from either the process count limit or the thread count limit; both mean that forking failed, and the kernel produces no text that distinguishes between them. ```python # Building on the previous block: EVENT, CAUSE come from there. # BEST diagnosis derivable from the message: the most frequent cause of each message. MESSAGE_DIAGNOSIS = {"resource temporarily unavailable": "process count limit", "too many open files": "per-process descriptor limit", "cannot allocate memory": "address space limit", "": "unknown"} def diagnose_by_message(events): return sum(1 for o in events if MESSAGE_DIAGNOSIS[o["message"]] != o["cause"]) def diagnose_by_limit(events): """Message + `ulimit` reading: which counter is sitting at its limit. If the counter is still at the limit the cause is read; if the event was transient the counter has dropped back.""" return sum(1 for o in events if o["transient"]) def diagnose_combined(events): """The limit is read first; if the counter has dropped back, fall back to the message.""" return sum(1 for o in events if o["transient"] and MESSAGE_DIAGNOSIS[o["message"]] != o["cause"]) print("diagnostic path wrong diagnoses /", len(EVENT)) print(f" message alone {diagnose_by_message(EVENT):9d}") print(f" limit reading alone {diagnose_by_limit(EVENT):9d}") print(f" limit reading, else message {diagnose_combined(EVENT):9d}") print(" transient events among these ", f"{sum(1 for o in EVENT if o['transient']):9d}") print("oracle: the real cause of each event is known, we generated the fiction") print() print("cause events correct from message?") for name, message, _ in CAUSE: n = sum(1 for o in EVENT if o["cause"] == name) print(f" {name:32s} {n:3d} {'yes' if MESSAGE_DIAGNOSIS[message] == name else 'no'}") ``` ``` diagnostic path wrong diagnoses / 60 message alone 21 limit reading alone 29 limit reading, else message 13 transient events among these 29 oracle: the real cause of each event is known, we generated the fiction cause events correct from message? process count limit 13 yes thread count limit 3 no per-process descriptor limit 16 yes system-wide descriptor limit 3 no address space limit 10 yes no free memory in the system 2 no file size limit 3 no CPU time limit 10 no ``` The lower table shows which causes can be read from the message: **three of the eight causes** can be correctly inferred, five cannot. The ones that cannot fall into two classes — the rarer member of a pair, and the causes that produce no message at all. **The CPU time limit alone accounts for 10 of the 60 events**, and not a single word is written about it. A diagnosis based on the message alone is wrong **21 times**. This is the best that a message-based diagnosis can realistically be: we picked the most frequent cause for each message. An operator who does not know the distribution picks worse. ## Evidence Vanishes on Its Own The correct response is to not trust the message and to read the limit instead: the process's counters are checked, and it is seen which one has hit the ceiling. This path has its own weakness, and it sits in the table. **A diagnosis based on the limit reading alone is wrong 29 times** — **worse** than the message-based diagnosis. The reason can be said in one word: **transience**. A failed attempt to fork does not raise the counter; the counter drops back right after the attempt. When the operator looks after the event, every counter sits below its limit and the reading says "everything is fine." **29 of the 60 events** are in this state. This is this lesson's version of the course's second claim: **a richer diagnostic path does not, on its own, give a better diagnosis.** The limit reading is far richer evidence than the message — it gives a number, a ceiling, a ratio — but the evidence lives only at the moment of the event and is gone afterward. The diagnosis that combines both paths is wrong **13 times**: it uses the limit reading when it works, and falls back to the message when the counter has dropped back. The remaining 13 events are the ones where both the evidence is lost and the message is misleading, and this number **cannot be reduced** — no reading taken afterward can bring back that moment's counter. The only fix is to catch the event **at the moment it happens**: watching the approach to the limit from a threshold, not waiting for the failure and looking afterward. ```python # Building on the previous blocks: failures, processes, and the three diagnostic paths. for seed in (SEED, SECOND_SEED): O2 = failures(processes(seed), seed) transient = sum(1 for o in O2 if o["transient"]) print(f"seed {seed}: events {len(O2)} | message {diagnose_by_message(O2):2d}" f" | limit {diagnose_by_limit(O2):2d} | combined {diagnose_combined(O2):2d}" f" | transient {transient:2d}" f" | silent {sum(1 for o in O2 if not o['message']):2d}") ``` ``` seed 20260218: events 60 | message 21 | limit 29 | combined 13 | transient 29 | silent 13 seed 20260219: events 60 | message 15 | limit 25 | combined 5 | transient 25 | silent 7 ``` With the second seed all the numbers drop: message goes from 21 to **15**, limit from 29 to **25**, combined from 13 to **5**. The numbers themselves depend on the fiction. What holds up is the **ranking**: in both seeds **the combined path is best, the limit reading alone is worst**, and the message-based diagnosis stays between the two. The richness of a diagnostic path does not, on its own, determine its place in the ranking. The practical management of limits follows from this measurement. Limits are not set so a failure occurs; they are set so **a process cannot consume the machine** — a good limit contains the failure but hides the cause. That is why setting a limit goes together with **watching the approach to it**. A per-process limit is also a blunt tool: a process count limit covering one user's processes can stop a faulty script while also blocking that user from logging in. ## Death Outside the Limit All eight causes measured here were a limit being exceeded, and in every one of them the process that died or failed was **the very process that exceeded the limit**. There is one more form of death that falls outside the limits, and it breaks the diagnosis from an entirely different direction. When the system's memory is truly exhausted, the kernel picks a **victim** and kills it with a forcing signal. The choice is not based on which process consumed the memory, but on a score in which current memory footprint weighs heavily. The result is often that **the process holding the most memory** dies — and that does not have to be the process that leaked it. A small leaking process can push the system under pressure and get a large, innocent process killed instead. The dying process leaves no trace in its own output; the only record of the event is in the system log. A second silencer is the dump itself. In the example transcript above, the core file size limit is **zero**, and this is a common default configuration. When it is zero, a process that dies by signal leaves nothing behind that can be examined. For the **13 silent events** in this lesson's first table, all that is left is the exit code; as long as the dump limit is zero, that code is left standing alone. Both observations lead to the same place. Finding a failure's cause **after** the event depends on the evidence still existing: the counter must stay at the limit, the dump must be written, the death must be recorded somewhere. All three default to **not** keeping the evidence. What belongs alongside a limit, then, is not a number but a recording path — a hook that writes the state at the moment of failure, or threshold monitoring. Where the death gets recorded, and how long that record lives, is measured in this course's logs topic. ## Summary - Every limit has two values, soft and hard; limits are inherited at the moment of forking, and a running process's limit cannot be changed from the shell afterward. - Eight failure causes collapse into **three error messages and one silence**; each message corresponds to **two distinct causes**, and **13 events** produce no text at all. - Only **three causes** can be correctly inferred from the message; the CPU time limit alone accounts for 10 of the 60 events and has no text about it at all. - The false-diagnosis count of the three diagnostic paths: message alone **21**, limit reading alone **29**, the combination of both **13**. Richer evidence does not, on its own, give a better diagnosis. - The limit reading's weakness is transience: in **29 of the 60 events** the counter has dropped back by the time it is checked. The remaining 13 false diagnoses cannot be reduced by any after-the-fact reading. - With the second seed the numbers drop to 15, 25, and 5; what holds up is the ranking — the combined path is best, the limit reading alone is worst. ## Next Step Throughout this topic, processes were seen from the operator's window: individual processes listed, searched, backgrounded, signaled, reprioritized, and limited. All of them shared one weakness — they were running while the operator was there to look. A server's real work, though, continues while no one is watching. The course's next topic covers the layer that binds that work to a durable object: the service manager, units, and the dependencies between them. The first thing measured there is what it means for a unit to appear "running."