Resource Limits

Priority affects how fast a process runs; it does not limit how much it can consume. The process pulled back in the previous lesson’s table could still open as many files as it liked, spawn as many children as it liked, request as much memory as it liked. Limiting consumption has its own mechanism.

This lesson measures that mechanism and the failure that appears when a limit is exceeded. The question is this: does the error message say which limit it was — or do distinct causes show up under the same sentence.

The Per-Process Constraint

Every process has a set of resource limits, and every limit has two values. The soft limit is the value in effect; a process can raise it itself, up to the hard limit. The hard limit is the ceiling and can only be raised by a privileged user. Limits are read and written with the shell’s ulimit builtin:

$ ulimit -a                # the limits of the server in the fiction, sample transcript
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
open files                      (-n) 1024
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
$ ulimit -Hn               # hard limit
4096
$ ulimit -Sn 2048          # raise the soft limit up to the hard limit
$ cat /proc/1014/limits
Limit                     Soft Limit  Hard Limit  Units
Max open files            2048        4096        files
Max processes             4096        8192        processes

This transcript is illustrative and has not been run. Two reading paths appear here: ulimit gives the limits of the calling shell, while the file under /proc gives the limits of a specific process. The two are not the same thing, and the difference is the source of part of this lesson’s diagnostic errors.

The critical rule is inheritance (at fork): limits are inherited from the parent process at fork time and cannot be changed from outside afterward. Three consequences follow. First, a limit raised with ulimit in a shell covers only that shell’s children from that point on. Second, a process started by the service manager inherits nothing from the operator’s shell; its limits are written in the unit definition. Third, raising a running process’s limit later cannot be done with a shell builtin — the process must be restarted. This is the measure behind “I raised the limit but the service still gives the same error.”

The per-process constraint is not the only kind of constraint. Constraints placed on the total of a group of processes are a separate mechanism; namespaces and control groups are not covered in this course and are left to the Kernel Interfaces and Isolation course. Every number here is the limit of a single process.

• PM30. 60 failure events are seen in the observation window.
• PM31. The failures have eight distinct causes, and the causes are not equally frequent; their relative frequencies are fixed in the model.
• PM32. Eight causes produce three error messages and one silence. Each message corresponds to two distinct causes.
• PM33. The file size and CPU time limits produce no message at all: the process is killed by signal and only an exit status remains.
• PM34. Six of the causes are per-process limits; two (the system-wide descriptor limit and free memory) are not per-process and do not appear in the process’s own limit reading.
• PM35. The first diagnostic path is the message alone. The best diagnosis that can be drawn from the message is the most frequent of the causes that produce it.
• PM36. The second diagnostic path is the limit reading: whether the counter is still sitting at its limit is checked.
• PM37. Part of the events are transient: by the time it is looked at, the counter has dropped back and the reading shows nothing. The evidence has vanished on its own.
• PM38. The second seed is 20260219.

# --- common definition, in the form used in the first lesson
SEED, SECOND_SEED = 20260218, 20260219
USER = ("root", "app", "backup", "monitor")
COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector",
           "queue-worker", "cache-cleaner")


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def processes(seed=SEED, count=24):
    r = generator(seed)
    result = []
    for i in range(count):
        kind = r(10)
        pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy")
        result.append({"pid": 1000 + i * 7 + r(5),
                      "user": USER[r(len(USER))],
                      "command": COMMAND[r(len(COMMAND))], "pattern": pattern,
                      "priority": r(11) - 5, "memory": 20 + r(400),
                      "ignores_signal": r(10) < 2})
    return result


# --- this lesson's layer: eight causes, three messages, one silence
# (name, message, is it a per-process limit)
CAUSE = (
    ("process count limit",          "resource temporarily unavailable", True),
    ("thread count limit",           "resource temporarily unavailable", True),
    ("per-process descriptor limit", "too many open files",              True),
    ("system-wide descriptor limit", "too many open files",              False),
    ("address space limit",          "cannot allocate memory",           True),
    ("no free memory in the system", "cannot allocate memory",           False),
    ("file size limit",              "",                                 True),
    ("CPU time limit",               "",                                 True),
)
FREQUENCY = (5, 2, 8, 1, 4, 1, 2, 3)      # relative frequency of causes
POOL = [i for i, w in enumerate(FREQUENCY) for _ in range(w)]


def failures(processes_, seed=SEED, count=60):
    """Failure events seen over the window. Oracle: the REAL cause of each event."""
    r = generator(seed)
    event = []
    for _ in range(count):
        s = processes_[r(len(processes_))]
        name, message, per_process = CAUSE[POOL[r(len(POOL))]]
        event.append({"pid": s["pid"], "cause": name, "message": message,
                     "per_process": per_process,
                     "transient": r(10) < 4})     # counter has dropped back by the time it is checked
    return event


S = processes()
EVENT = failures(S)
print("failure events:", len(EVENT), "| distinct causes:", len({o["cause"] for o in EVENT}))
counts = {}
for o in EVENT:
    counts.setdefault(o["message"] or "(no message, killed by signal)", []).append(o["cause"])
print("message                                events  distinct causes")
for i, n in sorted(counts.items()):
    print(f"  {i:38s} {len(n):4d}  {len(set(n)):16d}")

failure events: 60 | distinct causes: 8
message                                events  distinct causes
  (no message, killed by signal)           13                 2
  cannot allocate memory                   12                 2
  resource temporarily unavailable         16                 2
  too many open files                      19                 2

The table gives this lesson’s structure in one glance. Eight causes collapse onto four rows; each row carries two distinct causes. The top row has no message at all: 13 events produced no text whatsoever. The process that exceeds the file size limit gets SIGXFSZ, the one that exceeds the CPU time limit gets SIGXCPU; if no handler is defined the process dies silently and all that remains is an exit code in the form of 128 plus the signal number. The failure has no text, only a number.

The Cause the Message Hides

The message looks as if it shows a cause, but what it actually shows is the pair the cause belongs to. The “resource temporarily unavailable” message can come from either the process count limit or the thread count limit; both mean that forking failed, and the kernel produces no text that distinguishes between them.

# Building on the previous block: EVENT, CAUSE come from there.
# BEST diagnosis derivable from the message: the most frequent cause of each message.
MESSAGE_DIAGNOSIS = {"resource temporarily unavailable": "process count limit",
                "too many open files": "per-process descriptor limit",
                "cannot allocate memory": "address space limit",
                "": "unknown"}


def diagnose_by_message(events):
    return sum(1 for o in events if MESSAGE_DIAGNOSIS[o["message"]] != o["cause"])


def diagnose_by_limit(events):
    """Message + `ulimit` reading: which counter is sitting at its limit. If the
    counter is still at the limit the cause is read; if the event was transient
    the counter has dropped back."""
    return sum(1 for o in events if o["transient"])


def diagnose_combined(events):
    """The limit is read first; if the counter has dropped back, fall back to the message."""
    return sum(1 for o in events
               if o["transient"] and MESSAGE_DIAGNOSIS[o["message"]] != o["cause"])


print("diagnostic path                       wrong diagnoses  /", len(EVENT))
print(f"  message alone                          {diagnose_by_message(EVENT):9d}")
print(f"  limit reading alone                    {diagnose_by_limit(EVENT):9d}")
print(f"  limit reading, else message            {diagnose_combined(EVENT):9d}")
print("  transient events among these         ",
      f"{sum(1 for o in EVENT if o['transient']):9d}")
print("oracle: the real cause of each event is known, we generated the fiction")
print()
print("cause                             events  correct from message?")
for name, message, _ in CAUSE:
    n = sum(1 for o in EVENT if o["cause"] == name)
    print(f"  {name:32s} {n:3d}  {'yes' if MESSAGE_DIAGNOSIS[message] == name else 'no'}")

diagnostic path                       wrong diagnoses  / 60
  message alone                                 21
  limit reading alone                           29
  limit reading, else message                   13
  transient events among these                 29
oracle: the real cause of each event is known, we generated the fiction

cause                             events  correct from message?
  process count limit               13  yes
  thread count limit                 3  no
  per-process descriptor limit      16  yes
  system-wide descriptor limit       3  no
  address space limit               10  yes
  no free memory in the system       2  no
  file size limit                    3  no
  CPU time limit                    10  no

The lower table shows which causes can be read from the message: three of the eight causes can be correctly inferred, five cannot. The ones that cannot fall into two classes — the rarer member of a pair, and the causes that produce no message at all. The CPU time limit alone accounts for 10 of the 60 events, and not a single word is written about it.

A diagnosis based on the message alone is wrong 21 times. This is the best that a message-based diagnosis can realistically be: we picked the most frequent cause for each message. An operator who does not know the distribution picks worse.

Evidence Vanishes on Its Own

The correct response is to not trust the message and to read the limit instead: the process’s counters are checked, and it is seen which one has hit the ceiling. This path has its own weakness, and it sits in the table.

A diagnosis based on the limit reading alone is wrong 29 times — worse than the message-based diagnosis. The reason can be said in one word: transience. A failed attempt to fork does not raise the counter; the counter drops back right after the attempt. When the operator looks after the event, every counter sits below its limit and the reading says “everything is fine.” 29 of the 60 events are in this state.

This is this lesson’s version of the course’s second claim: a richer diagnostic path does not, on its own, give a better diagnosis. The limit reading is far richer evidence than the message — it gives a number, a ceiling, a ratio — but the evidence lives only at the moment of the event and is gone afterward.

The diagnosis that combines both paths is wrong 13 times: it uses the limit reading when it works, and falls back to the message when the counter has dropped back. The remaining 13 events are the ones where both the evidence is lost and the message is misleading, and this number cannot be reduced — no reading taken afterward can bring back that moment’s counter. The only fix is to catch the event at the moment it happens: watching the approach to the limit from a threshold, not waiting for the failure and looking afterward.

# Building on the previous blocks: failures, processes, and the three diagnostic paths.
for seed in (SEED, SECOND_SEED):
    O2 = failures(processes(seed), seed)
    transient = sum(1 for o in O2 if o["transient"])
    print(f"seed {seed}: events {len(O2)} | message {diagnose_by_message(O2):2d}"
          f" | limit {diagnose_by_limit(O2):2d} | combined {diagnose_combined(O2):2d}"
          f" | transient {transient:2d}"
          f" | silent {sum(1 for o in O2 if not o['message']):2d}")

seed 20260218: events 60 | message 21 | limit 29 | combined 13 | transient 29 | silent 13
seed 20260219: events 60 | message 15 | limit 25 | combined  5 | transient 25 | silent  7

With the second seed all the numbers drop: message goes from 21 to 15, limit from 29 to 25, combined from 13 to 5. The numbers themselves depend on the fiction. What holds up is the ranking: in both seeds the combined path is best, the limit reading alone is worst, and the message-based diagnosis stays between the two. The richness of a diagnostic path does not, on its own, determine its place in the ranking.

The practical management of limits follows from this measurement. Limits are not set so a failure occurs; they are set so a process cannot consume the machine — a good limit contains the failure but hides the cause. That is why setting a limit goes together with watching the approach to it. A per-process limit is also a blunt tool: a process count limit covering one user’s processes can stop a faulty script while also blocking that user from logging in.

Death Outside the Limit

All eight causes measured here were a limit being exceeded, and in every one of them the process that died or failed was the very process that exceeded the limit. There is one more form of death that falls outside the limits, and it breaks the diagnosis from an entirely different direction.

When the system’s memory is truly exhausted, the kernel picks a victim and kills it with a forcing signal. The choice is not based on which process consumed the memory, but on a score in which current memory footprint weighs heavily. The result is often that the process holding the most memory dies — and that does not have to be the process that leaked it. A small leaking process can push the system under pressure and get a large, innocent process killed instead. The dying process leaves no trace in its own output; the only record of the event is in the system log.

A second silencer is the dump itself. In the example transcript above, the core file size limit is zero, and this is a common default configuration. When it is zero, a process that dies by signal leaves nothing behind that can be examined. For the 13 silent events in this lesson’s first table, all that is left is the exit code; as long as the dump limit is zero, that code is left standing alone.

Both observations lead to the same place. Finding a failure’s cause after the event depends on the evidence still existing: the counter must stay at the limit, the dump must be written, the death must be recorded somewhere. All three default to not keeping the evidence. What belongs alongside a limit, then, is not a number but a recording path — a hook that writes the state at the moment of failure, or threshold monitoring. Where the death gets recorded, and how long that record lives, is measured in this course’s logs topic.

Summary

• Every limit has two values, soft and hard; limits are inherited at the moment of forking, and a running process’s limit cannot be changed from the shell afterward.
• Eight failure causes collapse into three error messages and one silence; each message corresponds to two distinct causes, and 13 events produce no text at all.
• Only three causes can be correctly inferred from the message; the CPU time limit alone accounts for 10 of the 60 events and has no text about it at all.
• The false-diagnosis count of the three diagnostic paths: message alone 21, limit reading alone 29, the combination of both 13. Richer evidence does not, on its own, give a better diagnosis.
• The limit reading’s weakness is transience: in 29 of the 60 events the counter has dropped back by the time it is checked. The remaining 13 false diagnoses cannot be reduced by any after-the-fact reading.
• With the second seed the numbers drop to 15, 25, and 5; what holds up is the ranking — the combined path is best, the limit reading alone is worst.

Next Step

Throughout this topic, processes were seen from the operator’s window: individual processes listed, searched, backgrounded, signaled, reprioritized, and limited. All of them shared one weakness — they were running while the operator was there to look. A server’s real work, though, continues while no one is watching. The course’s next topic covers the layer that binds that work to a durable object: the service manager, units, and the dependencies between them. The first thing measured there is what it means for a unit to appear “running.”