Foreground and Background Jobs

The previous lesson looked at processes from outside the machine: which exist, which are heavy. Processes also have a bond with the operator. When a script is started from a terminal, more than a single process is born: that process belongs to a process group, the group to a session, the session to a controlling terminal. When the terminal closes, events travel along this entire chain.

The Shell Programming course established job control at the level of usage: &, wait, the exit code. This lesson does not repeat that side. What it measures is a single question: which processes survive when the session closes, and how well jobs output predicts this in advance.

Session, Group, and Controlling Terminal

The commands in a pipeline are placed into a single process group; the shell sends signals per group, not to individual processes. A shell opened from a terminal and all its children share the same session; the shell stands at the head of the session, and that terminal is the session’s controlling terminal. These three identities can be listed:

$ ps -eo pid,ppid,pgid,sid,tty,stat,comm
    PID    PPID    PGID     SID TTY      STAT COMMAND
   1000     998    1000    1000 pts/4    Ss   bash
   1014    1000    1014    1000 pts/4    R    report-generator
   1021    1000    1014    1000 pts/4    R    metric-collector
   1043    1000    1043    1000 pts/4    T    backup-job
   1092       1    1092    1092 ?        Ss   queue-worker

This transcript is illustrative and has not been run. There are three things to read here. The first two processes carry the same PGID value: the two ends of a pipeline. The last process’s SID value equals its own process number, and its TTY column is empty — that process is in its own separate session and is attached to no terminal. The third process’s state is T: stopped.

When the terminal closes, two separate events travel, and either one can kill. The first is the hangup signal: the kernel delivers it to the head of the session, and the shell propagates it to its own jobs. The second is quieter — because the terminal no longer exists, a process trying to write to it gets an input/output error, and most programs stop on this error. The second event leaves no signal record at all.

• PM8. Part of the synthetic server’s 24 processes has been started from the operator’s shell; the rest belong to other sessions.
• PM9. Every process has two extra properties: whether its output still goes to the terminal (writes) and whether it waits for input from the terminal (reads).
• PM10. Five detachment methods are measured: foreground, background (&), disown, nohup, and separate session.
• PM11. A method determines three things: whether the process is in the session, whether the shell sends it the hangup signal, whether the process ignores that signal. Methods that redirect also turn off the writes property.
• PM12. The survival rule: no event reaches processes outside the session; those inside it are eliminated first by the signal, then by the write error.
• PM13. The common expectation is this: the foreground job dies, and all four of the other methods keep the job alive. The false diagnosis is the number of processes where this expectation and reality diverge.
• PM14. The second seed is 20260219.

# --- common definition, in the form used in the first lesson: synthetic server and oracle
SEED, SECOND_SEED = 20260218, 20260219
USER = ("root", "app", "backup", "monitor")
COMMAND = ("data-receiver", "report-generator", "backup-job", "metric-collector",
           "queue-worker", "cache-cleaner")


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def processes(seed=SEED, count=24):
    r = generator(seed)
    result = []
    for i in range(count):
        kind = r(10)
        pattern = "flat" if kind < 5 else ("spike" if kind < 8 else "heavy")
        result.append({"pid": 1000 + i * 7 + r(5),
                       "user": USER[r(len(USER))],
                       "command": COMMAND[r(len(COMMAND))], "pattern": pattern,
                       "priority": r(11) - 5, "memory": 20 + r(400),
                       "ignores_signal": r(10) < 2})
    return result


# --- this lesson's layer: the session bond
METHOD = ("foreground", "background", "disown", "nohup", "separate session")


def distribute_sessions(processes_, seed=SEED):
    """The shell's own jobs are separated from the rest of the machine.
    `writes`: its output still goes to the terminal. `reads`: it waits for input from the terminal."""
    r = generator(seed)
    for s in processes_:
        s["shell_job"] = r(10) < 5
        s["writes"] = r(10) < 6
        s["reads"] = r(10) < 3
    return processes_


def apply(process, method):
    """A method determines three properties of the process. The rest comes from the process itself."""
    in_session = method != "separate session"
    shell_sends = method in ("foreground", "background")
    protected = method in ("nohup", "separate session")
    return {"in_session": in_session, "shell_sends": shell_sends,
            "ignores": protected or process["ignores_signal"],
            "writes": process["writes"] and not protected}


def is_alive(d):
    """The terminal closing produces TWO separate events, and either one can kill."""
    if not d["in_session"]:
        return True                          # outside the session: neither event reaches it
    if d["shell_sends"] and not d["ignores"]:
        return False                         # first event: the hangup signal
    if d["writes"]:
        return False                         # second event: write error once the terminal is gone
    return True


S = distribute_sessions(processes())
SHELL_JOBS = [s for s in S if s["shell_job"]]
print("total processes:", len(S), "| shell's own jobs:", len(SHELL_JOBS),
      "| belong to another session:", len(S) - len(SHELL_JOBS))
print("shell jobs: ignore the signal", sum(1 for s in SHELL_JOBS if s["ignores_signal"]),
      "| write to the terminal", sum(1 for s in SHELL_JOBS if s["writes"]),
      "| read from the terminal", sum(1 for s in SHELL_JOBS if s["reads"]))

total processes: 24 | shell's own jobs: 13 | belong to another session: 11
shell jobs: ignore the signal 4 | write to the terminal 9 | read from the terminal 5

Backgrounding a Job

Job control’s commands are the shell’s own commands, and they do not appear in the process table. & starts a job in the background, Ctrl+Z stops the foreground job, bg resumes a stopped job in the background, fg brings it back to the foreground. jobs lists the shell’s job table:

$ jobs -l
[1]-  1014 Running                 ./report-generator --day 7 &
[2]+  1043 Stopped                 ./backup-job --full
[3]   1021 Running                 ./metric-collector | ./summarize &

This transcript, too, is illustrative and has not been run. Jobs are referred to with markers like %1, %2; fg %2 brings the second job to the foreground. The number in square brackets is not a process number — it is the shell’s own counter and has no meaning in another shell.

The jobs table has two limits, and both produce exactly what this lesson measures. First, the table carries only this shell’s jobs: no process started from another session, or by the service manager, appears here. Second, the table shows the bond as it is now, not the future outcome.

The practical consequence of these two limits is this: jobs is not the way to verify that a job has genuinely detached from the session. Verification is done by opening a second session and searching for the process there; in ps output, an empty TTY column and a PPID of 1 are the proof of detachment. jobs output can never answer this question, because what is being asked lies exactly outside jobs output’s scope.

The exit code contract established in the Shell Programming course holds here too: wait %1 waits for the job to end and returns its exit code. When a job is terminated by a signal, the code becomes 128 plus the signal number; whether a background job died silently is read from this code. If wait is not called and the code read, the information is lost — the job table goes away when the shell closes.

There are three ways to detach a process from the session. disown removes the job from the shell’s table; the shell no longer sends it the hangup signal. The nohup command starts the process so that it ignores the hangup signal, and redirects its output to a file. setsid places the process in a new session; that session has no controlling terminal, so there is no terminal to close either. The three methods do not do the same job, and the difference can be measured.

Five Methods, One Hangup

# On top of the previous block: SHELL_JOBS, METHOD, apply, is_alive come from there.
# Common expectation: the foreground job dies, all the others survive.
EXPECTATION = {"foreground": False, "background": True, "disown": True,
               "nohup": True, "separate session": True}

print("method        applied  expected  actual  died  false diagnosis")
for y in METHOD:
    actual = {s["pid"] for s in SHELL_JOBS if is_alive(apply(s, y))}
    expected = {s["pid"] for s in SHELL_JOBS} if EXPECTATION[y] else set()
    print(f"  {y:16s} {len(SHELL_JOBS):7d}  {len(expected):8d}  {len(actual):6d}"
          f"  {len(SHELL_JOBS) - len(actual):4d}  {len(expected ^ actual):15d}")
print("oracle: the `actual` column on every row, since we generated the fiction")

method        applied  expected  actual  died  false diagnosis
  foreground            13         0       0    13                0
  background            13        13       0    13               13
  disown                13        13       4     9                9
  nohup                 13        13      13     0                0
  separate session      13        13      13     0                0
oracle: the `actual` column on every row, since we generated the fiction

The second row is this lesson’s main result. & keeps nothing alive: all 13 of the 13 backgrounded jobs die, false diagnosis 13. The ampersand puts a job in the background; it does not take it out of the session. When the shell propagates the hangup signal to its own jobs, it includes the backgrounded ones too. The sentence “I put & after it, so it survives logout” is wrong for every job in the fiction.

The third row gives a finer result. disown genuinely blocks the hangup signal — but of 13 jobs, only 4 survive, 9 die. The ones that die are victims of the second event: their output was still going to the terminal, and once the terminal vanished, they stopped on their first write attempt. disown cuts the signal, not the stream. This is the cost of trying to meet two separate events with a single precaution.

In the last two rows, the false diagnosis is zero. nohup meets both events at once: it makes the signal ignored and redirects the output to a file. A separate session removes the problem entirely; there is no bond left to close. This is the right arrangement for lasting jobs — either a separate session, or not starting the job from the shell at all and turning it into a service unit instead. The second is the subject of this course’s next topic.

The detail can vary between shells; in some shells, whether the hangup signal propagates to jobs is controlled by an option. What is measured is the common behavior, and in the common behavior, & alone is not a protection.

A Job Sitting in the List, Not Progressing

The detachment problem has a counterpart. If a background process tries to read from the terminal, the kernel stops it; the process does not die, it waits. A stopped process appears with state T in ps output, and as “Stopped” in jobs output. Both confirm that the process exists; neither says that there is no progress.

# On top of the previous blocks: SHELL_JOBS, apply, is_alive, distribute_sessions, processes.
stalled = [s for s in SHELL_JOBS if s["reads"]]
print("backgrounded jobs trying to read from the terminal:", len(stalled))
print("  these get STOPPED: `jobs` lists them, `ps` shows them, progress is zero")
print("  `jobs` rows:", len(SHELL_JOBS), "| actually progressing:", len(SHELL_JOBS) - len(stalled),
      "| false diagnosis:", len(stalled))
print()
for seed in (SEED, SECOND_SEED):
    S2 = distribute_sessions(processes(seed), seed)
    K2 = [s for s in S2 if s["shell_job"]]
    d = {y: len({s["pid"] for s in K2} ^ {s["pid"] for s in K2
                                          if is_alive(apply(s, y))})
         for y in ("background", "disown", "nohup")}
    print(f"seed {seed}: shell jobs {len(K2):2d} | false diagnosis {d}"
          f" | stalled {sum(1 for s in K2 if s['reads'])}")

backgrounded jobs trying to read from the terminal: 5
  these get STOPPED: `jobs` lists them, `ps` shows them, progress is zero
  `jobs` rows: 13 | actually progressing: 8 | false diagnosis: 5

seed 20260218: shell jobs 13 | false diagnosis {'background': 13, 'disown': 9, 'nohup': 0} | stalled 5
seed 20260219: shell jobs 11 | false diagnosis {'background': 11, 'disown': 6, 'nohup': 0} | stalled 2

jobs shows 13 lines, the jobs actually progressing are 8. Five jobs are waiting on a password or a confirmation, and nobody can write to them; they are in the list, not in the progress. False diagnosis 5. The practical consequence of this is that every job to be backgrounded needs its input closed too — redirecting output alone is not enough. A process whose input is connected to an empty file is not stopped; a read attempt returns end-of-file immediately, and the process proceeds down its own error path. Stopping itself is not a visible event either: it is written nowhere except the status field in the job table.

With the second seed, the shell’s jobs drop to 11, stalled jobs to 2; the false diagnosis for & is again all the jobs, for disown 6 instead of 9, for nohup again 0. The numbers depend on the fiction; the order and direction do not change — & is worst, disown is half protection, nohup and a separate session are full protection.

Orphan and Zombie Processes

When a session closes, the parent process of any surviving processes has died. The kernel does not leave them orphaned: it reassigns their parent process number, and the PPID column in ps output becomes 1. Being orphaned is not a malfunction; three of the detachment methods work exactly this way.

The situation that creates real confusion is the opposite. When a process ends, its exit code is kept in the process table until its parent reaps it. If the parent does not do this reaping, the child stays in the zombie state: it runs no code, holds no memory, but its table entry remains. In ps output its state is Z, and its command name is in square brackets.

Sending a signal to a zombie has no effect at all — a signal is delivered to a process that runs code, and a zombie runs no code. What wait reaps in the Shell Programming course is exactly this; when a parent process is written that does not reap, the result is a zombie buildup. The fix is not to terminate the zombies, but to fix or terminate the parent process; once the parent is gone, the zombies are reassigned and reaped there.

The cost of the buildup is a countable resource: every zombie holds one process table entry and one process number. Once enough have built up, a new process cannot be created. Which error message this limit shows up as is the subject of this topic’s fifth lesson.

For lasting jobs, nohup and setsid are the right tools, but not the only option. Tools in the terminal multiplexer class carry the session themselves: the job does not die when the connection drops, because the terminal the job is attached to is the multiplexer’s, not the operator’s. The difference here is not one of measurement but of ownership — when the multiplexer itself closes, the same two events travel again.

Summary

• A process is bound to three identities: process group, session, controlling terminal. The shell sends signals to the process group; in ps output, the PGID, SID, and TTY columns show this bond.
• When the terminal closes, two separate events travel: the hangup signal, and the input/output error a process gets when it tries to write to the terminal. The second event leaves no signal record.
• & does not detach from the session: all 13 of 13 backgrounded jobs die, false diagnosis 13.
• disown cuts the signal but not the stream; of 13 jobs, 4 remain, 9 die because their output was going to the terminal. nohup and a separate session meet both events at once, false diagnosis 0.
• jobs shows 13 lines while the jobs actually progressing number 8; 5 jobs trying to read from the terminal have been stopped and appear to be running in the list.

Next Step

This lesson saw, from the side, how a signal reaches processes: the hangup signal propagated from the shell to its jobs, and some of them ignored it. Ignoring itself has not yet been measured. The next lesson takes up signals directly and counts a single sentence: a termination request is a request, and a command returning without error does not mean the target is gone.