Standard Input, Output, and Error

In the Introduction to Linux course, commands were run one at a time: a directory was listed, a file was copied, a permission was changed. Every command produced an output, and that output was written to the screen. The screen was where the job ended.

This course removes that assumption. The shell’s real power comes from being able to feed the data a command produces into another command; and for that, where the output goes must be decided not by the command but by whoever calls it. The first question is this: exactly where does a command write the data it writes?

The Three Streams

When a process is started, the shell hands it three open streams. Every stream is referred to by an integer, its file descriptor:

Number	Name	Direction	Default connection
0	standard input (stdin)	read	keyboard
1	standard output (stdout)	write	screen
2	standard error (stderr)	write	screen

These numbers are a convention, not a discovery: every program assumes, when it starts running, that descriptors 0, 1, and 2 are open. The program does not know, and should not know, what sits at the other end of these descriptors. The other end could be a terminal; it could just as well be a file, a pipe, or a network connection.

This ignorance is not a shortcoming; it is the design itself. A command does not say “write to the screen,” it says “write to descriptor 1.” Where that goes is decided by the caller.

Why Two Separate Output Streams

Having two streams in the write direction looks unnecessary at first glance. The reason is this: a command produces two kinds of text, and they must go to different places.

• Data is a command’s actual product: a file listing, a line count, filtered text. This is what can become the input of another command. It goes to standard output.
• A diagnostic message is for a human: an error report, a warning, progress information. It must not be used as input to the next command. It goes to standard error.

The distinction is immediately visible in practice. The following command lists an existing file together with a name that does not exist:

ls access.log missing.txt

ls: missing.txt: No such file or directory
access.log

Both lines appeared on the same screen, but they did not come from the same stream. Confirming this only requires discarding one of the streams. /dev/null is a special device file that swallows everything written to it:

ls access.log missing.txt 2>/dev/null

access.log

ls access.log missing.txt 1>/dev/null

ls: missing.txt: No such file or directory

The detail of the 2> and 1> notations is the next lesson’s subject; what matters here is the result. In the first call, only data remained; in the second, only the diagnostic message. This means ls deliberately wrote the message it produced for the file it could not find to the second stream.

This behavior is not a convention but a rule expected to be followed. If you write warnings to standard output in your own scripts, the next command processing your script’s output will mistake the warning text for data and break.

Writing Your Own Message to the Right Stream

The >&2 suffix connects a command’s standard output to standard error. This is the correct way to produce a warning in a script:

warn() { printf 'warning: %s\n' "$1" >&2; }
warn "log not found"
echo "report body"

The output of these two lines depends on which stream you look at. When you want only the data:

{ warn "log not found"; echo "report body"; } 2>/dev/null

report body

When you want only the diagnostic message:

{ warn "log not found"; echo "report body"; } 1>/dev/null

warning: log not found

This rule will not change in the script written throughout the course: the report itself goes to standard output, every explanation about the report goes to standard error.

Streams Are Buffered Independently

The order in which the two streams appear on the screen may not be the order the command wrote them in. In the ls call above, the error message appeared before the data line. The reason is a difference in buffering: standard error is typically unbuffered or line-buffered, while standard output is block-buffered when connected to a file or a pipe. A block-buffered stream is held back until its buffer fills or the process ends.

The result becomes visible when the two streams are collected into the same file:

ls access.log missing.txt > all.txt 2>&1
cat all.txt

ls: missing.txt: No such file or directory
access.log

The rule that follows: do not rely on the order of the two streams relative to each other. If you are writing a script’s output to a log together with error messages, do not assume the lines will be in chronological order; if order matters, write each line its own timestamp.

The Filter Model

Standard input is the least conspicuous of the streams, because most commands do not use it in interactive use. Yet the shell’s power of composition comes from exactly this.

A filter is a command that reads from its standard input and writes to its standard output. Filters know no file name, no location; they only transform the text that flows to them. sort, tr, wc, head, grep belong to this class.

tr 'a-z' 'A-Z' <<< 'shell programming'

SHELL PROGRAMMING

Commands fall into three groups by their input source:

1. Ones that work only from arguments. Commands like mkdir, chmod, rm need a file name to do their job; they do not read standard input.
2. Ones that work only from standard input. tr is the typical example; it takes no file name argument.
3. Ones that accept both. Commands like sort, wc, grep, cat read the file name if one is given, standard input if not.

The third group is the most common, and it has two different forms:

wc -l access.log

      30 access.log

wc -l < access.log

      30

The difference is not merely formal. In the first call, wc opens the file itself; since it knows the name, it writes it to its output. In the second, wc sees only a stream; it does not know the file’s name, and so cannot write it. Which one is wanted depends on the situation: a multi-file report needs the name, while capturing the number into a variable makes the name an obstacle.

Implementations that write the line count aligned put whitespace before the number; the GNU coreutils version does not. If you are going to use the number directly in a comparison, this whitespace causes trouble — in later lessons, arithmetic expansion absorbs this difference.

Most commands in the third group accept a single dash (-) instead of a file name and interpret it as “standard input.” It is used when a stream needs to be inserted in the middle of a file list:

head -2 access.log | cat -n -

     1	10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120
     2	10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048

This meaning of the dash is not defined at the kernel level but within each command itself; it is not a universal rule but a widespread convention.

The Course’s Working File

A single data set will be used throughout this course: a web server’s access log. Save the thirty lines below under the name access.log; every following lesson will build on this file.

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048
10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410
172.16.0.9 - - [07/Feb/2024:09:14:19 +0000] "GET /missing.html HTTP/1.1" 404 512
10.0.2.7 - - [07/Feb/2024:09:15:30 +0000] "POST /login HTTP/1.1" 302 0
10.0.2.7 - - [07/Feb/2024:09:15:31 +0000] "GET /panel HTTP/1.1" 200 12040
10.0.0.31 - - [07/Feb/2024:09:16:04 +0000] "GET /product/45 HTTP/1.1" 200 7330
192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180
10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300
172.16.0.9 - - [07/Feb/2024:09:18:47 +0000] "GET /secret HTTP/1.1" 403 290
10.0.0.31 - - [07/Feb/2024:09:19:12 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.2.7 - - [07/Feb/2024:09:20:05 +0000] "GET /product/12 HTTP/1.1" 200 8410
192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180
10.0.0.12 - - [07/Feb/2024:09:22:10 +0000] "GET /static/style.css HTTP/1.1" 304 0
172.16.0.9 - - [07/Feb/2024:09:23:55 +0000] "GET /missing.html HTTP/1.1" 404 512
10.0.0.31 - - [07/Feb/2024:09:24:41 +0000] "POST /api/data HTTP/1.1" 201 96
10.0.2.7 - - [07/Feb/2024:09:25:19 +0000] "GET /index.html HTTP/1.1" 200 5120
192.168.1.5 - - [07/Feb/2024:09:26:33 +0000] "GET /product/45 HTTP/1.1" 200 7330
10.0.0.12 - - [07/Feb/2024:09:27:02 +0000] "GET /panel HTTP/1.1" 200 12040
172.16.0.9 - - [07/Feb/2024:09:28:14 +0000] "GET /old-page HTTP/1.1" 301 0
10.0.0.31 - - [07/Feb/2024:09:29:48 +0000] "GET /api/data HTTP/1.1" 200 3300
10.0.2.7 - - [07/Feb/2024:09:30:26 +0000] "GET /static/style.css HTTP/1.1" 200 2048
192.168.1.5 - - [07/Feb/2024:09:31:07 +0000] "GET /missing.html HTTP/1.1" 404 512
10.0.0.12 - - [07/Feb/2024:09:32:53 +0000] "GET /product/12 HTTP/1.1" 200 8410
10.0.0.31 - - [07/Feb/2024:09:33:39 +0000] "GET /secret HTTP/1.1" 403 290
172.16.0.9 - - [07/Feb/2024:09:34:11 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.2.7 - - [07/Feb/2024:09:35:44 +0000] "POST /api/data HTTP/1.1" 500 180
192.168.1.5 - - [07/Feb/2024:09:36:58 +0000] "GET /panel HTTP/1.1" 200 12040
10.0.0.12 - - [07/Feb/2024:09:37:20 +0000] "GET /product/45 HTTP/1.1" 200 7330
10.0.0.31 - - [07/Feb/2024:09:38:05 +0000] "GET /missing.html HTTP/1.1" 404 512

Every line’s fields are separated by spaces: client address, two unused fields, a timestamp in square brackets, a request line in quotes (method, path, protocol), the response status code, and the number of bytes sent. By the end of the course, a script will have been written that reads this file, produces a summary report, and runs both fault-tolerant and scheduled.

Summary

• Every process is handed three streams: standard input at 0, standard output at 1, standard error at 2. A program does not know what sits at the other end of these descriptors.
• Data, a command’s actual product, is written to 1; diagnostic messages meant for a human go to 2; sending warnings to the second stream with >&2 in your own scripts keeps this convention.
• Because the two streams are buffered independently, their order relative to each other is not guaranteed.
• A filter is a command that reads from its standard input and writes to its standard output, knowing no file name; the shell’s power of composition comes from this model.
• Giving a command a file name is not equivalent to connecting a stream: a command that knows the name can also write it to its output.

Next Step

In this lesson, we changed the other end of streams with notations like 2>/dev/null but did not explain the notation itself. The next lesson covers redirection operators with their rules: the difference between writing to a file and appending, the correct order for collecting two streams into a single file, and why redirections are evaluated left to right.