--- title: 'Standard Input, Output, and Error' source: 'https://academia.sh/en/courses/shell-programming/standard-input-output-and-error' course: 'Shell Programming' language: en updated: '2026-08-17T18:10:03+00:00' license: 'CC BY-SA 4.0' --- # Standard Input, Output, and Error The three standard streams, file descriptor numbers, separating a command's data from its diagnostics, and the filter model. In the Introduction to Linux course, commands were run one at a time: a directory was listed, a file was copied, a permission was changed. Every command produced an output, and that output was written to the screen. The screen was where the job ended. This course removes that assumption. The shell's real power comes from being able to feed the data a command produces into another command; and for that, where the output goes must be decided not by the command but by whoever calls it. The first question is this: exactly where does a command write the data it writes? ## The Three Streams When a **process** is started, the shell hands it three open **streams**. Every stream is referred to by an integer, its **file descriptor**: | Number | Name | Direction | Default connection | |---|---|---|---| | 0 | standard input (stdin) | read | keyboard | | 1 | standard output (stdout) | write | screen | | 2 | standard error (stderr) | write | screen | These numbers are a convention, not a discovery: every program assumes, when it starts running, that descriptors 0, 1, and 2 are open. The program does not know, and should not know, what sits at the other end of these descriptors. The other end could be a terminal; it could just as well be a file, a pipe, or a network connection. This ignorance is not a shortcoming; it is the design itself. A command does not say "write to the screen," it says "write to descriptor 1." Where that goes is decided by the caller. ## Why Two Separate Output Streams Having two streams in the write direction looks unnecessary at first glance. The reason is this: a command produces two kinds of text, and they must go to different places. - **Data** is a command's actual product: a file listing, a line count, filtered text. This is what can become the input of another command. It goes to standard output. - A **diagnostic message** is for a human: an error report, a warning, progress information. It must not be used as input to the next command. It goes to standard error. The distinction is immediately visible in practice. The following command lists an existing file together with a name that does not exist: ```sh ls access.log missing.txt ``` ``` ls: missing.txt: No such file or directory access.log ``` Both lines appeared on the same screen, but they did not come from the same stream. Confirming this only requires discarding one of the streams. `/dev/null` is a special device file that swallows everything written to it: ```sh ls access.log missing.txt 2>/dev/null ``` ``` access.log ``` ```sh ls access.log missing.txt 1>/dev/null ``` ``` ls: missing.txt: No such file or directory ``` The detail of the `2>` and `1>` notations is the next lesson's subject; what matters here is the result. In the first call, only data remained; in the second, only the diagnostic message. This means `ls` deliberately wrote the message it produced for the file it could not find to the second stream. This behavior is not a convention but a rule expected to be followed. If you write warnings to standard output in your own scripts, the next command processing your script's output will mistake the warning text for data and break. ## Writing Your Own Message to the Right Stream The `>&2` suffix connects a command's standard output to standard error. This is the correct way to produce a warning in a script: ```sh warn() { printf 'warning: %s\n' "$1" >&2; } warn "log not found" echo "report body" ``` The output of these two lines depends on which stream you look at. When you want only the data: ```sh { warn "log not found"; echo "report body"; } 2>/dev/null ``` ``` report body ``` When you want only the diagnostic message: ```sh { warn "log not found"; echo "report body"; } 1>/dev/null ``` ``` warning: log not found ``` This rule will not change in the script written throughout the course: the report itself goes to standard output, every explanation about the report goes to standard error. ## Streams Are Buffered Independently The order in which the two streams appear on the screen may not be the order the command wrote them in. In the `ls` call above, the error message appeared **before** the data line. The reason is a difference in **buffering**: standard error is typically unbuffered or line-buffered, while standard output is block-buffered when connected to a file or a pipe. A block-buffered stream is held back until its buffer fills or the process ends. The result becomes visible when the two streams are collected into the same file: ```sh ls access.log missing.txt > all.txt 2>&1 cat all.txt ``` ``` ls: missing.txt: No such file or directory access.log ``` The rule that follows: **do not rely on the order of the two streams relative to each other.** If you are writing a script's output to a log together with error messages, do not assume the lines will be in chronological order; if order matters, write each line its own timestamp. ## The Filter Model Standard input is the least conspicuous of the streams, because most commands do not use it in interactive use. Yet the shell's power of composition comes from exactly this. A **filter** is a command that reads from its standard input and writes to its standard output. Filters know no file name, no location; they only transform the text that flows to them. `sort`, `tr`, `wc`, `head`, `grep` belong to this class. ```sh tr 'a-z' 'A-Z' <<< 'shell programming' ``` ``` SHELL PROGRAMMING ``` Commands fall into three groups by their input source: 1. **Ones that work only from arguments.** Commands like `mkdir`, `chmod`, `rm` need a file name to do their job; they do not read standard input. 2. **Ones that work only from standard input.** `tr` is the typical example; it takes no file name argument. 3. **Ones that accept both.** Commands like `sort`, `wc`, `grep`, `cat` read the file name if one is given, standard input if not. The third group is the most common, and it has two different forms: ```sh wc -l access.log ``` ``` 30 access.log ``` ```sh wc -l < access.log ``` ``` 30 ``` The difference is not merely formal. In the first call, `wc` opens the file itself; since it knows the name, it writes it to its output. In the second, `wc` sees only a stream; it does not know the file's name, and so cannot write it. Which one is wanted depends on the situation: a multi-file report needs the name, while capturing the number into a variable makes the name an obstacle. Implementations that write the line count aligned put whitespace before the number; the GNU coreutils version does not. If you are going to use the number directly in a comparison, this whitespace causes trouble — in later lessons, arithmetic expansion absorbs this difference. Most commands in the third group accept a single dash (`-`) instead of a file name and interpret it as "standard input." It is used when a stream needs to be inserted in the middle of a file list: ```sh head -2 access.log | cat -n - ``` ``` 1 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 2 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 ``` This meaning of the dash is not defined at the kernel level but within each command itself; it is not a universal rule but a widespread convention. ## The Course's Working File A single data set will be used throughout this course: a web server's access **log**. Save the thirty lines below under the name `access.log`; every following lesson will build on this file. ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410 172.16.0.9 - - [07/Feb/2024:09:14:19 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.2.7 - - [07/Feb/2024:09:15:30 +0000] "POST /login HTTP/1.1" 302 0 10.0.2.7 - - [07/Feb/2024:09:15:31 +0000] "GET /panel HTTP/1.1" 200 12040 10.0.0.31 - - [07/Feb/2024:09:16:04 +0000] "GET /product/45 HTTP/1.1" 200 7330 192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180 10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300 172.16.0.9 - - [07/Feb/2024:09:18:47 +0000] "GET /secret HTTP/1.1" 403 290 10.0.0.31 - - [07/Feb/2024:09:19:12 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.2.7 - - [07/Feb/2024:09:20:05 +0000] "GET /product/12 HTTP/1.1" 200 8410 192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180 10.0.0.12 - - [07/Feb/2024:09:22:10 +0000] "GET /static/style.css HTTP/1.1" 304 0 172.16.0.9 - - [07/Feb/2024:09:23:55 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.0.31 - - [07/Feb/2024:09:24:41 +0000] "POST /api/data HTTP/1.1" 201 96 10.0.2.7 - - [07/Feb/2024:09:25:19 +0000] "GET /index.html HTTP/1.1" 200 5120 192.168.1.5 - - [07/Feb/2024:09:26:33 +0000] "GET /product/45 HTTP/1.1" 200 7330 10.0.0.12 - - [07/Feb/2024:09:27:02 +0000] "GET /panel HTTP/1.1" 200 12040 172.16.0.9 - - [07/Feb/2024:09:28:14 +0000] "GET /old-page HTTP/1.1" 301 0 10.0.0.31 - - [07/Feb/2024:09:29:48 +0000] "GET /api/data HTTP/1.1" 200 3300 10.0.2.7 - - [07/Feb/2024:09:30:26 +0000] "GET /static/style.css HTTP/1.1" 200 2048 192.168.1.5 - - [07/Feb/2024:09:31:07 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.0.12 - - [07/Feb/2024:09:32:53 +0000] "GET /product/12 HTTP/1.1" 200 8410 10.0.0.31 - - [07/Feb/2024:09:33:39 +0000] "GET /secret HTTP/1.1" 403 290 172.16.0.9 - - [07/Feb/2024:09:34:11 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.2.7 - - [07/Feb/2024:09:35:44 +0000] "POST /api/data HTTP/1.1" 500 180 192.168.1.5 - - [07/Feb/2024:09:36:58 +0000] "GET /panel HTTP/1.1" 200 12040 10.0.0.12 - - [07/Feb/2024:09:37:20 +0000] "GET /product/45 HTTP/1.1" 200 7330 10.0.0.31 - - [07/Feb/2024:09:38:05 +0000] "GET /missing.html HTTP/1.1" 404 512 ``` Every line's fields are separated by spaces: client address, two unused fields, a timestamp in square brackets, a request line in quotes (method, path, protocol), the response status code, and the number of bytes sent. By the end of the course, a script will have been written that reads this file, produces a summary report, and runs both fault-tolerant and scheduled. ## Summary - Every process is handed three streams: standard input at 0, standard output at 1, standard error at 2. A program does not know what sits at the other end of these descriptors. - Data, a command's actual product, is written to 1; diagnostic messages meant for a human go to 2; sending warnings to the second stream with `>&2` in your own scripts keeps this convention. - Because the two streams are buffered independently, their order relative to each other is not guaranteed. - A filter is a command that reads from its standard input and writes to its standard output, knowing no file name; the shell's power of composition comes from this model. - Giving a command a file name is not equivalent to connecting a stream: a command that knows the name can also write it to its output. ## Next Step In this lesson, we changed the other end of streams with notations like `2>/dev/null` but did not explain the notation itself. The next lesson covers redirection operators with their rules: the difference between writing to a file and appending, the correct order for collecting two streams into a single file, and why redirections are evaluated left to right.