Redirection

The previous lesson showed that the other end of streams can be changed but did not explain the changing notation with its rule. What exactly does the shell do when 2>/dev/null is written, why are > and >> different, why does order matter when collecting two streams into one file?

Redirection is something the shell does before running the command. Descriptors are connected before the command starts; the command never notices that its ends were changed. Every rule in this lesson derives from this single sentence.

Truncating and Appending

There are two operators that connect standard output to a file.

> truncates: if the file does not exist, it is created; if it does, its contents are erased. Because the redirection is applied before the command runs, the file empties even if the command writes nothing at all.

echo "first" > note.txt
echo "second" > note.txt
cat note.txt

second

>> appends: the write position is moved to the end of the file every time.

echo "first" > note.txt
echo "second" >> note.txt
cat note.txt

first
second

Truncation being independent of the command gives the shortest way to empty a file. The : builtin does nothing and writes nothing; the redirection is still applied:

: > note.txt
wc -c note.txt

       0 note.txt

This is the correct way to reset a log file. Emptying its contents instead of deleting and recreating the file lets processes that hold the file open keep writing; a deleted file’s descriptor stays valid, but what gets written becomes unreachable from any directory.

Input Redirection

The < operator connects standard input to a file. As seen in the previous lesson, this is not the same thing as giving the file name as an argument: the command sees only a stream.

sort < access.log | head -2

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410

Input redirection opens the file only for reading; if a nonexistent file is given, the command never runs at all and the shell reports an error.

Redirecting by Descriptor Number

The number written before a redirection operator says which descriptor will be connected. If no number is written, 1 is assumed for output redirections, 0 for input redirection. So > file and 1> file are the same.

ls access.log missing.txt >present.txt 2>error.txt

present.txt gets only the data, error.txt only the diagnostic message:

access.log

ls: missing.txt: No such file or directory

There can be no space between the number and the operator: if 2 > error.txt is written, the shell passes the string 2 to the command as an argument and redirects only standard output.

Descriptor Duplication and Order

The notation n>&m is not redirection but duplication: descriptor n is set to point to wherever m currently points. This is copying a connection; even if m changes afterward, n stays where it was.

The rule follows from this: redirections are applied left to right. The correct way to collect two streams into the same file is to set the target first, then duplicate.

ls access.log missing.txt >all.txt 2>&1
cat all.txt

ls: missing.txt: No such file or directory
access.log

If the order is reversed, the result changes:

ls access.log missing.txt 2>&1 >separate.txt

In this call, 2 first gets connected to wherever 1 points at that moment — the terminal. Then 1 is turned into the file, but 2 has stayed at the terminal. This falls on the terminal:

ls: missing.txt: No such file or directory

separate.txt, however, gets only the data:

access.log

This is redirection’s most common mistake. Of two notations that look similar in writing, one means “everything to the file,” the other “errors to the old place, data to the file.”

Bash also recognizes the notation &> file as a shorthand for the first case. Although short, it is not defined in the POSIX shell language; in scripts that must be portable, the notation >file 2>&1 is preferred. Likewise, &>> appends.

Writing to Two Places at Once

Redirection takes a single target. When output needs to be both saved to a file and flow to the next command, tee is used: it writes what it reads from its standard input to both the given files and its own standard output.

head -2 access.log | tee first-two.txt | wc -l

       2

first-two.txt’s contents:

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120
10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048

tee -a writes by appending, not truncating. In the reporting script written at the end of the course, tee will be used to write to both the screen and the log.

Overwrite Protection

The > operator silently erasing an existing file is a common cause of data loss in hand-typed commands. The noclobber option turns this behavior off:

set -o noclobber
echo "a" > n.txt      # file does not exist, it is created
echo "b" > n.txt      # file exists, rejected

bash: n.txt: cannot overwrite existing file

While the protection is on, deliberate overwriting is done with >|:

echo "c" >| n.txt
cat n.txt

c

noclobber is useful for an interactive session; in scripts it is an unreliable foundation, because it depends on the setting of the shell running the script. If you want to prevent overwriting in a script, test explicitly for the file’s existence — condition tests are covered in the Script Structure topic.

Reading and Writing the Same File

Redirection being applied before the command makes one rule unavoidable: a command’s input file and output file cannot be the same.

printf 'c\nb\na\n' > small.txt
sort small.txt > small.txt
wc -c small.txt

       0 small.txt

The file emptied. The shell truncated small.txt, then sort ran and found nothing to read. The data is unrecoverable.

This is why tools that transform in place (like the stream editor’s -i option) do not modify the file directly; they write to a temporary file and swap it in at the end. When the same job needs to be done by hand, the pattern to follow is this:

sort small.txt > small.new && mv small.new small.txt

The && operator does not run the right-hand command if the left-hand one did not finish successfully; so when sort fails, the original file stays in place. This operator’s exact meaning is defined in this topic’s last lesson.

Redirecting an Entire Script

If the exec builtin is called with only redirections and no command name, it applies the redirections to the shell itself. Every command after that point uses the new ends.

exec 3>&1 1>record.txt    # save 1's old place in 3, turn 1 into the file
echo "to file"
exec 1>&3 3>&-           # restore the old place, close 3
echo "to screen"

Only the second line falls on the terminal:

to screen

record.txt’s contents:

to file

The notation 3>&- closes the descriptor. Descriptors numbered three and above are free for this kind of temporary storage; the shell assigns them no meaning. This pattern is used to capture a script’s entire output into a log while returning only a summary to the terminal, and it will be useful when writing the scheduled task at the end of the course.

Summary

• Redirection is applied before the command runs; the command does not know where its descriptors are connected.
• > truncates the target, >> appends to its end; because truncation is independent of the command, : > file empties a file.
• The number before the operator determines which descriptor is connected; n>&m is duplication, and because redirections are applied left to right, >file 2>&1 and 2>&1 >file have different meanings.
• tee is used when a single output needs to go to two targets.
• A command’s input and output file cannot be the same; the correct pattern is to write to a temporary file and swap it in.

Next Step

Redirection puts a file at the end of a stream. The real power of composition emerges when another command is put at that end instead. The next lesson covers the pipeline: commands running concurrently, data flowing without passing through a file in between, and the two traps this arrangement brings — the chain’s exit status and variables lost in a subshell.