The Pipeline

Redirection puts a file at the end of a stream. A file is a persistent intermediary: the reading command cannot start before the writing command finishes, and the intermediate result lands on disk. Yet in most cases the intermediate result is not wanted; what is wanted is for the text a command produces to pass directly to the next one.

The pipeline provides this. This lesson covers what the pipe operator does, why this arrangement is not merely a shorthand notation, and the two traps it brings.

The Pipe Operator

The | operator connects the standard output of the command on its left to the standard input of the command on its right. The connection between them is not a file but an in-memory buffer the kernel provides, called a pipe.

Deduplicating the client addresses in the log is a composition of two commands:

cut -d' ' -f1 access.log | sort -u

10.0.0.12
10.0.0.31
10.0.2.7
172.16.0.9
192.168.1.5

cut splits lines by whitespace and writes the first field; sort -u sorts and eliminates repeated lines. No file was created between the two.

The chain can be extended. The following pipeline sorts how many requests each address made from most to fewest, and forms the first body of the course’s reporting script:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn

   7 10.0.0.31
   7 10.0.0.12
   6 10.0.2.7
   5 192.168.1.5
   5 172.16.0.9

Building the same pipeline with the seventh field gives the distribution of requested paths:

cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -5

   6 /api/data
   4 /missing.html
   4 /index.html
   3 /static/style.css
   3 /product/45

This four-step pattern — extract, sort, count, sort — is shell text processing’s most common pattern. Because uniq -c counts only adjacent repeats, the sort before it is not optional; without it, the count comes out wrong.

Processes Run Concurrently

A pipeline does not work as “the left finishes first, then the right starts.” The shell starts every command in the pipeline at the same time; data flows as it is produced.

This is an observable fact. The pipeline below starts with a command that intends to produce a million lines, but it finishes instantly:

seq 1 1000000 | head -3

1
2
3

If it ran sequentially, all million lines would be produced, then the first three taken. What actually happens: head exits after reading three lines and closes the pipe’s read end. seq, trying to write to a closed pipe, receives a SIGPIPE signal from the kernel and terminates.

Concurrency has two practical consequences. First, the pipeline’s memory is bounded not by the size of the intermediate result but by the pipe buffer’s fixed size: data of unbounded size can be processed in constant memory. Second, the result’s first lines start appearing before the entire input has been read.

The second consequence has an exception: commands like sort cannot, by definition, write even a single line without seeing the entire input. If such a command sits in the middle of a pipeline, the flow stops there and resumes only after everything has accumulated. At that point, the pipeline’s memory becomes tied to the input size.

Backpressure

The pipe buffer’s fixed size also resolves the case where a fast producer meets a slow consumer. When the buffer fills, the writing process’s write call blocks; as the reading process drains the buffer, the writer continues. This arrangement is called backpressure.

The result: in a pipeline, a fast command can never drown a slow one in data. Memory consumption is fixed by the buffer size, not by the chain’s slowest link.

The Chain’s Exit Status

A pipeline’s exit status is, by default, only the last command’s code. Failures of the commands in between become invisible.

grep 'YOKBOYLE' access.log | sort > /dev/null
echo "status=$?"

status=0

grep reported failure because it found no match, but the pipeline’s code is sort’s code, and sort sorted the empty input without a problem. In a script, this is the most productive source of silent errors.

Bash offers two solutions. The first is the PIPESTATUS array; it holds every command’s code in the pipeline, in order:

grep -c 'YOKBOYLE' access.log | wc -l
echo "PIPESTATUS=${PIPESTATUS[*]}"

       1
PIPESTATUS=1 0

The second is the pipefail option. When it is on, the pipeline’s code is the code of the rightmost command that returned a nonzero code:

set -o pipefail
grep 'YOKBOYLE' access.log | sort > /dev/null
echo "status with pipefail=$?"

status with pipefail=1

PIPESTATUS and pipefail are bash extensions; they are not defined in the POSIX shell language. If portability is required, the intermediate result must be captured to a temporary file and each step’s code tested separately. In the Writing Robust Scripts topic, pipefail will be a fixed part of strict mode.

pipefail has a side effect: commands that terminate with SIGPIPE are also counted as failed. The seq | head pipeline above returns a nonzero code under pipefail. In pipelines that cut off early, this code being expected must be handled separately.

Variables Lost in a Subshell

Every component of a pipeline runs in a separate process. Even the shell’s own builtins, if they are inside the pipeline, run in a child process called a subshell. A subshell starts with a copy of the parent shell’s variables; the changes it makes vanish along with it.

count=0
cut -d' ' -f9 access.log | while read -r code; do
  if [ "$code" = "404" ]; then count=$((count + 1)); fi
done
echo "count after pipeline=$count"

count after pipeline=0

The loop counted four times, but the count stayed in the subshell. This is one of the most time-wasting behaviors in shell scripts; it produces no error message, it only gives a wrong result.

The solution is to take the loop out of the right end of the pipeline. In bash, process substitution presents a command’s output as if it were a file, and the loop stays in the parent shell:

count=0
while read -r code; do
  if [ "$code" = "404" ]; then count=$((count + 1)); fi
done < <(cut -d' ' -f9 access.log)
echo "count with redirection=$count"

count with redirection=4

In the notation < <(...), the two character sequences are separate: the <(...) on the right is process substitution, the < on the left is input redirection; the space between them is mandatory. Process substitution is also a bash extension. In the POSIX shell language, the same result is obtained by reading the count back as text from the subshell; this route will be shown in the Arithmetic and Command Substitution lesson.

In the Programming Fundamentals course, scope was defined along the axis of which region a name is visible in. In the shell, there is an additional axis: which process a name lives in. There is no variable passing between two processes; there is only text passing.

Chaining the Error Stream

| connects only standard output. If diagnostic messages are also wanted to go to the next command, the duplication is written before the pipe operator:

ls access.log missing.txt 2>&1 | sort

access.log
ls: missing.txt: No such file or directory

Bash recognizes |& as a shorthand for this; it is not found in the POSIX shell language.

This merging must be used cautiously: the moment you collect data and diagnostic messages into the same stream, the next command mistakes the error text for data too. The previous lesson’s rule holds here as well — the separation should be broken deliberately, not for convenience.

Summary

• The pipe operator connects the left command’s standard output to the right command’s standard input; between them is not a file but a fixed-size kernel buffer.
• Commands in the pipeline run concurrently; when the read end closes, the writing process terminates with SIGPIPE and unnecessary production stops.
• When the buffer fills, the writer is blocked; thanks to backpressure, memory consumption stays constant across the chain.
• A pipeline’s default exit status is the last command’s; intermediate failures are made visible with PIPESTATUS or pipefail.
• Because every component of the pipeline runs in a subshell, assignments at the right end do not return to the parent shell; the loop must be taken out of the pipeline.

Next Step

The inputs so far have come either from the keyboard or from an existing file. Scripts often need a third source: fixed text embedded within themselves — a configuration template, a multi-line message, test data. The next lesson covers the here-document and here-string notations, and shows how to reproduce this course’s access.log file with a single command.