--- title: 'Shell Programming' source: 'https://academia.sh/en/courses/shell-programming' language: en updated: '2026-08-17T04:41:53+00:00' license: 'CC BY-SA 4.0' lessons: 24 outcomes: - 'Combines commands by setting up input/output redirection and pipelines' - 'Writes scripts containing variables, conditionals, loops and functions' - 'Transforms data with regular expressions and text processing tools' --- # Shell Programming Streams and redirection, script structure, text processing with regular expressions and robust script writing; building re-runnable tools by combining commands. ## Streams and Pipelines The three standard streams, redirection operators, chaining commands with a pipeline, input embedded in a script, and the exit status convention. ### Standard Input, Output, and Error Source: https://academia.sh/en/courses/shell-programming/standard-input-output-and-error The three standard streams, file descriptor numbers, separating a command's data from its diagnostics, and the filter model. In the Introduction to Linux course, commands were run one at a time: a directory was listed, a file was copied, a permission was changed. Every command produced an output, and that output was written to the screen. The screen was where the job ended. This course removes that assumption. The shell's real power comes from being able to feed the data a command produces into another command; and for that, where the output goes must be decided not by the command but by whoever calls it. The first question is this: exactly where does a command write the data it writes? #### The Three Streams When a **process** is started, the shell hands it three open **streams**. Every stream is referred to by an integer, its **file descriptor**: | Number | Name | Direction | Default connection | |---|---|---|---| | 0 | standard input (stdin) | read | keyboard | | 1 | standard output (stdout) | write | screen | | 2 | standard error (stderr) | write | screen | These numbers are a convention, not a discovery: every program assumes, when it starts running, that descriptors 0, 1, and 2 are open. The program does not know, and should not know, what sits at the other end of these descriptors. The other end could be a terminal; it could just as well be a file, a pipe, or a network connection. This ignorance is not a shortcoming; it is the design itself. A command does not say "write to the screen," it says "write to descriptor 1." Where that goes is decided by the caller. #### The Course's Working File In this lesson and the following ones, the commands will work on a single data set: a web server's access **log**. The log is not downloaded from outside; the script below produces it under the name `access.log`, so every reader works with the same thirty lines and the numbers match exactly. The `<<'END'` notation in the script is a here document, which the fourth lesson explains in detail; for now, read it as "write these lines into the file". ```bash #!/usr/bin/env bash # make-data.sh cat > access.log <<'END' 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410 172.16.0.9 - - [07/Feb/2024:09:14:19 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.2.7 - - [07/Feb/2024:09:15:30 +0000] "POST /login HTTP/1.1" 302 0 10.0.2.7 - - [07/Feb/2024:09:15:31 +0000] "GET /panel HTTP/1.1" 200 12040 10.0.0.31 - - [07/Feb/2024:09:16:04 +0000] "GET /product/45 HTTP/1.1" 200 7330 192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180 10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300 172.16.0.9 - - [07/Feb/2024:09:18:47 +0000] "GET /secret HTTP/1.1" 403 290 10.0.0.31 - - [07/Feb/2024:09:19:12 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.2.7 - - [07/Feb/2024:09:20:05 +0000] "GET /product/12 HTTP/1.1" 200 8410 192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180 10.0.0.12 - - [07/Feb/2024:09:22:10 +0000] "GET /static/style.css HTTP/1.1" 304 0 172.16.0.9 - - [07/Feb/2024:09:23:55 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.0.31 - - [07/Feb/2024:09:24:41 +0000] "POST /api/data HTTP/1.1" 201 96 10.0.2.7 - - [07/Feb/2024:09:25:19 +0000] "GET /index.html HTTP/1.1" 200 5120 192.168.1.5 - - [07/Feb/2024:09:26:33 +0000] "GET /product/45 HTTP/1.1" 200 7330 10.0.0.12 - - [07/Feb/2024:09:27:02 +0000] "GET /panel HTTP/1.1" 200 12040 172.16.0.9 - - [07/Feb/2024:09:28:14 +0000] "GET /old-page HTTP/1.1" 301 0 10.0.0.31 - - [07/Feb/2024:09:29:48 +0000] "GET /api/data HTTP/1.1" 200 3300 10.0.2.7 - - [07/Feb/2024:09:30:26 +0000] "GET /static/style.css HTTP/1.1" 200 2048 192.168.1.5 - - [07/Feb/2024:09:31:07 +0000] "GET /missing.html HTTP/1.1" 404 512 10.0.0.12 - - [07/Feb/2024:09:32:53 +0000] "GET /product/12 HTTP/1.1" 200 8410 10.0.0.31 - - [07/Feb/2024:09:33:39 +0000] "GET /secret HTTP/1.1" 403 290 172.16.0.9 - - [07/Feb/2024:09:34:11 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.2.7 - - [07/Feb/2024:09:35:44 +0000] "POST /api/data HTTP/1.1" 500 180 192.168.1.5 - - [07/Feb/2024:09:36:58 +0000] "GET /panel HTTP/1.1" 200 12040 10.0.0.12 - - [07/Feb/2024:09:37:20 +0000] "GET /product/45 HTTP/1.1" 200 7330 10.0.0.31 - - [07/Feb/2024:09:38:05 +0000] "GET /missing.html HTTP/1.1" 404 512 END ``` Save the script as `make-data.sh` and run it once with `bash make-data.sh` to create the thirty-line `access.log` file in your working directory. Every line's fields are separated by spaces: client address, two unused fields, a timestamp in square brackets, a request line in quotes (method, path, protocol), the response status code, and the number of bytes sent. By the end of the course, a script will have been written that reads this file, produces a summary report, and runs both fault-tolerant and scheduled. #### Why Two Separate Output Streams Having two streams in the write direction looks unnecessary at first glance. The reason is this: a command produces two kinds of text, and they must go to different places. - **Data** is a command's actual product: a file listing, a line count, filtered text. This is what can become the input of another command. It goes to standard output. - A **diagnostic message** is for a human: an error report, a warning, progress information. It must not be used as input to the next command. It goes to standard error. The distinction is immediately visible in practice. The following command lists an existing file together with a name that does not exist: ```sh ls access.log missing.txt ``` ``` ls: cannot access 'missing.txt': No such file or directory access.log ``` Both lines appeared on the same screen, but they did not come from the same stream. Confirming this only requires discarding one of the streams. `/dev/null` is a special device file that swallows everything written to it: ```sh ls access.log missing.txt 2>/dev/null ``` ``` access.log ``` ```sh ls access.log missing.txt 1>/dev/null ``` ``` ls: cannot access 'missing.txt': No such file or directory ``` The detail of the `2>` and `1>` notations is the next lesson's subject; what matters here is the result. In the first call, only data remained; in the second, only the diagnostic message. This means `ls` deliberately wrote the message it produced for the file it could not find to the second stream. This behavior is not a convention but a rule expected to be followed. If you write warnings to standard output in your own scripts, the next command processing your script's output will mistake the warning text for data and break. #### Writing Your Own Message to the Right Stream The `>&2` suffix connects a command's standard output to standard error. This is the correct way to produce a warning in a script: ```sh warn() { printf 'warning: %s\n' "$1" >&2; } warn "log not found" echo "report body" ``` The output of these two lines depends on which stream you look at. When you want only the data: ```sh warn() { printf 'warning: %s\n' "$1" >&2; } { warn "log not found"; echo "report body"; } 2>/dev/null ``` ``` report body ``` When you want only the diagnostic message: ```sh warn() { printf 'warning: %s\n' "$1" >&2; } { warn "log not found"; echo "report body"; } 1>/dev/null ``` ``` warning: log not found ``` This rule will not change in the script written throughout the course: the report itself goes to standard output, every explanation about the report goes to standard error. #### Streams Are Buffered Independently The order in which the two streams appear on the screen may not be the order the command wrote them in. The difference comes from the streams being **buffered independently**: standard error is typically unbuffered or line-buffered, while standard output is block-buffered when connected to a file or a pipe. A block-buffered stream is held back until its buffer fills or the process ends. The result becomes visible when the two streams are collected into the same file: ```sh ls access.log missing.txt > all.txt 2>&1 cat all.txt ``` ``` ls: cannot access 'missing.txt': No such file or directory access.log ``` The rule that follows: **do not rely on the order of the two streams relative to each other.** If you are writing a script's output to a log together with error messages, do not assume the lines will be in chronological order; if order matters, write each line its own timestamp. #### The Filter Model Standard input is the least conspicuous of the streams, because most commands do not use it in interactive use. Yet the shell's power of composition comes from exactly this. A **filter** is a command that reads from its standard input and writes to its standard output. Filters know no file name, no location; they only transform the text that flows to them. `sort`, `tr`, `wc`, `head`, `grep` belong to this class. ```sh tr 'a-z' 'A-Z' <<< 'shell programming' ``` ``` SHELL PROGRAMMING ``` Commands fall into three groups by their input source: 1. **Ones that work only from arguments.** Commands like `mkdir`, `chmod`, `rm` need a file name to do their job; they do not read standard input. 2. **Ones that work only from standard input.** `tr` is the typical example; it takes no file name argument. 3. **Ones that accept both.** Commands like `sort`, `wc`, `grep`, `cat` read the file name if one is given, standard input if not. The third group is the most common, and it has two different forms: ```sh wc -l access.log ``` ``` 30 access.log ``` ```sh wc -l < access.log ``` ``` 30 ``` The difference is not merely formal. In the first call, `wc` opens the file itself; since it knows the name, it writes it to its output. In the second, `wc` sees only a stream; it does not know the file's name, and so cannot write it. Which one is wanted depends on the situation: a multi-file report needs the name, while capturing the number into a variable makes the name an obstacle. Implementations that write the line count aligned put whitespace before the number; the BSD `wc` on macOS does, while the GNU coreutils version, the target of this course, does not for a single input. If you are going to use the number directly in a comparison, this whitespace causes trouble — in later lessons, arithmetic expansion absorbs this difference. Most commands in the third group accept a single dash (`-`) instead of a file name and interpret it as "standard input." It is used when a stream needs to be inserted in the middle of a file list: ```sh head -2 access.log | cat -n - ``` ``` 1 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 2 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 ``` This meaning of the dash is not defined at the kernel level but within each command itself; it is not a universal rule but a widespread convention. #### Summary - Every process is handed three streams: standard input at 0, standard output at 1, standard error at 2. A program does not know what sits at the other end of these descriptors. - Data, a command's actual product, is written to 1; diagnostic messages meant for a human go to 2; sending warnings to the second stream with `>&2` in your own scripts keeps this convention. - Because the two streams are buffered independently, their order relative to each other is not guaranteed. - A filter is a command that reads from its standard input and writes to its standard output, knowing no file name; the shell's power of composition comes from this model. - Giving a command a file name is not equivalent to connecting a stream: a command that knows the name can also write it to its output. #### Next Step In this lesson, we changed the other end of streams with notations like `2>/dev/null` but did not explain the notation itself. The next lesson covers redirection operators with their rules: the difference between writing to a file and appending, the correct order for collecting two streams into a single file, and why redirections are evaluated left to right. ### Redirection Source: https://academia.sh/en/courses/shell-programming/redirection Redirection operators, the difference between truncating and appending, descriptor duplication and the importance of order, overwrite-protection traps. The previous lesson showed that the other end of streams can be changed but did not explain the changing notation with its rule. What exactly does the shell do when `2>/dev/null` is written, why are `>` and `>>` different, why does order matter when collecting two streams into one file? Redirection is something the shell does **before** running the command. Descriptors are connected before the command starts; the command never notices that its ends were changed. Every rule in this lesson derives from this single sentence. #### Truncating and Appending There are two operators that connect standard output to a file. `>` **truncates**: if the file does not exist, it is created; if it does, its contents are erased. Because the redirection is applied before the command runs, the file empties even if the command writes nothing at all. ```sh echo "first" > note.txt echo "second" > note.txt cat note.txt ``` ``` second ``` `>>` **appends**: the write position is moved to the end of the file every time. ```sh echo "first" > note.txt echo "second" >> note.txt cat note.txt ``` ``` first second ``` Truncation being independent of the command gives the shortest way to empty a file. The `:` builtin does nothing and writes nothing; the redirection is still applied: ```sh : > note.txt wc -c note.txt ``` ``` 0 note.txt ``` This is the correct way to reset a log file. Emptying its contents instead of deleting and recreating the file lets processes that hold the file open keep writing; a deleted file's descriptor stays valid, but what gets written becomes unreachable from any directory. #### Input Redirection The `<` operator connects standard input to a file. As seen in the previous lesson, this is not the same thing as giving the file name as an argument: the command sees only a stream. ```sh sort < access.log | head -2 ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410 ``` Input redirection opens the file only for reading; if a nonexistent file is given, the command never runs at all and the shell reports an error. #### Redirecting by Descriptor Number The number written before a redirection operator says which descriptor will be connected. If no number is written, 1 is assumed for output redirections, 0 for input redirection. So `> file` and `1> file` are the same. ```sh ls access.log missing.txt >present.txt 2>error.txt cat present.txt ``` `present.txt` gets only the data: ``` access.log ``` `error.txt` gets only the diagnostic message (`cat error.txt`): ``` ls: cannot access 'missing.txt': No such file or directory ``` There can be no space between the number and the operator: if `2 > error.txt` is written, the shell passes the string `2` to the command as an argument and redirects only standard output. #### Descriptor Duplication and Order The notation `n>&m` is not redirection but **duplication**: descriptor `n` is set to point to wherever `m` currently points. This is copying a connection; even if `m` changes afterward, `n` stays where it was. The rule follows from this: **redirections are applied left to right.** The correct way to collect two streams into the same file is to set the target first, then duplicate. ```sh ls access.log missing.txt >all.txt 2>&1 cat all.txt ``` ``` ls: cannot access 'missing.txt': No such file or directory access.log ``` If the order is reversed, the result changes: ```sh ls access.log missing.txt 2>&1 >separate.txt ``` In this call, `2` first gets connected to wherever `1` points at that moment — the terminal. Then `1` is turned into the file, but `2` has stayed at the terminal. This falls on the terminal: ``` ls: cannot access 'missing.txt': No such file or directory ``` `separate.txt`, however, gets only the data: ``` access.log ``` This is redirection's most common mistake. Of two notations that look similar in writing, one means "everything to the file," the other "errors to the old place, data to the file." Bash also recognizes the notation `&> file` as a shorthand for the first case. Although short, it is not defined in the POSIX shell language; in scripts that must be portable, the notation `>file 2>&1` is preferred. Likewise, `&>>` appends. #### Writing to Two Places at Once Redirection takes a single target. When output needs to be both saved to a file and flow to the next command, `tee` is used: it writes what it reads from its standard input to both the given files and its own standard output. ```sh head -2 access.log | tee first-two.txt | wc -l ``` ``` 2 ``` `first-two.txt`'s contents: ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 ``` `tee -a` writes by appending, not truncating. In the reporting script written at the end of the course, `tee` will be used to write to both the screen and the log. #### Overwrite Protection The `>` operator silently erasing an existing file is a common cause of data loss in hand-typed commands. The `noclobber` option turns this behavior off: ```sh # try.sh set -o noclobber echo "a" > n.txt # file does not exist, it is created echo "b" > n.txt # file exists, rejected ``` ``` try.sh: line 4: n.txt: cannot overwrite existing file ``` This message is the output of running the block as a script: it starts with the script's name and the line number where the error occurred. If you type the same command in an interactive shell, the prefix is `bash:`. While the protection is on, deliberate overwriting is done with `>|`: ```sh echo "c" >| n.txt cat n.txt ``` ``` c ``` `noclobber` is useful for an interactive session; in scripts it is an unreliable foundation, because it depends on the setting of the shell running the script. If you want to prevent overwriting in a script, test explicitly for the file's existence — condition tests are covered in the Script Structure topic. #### Reading and Writing the Same File Redirection being applied before the command makes one rule unavoidable: **a command's input file and output file cannot be the same.** ```sh printf 'c\nb\na\n' > small.txt sort small.txt > small.txt wc -c small.txt ``` ``` 0 small.txt ``` The file emptied. The shell truncated `small.txt`, then `sort` ran and found nothing to read. The data is unrecoverable. This is why tools that transform in place (like the stream editor's `-i` option) do not modify the file directly; they write to a temporary file and swap it in at the end. When the same job needs to be done by hand, the pattern to follow is this: ```sh sort small.txt > small.new && mv small.new small.txt ``` The `&&` operator does not run the right-hand command if the left-hand one did not finish successfully; so when `sort` fails, the original file stays in place. This operator's exact meaning is defined in this topic's last lesson. #### Redirecting an Entire Script If the `exec` builtin is called with only redirections and no command name, it applies the redirections to the shell itself. Every command after that point uses the new ends. ```sh exec 3>&1 1>record.txt # save 1's old place in 3, turn 1 into the file echo "to file" exec 1>&3 3>&- # restore the old place, close 3 echo "to screen" ``` Only the second line falls on the terminal: ``` to screen ``` `record.txt`'s contents: ``` to file ``` The notation `3>&-` closes the descriptor. Descriptors numbered three and above are free for this kind of temporary storage; the shell assigns them no meaning. This pattern is used to capture a script's entire output into a log while returning only a summary to the terminal, and it will be useful when writing the scheduled task at the end of the course. #### Summary - Redirection is applied before the command runs; the command does not know where its descriptors are connected. - `>` truncates the target, `>>` appends to its end; because truncation is independent of the command, `: > file` empties a file. - The number before the operator determines which descriptor is connected; `n>&m` is duplication, and because redirections are applied left to right, `>file 2>&1` and `2>&1 >file` have different meanings. - `tee` is used when a single output needs to go to two targets. - A command's input and output file cannot be the same; the correct pattern is to write to a temporary file and swap it in. #### Next Step Redirection puts a file at the end of a stream. The real power of composition emerges when another command is put at that end instead. The next lesson covers the pipeline: commands running concurrently, data flowing without passing through a file in between, and the two traps this arrangement brings — the chain's exit status and variables lost in a subshell. ### The Pipeline Source: https://academia.sh/en/courses/shell-programming/pipeline The pipe operator, concurrent process execution, pipe buffering and backpressure, the chain's exit status, and variables lost in a subshell. Redirection puts a file at the end of a stream. A file is a persistent intermediary: the reading command cannot start before the writing command finishes, and the intermediate result lands on disk. Yet in most cases the intermediate result is not wanted; what is wanted is for the text a command produces to pass directly to the next one. The **pipeline** provides this. This lesson covers what the pipe operator does, why this arrangement is not merely a shorthand notation, and the two traps it brings. #### The Pipe Operator The `|` operator connects the standard output of the command on its left to the standard input of the command on its right. The connection between them is not a file but an in-memory buffer the kernel provides, called a **pipe**. Deduplicating the client addresses in the log is a composition of two commands: ```sh cut -d' ' -f1 access.log | sort -u ``` ``` 10.0.0.12 10.0.0.31 10.0.2.7 172.16.0.9 192.168.1.5 ``` `cut` splits lines by whitespace and writes the first field; `sort -u` sorts and eliminates repeated lines. No file was created between the two. The chain can be extended. The following pipeline sorts how many requests each address made from most to fewest, and forms the first body of the course's reporting script: ```sh cut -d' ' -f1 access.log | sort | uniq -c | sort -rn ``` ``` 7 10.0.0.31 7 10.0.0.12 6 10.0.2.7 5 192.168.1.5 5 172.16.0.9 ``` Building the same pipeline with the seventh field gives the distribution of requested paths: ```sh cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -5 ``` ``` 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /product/45 ``` This four-step pattern — **extract, sort, count, sort** — is shell text processing's most common pattern. Because `uniq -c` counts only **adjacent** repeats, the `sort` before it is not optional; without it, the count comes out wrong. #### Processes Run Concurrently A pipeline does not work as "the left finishes first, then the right starts." The shell starts every command in the pipeline **at the same time**; data flows as it is produced. This is an observable fact. The pipeline below starts with a command that intends to produce a million lines, but it finishes instantly: ```sh seq 1 1000000 | head -3 ``` ``` 1 2 3 ``` If it ran sequentially, all million lines would be produced, then the first three taken. What actually happens: `head` exits after reading three lines and closes the pipe's read end. `seq`, trying to write to a closed pipe, receives a `SIGPIPE` signal from the kernel and terminates. Concurrency has two practical consequences. First, the pipeline's memory is bounded not by the size of the intermediate result but by the pipe buffer's fixed size: data of unbounded size can be processed in constant memory. Second, the result's first lines start appearing before the entire input has been read. The second consequence has an exception: commands like `sort` cannot, by definition, write even a single line without seeing the entire input. If such a command sits in the middle of a pipeline, the flow stops there and resumes only after everything has accumulated. At that point, the pipeline's memory becomes tied to the input size. #### Backpressure The pipe buffer's fixed size also resolves the case where a fast producer meets a slow consumer. When the buffer fills, the writing process's write call blocks; as the reading process drains the buffer, the writer continues. This arrangement is called **backpressure**. The result: in a pipeline, a fast command can never drown a slow one in data. Memory consumption is fixed by the buffer size, not by the chain's slowest link. #### The Chain's Exit Status A pipeline's exit status is, by default, **only the last command's** code. Failures of the commands in between become invisible. ```sh grep 'YOKBOYLE' access.log | sort > /dev/null echo "status=$?" ``` ``` status=0 ``` `grep` reported failure because it found no match, but the pipeline's code is `sort`'s code, and `sort` sorted the empty input without a problem. In a script, this is the most productive source of silent errors. Bash offers two solutions. The first is the `PIPESTATUS` array; it holds every command's code in the pipeline, in order: ```sh grep -c 'YOKBOYLE' access.log | wc -l echo "PIPESTATUS=${PIPESTATUS[*]}" ``` ``` 1 PIPESTATUS=1 0 ``` The second is the `pipefail` option. When it is on, the pipeline's code is the code of the rightmost command that returned a nonzero code: ```sh set -o pipefail grep 'YOKBOYLE' access.log | sort > /dev/null echo "status with pipefail=$?" ``` ``` status with pipefail=1 ``` `PIPESTATUS` and `pipefail` are bash extensions; they are not defined in the POSIX shell language. If portability is required, the intermediate result must be captured to a temporary file and each step's code tested separately. In the Writing Robust Scripts topic, `pipefail` will be a fixed part of strict mode. `pipefail` has a side effect: commands that terminate with `SIGPIPE` are also counted as failed. The `seq | head` pipeline above returns a nonzero code under `pipefail`. In pipelines that cut off early, this code being expected must be handled separately. #### Variables Lost in a Subshell Every component of a pipeline runs in a separate process. Even the shell's own builtins, if they are inside the pipeline, run in a child process called a **subshell**. A subshell starts with a copy of the parent shell's variables; the changes it makes vanish along with it. ```sh count=0 cut -d' ' -f9 access.log | while read -r code; do if [ "$code" = "404" ]; then count=$((count + 1)); fi done echo "count after pipeline=$count" ``` ``` count after pipeline=0 ``` The loop counted four times, but the count stayed in the subshell. This is one of the most time-wasting behaviors in shell scripts; it produces no error message, it only gives a wrong result. The solution is to take the loop out of the right end of the pipeline. In bash, **process substitution** presents a command's output as if it were a file, and the loop stays in the parent shell: ```sh count=0 while read -r code; do if [ "$code" = "404" ]; then count=$((count + 1)); fi done < <(cut -d' ' -f9 access.log) echo "count with redirection=$count" ``` ``` count with redirection=4 ``` In the notation `< <(...)`, the two character sequences are separate: the `<(...)` on the right is process substitution, the `<` on the left is input redirection; the space between them is mandatory. Process substitution is also a bash extension. In the POSIX shell language, the same result is obtained by reading the count back as text from the subshell; this route will be shown in the Arithmetic and Command Substitution lesson. In the Programming Fundamentals course, scope was defined along the axis of which region a name is visible in. In the shell, there is an additional axis: **which process** a name lives in. There is no variable passing between two processes; there is only text passing. #### Chaining the Error Stream `|` connects only standard output. If diagnostic messages are also wanted to go to the next command, the duplication is written **before** the pipe operator: ```sh ls access.log missing.txt 2>&1 | sort ``` ``` access.log ls: cannot access 'missing.txt': No such file or directory ``` Bash recognizes `|&` as a shorthand for this; it is not found in the POSIX shell language. This merging must be used cautiously: the moment you collect data and diagnostic messages into the same stream, the next command mistakes the error text for data too. The previous lesson's rule holds here as well — the separation should be broken deliberately, not for convenience. #### Summary - The pipe operator connects the left command's standard output to the right command's standard input; between them is not a file but a fixed-size kernel buffer. - Commands in the pipeline run concurrently; when the read end closes, the writing process terminates with `SIGPIPE` and unnecessary production stops. - When the buffer fills, the writer is blocked; thanks to backpressure, memory consumption stays constant across the chain. - A pipeline's default exit status is the last command's; intermediate failures are made visible with `PIPESTATUS` or `pipefail`. - Because every component of the pipeline runs in a subshell, assignments at the right end do not return to the parent shell; the loop must be taken out of the pipeline. #### Next Step The inputs so far have come either from the keyboard or from an existing file. Scripts often need a third source: fixed text embedded within themselves — a configuration template, a multi-line message, test data. The next lesson covers the here-document and here-string notations, and shows how to reproduce this course's `access.log` file with a single command. ### Here Documents and Here Strings Source: https://academia.sh/en/courses/shell-programming/here-documents-and-here-strings Input embedded in a script, the effect of quoting the delimiter on expansion, indentation removal, and single-line input notation. The previous three lessons showed two sources of input: the keyboard and a file. Scripts need a third source as well — fixed content embedded in the script's own text. A configuration template, a multi-line usage message, test data, and a database query all belong to this class. Keeping this content in a separate file makes the script unable to travel on its own. Writing every line with a separate `echo` requires managing line endings, quotes, and spaces by hand. The shell defines two notations for this job. #### The Here Document A **here document** begins by writing a **delimiter** after the `<<` operator; the text up to the line where the delimiter appears alone becomes the command's standard input. ```sh code=404 cat < mini.log <<'END' 10.0.0.1 GET /a 200 10.0.0.2 GET /b 404 END ``` ```sh wc -l mini.log ``` ``` 2 mini.log ``` #### Reproducing the Log This course's `access.log` file is not downloaded from an external source; it is produced by a script. This way, every reader works with the same data, and results become directly comparable. The `make-data.sh` script below writes the log from scratch. The delimiter is quoted: even though `$` does not appear in the content, not leaving data text open to expansion is the correct default. ```sh cat > make-data.sh <<'SCRIPT_END' cat > access.log <<'END' 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP/1.1" 200 2048 END SCRIPT_END ``` The outer delimiter (`SCRIPT_END`) and the inner one (`END`) were chosen differently. If they were the same, the outer document would end at the first `END` line, and the rest of the text would be interpreted as commands. When writing nested here documents, the delimiters must be different. The fragment above contains only the first two of the log's thirty lines; the script carrying all thirty lines was given in the Standard Input, Output, and Error lesson. The `make-data.sh` you saved from there restores the log with the single command `bash make-data.sh` whenever it gets corrupted in later lessons. If you try the fragment, it overwrites your script of the same name; to avoid losing the full script, try it in a separate directory. #### Which One, When | Notation | Input | Expansion | Typical use | |---|---|---|---| | `<< END` | multi-line | applied | template, formatted report header | | `<< 'END'` | multi-line | not applied | embedded source code, fixed data | | `<<<` | single-line | applied | splitting a string into fields, filtering one line | As a fourth option, a `printf` pipeline can always be used and is fully portable; in exchange, it requires managing newlines by hand. #### Summary - A here document begins with `<<` and makes the text up to the line where the delimiter appears alone the command's standard input. - If the delimiter is not quoted, the content is expanded by the double-quote rule; if quoted, it passes through literally. Quoting is mandatory for embedded source code. - The `<<-` form strips only tab indentation; it has no effect on space indentation. - A here string gives single-line input and appends a newline to its end; unlike a pipeline, it does not push the command into a subshell. - In nested here documents, the delimiters must be chosen differently. #### Next Step Every example in this topic assumed the commands succeeded. In reality, `grep` reports a result when it finds no match, `cut` when it cannot open the file, or `sort` when it finds no disk space. The next lesson defines that report — the exit status: why zero means success, how the `&&` and `||` operators tie to this code, and the code ranges the shell reserves for itself. ### Exit Status Source: https://academia.sh/en/courses/shell-programming/exit-status The success/failure convention, reserved code ranges, conditional chaining operators, and why these operators are not confused with conditional branching. In the previous lessons, commands' results always came out as expected. In reality, `grep` may find no match, `cut` may fail to open the file, `sort` may not find enough temporary space. The shell's way of learning about these situations is a single number. Every process leaves an integer with the kernel when it terminates. The shell takes this number as the **exit status** and stores it in the special variable `$?`. Shell programming's control flow — conditions, loop conditions, error handling — is built entirely on this number. #### Why Zero Is Success The convention is this: **0 is success, every nonzero value is failure.** In the Programming Fundamentals course, the value `true` was used for logical correctness; the shell's convention looks like the reverse of this. The reason is simple: a job succeeding has exactly one meaning, while its failing has many causes. The 255 nonzero values can be used to distinguish different kinds of failure; a single value suffices for success. `grep` is a clear example of this distinction, and it separates three codes from each other: ```sh grep -q '404' access.log; echo "found -> $?" grep -q '999' access.log; echo "not found -> $?" grep -q '404' missing-file.log 2>/dev/null; echo "file missing -> $?" ``` ``` found -> 0 not found -> 1 file missing -> 2 ``` All three differ from each other, and this difference does real work. "No match" is an expected result; "file missing" is a situation where the script should stop. A script that looks only at "zero or not" confuses the two. The same rigor is expected in your own scripts: give different codes to different error classes, do not use code `1` for everything. #### Commands That Embody the Convention The `true` and `false` builtins do no work at all; they only return a code. ```sh true; echo "true -> $?" false; echo "false -> $?" ``` ``` true -> 0 false -> 1 ``` The `:` command is equivalent to `true`, and it was used in the Redirection lesson to empty a file. The `test` command, and its bracket notation `[ ... ]`, tests a condition and returns the result as a code. This command has no output at all; all the information is in the code. ```sh [ -f access.log ]; echo "file exists -> $?" [ -f missing.log ]; echo "file missing -> $?" ``` ``` file exists -> 0 file missing -> 1 ``` `[` is not a syntax element but an executable command name; this is why the spaces around it are mandatory, and the closing `]` is its last argument. The detail of tests will be covered in the Conditions and Tests lesson. #### The Code's Numeric Limits An exit status fits into a byte, in the range 0–255. Values outside this range are reduced modulo 256: ```sh (exit 300); echo "300 -> $?" (exit -1); echo "-1 -> $?" ``` ``` 300 -> 44 -1 -> 255 ``` $300 \bmod 256 = 44$ and $-1 \bmod 256 = 255$. Returning a code greater than 255 in your script has no point; the number silently turns into something else. #### Reserved Codes The shell reserves some codes for its own use. Giving these codes to your own error classes leads your script to mislead its caller. ```sh echo text > unrunnable.txt # a file without execute permission nosuchcommand 2>/dev/null; echo "command not found -> $?" ./unrunnable.txt 2>/dev/null; echo "not executable -> $?" ``` ``` command not found -> 127 not executable -> 126 ``` `127` reports that the command was not found in the search path; `126` that it was found but could not be run (no execute permission, or it is a directory). If you get 127 when calling a script, the problem is not inside the script but in its name or the search path. For processes that terminate by a signal, the code is **greater than 128**; in common shells it is **128 + the signal number**, but the standard only requires it to exceed 128: ```sh bash -c 'trap - INT; kill -INT $$'; echo "INT -> $?" bash -c 'kill -TERM $$'; echo "TERM -> $?" ``` ``` INT -> 130 TERM -> 143 ``` `SIGINT` is number 2 ($128 + 2 = 130$), `SIGTERM` is 15 ($128 + 15 = 143$). This is why a script interrupted from the keyboard returns 130. Signals and catching them are covered in detail in the Writing Robust Scripts topic. The usable range in practice is 1–125. If the `exit` command is called with no argument, it returns the code of the last command that ran in the script. #### Conditional Chaining The shell defines two chaining operators that depend on the exit status. The `&&` operator runs the right-hand command if the left-hand one **returns 0**. The `||` operator runs the right-hand command if the left-hand one returns **nonzero**. In both cases, if the right-hand command does not run, the chain's code is the left-hand one's code. ```sh grep -q '404' access.log && echo "log has 404" grep -q '999' access.log || echo "999 does not appear" ``` ``` log has 404 999 does not appear ``` These operators perform **short-circuit evaluation** — the same behavior defined for logical operators in the Programming Fundamentals course. The difference is that what is being evaluated here is not an expression but a process being run: short-circuiting does not just save time, it ensures a job is never done at all. The safe in-place transformation pattern from the Redirection lesson was written with this operator: ```sh printf 'c\nb\na\n' > small.txt sort small.txt > small.new && mv small.new small.txt ``` If `sort` fails, `mv` never runs, and the original file stays in place. Tying a destructive operation to a success condition is the plainest form of error handling in the shell. #### Chaining Is Not Conditional Branching `&&` and `||` have **the same precedence** and associate **left to right**. These two rules give rise to a common misunderstanding: the notation `condition && body || fallback` is thought to be equivalent to `if`–`else`. It is not. ```sh fail() { echo "body ran"; return 1; } true && fail || echo "FALLBACK RAN" ``` ``` body ran FALLBACK RAN ``` The condition was true, the body ran — but because the body reported failure, the fallback ran too. `if`–`else` does not behave this way: ```sh fail() { echo "body ran"; return 1; } if true; then fail; else echo "FALLBACK"; fi ``` ``` body ran ``` Rule: **an `&&` … `||` chain substitutes for `if`–`else` only when the body can never fail.** Because this guarantee can rarely be given, writing `if` for a two-branch choice is the correct default. `&&` and `||` are suited to single-branch guards — "continue if it succeeded," "warn and exit if it failed." Left-to-right associativity also shows itself in mixed chains: ```sh false || echo "first" && echo "second" ``` ``` first second ``` `false || echo "first"` is evaluated first; because `echo` succeeds, the chain's code is 0, and `&&` runs the right-hand side. Use grouping instead of hard-to-read chains: the code of a command list inside `{ ...; }` is the code of the list's last command. #### Negation The `!` prefix reverses a command's code: it turns 0 into nonzero, and nonzero into 0. ```sh ! grep -q '999' access.log; echo "negation -> $?" ``` ``` negation -> 0 ``` This is the direct way to express the condition "if 999 does not appear in the log." Negation has a second function too: suppressing expected failures under strict mode (`set -e`). This use is covered in the Error Handling lesson. #### Wrapping Up the Topic The four concepts established in this topic form the vocabulary for the rest of the course. The pipeline that extracts the five most requested paths from the log uses this entire vocabulary: ```sh [ -r access.log ] || { echo 'log unreadable' >&2; exit 2; } cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -5 ``` The first line contains an exit status test, a chaining operator, a grouping, and a redirection to standard error. The second line is a five-command pipeline. Together, the two form the core of the `report.sh` script to be written in the next topic. #### Summary - Every process leaves an exit status in the range 0–255 when it terminates; 0 is success, every nonzero value is failure. The code is read with `$?`. - Different kinds of failure are distinguished by different codes; `grep` returns 1 for no match, 2 for a file error. - 126, 127, and 128 + signal number are reserved by the shell; the range 1–125 is used for your own codes. - `&&` and `||` perform short-circuit evaluation; they have the same precedence and associate left to right. - The notation `condition && body || fallback` is not equivalent to `if`–`else` when the body can fail. #### Next Step Everything in this topic was done with single-line commands. When the same pipeline needs to run every day, gain options, and be handed off to someone else, a single line is not enough. The next topic builds the script file: it opens by covering what the interpreter declaration does, how a script becomes executable, and which shell is used when the system runs a file. ## Script Structure The building blocks of a shell script, from the shebang to variable expansion, from quoting rules to conditions, loops, functions, and arrays. ### Script Anatomy Source: https://academia.sh/en/courses/shell-programming/script-anatomy The shebang, execute permission, the three ways to run a script, and the difference between a subprocess and sourcing. The previous topic established chaining commands together with streams and exit codes. The resulting pipelines were one-liners: they ran the moment they were written and then vanished. When the same pipeline needs to run every day, gain options, and be handed off to someone else, a single line is not enough. A **script** is an ordinary text file holding commands that the shell reads and executes in sequence. This lesson takes up what the file is, how the kernel runs it, and what the three different ways of running it mean in terms of processes. #### The First Script The previous lesson's final pipeline is put into a file: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. echo "== Most requested paths ==" cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -5 ``` This file is saved as `report.sh`. Throughout the course, every topic will advance this script one step further; by the end it will become a tool that takes arguments, tolerates errors, and runs on a schedule. Lines beginning with `#` are **comments**; the shell ignores them. The comment marker does not have to be at the start of a line, but it does not start a comment in the middle of a word: ```sh echo abc#def echo abc #def ``` ``` abc#def abc ``` For a comment to start, the `#` mark must sit at the start of a word. This is a rule that matters when `#` appears in file names and URLs. #### Execute Permission When a script file is created, it does not carry execute permission. If it is invoked directly: ```sh # try.sh ./report.sh echo $? ``` ``` try.sh: line 2: ./report.sh: Permission denied 126 ``` The `try.sh: line 2:` prefix at the start of the error message shows that the command was invoked from inside a script: the script's name and the line number are written. Had the same command been typed in an interactive shell, the prefix would have been `bash:`. The code `126` means "file found but could not be executed"; it appeared among the reserved codes in the previous lesson. Permission is granted: ```sh chmod +x report.sh ./report.sh ``` ``` == Most requested paths == 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /product/45 ``` If the `./` in front of the script's name is dropped, the result changes: ```sh # try.sh report.sh echo $? ``` ``` try.sh: line 2: report.sh: command not found 127 ``` The rule established in the command path lesson of the Introduction to Linux course applies here: a name without a slash is searched for only in the directories listed in `PATH`, and the current directory is not on that list. Writing `./report.sh` turns the name into a path, and no search happens. Adding the current directory to the `PATH` list is not a widely recommended practice: anyone who drops a file named `ls` into that directory can hijack every command you run there. #### The Shebang The `#!` sequence on a script's first line looks like a comment, but its function belongs to the kernel's domain, not the shell's. If the file's first two bytes are `#!`, the kernel does not attempt to execute it directly; it reads the rest of the line as a program path, starts that program, and gives it the script file's path as an argument. This mechanism is called the **shebang**. It has three details. **It is valid only on the first line.** A `#!` on the second line is an ordinary comment. If the first line's first two bytes are not `#!`, there is no shebang. **The path is absolute; no search happens.** The kernel does not look at `PATH`. A script that writes `#!/bin/bash` does not run on a system where bash is not at the path `/bin/bash`. **Without a shebang, the behavior is left to the caller.** The kernel cannot execute the file and returns an error; the calling shell then, in most cases, has the file read into its own subshell. The result depends on the shell running the script — that is, it is not portable. ```sh printf 'echo "ran without a shebang: $0"\n' > plain.sh chmod +x plain.sh ./plain.sh ``` ``` ran without a shebang: ./plain.sh ``` It ran, but which shell it ran with is not guaranteed. Writing a shebang on every script is a mandatory habit. #### Which Interpreter There are two common forms, and the choice depends on what the script uses. `#!/bin/sh` declares the POSIX shell language. Which shell sits at the end of this path varies by system; the only guarantee given is that it will interpret the POSIX shell language. Bash extensions such as arrays, `[[ ]]`, `<<<`, `${var,,}` cannot be used with this declaration — that they appear to work on some systems is misleading, because there `/bin/sh` is bash's POSIX mode. `#!/usr/bin/env bash` calls the `env` command to search for bash in `PATH`. This form absorbs bash being installed in different places on different systems. Its cost is that an indirect search runs through `PATH`; in strict environments that require a fixed path, `#!/bin/bash` is preferred. Decision rule: **if the script uses even a single bash-specific feature, declare bash.** Declaring POSIX while using a bash feature is the most common reason a script silently breaks on another system. In this course, every bash-specific feature will be marked explicitly in the text. There is also a difference between bash versions. On a given system, `/bin/bash` may be an old major version; features such as associative arrays do not exist there. The Arrays and Associative Arrays lesson will give a measurable example of this difference. #### The Three Ways to Run It The same script can be run in three different ways, and the three are not the same thing. **Direct execution** (`./report.sh`) requires execute permission and a shebang. A new process is created. **Giving it as an argument to the interpreter** (`bash report.sh`) requires neither permission nor a shebang; even if a shebang is present, it is ignored, because the file is not executed, it is read. A new process is created here too. **Sourcing** (`. report.sh` or `source report.sh`) creates no new process. The commands run inside the current shell. The difference of the third can be observed. Let `config.sh` just assign a variable: ```sh # config.sh LOG_FILE=access.log ``` ```sh bash config.sh echo "after subprocess: [${LOG_FILE-undefined}]" . ./config.sh echo "after source: [${LOG_FILE-undefined}]" ``` ``` after subprocess: [undefined] after source: [access.log] ``` The assignment made in the subprocess vanished along with the process. This is the same subshell behavior as in the pipeline lesson: **variables do not pass between processes.** Sourcing is therefore used for configuration files and function libraries; it is not suitable for scripts that do work, because the script's `exit` call closes the calling shell. The `.` form is POSIX; `source` is a bash extension; the two do the same thing. In the POSIX form, if the file name contains no slash, it is searched for in `PATH` — this is why the `./` in `. ./config.sh` is required. #### The Script's Exit Code A script returns the code of the last command it ran. If the following script's last line is a `grep` that finds no match: ```sh #!/usr/bin/env bash # finalcode.sh grep -q '999' access.log ``` ```sh chmod +x finalcode.sh ./finalcode.sh; echo "code=$?" ``` ``` code=1 ``` The script ran successfully but reported failure. If this behavior is not wanted, the code is written explicitly: `exit 0`. Deliberately setting the exit code is what makes a script usable in a chain or a scheduled job. #### Syntax Checking Without Running The `bash -n` option reads and parses the script but does not run it. If a construct is left incomplete, it reports it: Let `broken.sh` be a file that opens an `if` and forgets to close it with `fi`: ```sh # broken.sh if [ -f access.log ]; then echo "log exists" ``` ```sh bash -n broken.sh ``` ``` broken.sh: line 4: syntax error: unexpected end of file ``` This check sees only syntax; it does not catch nonexistent commands, wrong options, or logic errors. Still, it is the cheapest test to run before running a script with destructive operations for the first time. The full set of debugging tools will be taken up in the Script Debugging lesson. #### Summary - A script is a text file holding commands that the shell runs in sequence; a word starting with `#` opens a comment. - Direct execution requires execute permission; if permission is missing, code 126 is returned, and if the name is not found in `PATH`, 127. - For a file whose first two bytes are `#!`, the kernel starts the absolute path in the rest of the line as the interpreter; it does not search, and it looks only at the first line. - `#!/bin/sh` guarantees only the POSIX shell language; a script using a bash extension must declare bash. - `bash script` creates a new process, `. script` does not; in the latter, assignments stay in the calling shell. #### Next Step The log's name is hardcoded in the script's body for now. Taking it into a variable looks, at first glance, like just a matter of tidiness; but in the shell, the result of variable expansion depends on quoting, and an unquoted expansion silently makes the script run wrong on file names that contain spaces. The next lesson takes up expansion, word splitting, and quoting rules. ### Variables and Quoting Rules Source: https://academia.sh/en/courses/shell-programming/variables-and-quoting-rules Assignment syntax, expansion order, word splitting and pathname expansion, the three quoting forms, and default-value expansions. The log's name in the script is hardcoded for now. Taking it into a variable looks, at first glance, like a matter of tidiness. In the shell, though, this means stepping into the language's most error-prone area: expanding a variable is not a single operation but a multi-stage process, and which stages apply depends on quoting. In the Programming Fundamentals course, a variable was a name bound to a value; reading it gave that value. In the shell, the situation is different: the text that comes out when a variable is read gets re-resolved together with the rest of the command line. This lesson's subject is exactly that re-resolution. #### Assignment Assignment is written without spaces on either side of the equals sign. ```sh # try.sh name = value ``` ``` try.sh: line 2: name: command not found ``` The shell took the word `name` as a command name, and `=` and `value` as arguments. Because the shell splits the line into words by whitespace first, it could not have behaved otherwise. The correct form has no spaces: ```sh name=value echo "[$name] [${name}] [${name}less]" ``` ``` [value] [value] [valueless] ``` The braced form marks the boundary between the name and the text that follows it. If `$nameless` had been written, the shell would look for a variable named `nameless`. Variables in the shell have no type; every value is a character string. The assignment `n=5` does not produce an integer, it produces a two-character string. A separate notation is used when a numeric operation is needed; this is the subject of the Arithmetic and Command Substitution lesson. #### What Happens After Expansion Expanding a variable does not finish the job. The shell applies two more operations to the result of an unquoted expansion: 1. **Word splitting.** The resulting text is split into words according to the characters in the `IFS` variable. `IFS`'s default value is space, tab, and newline. 2. **Pathname expansion.** If words contain `*`, `?`, `[...]` characters, they are replaced with matching file names. The result of both operations can be observed. The helper below prints the number of arguments it receives and each argument separately: ```sh show_args() { printf 'arguments: %d\n' "$#"; printf ' <%s>\n' "$@"; } ``` The definition is repeated so that each block can run on its own: ```sh show_args() { printf 'arguments: %d\n' "$#"; printf ' <%s>\n' "$@"; } text="two spaced" show_args $text show_args "$text" ``` ``` arguments: 2 arguments: 1 ``` The unquoted expansion produced two words, the quoted expansion produced one. The three spaces between them also disappeared in the unquoted case — word splitting does not preserve separators. #### The Cost of Unquoted Expansion This behavior has a concrete counterpart in scripts. Consider a file name that contains a space: ```sh show_args() { printf 'arguments: %d\n' "$#"; printf ' <%s>\n' "$@"; } log_file="access log.log" show_args $log_file show_args "$log_file" ``` ``` arguments: 2 arguments: 1 ``` In the unquoted form, the command received two nonexistent file names. In a line like `rm $log_file`, this means the wrong file gets deleted. Pathname expansion adds a second layer. If the variable contains a pattern character, unquoted expansion looks at the file system: ```sh show_args() { printf 'arguments: %d\n' "$#"; printf ' <%s>\n' "$@"; } cd "$(mktemp -d)" # an empty directory containing only two files : > "access log.log" : > missing.log pattern="*.log" show_args $pattern show_args "$pattern" ``` ``` arguments: 2 arguments: 1 <*.log> ``` Note the order: the variable's value was `*.log`; after expansion, word splitting, and pathname expansion, two file names emerged. And `access log.log` arrived as a single argument — because **the result of pathname expansion is not subject to word splitting.** Splitting applies only to the variable's own value. From this comes the rule that will hold for the rest of the course: > **Variable expansions are always enclosed in double quotes.** Leaving one > unquoted is done only when word splitting or pathname expansion is > deliberately wanted. This rule also applies to `"$var"`, `"$@"`, `"$(command)"`, and `"${array[@]}"`. #### Three Quoting Forms **Double quotes** apply expansions but block word splitting and pathname expansion. Variable, command, and arithmetic expansion still work inside. **Single quotes** expand nothing; every character inside passes through literally. A single quote cannot be used inside single quotes — not even a backslash escapes it. **Backslash** strips the single character after it of its special meaning. Inside double quotes it is effective only before `$`, `` ` ``, `"`, `\`, and a newline; in front of other characters it stays literal. ```sh text="two spaced" echo $text echo "$text" echo '$text' ``` ``` two spaced two spaced $text ``` On the first line, `echo` received two separate arguments and printed them with a single space between; on the second line, the spaces were preserved; on the third, no expansion happened at all. Regular expressions and `sed` patterns are written in single quotes: the `$`, `*`, and `\` characters that appear in patterns are not meant to be interpreted by the shell. This rule will hold throughout the Text Processing topic. #### IFS and Deliberate Splitting The `IFS` (internal field separator) variable determines which characters word splitting happens at. Changing it is one of the ways to split a delimited line into its fields: ```sh show_args() { printf 'arguments: %d\n' "$#"; printf ' <%s>\n' "$@"; } line="10.0.0.12:404:512" IFS=: show_args $line ``` ``` arguments: 3 <10.0.0.12> <404> <512> ``` `IFS` is a global setting; once changed, it affects every unquoted expansion in that shell from then on. Forgetting to restore its old value produces hard-to-explain errors in the script's later lines. The correct pattern is to set `IFS` for a single command only: ```sh IFS=: read -r address code bytes <<< "$line" ``` An assignment written before a command is valid only in that command's environment; once the command finishes, the old value returns. This pattern will be used in the Loops lesson when reading the log line by line. #### Default-Value Expansions The shell offers a family of expansions for handling undefined or empty variables on the spot. Whether a colon is present determines whether an "empty" value also counts as missing. | Expansion | If the variable is undefined | If the variable is empty | |---|---|---| | `${v-fallback}` | `fallback` | empty string | | `${v:-fallback}` | `fallback` | `fallback` | | `${v=fallback}` | assigns and gives `fallback` | empty string | | `${v:=fallback}` | assigns and gives `fallback` | assigns and gives `fallback` | | `${v:?message}` | errors and exits | errors and exits | | `${v:+value}` | empty string | empty string | | `${#v}` | 0 | 0 | ```sh unset LOG_FILE echo "1 undefined, with :-: [${LOG_FILE:-access.log}]" LOG_FILE="" echo "2 empty, with :-: [${LOG_FILE:-access.log}]" echo "3 empty, with -: [${LOG_FILE-access.log}]" ``` ``` 1 undefined, with :-: [access.log] 2 empty, with :-: [access.log] 3 empty, with -: [] ``` The distinction matters: an option being deliberately left empty and not being given at all are different situations. `:-` treats them the same, `-` separates them. The `:?` form is the shortest way to declare required variables: ```sh # try.sh unset REQUIRED echo "[${REQUIRED:?must be set}]" ``` ``` try.sh: line 3: REQUIRED: must be set ``` The script stops at this point with a nonzero code. In the Error Handling lesson, this expansion will be used together with strict mode. #### Environment Variables and Scope Shell variables belong, by default, only to that shell. `export` puts the variable into the **environment**; subprocesses started after that point can see it. ```sh export LANG=en bash -c 'echo "in subprocess LANG=$LANG"' LOCAL=x bash -c 'echo "in subprocess LOCAL=[${LOCAL-none}]"' ``` ``` in subprocess LANG=en in subprocess LOCAL=[none] ``` Propagation is one-way: even if the subprocess changes the environment, the parent process is unaffected by it. The rule from the Pipeline lesson holds here too. `readonly` makes a variable unchangeable: ```sh # try.sh readonly CONST=5 CONST=6 ``` ``` try.sh: line 3: CONST: readonly variable ``` A naming convention is widespread and worth following: **names that enter the environment and are shared with the outside world are written uppercase, names specific to the script are written lowercase.** Using lowercase also prevents accidentally overwriting shell variables like `PATH`, `HOME`, `IFS`. #### Applying It to the Script The `report.sh` script gets rid of the hardcoded log name: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. log_file="${LOG_FILE:-access.log}" echo "== Log: $log_file ==" echo "== Most requested paths ==" cut -d' ' -f7 "$log_file" | sort | uniq -c | sort -rn | head -5 ``` ``` == Log: access.log == == Most requested paths == 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /product/45 ``` Three decisions were made together: the name was pulled into a variable, the `LOG_FILE` value coming from the environment was given priority, and the `"$log_file"` expansion was quoted. Without the third, the script would silently try to read the wrong file when given a path containing a space. #### Summary - Assignment leaves no space on either side of the equals sign; the shell cannot parse it any other way, because it splits the line into words. - Word splitting and pathname expansion apply to the result of an unquoted expansion; the result of pathname expansion is not split again. - Variable expansions are always enclosed in double quotes; leaving one unquoted is a deliberate decision. - Double quotes expand and block splitting, single quotes expand nothing, a backslash strips one character. - The difference between `-` and `:-` is whether an empty value counts as missing; `:?` declares a required variable. - `export` propagates a variable to subprocesses; propagation is one-way. #### Next Step The log's name can now come from an environment variable, but this is not the natural path for whoever calls the script: writing `./report.sh access.log` is what is expected. The next lesson takes up passing values to a script from the command line — positional parameters, quoting the argument list, and the difference between `"$@"` and `"$*"` that shows up with arguments containing spaces. ### Special Variables and Arguments Source: https://academia.sh/en/courses/shell-programming/special-variables-and-arguments Positional parameters, the two representations of the argument list, the loop that consumes the list, and option parsing. The log's name now comes from a variable, but this is not the natural path for whoever calls the script. Instead of writing `LOG_FILE=access.log ./report.sh`, writing `./report.sh access.log` is expected. Values coming from the command line are handled in the shell by a separate mechanism — **positional parameters**. In the Programming Fundamentals course, function parameters were defined by name. There is no naming in the shell: parameters are numbered and read by their numbers. This lesson establishes those numbers, the two ways of treating the list as a whole, and option parsing. #### Positional Parameters `$1`, `$2`, … give the arguments the script received, in order. `$0` holds the name the script was called with, `$#` the argument count. ```sh #!/usr/bin/env bash # args.sh echo "script name : $0" echo "argument count : $#" echo "first : $1" echo "second : $2" echo "tenth : ${10}" ``` ```sh chmod +x args.sh ./args.sh a b c d e f g h i j ``` ``` script name : ./args.sh argument count : 10 first : a second : b tenth : j ``` Numbers greater than nine require braces. If `$10` had been written, the shell would read it as `$1` and `0`, producing `a0` by appending a zero after the first argument. Using braces is therefore mandatory. `$0` is the name the script was invoked with; it does not have to be the file's real name, and it may not contain an absolute path. Writing `$0` in usage messages is common, but hardcoding the script's name gives more readable output. #### The Argument List: `"$@"` and `"$*"` There are two representations for passing all arguments together, and the difference between them shows up only inside quotes. - `"$@"` expands every argument as a **separate word**. - `"$*"` joins all arguments with `IFS`'s first character and produces a **single word**. - Unquoted `$@` and `$*` both undergo word splitting, and the distinction disappears. ```sh #!/usr/bin/env bash # list.sh show_args() { printf 'count=%d\n' "$#"; printf ' <%s>\n' "$@"; } echo '--- "$@" ---'; show_args "$@" echo '--- "$*" ---'; show_args "$*" echo '--- $@ ---'; show_args $@ ``` ```sh chmod +x list.sh ./list.sh "access log.log" 404 ``` ``` --- "$@" --- count=2 <404> --- "$*" --- count=1 --- $@ --- count=3 <404> ``` The three results carry three different meanings. The first passes the list while preserving the boundaries the caller gave. The second turns the list into a single piece of text — useful for nothing but printing a message. The third splits the space-containing argument in two and breaks the caller's intent. Rule: **always use `"$@"` when passing arguments on to another command.** This is the quoting rule from the previous lesson extended to positional parameters. #### Consuming the List The `shift` command shifts positional parameters one to the left: `$2` becomes the new `$1`, `$#` decreases by one. It is the foundation of loops that process arguments in order. ```sh #!/usr/bin/env bash # shift.sh while [ "$#" -gt 0 ]; do echo "processing: $1 (remaining: $#)" shift done ``` ```sh chmod +x shift.sh ./shift.sh one two three ``` ``` processing: one (remaining: 3) processing: two (remaining: 2) processing: three (remaining: 1) ``` The form `shift N` discards `N` parameters at once. Trying to discard more than the remaining parameter count returns a nonzero code and leaves the list unchanged. #### Script-Related Special Variables The shell holds several values about the running script under special names. | Name | Content | |---|---| | `$?` | the last command's exit code | | `$$` | the running shell's process number | | `$!` | the number of the last process started in the background | | `$-` | active shell options | | `$0` | the name the script was called with | | `$#` | positional parameter count | `$$` is often used to generate temporary file names; but because process numbers can be reused, it is not safe on its own. The Traps and Cleanup lesson will show the correct way with `mktemp`. `$!` is needed for waiting on and stopping background processes; in the Exit Codes lesson, signal codes were produced using `$$`. #### Option Parsing Parsing arguments by hand — if the first argument is an option, if the second is its value, and so on — quickly becomes unreadable. The `getopts` builtin parses short options in a way that follows the convention. `getopts` takes two arguments: the string of recognized options, and the name of the variable the found option is written to. If a letter in the string is followed by `:`, that option expects a value; the value is placed in the `OPTARG` variable. A `:` at the start of the string declares that error messages will be produced by the script, not by `getopts`. ```sh #!/usr/bin/env bash # opts.sh count=5 verbose=0 while getopts ':n:va' option; do case "$option" in n) count="$OPTARG" ;; v) verbose=1 ;; a) echo "usage: opts.sh [-n NUMBER] [-v] LOG_FILE"; exit 0 ;; :) echo "error: -$OPTARG expects a value" >&2; exit 2 ;; \?) echo "error: unknown option -$OPTARG" >&2; exit 2 ;; esac done shift $((OPTIND - 1)) echo "count=$count verbose=$verbose remaining=$# first=[${1-none}]" ``` Four calls show four behaviors: ```sh chmod +x opts.sh ./opts.sh -n 3 -v access.log ./opts.sh access.log ./opts.sh -n ./opts.sh -z ``` ``` count=3 verbose=1 remaining=1 first=[access.log] count=5 verbose=0 remaining=1 first=[access.log] error: -n expects a value error: unknown option -z ``` The last two calls ended with code `2`. Reporting a usage error with a code separate from another error class is what makes the script usable in a chain. `OPTIND` holds the index of the last argument `getopts` processed. The `shift $((OPTIND - 1))` line after the loop discards the options from the list; only the non-option arguments remain. If this line is forgotten, `$1` still points at the first option. `getopts` recognizes only single-letter options; there is no support for long options like `--verbose`. If a long option is needed, hand-written parsing with `while`–`case` is written instead. Also, the backslash in the `\?` written in the `case` branches is required: `?` is a special symbol in `case` patterns that matches a single character. #### The End-of-Options Marker A value starting with a dash can be mistaken for an option. For example, a file named `-old.log` is interpreted as an option by most commands. The convention is that once `--` is seen in the argument list, everything after it is accepted as not being an option. The pattern below starts with a dash and looks for the `- - [` sequence that appears on every line of the log; without `--`, `grep` would take the pattern for an option: ```sh grep -c -- '- - \[' access.log ``` ``` 30 ``` In your own scripts, `getopts` also recognizes this marker and stops there. Writing `--` when passing file names coming from a variable to another command prevents user input from being confused with an option. #### Applying It to the Script `report.sh` now gains an interface: how many lines to list is given with `-n`, the log name with the first positional parameter. ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. # Usage: report.sh [-n NUMBER] [LOG_FILE] count=5 while getopts ':n:' option; do case "$option" in n) count="$OPTARG" ;; :) echo "error: -$OPTARG expects a value" >&2; exit 2 ;; \?) echo "error: unknown option -$OPTARG" >&2; exit 2 ;; esac done shift $((OPTIND - 1)) log_file="${1:-access.log}" echo "== Log: $log_file (top $count paths) ==" cut -d' ' -f7 "$log_file" | sort | uniq -c | sort -rn | head -"$count" ``` ```sh chmod +x report.sh ./report.sh -n 3 ``` ``` == Log: access.log (top 3 paths) == 6 /api/data 4 /missing.html 4 /index.html ``` The script is still missing something: if the given log does not exist, `cut` errors out and the report header has been printed for nothing. This gap is the next lesson's subject. #### Summary - Positional parameters start at `$1`; numbers greater than nine require braces. `$#` gives the count, `$0` the name it was called with. - `"$@"` expands arguments as separate words, `"$*"` as a single word; unquoted, both undergo word splitting. `"$@"` is used for passing them on. - `shift` shifts the list one to the left and makes a loop that consumes arguments in order possible. - `getopts` parses short options; `OPTARG` holds the value, `OPTIND` the processed index, and `shift $((OPTIND - 1))` is required after the loop. - The `--` marker declares that what follows is not an option, and protects values that begin with a dash. #### Next Step The script assumes that the log given to it exists and is readable. Testing this assumption requires a condition; in the shell, a condition is a command's exit code. The next lesson takes up file, string, and number tests, the difference between `[ ]` and `[[ ]]`, and why this difference is a matter of correctness rather than syntax preference. ### Conditions and Tests Source: https://academia.sh/en/courses/shell-programming/conditions-and-tests A condition being an exit code, file/string/number tests, the parsing difference between single and double brackets, pattern matching, and multi-way selection. The script assumes that the log given to it exists and is readable. Testing this assumption requires a condition. In the Programming Fundamentals course, a condition was a logical expression, and it reduced to a true or false value. There is no such type in the shell. **A condition is a command that gets executed; its result is the exit code.** The `if` construct does not evaluate an expression, it runs a command and executes the body if its code is 0. This single difference explains all the syntax oddities of shell conditions. #### `if` Takes a Command The general form is as follows; any command takes the place of `COMMAND`, and the matching body runs when its exit code is 0: ```text if COMMAND; then ... elif COMMAND; then ... else ... fi ``` A concrete example: ```sh if grep -q ' 404 ' access.log; then echo "404 present" elif grep -q ' 500 ' access.log; then echo "no 404, 500 present" else echo "neither present" fi ``` ``` 404 present ``` Any command can be a condition: ```sh if grep -q '404' access.log; then echo "the log has a 404" fi ``` The `[ ... ]` form used for conditions is not a syntax element, it is a second name for the command called `test`. This is why a space between `[` and the first argument is mandatory, and the closing `]` is this command's last argument. Writing `[-f file]` looks for a command named `[-f` and does not find it. #### File Tests `test` defines a family of operators related to the file system. | Operator | True when | |---|---| | `-e` | the path exists (regardless of type) | | `-f` | regular file | | `-d` | directory | | `-r`, `-w`, `-x` | read / write / execute permission exists | | `-s` | the file exists and its size is greater than zero | | `-L` | symbolic link | | `a -nt b` | `a` was modified more recently than `b` | ```sh [ -f access.log ]; echo "regular file -> $?" [ -d access.log ]; echo "directory -> $?" [ -x access.log ]; echo "executable -> $?" ``` ``` regular file -> 0 directory -> 1 executable -> 1 ``` Permission tests do not check the file's permission bits, they check **whether the calling process can perform that operation on that file**; the result can come out different for the superuser. The permission model established in the Introduction to Linux course turns into behavior here. The distinction between `-e` and `-f` must not be neglected: a directory also passes the `-e` test, but it cannot be read with `cut`. #### String and Number Tests String tests: `-z` is it empty, `-n` is it non-empty, `=` are they equal, `!=` are they different. Number tests use separate operators: `-eq`, `-ne`, `-lt`, `-le`, `-gt`, `-ge`. The two families need to be separate, because in the shell every value is a character string, and "equality" carries two different meanings: ```sh [ "007" = "7" ]; echo "string equality -> $?" [ "007" -eq "7" ]; echo "number equality -> $?" ``` ``` string equality -> 1 number equality -> 0 ``` `007` and `7` are different strings but the same number. It is the one writing the code who decides which meaning is wanted; the choice of operator is the declaration of that decision. Number operators work only with integers. If a decimal value is given, `test` reports an error. If a decimal comparison is needed, a calculation tool is used instead; this is taken up in the Arithmetic and Command Substitution lesson. #### The Single Bracket's Parsing Problem Because `[` is an ordinary command, its arguments are first expanded by the shell and split into words. The result of splitting can change the argument count `test` expects. ```sh # try.sh empty="" [ $empty = "x" ] ``` ``` try.sh: line 3: [: =: unary operator expected ``` After expansion, only the arguments `=`, `x`, `]` reached the `[` command; `test` took this as an expression with a single operand. The same problem shows up in the opposite direction with a value containing a space: ```sh # try.sh d="two words" [ $d = "two words" ] ``` ``` try.sh: line 3: [: too many arguments ``` Both errors are fixed with quoting: ```sh [ "$empty" = "x" ]; echo "code=$?" ``` ``` code=1 ``` In older scripts, a pattern of the form `[ "x$a" = "xb" ]` is seen. Its purpose is to prevent the variable's value from being confused with an operator; since quoting is enough, this pattern is unnecessary now, but it should be recognized for what it does when encountered. #### Double Brackets Bash and ksh define a separate syntax element called `[[ ... ]]`. This is not a command; the shell recognizes it at the parsing stage and **does not apply word splitting or pathname expansion** to the expansions inside it. ```sh empty=""; d="two words" [[ $empty = "x" ]]; echo "empty -> $?" [[ $d = "two words" ]]; echo "spaced -> $?" ``` ``` empty -> 1 spaced -> 0 ``` The unquoted forms did not error. This means quotes are not required inside `[[ ]]` — but writing them does no harm either, and can be preferred so as not to break the habit. Double brackets have three additional capabilities. **Pattern matching.** If the right side of the `=` or `==` operator is unquoted, it is interpreted as a pathname pattern: ```sh path="/api/data" [[ $path == /api/* ]] && echo "under api" [[ $path == "/api/*" ]] || echo "quoted pattern is literal" ``` ``` under api quoted pattern is literal ``` Quoting the right side turns the pattern into an ordinary string. This is the control needed in situations where pattern matching is not wanted. **Regular expression matching.** The `=~` operator treats the right side as an extended regular expression: ```sh code="404" [[ $code =~ ^[45][0-9][0-9]$ ]] && echo "error-class code" ``` ``` error-class code ``` Regular expression syntax will be taken up in detail in the Text Processing topic. What matters here is that the pattern is written unquoted: a quoted right side looks for a literal match. **Logical operators.** `&&` and `||` combine conditions inside the brackets: ```sh [[ -f access.log && -r access.log ]] && echo "readable file" ``` ``` readable file ``` Inside `[ ]`, the `-a` and `-o` operators exist for the same job, but they are not recommended because they produce parsing ambiguity. In portable code, two separate tests are chained with `&&`: ```sh [ -f access.log ] && [ -r access.log ] && echo "readable file" ``` **`[[ ]]` is a bash extension; it does not exist in the POSIX shell language.** It cannot be used in a script declaring `#!/bin/sh`. Decision rule: if bash is declared, `[[ ]]` is preferred — it is immune to the parsing mistakes that word splitting and pathname expansion cause, though quoting the right-hand side is still needed for the pattern/string distinction; if POSIX is required, `[ ]` and careful quoting are used. #### Multi-Way Selection Instead of `if`–`elif` chains that test the same value against many possibilities, `case` is written. The patterns use pathname pattern syntax. ```sh for code in 200 301 404 500 xyz; do case "$code" in 2??) class="successful" ;; 3??) class="redirect" ;; 4??) class="client error" ;; 5??) class="server error" ;; *) class="unknown" ;; esac printf '%-4s -> %s\n' "$code" "$class" done ``` ``` 200 -> successful 301 -> redirect 404 -> client error 500 -> server error xyz -> unknown ``` `case` runs the first matching branch and stops; there is no fall-through between branches. Multiple options are added to a pattern with `|`: `200|201|204)`. The `*)` written as the last branch catches values that match no pattern, and it should be present in every `case` construct — if it is missing, an unrecognized value is silently ignored. `case` is POSIX and provides pattern matching without requiring `[[ ]]`; it is the correct tool for pattern testing in portable scripts. #### Applying It to the Script `report.sh` is extended to validate its inputs. Giving separate codes to error classes continues the contract from the previous lesson. ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. # Usage: report.sh [-n NUMBER] [LOG_FILE] count=5 while getopts ':n:' option; do case "$option" in n) count="$OPTARG" ;; :) echo "error: -$OPTARG expects a value" >&2; exit 2 ;; \?) echo "error: unknown option -$OPTARG" >&2; exit 2 ;; esac done shift $((OPTIND - 1)) log_file="${1:-access.log}" if ! [[ "$count" =~ ^[0-9]+$ ]] || [ "$count" -lt 1 ]; then echo "error: -n expects a positive integer: $count" >&2 exit 2 fi if [ ! -e "$log_file" ]; then echo "error: log does not exist: $log_file" >&2 exit 3 elif [ ! -f "$log_file" ]; then echo "error: not a regular file: $log_file" >&2 exit 3 elif [ ! -r "$log_file" ]; then echo "error: no read permission: $log_file" >&2 exit 3 elif [ ! -s "$log_file" ]; then echo "warning: log is empty: $log_file" >&2 fi echo "== Log: $log_file (top $count paths) ==" cut -d' ' -f7 "$log_file" | sort | uniq -c | sort -rn | head -"$count" ``` The reason the regular-expression test runs before the `[ "$count" -lt 1 ]` test in the number check is order: if `count` were not a number, `-lt` would error out. Because `||` short-circuits, the second test is reached only when the first one passes. Three calls follow three paths: ```sh chmod +x report.sh ./report.sh missing.log 2>&1; echo "code=$?" ./report.sh -n abc 2>&1; echo "code=$?" ./report.sh /etc 2>&1; echo "code=$?" ``` ``` error: log does not exist: missing.log code=3 error: -n expects a positive integer: abc code=2 error: not a regular file: /etc code=3 ``` #### Summary - In the shell, a condition is a command; `if` looks at the exit code. `[ ]` is a second name for the `test` command, and the spaces around it are mandatory. - String equality is tested with `=`, number equality with `-eq`; `007` and `7` are different as strings, the same as numbers. - Because expansions inside `[ ]` undergo word splitting, quoting is mandatory; an empty or space-containing value otherwise produces a parsing error. - `[[ ]]` is recognized at the parsing level, applies no splitting, and offers pattern and regular expression matching along with logical operators; it is a bash extension. - `case` runs the first matching branch, there is no fall-through, and the `*)` branch should always be written. #### Next Step The report currently produces a single list. When a separate count is wanted for each status-code class, the same pipeline has to be duplicated by hand. The next lesson takes up iteration: loops that run over a list, counted loops, and the correct form of reading a file line by line — which is anything but obvious in the shell. ### Loops Source: https://academia.sh/en/courses/shell-programming/loops List-, counter-, and condition-based iteration, safely iterating over file names, the correct form of reading line by line, and loop control. The report currently produces a single list. When a separate count is wanted for each status-code class, the same pipeline has to be written by hand four times. In the Programming Fundamentals course, this was the classic justification for iteration. Loops in the shell build the same structures, but where the list comes from depends on the shell's expansion rules. This lesson takes up the loop forms and — the most commonly miswritten part of shell programming — the pattern for reading a file line by line. #### Iterating Over a List The `for` loop runs the body for every element of the word list given to it. ```sh for class in 2 3 4 5; do count=$(cut -d' ' -f9 access.log | grep -c "^${class}") echo "${class}xx -> $count" done ``` ``` 2xx -> 18 3xx -> 3 4xx -> 6 5xx -> 3 ``` The `$( ... )` notation turns a command's output into a value; the details will be taken up in this topic's final lesson. The concept of a "word" in the list is the same word-splitting concept established in the previous lessons. If the list comes from a variable, quoting directly determines the loop's iteration count: ```sh codes="200 404 500" for code in $codes; do printf '<%s> ' "$code"; done; echo for code in "$codes"; do printf '<%s> ' "$code"; done; echo ``` ``` <200> <404> <500> <200 404 500> ``` This is one of the rare cases where word splitting is deliberately wanted. Still, keeping a value in a single string and relying on it being split is fragile; if more than one value needs to be carried, an array is the correct tool. Arrays are the subject of this topic's seventh lesson. #### Iterating Over File Names The correct way to iterate over the files in a directory is to write the pathname pattern directly into the list: ```sh mkdir -p logs; : > 'logs/a.log'; : > 'logs/b c.log' for f in logs/*.log; do echo " [$f]"; done ``` ``` [logs/a.log] [logs/b c.log] ``` The name containing a space stayed as a single piece; because the result of pathname expansion is not subject to word splitting. This is the practical counterpart of the rule established in the second lesson. Writing the same job as `for f in $(ls logs)` is wrong: `ls`'s output is text and undergoes word splitting; every name containing a space gets split apart. In the shell, the tool for producing a file list is not `ls`, it is pathname expansion. A pattern matching no file at all is a separate trap: ```sh for f in logs/*.missing; do echo " [$f]"; done ``` ``` [logs/*.missing] ``` If there is no match, the shell leaves the pattern literal, and the loop runs once, with a nonexistent file name. There are two solutions. The portable one is to put an existence test at the start of the body: ```sh for f in logs/*.missing; do [ -e "$f" ] || continue echo " [$f]" done ``` The bash-specific one is the `nullglob` option; a pattern with no match expands to an empty list, and the loop does not run at all: ```sh shopt -s nullglob for f in logs/*.missing; do echo " [$f]"; done shopt -u nullglob ``` The option is a global setting and affects every pattern while it is active; turning it on at the start of the script and leaving it on can lead to unexpected behavior in later lines. #### Counted Loops There are three forms for a fixed number of iterations. **Brace expansion** produces a fixed list at parse time: ```sh for i in {1..5}; do printf '%s ' "$i"; done; echo ``` ``` 1 2 3 4 5 ``` Brace expansion cannot contain a variable — `{1..$n}` does not work as expected, because brace expansion is applied **before** variable expansion. This is one of the observable consequences of expansion order. **The C-style loop** is a bash extension and can take a variable bound: ```sh for ((i = 1; i <= 5; i++)); do printf '%s ' "$i"; done; echo ``` ``` 1 2 3 4 5 ``` **A command that produces a sequence** is the portable third way: `for i in $(seq 1 "$n")`. Because of the command invocation, it is the most expensive option, but it works in the POSIX shell language. #### Condition-Based Loops `while` repeats the body as long as the condition command returns 0; `until` as long as it returns nonzero. ```sh counter=0 until [ "$counter" -ge 3 ]; do echo "counter=$counter" counter=$((counter + 1)) done ``` ``` counter=0 counter=1 counter=2 ``` The two can be used interchangeably; `until` saves writing a negation of the condition, and is preferred in situations where it improves readability. #### Reading a File Line by Line This is the most commonly miswritten pattern in shell programming. The correct form has three parts, and each part has its own justification: ```sh counter=0 while IFS= read -r line; do counter=$((counter + 1)) done < access.log echo "lines read: $counter" ``` ``` lines read: 30 ``` **The `-r` option** turns off backslash processing. Without this option, `read` counts the backslash as an escape character and corrupts the data. **The `IFS=` assignment** prevents the leading and trailing whitespace of the line from being trimmed. **Redirection, not a pipe.** A loop at the right end of a pipeline runs in a subshell, and the variables it assigns are lost; this was shown in the Pipeline lesson. The effect of the three parts can be measured. The following file has three lines: the first has two spaces, the second starts with a backslash, the third has whitespace on both ends. ```sh printf 'one two\n\\escaped\n spaced edge \n' > odd.txt while read line; do printf '<%s>\n' "$line"; done < odd.txt ``` ``` ``` ```sh while IFS= read -r line; do printf '<%s>\n' "$line"; done < odd.txt ``` ``` <\escaped> < spaced edge > ``` In the first form, the backslash was swallowed and the edge whitespace was trimmed. The second gave the file as it is. A fourth detail is the last line lacking a newline. If `read` reaches the end of the file without seeing a newline, it writes what it read into the variable but returns a nonzero code; the loop condition misreads this code, and the last line does not get processed: ```sh printf 'first\nsecond' > missing.txt while IFS= read -r s; do echo "read: $s"; done < missing.txt ``` ``` read: first ``` An addition is made to the condition: if the variable is not empty, the loop runs one more iteration. ```sh while IFS= read -r s || [ -n "$s" ]; do echo "read: $s"; done < missing.txt ``` ``` read: first read: second ``` This pattern is required in every script that reads externally sourced data; files that do not end with a newline are not rare. #### Reading a Line Split into Fields If `read` takes multiple variables, it splits the line according to `IFS` and gives the last variable everything that remains. Because the log's fields are separated by spaces, it can be read directly: ```sh head -3 access.log | while read -r address _ _ _ _ method path _ code bytes; do printf '%-12s %-4s %-18s %s %s\n' "$address" "${method#\"}" "$path" "$code" "$bytes" done ``` ``` 10.0.0.12 GET /index.html 200 5120 10.0.0.31 GET /static/style.css 200 2048 10.0.0.12 GET /product/12 200 8410 ``` The name `_` is a convention, not a language element: fields that are not of interest get assigned to the same variable. The `${method#\"}` expansion strips the leading quote; string-processing expansions will be gathered together in the Arithmetic and Command Substitution lesson. This loop is sufficient for data with a fixed field count. When the field count varies or a calculation is needed, a field-based tool is more suitable; in the Text Processing topic, the same job will come down to a single line. #### Loop Control `break` terminates the loop it is in, `continue` skips the iteration. Both can take a number argument to affect outer loops. ```sh for code in 200 301 404 500 xyz 200; do case "$code" in [0-9][0-9][0-9]) : ;; *) echo "invalid code, skipping: $code"; continue ;; esac if [ "$code" -ge 500 ]; then echo "server error found: $code"; break; fi echo "processed: $code" done ``` ``` processed: 200 processed: 301 processed: 404 server error found: 500 ``` `break 2` exits two levels out: ```sh for a in 1 2; do for b in x y; do [ "$b" = y ] && break 2 echo "$a$b" done done ``` ``` 1x ``` A numbered `break` quickly hurts readability; beyond two levels, putting the loop into a function and using `return` becomes clearer. Functions are the subject of the next lesson. #### Applying It to the Script `report.sh` gains a second section: the distribution of status-code classes. ```sh log_file=access.log echo "== Status code classes ==" for class in 2 3 4 5; do count=$(cut -d' ' -f9 "$log_file" | grep -c "^${class}") case "$class" in 2) name="successful" ;; 3) name="redirect" ;; 4) name="client error" ;; 5) name="server error" ;; esac printf ' %sxx %-16s %3d\n' "$class" "$name" "$count" done ``` ``` == Status code classes == 2xx successful 18 3xx redirect 3 4xx client error 6 5xx server error 3 ``` The numbers add up to thirty; equal to the number of lines in the log. This kind of internal consistency check is a cheap way to verify a report script's correctness. #### Summary - A `for` list is formed by the shell's expansion rules; an unquoted variable is split, a quoted one is not. - A file list is produced from pathname expansion, not from `ls` output; if there is no match, the pattern stays literal, and `[ -e "$f" ] || continue` or `nullglob` is required. - Brace expansion cannot take a variable, because it is applied before variable expansion. - The correct form for reading line by line is `while IFS= read -r line; do ... done < file`; if a pipe is used, the loop falls into a subshell. - In files that do not end with a newline, the last line is processed only with the `|| [ -n "$line" ]` addition. #### Next Step As the script grows, the same work — reporting errors, printing a section header, counting a field — repeats in more than one place. The next lesson takes up functions: what a function definition is in the shell, how arguments pass, the difference between `return` and `exit`, and where the shell's scope model departs from the Programming Fundamentals course's. ### Functions and Scope Source: https://academia.sh/en/courses/shell-programming/functions-and-scope Function definition, argument passing, the distinction between returning an exit code and returning a value, local variables, and dynamic scope. As the script grows, the same work repeats in more than one place: reporting errors, printing a section header, counting and sorting a field. In the Programming Fundamentals course, the counterpart of this was the function, and a function was a construct that took named parameters and returned a value. A function in the shell is a different thing. **It is a named command**: it is called with positional parameters, it returns an exit code, and it writes its result — if it produces a value — to its standard output. That is, a shell function is a program's counterpart inside the shell. #### Definition and Invocation ```sh section() { printf '\n== %s ==\n' "$*"; } section "Most requested paths" ``` ``` == Most requested paths == ``` The body sits between braces. Because braces are recognized as words, spaces around them are mandatory, and a `;` or newline is required after the last command. Bash also recognizes the `function name { ... }` form; the form above is the POSIX one. A function must be defined at the point it is called. The shell reads the file from top to bottom; a call before the definition gives a "command not found" error: ```sh # try.sh call_first echo "code=$?" call_first() { echo "you are late"; } ``` ``` try.sh: line 2: call_first: command not found code=127 ``` This is why functions are gathered at the start of the script, above the code that does the work. #### Arguments When a function is called, positional parameters are replaced with the function's own: `$1`, `$2`, `$#`, `"$@"` point at the arguments given to the function. There is one exception — `$0` does not change, it keeps pointing at the script's name. ```sh #!/usr/bin/env bash # f.sh who() { echo "0=$0 1=$1 count=$#"; } who a b ``` ```sh chmod +x f.sh ./f.sh ``` ``` 0=./f.sh 1=a count=2 ``` The rule from the third lesson applies to argument passing: `"$@"` is used, `$*` is written only to produce message text. Shell functions have no signature; a function called with a missing argument does not error, `$1` stays empty. Required arguments must be tested at the start of the body: ```sh class_name() { [ "$#" -eq 1 ] || { echo "class_name: expects one argument" >&2; return 2; } ... } ``` #### Returning an Exit Code `return` ends the function and sets the exit code. The code is subject to the same contract as process exit codes: 0 success, nonzero failure, range 0–255. ```sh valid_code() { case "$1" in [1-5][0-9][0-9]) return 0 ;; *) return 1 ;; esac } valid_code 404; echo "404 -> $?" valid_code abc; echo "abc -> $?" ``` ``` 404 -> 0 abc -> 1 ``` Out-of-range values are reduced modulo 256; a function that writes `return 300` returns `44`. `return` should not be confused with `exit`. `return` only exits the function; `exit` terminates the entire script. ```sh #!/usr/bin/env bash # exit.sh f() { echo "inside f"; return 1; } g() { echo "inside g"; exit 4; } f; echo "continuing after f, code=$?" g; echo "THIS LINE IS NOT WRITTEN" ``` ``` inside f continuing after f, code=1 inside g ``` ```sh bash exit.sh > /dev/null; echo $? ``` ``` 4 ``` Rule: library-like functions use `return` and leave the decision to the caller; only helpers that function as "fatal error" handlers call `exit`. #### Returning a Value `return` is not a tool for returning a value; a code in the range 0–255 cannot carry a string or a large number. In the shell, the way to return a value is to write to standard output, with the caller retrieving it through command substitution. ```sh class_name() { case "$1" in 2) echo "successful" ;; 3) echo "redirect" ;; 4) echo "client error" ;; 5) echo "server error" ;; *) echo "unknown"; return 1 ;; esac } name=$(class_name 4); echo "4 -> $name" ``` ``` 4 -> client error ``` This contract has a direct consequence: **a function that returns a value cannot write anything else to standard output.** If it needs to print a progress message or a warning, it writes to standard error. This is the clearest example of why the stream distinction from the topic's first lesson is needed in practice. The second consequence is cost: command substitution creates a subshell. In scripts that make many calls inside a loop, this leads to a measurable slowdown. Writing the value to a global variable is the alternative; because it lowers readability, it is used only when needed. #### Scope: Global by Default Variables in the shell are **global** by default. An assignment made inside a function is also visible outside. ```sh counter=0 increment() { counter=$((counter + 1)); } increment; increment; echo "counter=$counter" ``` ``` counter=2 ``` This is the reverse of the default in the Programming Fundamentals course; there, an assignment in a function body created a local name. The shell's behavior is a consequence of the function being designed as a "command" — commands see shell variables too. The `local` declaration binds the name to the function: ```sh x="outside" demo() { local x="inside"; echo "inside the function: $x"; } demo; echo "outside the function: $x" ``` ``` inside the function: inside outside the function: outside ``` `local` is not defined in the POSIX shell language; it exists in almost all common shells, but using it in a script declaring `#!/bin/sh` removes the portability guarantee. Rule: **every temporary variable used inside a function is declared `local`.** An undeclared name silently overwrites the same-named variable in the calling context; this is the hardest class of bug to find in long scripts. #### Dynamic Scope A name declared `local` is visible not only to the function it is defined in, but to **every function that function calls**. The shell's scope model is not lexical, it is **dynamic**. ```sh inner() { echo "inner sees: $secret"; } outer() { local secret="from the outer function"; inner; } outer echo "main shell sees: [${secret-undefined}]" ``` ``` inner sees: from the outer function main shell sees: [undefined] ``` The `inner` function saw the name `secret` even though it never defined it, because that name was active along the call chain. This visibility would not occur under the name lookup order introduced in the Programming Fundamentals course; there, which definition a name bound to was determined by looking at where the code was written. Practical consequence: your functions' behavior can become dependent on the local variables of a caller they never see. The way to prevent this is for functions to take every value they need as an argument. #### `local` and the Exit Code `local` is a command, and it produces its own exit code. Doing the declaration and the assignment on the same line makes the right-hand command's code invisible: ```sh fail() { return 3; } try1() { local d; d=$(fail); echo "separate assignment -> $?"; } try2() { local d=$(fail); echo "combined with local -> $?"; } try1; try2 ``` ``` separate assignment -> 3 combined with local -> 0 ``` The `0` seen in the second function is the `local` command's own code. Under strict mode, this leads to a failed command not stopping the script — the same trap will be taken up again in the Error Handling lesson. Rule: **declaration and value assignment are written on separate lines.** #### Functions with a Subshell Body If the body is written with parentheses instead of braces, the function runs in a subshell. Side effects such as changing directory, setting `IFS`, or turning on a shell option do not leak to the caller: ```sh temp() ( cd /tmp && pwd ) ``` Its cost is the cost of creating a process and the loss of global variable assignments. If isolation is genuinely needed, it is the right tool. #### Applying It to the Script `report.sh` takes the repeated work into functions: ```sh SCRIPT_NAME="report.sh" error() { printf '%s: error: %s\n' "$SCRIPT_NAME" "$*" >&2; } warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; } die() { error "$1"; exit "${2:-1}"; } section() { printf '\n== %s ==\n' "$*"; } class_name() { case "$1" in 2) echo "successful" ;; 3) echo "redirect" ;; 4) echo "client error" ;; 5) echo "server error" ;; *) echo "unknown"; return 1 ;; esac } field_count() { local field="$1" file="$2" cut -d' ' -f"$field" "$file" | sort | uniq -c | sort -rn } ``` The checks come down to a single line each, and error codes keep their meaning: ```sh log_file=access.log [[ "$count" =~ ^[1-9][0-9]*$ ]] || die "-n expects a positive integer: $count" 2 [ -f "$log_file" ] || die "not a regular file or does not exist: $log_file" 3 [ -r "$log_file" ] || die "no read permission: $log_file" 3 [ -s "$log_file" ] || warn "log is empty: $log_file" ``` The assembled script looks like this: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. # Usage: report.sh [-n NUMBER] [LOG_FILE] SCRIPT_NAME="report.sh" error() { printf '%s: error: %s\n' "$SCRIPT_NAME" "$*" >&2; } warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; } die() { error "$1"; exit "${2:-1}"; } section() { printf '\n== %s ==\n' "$*"; } class_name() { case "$1" in 2) echo "successful" ;; 3) echo "redirect" ;; 4) echo "client error" ;; 5) echo "server error" ;; *) echo "unknown"; return 1 ;; esac } field_count() { local field="$1" file="$2" cut -d' ' -f"$field" "$file" | sort | uniq -c | sort -rn } count=5 while getopts ':n:' option; do case "$option" in n) count="$OPTARG" ;; :) die "-$OPTARG expects a value" 2 ;; \?) die "unknown option -$OPTARG" 2 ;; esac done shift $((OPTIND - 1)) log_file="${1:-access.log}" [[ "$count" =~ ^[1-9][0-9]*$ ]] || die "-n expects a positive integer: $count" 2 [ -f "$log_file" ] || die "not a regular file or does not exist: $log_file" 3 [ -r "$log_file" ] || die "no read permission: $log_file" 3 [ -s "$log_file" ] || warn "log is empty: $log_file" section "Log: $log_file" section "Most requested $count paths" field_count 7 "$log_file" | head -"$count" section "Status code classes" cut -d' ' -f9 "$log_file" | cut -c1 | sort | uniq -c | while read -r n class; do printf ' %sxx %-17s%3d\n' "$class" "$(class_name "$class")" "$n" done ``` ```sh chmod +x report.sh ./report.sh -n 3 ``` ``` == Log: access.log == == Most requested 3 paths == 6 /api/data 4 /missing.html 4 /index.html == Status code classes == 2xx successful 18 3xx redirect 3 4xx client error 6 5xx server error 3 ``` ```sh ./report.sh missing.log; echo "code=$?" ``` ``` report.sh: error: not a regular file or does not exist: missing.log code=3 ``` Because the `die` function calls `exit`, the script stops there; `error` and `warn` only write and leave the decision to the caller. Separating the three functions this way will provide a single touch point when strict mode is added in later lessons. #### Summary - A shell function is a named command: it is called with positional parameters, returns an exit code, and writes its result to standard output. `$0` does not change inside a function. - `return` sets the exit code and only exits the function; `exit` terminates the script. - The way to return a value is to write to standard output; because of this, a function that returns a value must write its warnings to standard error. - Variables are global by default; the `local` declaration binds the name to the function, and should be written for every temporary variable. - The shell's scope is dynamic: a `local` name is also visible to called functions. - The form `local d=$(command)` hides the command's exit code; declaration and assignment are kept separate. #### Next Step Every count in the report reads the log from start to end once more. Five counts mean five scans. Bringing this down to a single scan requires keeping intermediate results in memory. The next lesson takes up arrays and associative arrays; from which bash version the latter is available, and what to do when it is not, will be shown there too. ### Arrays and Associative Arrays Source: https://academia.sh/en/courses/shell-programming/arrays-and-associative-arrays Definition of indexed and associative arrays, bulk expansion, sparseness, single-pass counting, and checking version dependency. Every count in the report reads the log from start to end once more: five counts mean five scans. This is not noticeable on small files, and becomes apparent as the log grows. Bringing it down to a single scan requires keeping intermediate results in memory. In the Data Structures course, two structures were defined for this job: the **array**, accessed by index, and the **hash table**, accessed by key. The shell offers a form of both. This lesson takes them up, along with their syntax traps and — unavoidable for associative arrays — version dependency. #### Indexed Arrays An array is defined with a word list in parentheses. Indices start at zero. ```sh classes=(2 3 4 5) echo "first=${classes[0]} last=${classes[3]} count=${#classes[@]}" ``` ``` first=2 last=5 count=4 ``` Braces are mandatory: writing `$classes[0]` appends the text `[0]` after the value of the `classes` variable. Without braces, `$classes` alone gives **the first element** — not the whole array. #### Bulk Expansion There are two forms for expanding the array as a whole, and the difference is the same as the `"$@"`–`"$*"` distinction in argument lists. ```sh show_args() { printf 'count=%d\n' "$#"; printf ' <%s>\n' "$@"; } paths=("/api/data" "/product/12" "three word path") show_args "${paths[@]}" show_args "${paths[*]}" show_args ${paths[@]} ``` ``` count=3 count=1 count=5 ``` The rule has not changed: **`"${array[@]}"` is written.** The unquoted form splits the elements, the `[*]` form collapses them all into a single piece of text. Arrays cannot be passed directly to functions; in the shell, function arguments are only a word list. Passing is done through expansion: ```sh paths=("/api/data" "/product/12" "three word path") show() { printf 'received: %s\n' "$@"; } show "${paths[@]}" ``` ``` received: /api/data received: /product/12 received: three word path ``` This means the array passes by being copied; changes inside the function do not propagate back to the caller. #### Appending, Indices, and Sparseness The `+=` operator appends to the end of the array. `${!array[@]}` gives the indices in use. ```sh paths=("/api/data" "/product/12" "three word path") paths+=("/panel") echo "count=${#paths[@]} indices=${!paths[@]}" ``` ``` count=4 indices=0 1 2 3 ``` Shell arrays can be **sparse**: when an element is deleted, the remaining ones do not shift, and a gap forms in the indices. ```sh paths=("/api/data" "/product/12" "three word path" "/panel") unset 'paths[1]' echo "count=${#paths[@]} indices=${!paths[@]}" echo "last element=${paths[-1]}" ``` ``` count=3 indices=0 2 3 last element=/panel ``` This shows why iterating over an array with `for ((i = 0; i < ${#array[@]}; i++))` is wrong: the element count is 3, but the valid indices are 0, 2, and 3. The correct forms are `for e in "${array[@]}"` or `for i in "${!array[@]}"`. The quote in `unset 'paths[1]'` is required; if left unquoted, `[1]` can be interpreted as a pathname pattern. Slicing is done with `${array[@]:start:count}`: ```sh full=(a b c d e) echo "1..3: ${full[@]:1:3}" ``` ``` 1..3: b c d ``` #### Building an Array from Command Output `mapfile` (a synonym is `readarray`) reads a stream line by line into an array. The `-t` option discards line endings. ```sh mapfile -t codes < <(cut -d' ' -f9 access.log | sort -u) echo "distinct code count=${#codes[@]} -> ${codes[*]}" ``` ``` distinct code count=8 -> 200 201 301 302 304 403 404 500 ``` Writing `array=( $(command) )` is common but wrong: the output undergoes word splitting and pathname expansion, and lines containing spaces get split apart. `mapfile` does not apply either operation. It is available from bash 4.0 onward. #### Associative Arrays The `declare -A` declaration creates an array whose keys are strings. This is the shell's counterpart to a hash table, and it makes single-pass counting possible. ```sh declare -A count while read -r code; do count["$code"]=$(( ${count["$code"]:-0} + 1 )) done < <(cut -d' ' -f9 access.log) for key in "${!count[@]}"; do printf '%s -> %s\n' "$key" "${count[$key]}"; done | sort ``` ``` 200 -> 17 201 -> 1 301 -> 1 302 -> 1 304 -> 1 403 -> 2 404 -> 4 500 -> 3 ``` The expression `${count["$code"]:-0}` makes a key seen for the first time start from zero; it shows that default-value expansions also work on array elements. The order of keys is undefined — a natural consequence of a hash table. If sorted output is wanted, `sort` is required; this is why it is there in the example above. A key's presence is tested with the `-v` test: ```sh declare -A count while read -r code; do count["$code"]=$(( ${count["$code"]:-0} + 1 )) done < <(cut -d' ' -f9 access.log) [[ -v count[404] ]] && echo "404 key exists" [[ -v count[999] ]] || echo "999 key does not exist" ``` ``` 404 key exists 999 key does not exist ``` Taking a total per key is the same pattern. The loop below computes the total number of bytes sent to each path in a single scan: ```sh declare -A bytes while read -r path b; do bytes["$path"]=$(( ${bytes["$path"]:-0} + b )) done < <(cut -d' ' -f7,10 access.log) for path in "${!bytes[@]}"; do printf '%8d %s\n' "${bytes[$path]}" "$path"; done | sort -rn | head -5 ``` ``` 36120 /panel 25230 /product/12 21990 /product/45 20480 /index.html 7236 /api/data ``` #### Version Dependency Associative arrays are available from bash's 4.0 version onward. On an older bash, `declare -A` is not recognized — and the result is dangerous because it is silent. ```sh #!/usr/bin/env bash declare -A count count[404]=4 echo "404 -> ${count[404]}" ``` Bash 4 and later: ``` 404 -> 4 ``` Bash 3.2: ``` assoc.sh: line 2: declare: -A: invalid option declare: usage: declare [-afFirtx] [-p] [name[=value] ...] 404 -> 4 ``` The script printed an error message but **kept running and produced a result that looked correct.** The reason is this: once the declaration failed, `count` became an ordinary indexed array, and the key `404` was interpreted as a numeric index. As long as the keys are integers, the result comes out correct by coincidence. Once the keys are strings, the concealment ends: ``` text-key.sh: line 3: /api/data: syntax error: operand expected (error token is "/api/data") ``` An indexed array treats the text inside the brackets as an arithmetic expression, and cannot evaluate a path name. The solution is to test the requirement at the start of the script. The `BASH_VERSINFO` array holds the version components as numbers: ```sh #!/usr/bin/env bash # version-check.sh if ((BASH_VERSINFO[0] < 4)); then echo "this script requires bash 4 or later (found: $BASH_VERSION)" >&2 exit 4 fi declare -A count count[404]=4 echo "404 -> ${count[404]}" ``` The script stops immediately when called with an interpreter older than version four; for example, on a system shell that has stayed at version 3.2, it writes the message `this script requires bash 4 or later (found: 3.2.57(1)-release)` and exits with code 4. On bash 4 and later the test passes and the script keeps running: ```sh /bin/bash version-check.sh echo "exit code: $?" ``` ``` 404 -> 4 exit code: 0 ``` This test reports the error at its source, not where it surfaces. It should be present in every script that requires a specific version. #### The POSIX Shell Has No Arrays The POSIX shell language defines neither indexed nor associative arrays. Its only list structure is positional parameters, and they can be filled with `set --`: ```sh set -- 2 3 4 5 echo "count=$# first=$1 last=$4" for s in "$@"; do printf '%s ' "$s"; done; echo ``` ``` count=4 first=2 last=5 2 3 4 5 ``` There is only one portable list, and it overwrites the script's own arguments; this is a serious limitation. The POSIX solution for per-key counting is not to build a data structure in the shell, but to hand the job off to a tool designed for it. One line of a field-based processing tool takes the place of this entire lesson's loop, and works the same everywhere: ```sh awk '{ count[$9]++ } END { for (k in count) print k, count[k] }' access.log | sort ``` ``` 200 17 201 1 301 1 302 1 304 1 403 2 404 4 500 3 ``` This tool will be taken up in detail in the Text Processing topic. The lesson here is about choice: **building a data structure in the shell makes sense when the data needs to stay in the shell;** if only counting is needed, an external tool is both faster and more portable. #### Summary - Array elements are accessed with `${array[i]}`; unbraced `$array` gives only the first element. - `"${array[@]}"` expands elements as separate words; `[*]` collapses them into one piece of text, the unquoted form splits them. - Shell arrays can be sparse; because the element count and the indices may not line up, `"${!array[@]}"` is used instead of a counted loop. - `mapfile -t` reads a stream into an array without applying splitting or expansion. - Associative arrays are available from bash 4.0 onward; when missing, the declaration fails but the script keeps running. A version test with `BASH_VERSINFO` is mandatory. - The POSIX shell language has no arrays; a field-based tool is used for per-key counting. #### Next Step In this lesson's counts, the notations `$(( ... ))` and `$( ... )` were used without justification. The next lesson defines these two notations: why arithmetic in the shell is integer-only, how decimal calculation is done, which characters are lost when a command's output is turned into a value, and the cost of that conversion. ### Arithmetic and Command Substitution Source: https://academia.sh/en/courses/shell-programming/arithmetic-and-command-substitution Integer arithmetic and its traps, the arithmetic command's inverted exit code, offloading decimal calculation, and turning command output into a value. In the previous two lessons, the notations `$(( ... ))` and `$( ... )` were used without justification. Both are fundamental shell tools, and both have corners that silently produce a wrong result. The principle established in the Variables lesson still holds: every value in the shell is a character string. These two notations temporarily carry those strings into another domain and bring them back — one into the domain of numbers, the other into the domain of command output. #### Arithmetic Expansion The `$(( expression ))` notation evaluates the expression inside as an integer and substitutes the result as text. ```sh echo "$(( 7 / 2 )) $(( -7 / 2 )) $(( 7 % 2 )) $(( 2 ** 10 ))" ``` ``` 3 -3 1 1024 ``` Division **truncates toward zero**: it gives 3 for $7/2$, $-3$ for $-7/2$. This is different from languages that round down; the distinction matters when working with negative numbers. Variables can be written without a dollar sign inside the expression; the inside of the parentheses is already the number domain: ```sh a=6; b=4 echo "$(( a + b )) $(( $a * $b ))" ``` ``` 10 24 ``` Whitespace inside the parentheses is free, and word splitting does not apply. The expression `$(( 1+2 * 3 ))` gives 7; precedence rules are the same as the operator precedence in the Programming Fundamentals course. #### The Base Trap The shell treats a number starting with zero as **octal**. Values coming from zero-padded fields — hour, minute, sequence number — therefore produce an error: ```sh # try.sh echo "$(( 08 ))" ``` ``` try.sh: line 2: 08: value too great for base (error token is "08") ``` `08` is not valid in base eight. The base needs to be declared explicitly: ```sh echo "$(( 10#08 ))" ``` ``` 8 ``` The `10#` prefix reads the value in base ten. It should be used on every value coming from an external source that might be zero-padded. #### The Arithmetic Command and the Inverted Exit Code `(( expression ))` is not an expansion, it is a command: it does not substitute a value, it only calculates and produces an exit code. The code is **the inverse of the value**: if the result is nonzero, 0 (success); if the result is zero, 1 (failure). ```sh (( 1 + 1 )); echo "result 2 -> code=$?" (( 0 )); echo "result 0 -> code=$?" ``` ``` result 2 -> code=0 result 0 -> code=1 ``` This inversion is not arbitrary, it is the reconciliation of two conventions: in C-like languages, a nonzero value is "true," while in the shell, code 0 is "success." The result is that arithmetic conditions can be written naturally: ```sh (( 5 > 3 )) && echo "comparison is true" ``` ``` comparison is true ``` The trap shows up under strict mode. The expression `(( counter++ ))` returns code 1 while `counter` is zero; in a script with `set -e` active, this stops the script. The safe form for incrementing a counter is `counter=$(( counter + 1 ))`. This trap will be taken up again in the Error Handling lesson. Operators such as `++`, `--`, `+=` can be used inside `(( ))`: ```sh counter=0; (( counter++ )); (( counter += 5 )); echo "counter=$counter" ``` ``` counter=6 ``` In older scripts, `let` and `expr` are seen for the same job. `let` is a bash extension and is equivalent to `(( ))`. `expr` is a separate program; it creates a process on every call, and its operators need to be escaped so the shell does not interpret them. Neither is needed in new scripts. #### The Shell Has No Decimal Calculation Shell arithmetic is integer-only. When a decimal result is needed, the work is handed off externally. ```sh awk 'BEGIN { printf "%.2f\n", 4 / 30 * 100 }' ``` ``` 13.33 ``` A calculator tool that computes with arbitrary precision can also be used, but it should be kept in mind that the `scale` setting is applied **at every division**: ```sh echo 'scale=2; 4 / 30 * 100' | bc echo 'scale=2; 4 * 100 / 30' | bc ``` ``` 13.00 13.33 ``` In the first form, the division happened first and the result was truncated to 0.13; the multiplication proceeded with this truncated value. Arranging the order of operations so that multiplication comes before division is a general rule in fixed-precision calculation. The third way is scaled integer arithmetic. Placing the decimal point by hand gives one digit of precision without calling an external tool: ```sh part=4; total=30 printf '%d.%d\n' $(( part * 1000 / total / 10 )) $(( part * 1000 / total % 10 )) ``` ``` 13.3 ``` This method truncates, it does not round. It is sufficient for report-style output; not for monetary or scientific calculation. #### Command Substitution The `$( command )` notation runs the command, captures its standard output, and substitutes that text. ```sh count=$(wc -l < access.log) echo "[$count]" ``` ``` [30] ``` There is no leading space in this output, because GNU `wc` writes the count without padding when it reads the file from standard input. The BSD-derived `wc` (macOS) aligns the count with spaces in the same command and produces `[ 30]`; this difference, mentioned in the first lesson, silently breaks scripts that compare the number as text. Arithmetic expansion absorbs the spaces in both cases: ```sh count=$(( $(wc -l < access.log) )) echo "[$count]" ``` ``` [30] ``` This nested form is the standard way to turn a command's output into a number. #### The Transformations Substitution Applies Three behaviors need to be known. **Trailing newlines are stripped.** All trailing newlines in the command's output are discarded; the ones in between are preserved. ```sh d=$(printf 'a\n\n\n') printf '<%s>\n' "$d" ``` ``` ``` This behavior is wanted most of the time, but it leads to loss in data where the newline is meaningful. **Unquoted writing is split.** The rule that holds for variable expansion holds here too: ```sh show_count() { printf 'count=%d\n' "$#"; } show_count $(head -2 access.log) show_count "$(head -2 access.log)" ``` ``` count=20 count=1 ``` In the unquoted form, the two lines were split into twenty words. **Command substitution is always quoted.** **The exit code is not lost by assignment, but it can be hidden.** An assignment that only does substitution leaves the command's code in `$?`: ```sh d=$(grep '999' access.log); echo "code=$?" ``` ``` code=1 ``` But if `local`, `export`, or `declare` precedes the assignment, the code belongs to that command instead, and the real result does not show; this trap was shown in the Functions and Scope lesson. #### Syntax Forms The older counterpart of the `$( ... )` form is the backtick. It is not preferred for two reasons: nested use requires a backslash for every level, and a backslash's meaning inside backticks is different. `$( ... )` requires no escaping at all in nested use: ```sh echo "most requested: $(cut -d' ' -f7 "$(echo access.log)" | sort | uniq -c | sort -rn | head -1)" ``` ``` most requested: 6 /api/data ``` To read an entire file, bash offers the `$(< file)` shortcut; because it creates no process, it is cheaper than the `$(cat file)` form: ```sh printf 'hello\n' > single.txt echo "[$(< single.txt)]" ``` ``` [hello] ``` #### Cost Command substitution creates a subshell and, usually, an external process. This is unnoticeable on a thirty-line log; in a script that makes one call per line inside a loop, process creation alone can make up the entire workload. Rule: **if command substitution is inside a loop body, a way to do the same job with a single external tool call is sought.** This entire next topic is the application of this principle. #### Applying It to the Script `report.sh` gains a summary block and a percentage column: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. # Usage: report.sh [-n NUMBER] [LOG_FILE] SCRIPT_NAME="report.sh" error() { printf '%s: error: %s\n' "$SCRIPT_NAME" "$*" >&2; } warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; } die() { error "$1"; exit "${2:-1}"; } section() { printf '\n== %s ==\n' "$*"; } class_name() { case "$1" in 2) echo "successful" ;; 3) echo "redirect" ;; 4) echo "client error" ;; 5) echo "server error" ;; *) echo "unknown"; return 1 ;; esac } field_count() { local field="$1" file="$2" cut -d' ' -f"$field" "$file" | sort | uniq -c | sort -rn } percent() { # percent PART TOTAL -> "13.3" local part="$1" total="$2" (( total == 0 )) && { echo "0.0"; return; } printf '%d.%d' $(( part * 1000 / total / 10 )) $(( part * 1000 / total % 10 )) } count=5 while getopts ':n:' option; do case "$option" in n) count="$OPTARG" ;; :) die "-$OPTARG expects a value" 2 ;; \?) die "unknown option -$OPTARG" 2 ;; esac done shift $((OPTIND - 1)) log_file="${1:-access.log}" [[ "$count" =~ ^[1-9][0-9]*$ ]] || die "-n expects a positive integer: $count" 2 [ -f "$log_file" ] || die "not a regular file or does not exist: $log_file" 3 [ -r "$log_file" ] || die "no read permission: $log_file" 3 [ -s "$log_file" ] || warn "log is empty: $log_file" requests=$(( $(wc -l < "$log_file") )) bytes=$(awk '{ t += $10 } END { print t + 0 }' "$log_file") section "Log: $log_file" printf ' request count : %d\n' "$requests" printf ' total bytes : %d\n' "$bytes" printf ' per request : %d bytes\n' $(( requests > 0 ? bytes / requests : 0 )) section "Most requested $count paths" field_count 7 "$log_file" | head -"$count" section "Status code classes" cut -d' ' -f9 "$log_file" | cut -c1 | sort | uniq -c | while read -r n class; do printf ' %sxx %-17s%3d %%%s\n' "$class" "$(class_name "$class")" "$n" "$(percent "$n" "$requests")" done ``` ```sh chmod +x report.sh ./report.sh -n 3 ``` ``` == Log: access.log == request count : 30 total bytes : 117780 per request : 3926 bytes == Most requested 3 paths == 6 /api/data 4 /missing.html 4 /index.html == Status code classes == 2xx successful 18 %60.0 3xx redirect 3 %10.0 4xx client error 6 %20.0 5xx server error 3 %10.0 ``` The ternary operator in `$(( requests > 0 ? bytes / requests : 0 ))` prevents division by zero. In shell arithmetic, division by zero is an error that stops the script; the divisor must always be tested. The percentages add up to 100.0, and the numbers add up to the request count. This internal consistency is the cheapest test of the report's correctness. #### Summary - `$(( ))` does integer arithmetic, truncates toward zero, and lets variables inside be written without a dollar sign; numbers starting with zero are read as octal, requiring the `10#` prefix. - `(( ))` is a command, and its exit code is the inverse of the value; a zero result reports failure. - Decimal calculation does not exist in the shell; it is handed off externally, or done with scaled integers. - Command substitution strips trailing newlines and gets split in unquoted form; it is always quoted. - Every substitution costs a subshell; calls inside a loop body should be replaced with a single bulk external tool call. #### Next Step This topic completed the script's skeleton: the shebang, the argument interface, input validation, functions, loops, and calculation. The script works, but it does most of its job with blunt tools: it splits paths with `cut`, counts status codes with `grep -c`, and reads the log from the start for every count. The next topic hands these jobs off to their real tools. It starts with regular expressions — the common language for splitting the log's request line into its fields, filtering out invalid lines, and reformatting the timestamp. ## Text Processing Regular expressions, line filtering, stream editor transformation, field-based processing, sorting and deduplication, and column-level transformations. ### Regular Expressions Source: https://academia.sh/en/courses/shell-programming/regular-expressions The basic-versus-extended syntax distinction, quantifiers, anchors and bracket expressions, greediness, and portable character classes. The script has so far split paths with `cut` and counted status codes with fixed strings. Both approaches assume the data's format is flawless. Real logs contain broken lines, missing fields, and unexpected formats; and patterns such as "codes starting with four hundred" cannot be expressed with a fixed string. A **regular expression** is a pattern language that describes a set of strings. In the text algorithms topic of the Algorithms course, pattern matching was defined for a single fixed string; a regular expression sets up the same problem for a set that can be infinite, instead of a single string. This lesson builds the pattern language. The following lessons connect it to tools. #### Two Syntaxes POSIX defines two regular expression syntaxes, and tools state which one they use through an option. **Basic regular expression (BRE)** is the default syntax for the `grep` and `sed` commands. Grouping, alternation, and some quantifiers are written with a backslash. **Extended regular expression (ERE)** is used by `grep -E` and `awk`. The same operators are written without the backslash. ```sh grep -c '\(404\|500\)' access.log grep -Ec '(404|500)' access.log ``` ``` 7 7 ``` The two patterns describe the same set; only their spelling differs. This course prefers ERE: it is easier to read and avoids backslash pileup. Patterns are written **inside single quotes**. The `$`, `*`, `\`, `[` characters in the pattern language also carry meaning for the shell; single quotes prevent the shell from interpreting them. This is a direct application of the rule from the Variables and Quoting Rules lesson. #### Literal Matching and the Dot Every character with no special meaning matches itself. `.`, on the other hand, matches **any single character** — this is the most frequently overlooked special character. ```sh printf '%s\n' 'style.css' 'styleXcss' 'style-css' > names.txt grep 'style.css' names.txt ``` ``` style.css styleXcss style-css ``` Once the dot is escaped, only a literal dot matches: ```sh grep 'style\.css' names.txt ``` ``` style.css ``` In data containing dots — file names, domain names, IP addresses — this distinction is a matter of correctness. #### Quantifiers A quantifier states how many times the element before it repeats. | ERE | BRE | Meaning | |---|---|---| | `*` | `*` | zero or more | | `+` | `\+` | one or more | | `?` | `\?` | zero or one | | `{n}` | `\{n\}` | exactly `n` | | `{n,m}` | `\{n,m\}` | at least `n`, at most `m` | ```sh printf '%s\n' 'color' 'colorr' 'colorrr' > rep.txt ``` ```sh grep -Ec 'color+' rep.txt grep -Ec 'color{2}' rep.txt grep -Ec 'color{1,2}' rep.txt ``` ``` 3 2 3 ``` `color{2}` matched two lines: `colorr` and `colorrr` — in the second, the first two `r` characters are enough, because the pattern does not have to cover the entire line. This last sentence is critical. **A regular expression looks for a match anywhere in the line.** If a full-line match is wanted, an anchor is used. #### Anchors `^` marks the start of the line, `$` the end. Neither matches a character; both match a **position** and have no width. ```sh grep -c '^10\.0\.0' access.log ``` ``` 14 ``` When the two are used together, the pattern covers the entire line. This is how every line of a log is confirmed to be in the expected format: ```sh PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$' grep -Ec "$PATTERN" access.log ``` ``` 30 ``` All thirty lines fit the pattern. When broken lines mix into the log, the `-v` option filters them out: ```sh PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$' { cat access.log echo 'broken line' echo '10.0.0.1 - - [x] "GET /a HTTP/1.1" 200 5' } > all.log grep -Ev "$PATTERN" all.log ``` ``` broken line ``` Two broken lines were added, but the pattern caught only one. The second one's timestamp is not realistic, yet the `\[[^]]+\]` part says "at least one character between the square brackets," so it still fits the pattern. A pattern validates only as much as it describes; a complementary check based on field count will be added in the Field-Based Processing lesson. Because this pattern is held in a variable, it is expanded with **double quotes**; had single quotes been used, the variable name would have been passed through literally. The pattern itself was inside single quotes at the point of assignment. #### Bracket Expressions A bracket expression defines a set of characters and matches **one** of the characters inside it. - `[abc]` — one of three letters - `[0-9]` — a range - `[^0-9]` — not in the set (negation) - `[.-]` — most special characters lose their meaning inside a bracket expression ```sh cut -d' ' -f9 access.log | grep -c '^[^2]' cut -d' ' -f9 access.log | grep -Ec '^[45]' ``` ``` 12 9 ``` Of thirty requests, twelve are outside the success class, nine are in the error class. Negation carries its special meaning only in the **first** position of the bracket expression; `[a^b]` matches one of three characters. The hyphen follows the same rule: in the last position it is literal. #### Portable Character Classes POSIX defines named character classes. They are written inside a bracket expression and are themselves bracketed: | Class | Matches | |---|---| | `[[:digit:]]` | digits | | `[[:alpha:]]` | letters | | `[[:alnum:]]` | letters and digits | | `[[:space:]]` | whitespace kinds | | `[[:upper:]]`, `[[:lower:]]` | uppercase and lowercase letters | ```sh grep -Eo '^[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+' access.log | sort -u ``` ``` 10.0.0.12 10.0.0.31 10.0.2.7 172.16.0.9 192.168.1.5 ``` The `-o` option writes not the whole line but only the matched portion. Some implementations offer shortcuts such as `\d`, `\w`, `\s`. These are not defined in POSIX; tools that support them and tools that do not exist side by side. This is the most common reason for getting a different result when running the same pattern on two different systems. **A portable script writes `[[:digit:]]`.** The named classes' second advantage is that ranges such as `[a-z]` depend on the locale: when the sort order changes, the characters a range covers change too. `[[:lower:]]` carries no such ambiguity. #### Greediness Quantifiers are **greedy**: they take the longest text they can match. ```sh echo 'a"one" and "two"z' | grep -Eo '".*"' echo 'a"one" and "two"z' | grep -Eo '"[^"]*"' ``` ``` "one" and "two" "one" "two" ``` The `.*` in the first pattern swallowed everything between the two quote pairs. The second pattern, by saying "characters that are not a quote" with `[^"]*`, matched each quoted piece separately. Common implementations **have no** non-greedy quantifier (such as `*?`); the latest POSIX edition defines one, but the tools do not implement it. The solution is always the same: a character class that negates the delimiter. This pattern — opening delimiter, repetition of non-delimiter characters, closing delimiter — is the basic form of field extraction, and it is also used to capture the square-bracketed timestamp in the log. #### Grouping and Backreferences Parentheses group a subpattern; they are used both to apply a quantifier and to store the matched text. `\1`, `\2` … **refer back** to the stored text (a **backreference**). ```sh printf '%s\n' 'aa' 'ab' 'abab' > repeat.txt grep -E '(.)\1' repeat.txt grep -E '(..)\1' repeat.txt ``` ``` aa abab ``` The first pattern describes "the same character repeated twice," the second "the same two characters repeated twice." The backreference is the point where regular expressions go beyond their definition in formal language theory: patterns with backreferences exceed the regular language class, and matching cost can become exponential. The real use of groups is substituting the matched portions; this is taken up in the Transforming with the Stream Editor lesson. #### A Method for Writing Patterns Complex patterns are not written in one pass. The sequence to follow is this: 1. Take one of the lines you want matched and search for it literally. 2. Replace the variable parts, one at a time, with a class and a quantifier; watch the match count with `grep -c` at every step. 3. If the count is larger than expected, the pattern is too wide; if smaller, it is missing a case. 4. Run the pattern with `-o` and visually confirm the matched portion. 5. In the last step, add anchors and move to a full-line match. The line-validation pattern above was built exactly this way. Producing a measurable number at every step is the cheapest way to test a pattern. #### Summary - A regular expression describes a set of strings; POSIX defines two syntaxes, and tools state which one they use through an option. - Patterns are written inside single quotes; `.` matches any character and must be escaped to match a literal dot. - A pattern matches anywhere in the line; a full-line match is built with the `^` and `$` anchors. - Named character classes are portable and unaffected by locale; shortcuts such as `\d` are not defined in POSIX. - Quantifiers are greedy and common tools have no non-greedy form; the solution is a character class that negates the delimiter. #### Next Step The pattern language is built; now it needs to be connected to a tool. The next lesson takes up line filtering: beyond selecting matching lines, counting them, inverting the selection, writing only the matched portion, listing file names, and using `grep`'s exit code as a condition. ### Line Filtering Source: https://academia.sh/en/courses/shell-programming/line-filtering Selecting, counting, and inverting lines by pattern, word and full-line matching, fixed string mode, multi-file search, and using the exit code as a condition. The pattern language is built. The first tool that applies it is the filter that selects, from a stream, the lines that fit a pattern. This tool is the most frequently called of shell text processing, and most of its options turn the behavior "write the matching lines" into a different question. This lesson takes up those questions and the correct option for each. #### Selecting, Counting, Locating The default behavior is to write the matching lines. The `-n` option adds the line number: ```sh grep -n '500' access.log ``` ``` 8:192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180 13:192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180 27:10.0.2.7 - - [07/Feb/2024:09:35:44 +0000] "POST /api/data HTTP/1.1" 500 180 ``` `-c` gives only the number. What is counted is the **number of matching lines**, not the number of matches: ```sh grep -Ec '\.' access.log grep -Eo '\.' access.log | wc -l ``` ``` 30 131 ``` All thirty lines have at least one dot; the total dot count is 131. The distinction separates the questions "how many requests got a 404" from "how many times does the string 404 occur." If a match count is wanted, `-o` spills each match onto its own line and it is counted. The `grep -c ''` form gives a line count, because the empty pattern matches every line, and it does not produce `wc -l`'s alignment padding: ```sh grep -c '' access.log ``` ``` 30 ``` #### Inverting The `-v` option selects lines that do **not** match. It is the most-used option in filtering jobs; it is the direct way to drop noise lines. ```sh grep -vc '" 200 ' access.log ``` ``` 13 ``` Thirteen of thirty requests got a code other than `200`. `-v` can be combined with `-c`; combining it with `-o` is meaningless, because a non-matching line has no match to write. #### Narrowing the Boundaries of a Match Three options restrict which part of the line a pattern can match against. `-i` removes case sensitivity. `-w` requires the pattern to start and end at **word boundaries**. `-x` requires the pattern to match the **entire line**. ```sh printf '%s\n' 'GET /a' 'get /b' 'GETTER /c' > case.txt grep -c 'GET' case.txt grep -ic 'GET' case.txt grep -wc 'GET' case.txt ``` ``` 2 3 1 ``` The file contains the lines `GET /a`, `get /b`, and `GETTER /c`. The default search selected two, the case-insensitive search three, the word search only one — `GET` inside `GETTER` is not a word. `-x` is a shortcut for writing anchors: ```sh printf '%s\n' '404' '4040' > code.txt grep -c '404' code.txt grep -xc '404' code.txt ``` ``` 2 1 ``` The line `4040` in the file fit the unanchored pattern. In checks that want an exact match, forgetting `-x` silently produces an inflated result. #### Fixed String Mode The `-F` option interprets the pattern not as a regular expression but as a literal string. ```sh printf '%s\n' 'a.b' 'axb' > dot.txt grep -c 'a.b' dot.txt grep -Fc 'a.b' dot.txt ``` ``` 2 1 ``` `-F` matters for two reasons. First, correctness: if the pattern comes from a variable — user input, a file name, a configuration value — the `.`, `*`, `[` characters inside it are interpreted as pattern syntax, and the search gives a wider result than expected. Second, using user input as a pattern is a security boundary: a crafted pattern can deliberately blow up the matching cost. Rule: **if you did not write the pattern yourself, use `-F`.** Patterns can be kept in a file; the `-f` option counts each line as a separate pattern: ```sh printf '%s\n' '403' '500' > patterns.txt grep -Fcf patterns.txt access.log ``` ``` 5 ``` #### Multi-File Search When more than one file is given, the file name is prefixed to the output: ```sh mkdir -p log-dir head -10 access.log > log-dir/monday.log tail -10 access.log > log-dir/tuesday.log grep -c '404' log-dir/*.log ``` ``` log-dir/monday.log:1 log-dir/tuesday.log:2 ``` `-h` suppresses this prefix, `-H` adds it even for a single file. If the output is going to be parsed in a script, the option must be written explicitly; an output whose prefix changes with the file count is a fragile foundation. `-l` gives the names of files where a match was found, `-L` the names where it was not: ```sh grep -l '500' log-dir/*.log ``` ``` log-dir/monday.log log-dir/tuesday.log ``` `-r` searches a directory tree recursively. The order files are visited is whatever order the file system gives, and it varies from implementation to implementation; if sorted output is needed, the result must be sorted separately with `sort`. `-m N` stops after the first `N` matches in each file. For the question "does it exist" on large files, this avoids reading the whole file. #### Context Lines `-A N` writes the `N` lines after a match, `-B N` before, `-C N` on both sides. A `--` separator is placed between blocks. ```sh grep -A 1 '" 500 ' access.log | head -4 ``` ``` 192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180 10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300 -- 192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180 ``` This is used in debugging to look at the requests before and after a fault. Because the separator line breaks the output's format, context options are not used on lines whose output will flow into another tool. #### Using the Exit Code as a Condition The `-q` option writes nothing; it stops at the first match and reports the result only through its exit code. This is the correct form of an "does it exist" question in a script: ```sh if grep -q '" 500 ' access.log; then echo "server error found" fi ``` ``` server error found ``` The form `grep -c ... > /dev/null` does not do the same job: `-c` writes `0` even without a match, and the exit code still depends on whether there was a match, but the entire file is read regardless. `-q` stops at the first match. The three-code contract built in the Exit Codes lesson applies here: `0` found, `1` not found, `2` file error. In scripts running under strict mode, the "not found" case must be used as a condition, or `|| true` must be added, so it does not stop the script. #### Applying to the Script `report.sh` gains two new sections. The first counts the log's broken lines; the second lists the paths that returned an error. ```sh #!/usr/bin/env bash # report.sh SCRIPT_NAME="report.sh" warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; } section() { printf '\n== %s ==\n' "$*"; } log_file="${1:-access.log}" LINE_PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$' malformed=$(grep -Evc "$LINE_PATTERN" "$log_file") if [ "$malformed" -gt 0 ]; then warn "$malformed line(s) not in the expected format and included in the counts" fi section "Paths returning errors" grep -E '" [45][0-9]{2} ' "$log_file" | cut -d' ' -f7 | sort | uniq -c | sort -rn ``` The block also contains the `warn` and `section` helpers and takes the log name from the first argument, so it runs on its own. Run with a log that has a broken line: ```sh { cat access.log; echo 'broken line'; } > all.log chmod +x report.sh ./report.sh all.log ``` The warning falls to standard error, the report continues to standard output. A broken line carries no error code, so the error-returning-paths section is the same as for a clean log: ``` report.sh: warning: 1 line(s) not in the expected format and included in the counts == Paths returning errors == 4 /missing.html 3 /api/data 2 /secret ``` The quotes and spaces in the `grep -E '" [45][0-9]{2} '` pattern separate the status code field from another number in the log. This is imitating the concept of a field with a pattern; in the following lessons, field-based tools will do the same job by position, and the pattern will no longer be needed. #### Filtering's Place in the Pipeline Concurrency was established in the pipeline lesson. Its practical consequence is an ordering rule: **put the step that shrinks the data at the start of the pipeline.** ```sh grep -E '" [45][0-9]{2} ' "$log_file" | cut -d' ' -f7 | sort | uniq -c ``` In this pipeline, `grep` brings thirty lines down to nine; `sort` sorts nine lines. Had sorting come last, thirty lines would have been sorted. That sorting cost depends on input size — as shown in the Algorithms course — is the justification for this rule. The rule carries one exception: filtering must not drop lines the next step needs. In transformations that require context, filtering is deferred to later. #### Summary - `-c` counts matching lines, not matches; a match count is spilled into lines with `-o` and counted. - `-v` inverts the selection; `-w` requires a word match, `-x` a full-line match. - `-F` interprets the pattern literally and is required for patterns coming from outside. - In multi-file search, the output prefix changes with the file count; in a script, `-h` or `-H` is written explicitly. - `-q` writes nothing and stops at the first match; it is the correct form of an existence check in a script. - The filter that shrinks the data is placed at the start of the pipeline. #### Next Step Filtering selects lines but does not change them. Reformatting the timestamp in the log, simplifying field delimiters, or masking a field requires transforming the line itself. The next lesson takes up the stream editor: substitution, address and range selection, rewriting with backreferences, and the mismatch between implementations in the in-place editing option. ### Transforming with the Stream Editor Source: https://academia.sh/en/courses/shell-programming/transforming-with-the-stream-editor The substitution command and its flags, rewriting with backreferences, address and range selection, and the mismatch between implementations in in-place editing. Filtering selects lines but does not change them. Taking only the hour from the log's timestamp, reducing paths with a numeric identifier to a single pattern, or turning the line into a delimited record requires transforming the line itself. A **stream editor** is a filter that reads its input line by line, applies a sequence of commands to each line, and writes the result. Its difference from an interactive editor is that the decisions are made in advance: no file is opened, nothing is displayed, the stream only flows through. #### Substitution The most frequently used command is `s`: `s/pattern/replacement/flags`. ```sh head -2 access.log | sed 's/HTTP\/1.1/HTTP/' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP" 200 2048 ``` Because the pattern contains a slash, it had to be escaped. The delimiter is free; the first character after the `s` command is taken as the delimiter: ```sh head -1 access.log | sed 's|HTTP/1.1|HTTP|' ``` Choosing `|` or `#` in patterns that contain a path removes the backslash pileup. The default behavior is to change **only the first match on each line**. Flags change this: ```sh echo 'a-b-c-d' | sed 's/-/+/' echo 'a-b-c-d' | sed 's/-/+/g' echo 'a-b-c-d' | sed 's/-/+/3' ``` ``` a+b-c-d a+b+c+d a-b-c+d ``` The `g` flag changes every match, a number flag changes only the `N`th match. The `I` flag removes case sensitivity. POSIX defined this feature in its latest edition in the lowercase form; the uppercase spelling used here is an implementation extension. It is present in common implementations. #### Reusing the Matched Text Inside the replacement, `&` stands for the entire match: ```sh head -1 access.log | sed -E 's/ [0-9]{3} [0-9]+$/[&]/' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1"[ 200 5120] ``` Groups are recalled with `\1`, `\2` … This is the way to rewrite a line into a desired format: ```sh sed -E 's/^([0-9.]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) [^"]*" ([0-9]{3}) ([0-9]+)$/\1|\3|\4|\5|\6/' access.log | head -3 ``` ``` 10.0.0.12|GET|/index.html|200|5120 10.0.0.31|GET|/static/style.css|200|2048 10.0.0.12|GET|/product/12|200|8410 ``` A space-separated log line containing quotes turned into a five-field record separated by pipes. The second group (the timestamp) was captured but not used; capturing it is required, because the pattern has to cover the entire line. #### Choosing the Syntax `sed` uses basic regular expression syntax by default. The `-E` option switches to the extended syntax. In the basic syntax, support for some operators depends on the implementation: ```sh echo 'aaa' | sed 's/a\+/X/' echo 'aaa' | sed -E 's/a+/X/' ``` ``` X X ``` In the GNU version both commands work; the `\+` operator is recognized as an extension in the basic syntax. In the BSD version (macOS) `\+` is not recognized: the first command reads the pattern as "an `a` and a plus sign," finds no match and writes the line `aaa` unchanged. Conclusion: **if `+`, `?`, `|`, and `{n,m}` are going to be used, write `-E`.** The `-E` option is present in both the BSD and GNU versions; the `-r` option seen in older GNU scripts is GNU-specific. #### Addresses An **address** written before a command determines which lines the command applies to. | Address | Meaning | |---|---| | `5` | the fifth line | | `$` | the last line | | `/pattern/` | lines matching the pattern | | `2,4` | from the second line to the fourth | | `/a/,/b/` | from the line matching `a` to the line matching `b` | | `address!` | lines **outside** the address | The `d` command deletes the line. In a file containing the lines `one`, `two`, `three`, `four`: ```sh printf 'one\ntwo\nthree\nfour\n' > words.txt sed '2,3d' words.txt ``` ``` one four ``` ```sh sed '$d' words.txt ``` ``` one two three ``` ```sh sed '/tw/d' words.txt ``` ``` one three four ``` ```sh sed '/tw/!d' words.txt ``` ``` two ``` The `!` mark inverts the selection; the last call, by saying "delete the ones that do not match," left only the one that did. The `p` command writes the line. Because the stream editor already writes every line by default, `p` alone produces a repeat; the `-n` option turns off the default writing and `p` becomes the selector: ```sh sed -n '2,3p' words.txt ``` ``` two three ``` This pair — `-n` and `p` — is filtering written with the stream editor. For selecting lines alone, the filter tool is more suitable; when transformation and selection are both needed, the stream editor does both in a single pass. #### Other Commands `y/abc/xyz/` transliterates characters; it uses literal, character-for-character mapping, not a pattern: ```sh echo 'abc' | sed 'y/abc/xyz/' ``` ``` xyz ``` The `q` command ends the stream once its address is reached: `sed '100q'` writes the first hundred lines and stops, never reading the rest of the file. Multiple commands are chained with `-e` or with semicolons: ```sh head -1 access.log | sed -e 's/HTTP\/1.1//' -e 's/ */ /g' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html " 200 5120 ``` Commands are applied to each line **in sequence**; the second command works on the first one's output. This means one substitution can feed another, and the result changes if the order changes. #### In-Place Editing: Not Portable The `-i` option changes the file in place. How this option takes its argument is incompatible between the two major implementations, and this is one of the most common portability bugs in shell scripts. In the GNU version the suffix is written **attached to the option** and is optional; if no suffix is given, no backup is made: ```sh sed -i 's/one/ONE/' words.txt cat words.txt ``` ``` ONE two three four ``` In the BSD version (macOS), `-i` **mandatorily** wants a separate backup-suffix argument. The same command fails there with `sed: -I or -i may not be used with stdin`: the option took the following `s/one/ONE/` as the backup suffix, `words.txt` became the script, and no file operand was left, so `sed` tried to read standard input, which `-i` does not allow. The correct BSD spelling gives the empty suffix explicitly: `sed -i '' 's/one/ONE/' words.txt`. In the GNU version that spelling does not work: if a separate empty argument is given, that argument is taken as the script and the string after it is taken for a file name: ```sh sed -i '' 's/one/ONE/' words.txt ``` ``` sed: can't read s/one/ONE/: No such file or directory ``` Neither spelling works on both versions. Not using `-i` in scripts is the right decision whenever portability is needed. In its place, the pattern built in the Redirection lesson is written: ```sh sed 's/one/ONE/' words.txt > words.new && mv words.new words.txt ``` This pattern carries three advantages: it works the same way everywhere, the original file is not corrupted if `sed` fails, and `mv` is atomic on the same file system — a process reading the file sees either the old or the new content, never a half-written one. The `-i` option itself does the same thing internally: it writes to a temporary file and swaps it in. The only thing it offers is brevity of notation, and its cost is portability. #### Applying to the Script Part of the paths in the log contain a numeric identifier: when `/product/12` and `/product/45` are counted separately, both stay near the bottom of the list. Reducing the identifiers to a single symbol makes visible which **endpoint** is busy: ```sh log_file=access.log cut -d' ' -f7 "$log_file" | sed -E 's#/[0-9]+#/:id#g' | sort | uniq -c | sort -rn ``` ``` 6 /product/:id 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /panel 2 /secret 1 /old-page 1 /login ``` `/product/:id` rose to the top of the list with six requests; counted separately, it sat in fourth and fifth place with three requests each. Normalization changes what the count measures — this is not a formatting fix but an analysis decision, and it should be stated in the report output. The second application pulls the hour field out of the timestamp: ```sh log_file=access.log sed -E 's/^.*:([0-9]{2}):[0-9]{2}:[0-9]{2} .*$/\1/' "$log_file" | sort | uniq -c ``` ``` 30 09 ``` Because the sample log falls entirely within a single hour, the output is a single line; in a real log spread across days, this produces a column giving the load distribution. The `^.*:` part of the pattern is greedy and advances up to the **last** colon; the following groups therefore match the minute and second. Greediness working here is not a coincidence, but the pattern's correctness depends on it — patterns that rely on greediness are fragile, and a field-based solution is safer. #### Summary - Substitution has the form `s/pattern/replacement/flag`; the delimiter is free, and the default behavior is one match per line. - `&` stands for the entire match, `\1`, `\2` for groups; this is the tool for rewriting a line. - In the basic syntax, support for operators such as `\+` depends on the implementation; `-E` is present in both common versions and is preferred. - An address determines a command's scope; `!` inverts the selection, `-n` with `p` gives selective writing. - The `-i` option's argument handling is incompatible between the BSD and GNU versions; the portable form is to write to a temporary file and swap it in. #### Next Step The stream editor sees a line as text; it can only imitate fields with a pattern. In column-shaped data such as a log, this leads to fragile patterns. The next lesson takes up field-based processing: the tool that automatically splits a line into fields, can compute over fields, and can keep an accumulation per key — so the counts written with associative arrays in the previous topic collapse into a single line. ### Field-Based Processing Source: https://academia.sh/en/courses/shell-programming/field-based-processing The record and field model, the pattern–action structure, built-in variables, accumulation with associative arrays, and producing a report in a single pass. The stream editor sees a line as text; it can only imitate fields with a pattern. Taking the ninth field from the log required writing a pattern that described the entire line, and that pattern breaks the moment the format changes even slightly. There is a separate class of tool for column-shaped data. The **field-based processing tool** splits input into records, records into fields, and runs a program made of pattern–action pairs. This lesson builds that model and collapses the counts written with associative arrays in the previous topic into a single line. #### The Record and Field Model Input is split, by default, into lines — **records**. Each record is separated into **fields** by runs of whitespace. `$1` gives the first field, `$0` the entire record, `$NF` the last field. ```sh head -2 access.log | awk '{ print NR, NF, $1, $9, $NF }' ``` ``` 1 10 10.0.0.12 200 5120 2 10 10.0.0.31 200 2048 ``` Built-in variables complete the model's surroundings: | Variable | Content | |---|---| | `NR` | number of records read (accumulates across files) | | `FNR` | record's position within that file | | `NF` | number of fields in that record | | `FS` | field separator (default: runs of whitespace) | | `OFS` | output field separator (default: a single space) | | `FILENAME` | name of the file being read | That the default separator is "runs of whitespace" matters: consecutive spaces count as a single separator, and leading whitespace on a line is ignored. This is the point where it departs from the `cut` command — `cut` counts every separator individually and produces empty fields in aligned text. #### Pattern and Action A program is made of `pattern { action }` pairs. If the record fits the pattern, the action runs. ```sh awk '$9 >= 500 { print $1, $7, $9 }' access.log ``` ``` 192.168.1.5 /api/data 500 192.168.1.5 /api/data 500 10.0.2.7 /api/data 500 ``` Both parts are optional, and a default fills in whichever is missing: - If no pattern is written, the action applies to **every record**. - If no action is written, the default action is `{ print }`. ```sh awk '$9 == 404' access.log | grep -c '' ``` ``` 4 ``` A pattern can also be a regular expression. `/pattern/` tests the whole record, `$9 ~ /pattern/` tests a specific field: ```sh awk '$9 ~ /^[45]/ { n++ } END { print n }' access.log ``` The entire program is written **inside single quotes**. The `$` sign in the program is a field reference; if double quotes are used, the shell mistakes it for variable expansion and the program breaks. This is the place where the rule from the Variables and Quoting Rules lesson is most often needed. #### `BEGIN` and `END` The `BEGIN` block runs before the first record is read, the `END` block after the last. They are used to write a header, set up configuration, and report accumulations. ```sh awk ' BEGIN { print "address bytes" } { t += $10 } END { printf "total: %d bytes, %d requests, average %.1f\n", t, NR, t / NR } ' access.log ``` ``` address bytes total: 117780 bytes, 30 requests, average 3926.0 ``` Variables are used without being declared; on first use they hold zero in numeric context, the empty string in string context. This is why the line `t += $10` needed no separate initialization. `printf` formatting uses the same specifiers as the shell's `printf` command, and it can print a decimal number — the integer restriction of shell arithmetic does not apply here. #### Associative Arrays Arrays are only associative and need no declaration. A per-key count is the one-line equivalent of the shell loop from the previous topic: ```sh awk '{ count[$7]++ } END { for (y in count) printf "%3d %s\n", count[y], y }' access.log \ | sort -rn | head -5 ``` ``` 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /product/45 ``` A per-key sum follows the same pattern: ```sh awk '{ bytes[$7] += $10 } END { for (y in bytes) printf "%8d %s\n", bytes[y], y }' access.log \ | sort -rn | head -5 ``` ``` 36120 /panel 25230 /product/12 21990 /product/45 20480 /index.html 7236 /api/data ``` The order produced by a `for (key in array)` loop is undefined; sorted output needs an external `sort`. This is a direct consequence of associative arrays being implemented with a hash table, defined in the Data Structures course. #### Changing the Field Separator The `-F` option sets the field separator. The value can be a single character or a regular expression. ```sh printf 'a:b:c\nd:e:f\n' | awk -F: '{ print $2 }' ``` ``` b e ``` The output separator is a separate variable: ```sh head -2 access.log | awk 'BEGIN { OFS="|" } { print $1, $7, $9 }' ``` ``` 10.0.0.12|/index.html|200 10.0.0.31|/static/style.css|200 ``` In the `print` command, fields are separated by **commas**; the comma is what `OFS` replaces. If a space is written instead of a comma, the fields are concatenated with no separator at all — this is string-concatenation syntax, and it is a silent source of bugs. Assigning to a field rebuilds the record: ```sh head -1 access.log | awk '{ $1="MASKED"; print }' | cut -c1-46 ``` ``` MASKED - - [07/Feb/2024:09:12:44 +0000] "GET / ``` When the record is rebuilt, the fields are joined with `OFS`; the multiple spaces in the original line are not preserved. This is the expected behavior in field masking and redaction operations. #### Numeric and String Context A field can be interpreted both as a number and as a string; which interpretation applies depends on the operator and on where the operands came from. A field coming from input that looks like a number is compared numerically; but when compared against a string constant, it can take on string context. ```sh awk 'BEGIN { x="10"; y="9"; print (x > y ) ? "string: 10>9" : "string: 10<9" }' awk 'BEGIN { x="10"; y="9"; print (x+0 > y+0) ? "number: 10>9" : "number: 10<9" }' ``` ``` string: 10<9 number: 10>9 ``` The way to remove the ambiguity is to force the context explicitly: `+0` for a numeric comparison, concatenation with `""` for a string comparison. In programs working with externally sourced data, this habit prevents results that are hard to explain. #### Passing a Shell Variable Shell variables are not embedded in the program text; they are passed with the `-v` option. ```sh threshold=400 awk -v threshold="$threshold" '$9 >= threshold { n++ } END { print n " requests " threshold " and above" }' access.log ``` ``` 9 requests 400 and above ``` Embedding the value in the program text — by closing and reopening quotes — is wrong for two reasons: quote and backslash characters in the value break the program, and a value coming from outside becomes program code and can run unwanted commands. `-v` preserves this boundary. #### Catching Broken Records Field count is the cheapest criterion for format validation: ```sh { cat access.log echo 'broken line' echo '10.0.0.1 - - [x] "GET /a HTTP/1.1" 200 5' } > all.log awk 'NF != 10 { print FILENAME ":" FNR ": field count " NF }' all.log ``` ``` all.log:31: field count 2 all.log:32: field count 9 ``` The Regular Expressions lesson did the same job with a pattern describing the entire line. A field-count check performs a narrower validation but is much shorter to write and is resilient to format changes. The two can be used together: field count for fast elimination, the pattern for detailed validation. #### A Multi-Pattern Program A program can hold more than one pattern–action pair; every record is tested against all of them in sequence. ```sh awk ' $9 ~ /^2/ { success++ } $9 ~ /^[45]/ { error++; error_path[$7]++ } END { printf "success=%d error=%d\n", success, error for (y in error_path) printf " %-14s %d\n", y, error_path[y] } ' access.log | LC_ALL=C sort ``` ``` /api/data 3 /missing.html 4 /secret 2 success=18 error=9 ``` `sort` ordering depends on the locale; `LC_ALL=C` pins the output to byte order, and the lines with leading spaces come before the `success` line. If a record fits more than one pattern, every corresponding action runs; the "first matching branch" rule from a `case` structure does not apply here. #### Applying to the Script Until now, `report.sh` reread the log from scratch for every count: the line count, the byte total, and six separate passes for four status classes. All of it collapses into one pass: ```sh log_file=access.log awk -v count="$count" ' { requests++; bytes += $10; path[$7]++; class[substr($9, 1, 1)]++ } END { printf "requests: %d\n", requests printf "bytes : %d\n", bytes printf "average : %.1f bytes\n", (requests ? bytes / requests : 0) print "" print "status code classes:" for (s = 2; s <= 5; s++) printf " %sxx %4d %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0) } ' "$log_file" ``` ``` requests: 30 bytes : 117780 average : 3926.0 bytes status code classes: 2xx 18 %60.0 3xx 3 %10.0 4xx 6 %20.0 5xx 3 %10.0 ``` There are three gains. The log is read once. Percentages are decimal and need no scaling trick. The divide-by-zero guard is written the same way as the shell's ternary operator. There is also a loss: part of the report logic is no longer in the shell but in an embedded program. The script's readability depends on this program staying short and single-purpose. Two-hundred-line embedded programs are a sign the shell script's boundary has been crossed. #### Summary - Input is split into records, records into fields; `$0` is the entire record, `NF` the field count, `NR` the record position. The default separator is runs of whitespace. - A program is made of `pattern { action }` pairs; with no pattern, every record is matched, with no action, `print` applies, and a record is tested against every pair. - `BEGIN` and `END` blocks run outside the record stream; variables need no declaration and start at zero. - A per-key count and sum with associative arrays are written in a single line; traversal order is undefined. - Shell values are not embedded in the program text, they are passed with `-v`. - Numeric and string context depends on the operands; it is forced explicitly with `+0` and `""`. #### Next Step The counts have been produced, but sorting them has always been left to an outside command. The next lesson takes up that command and two tools alongside it: sorting by key, deduplicating adjacent repeats, and comparing two sorted streams with set operations — and also why sorting depends on the locale, and why that is a source of bugs in scripts. ### Sorting, Deduplication, and Joining Source: https://academia.sh/en/courses/shell-programming/sorting-deduplication-and-joining Sorting by key, the effect of the locale on sort order, deduplication that requires adjacency, and set operations over sorted streams. In previous lessons, every count ended with `sort`, but the command itself was never explained. Sorting is the second most frequently called tool in shell text processing, and it deserves a lesson of its own for two reasons: its options change the result entirely, and the three tools alongside it — deduplication, comparison, joining — depend on their input being sorted. The Algorithms course defined sorting's cost and the stability criterion. This lesson ties those concepts to a command's options. #### Lexicographic Order and Numeric Order The default comparison is **lexicographic order**: values are compared character by character. ```sh printf '10\n9\n100\n2\n' | sort ``` ``` 10 100 2 9 ``` ```sh printf '10\n9\n100\n2\n' | sort -n ``` ``` 2 9 10 100 ``` In lexicographic order, `10` comes before `9` because its first character, `1`, is smaller than `9`. Wherever numbers are sorted, `-n` is required; forgetting it is the most common silent bug in reports. `-r` reverses the order. `-n -r` together is the standard spelling for sorting from largest to smallest. #### Key Definition The `-k` option states which field the comparison is done over. Its form is `-k start,end`, and **writing the end is mandatory**: `-k2` means "from the second field to the end of the line," `-k2,2` selects only the second field. ```sh awk '{ print $7, $10 }' access.log | sort -k2,2n | head -3 ``` ``` /login 0 /old-page 0 /static/style.css 0 ``` Numeric mode can also be attached to the key: the form `-k2,2n` applies only to that key. When more than one `-k` is given, the keys apply in sequence, moving to the next key on a tie. `-t` sets the delimiter. The default delimiter differs from the field-based tool's: the point where a run of whitespace **transitions into non-whitespace** counts as the separator. ```sh printf 'c:3\na:1\nb:2\n' | sort -t: -k2,2n ``` ``` a:1 b:2 c:3 ``` #### Stability Sorting is **not stable** by default: the relative order of lines with equal keys is not preserved. On a tie, the entire line is compared as a last resort. ```sh printf 'b 1\na 1\nc 0\n' | sort -k2,2n ``` ``` c 0 a 1 b 1 ``` ```sh printf 'b 1\na 1\nc 0\n' | sort -s -k2,2n ``` ``` c 0 b 1 a 1 ``` The `-s` option turns off the last-resort comparison and preserves the input order. In multi-key sorts — first by city, then stably by date — the stability requirement shown in the Algorithms course is met with this option. #### Locale The comparison order depends on the runtime environment's locale. ```sh printf 'a\nB\nb\nA\n' | LC_ALL=C sort ``` ``` A B a b ``` ```sh printf 'a\nB\nb\nA\n' | LC_ALL=en_US.UTF-8 sort ``` ``` a A b B ``` The first order sorts by byte value, the second on a letter basis. Both are correct; the problem is that which one applies depends on the environment the script runs in. This is the primary source of unreproducible results in scripts. The same script gives differently sorted output on two machines; a check that compares one output against another fails despite there being no real difference. Rule: **if a script does sorting, the locale is fixed explicitly.** ```sh LC_ALL=C sort access.log | head -3 ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1" 200 5120 10.0.0.12 - - [07/Feb/2024:09:13:02 +0000] "GET /product/12 HTTP/1.1" 200 8410 10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300 ``` `LC_ALL=C` selects byte order; it is fast, predictable, and identical across machines. If a letter-based order is wanted for a human reader, the locale must still be written explicitly — it must not be left ambiguous. Because operations such as `sort -u` and `uniq` also take their definition of "equality" from the locale, they can produce a different number of lines under different settings. Fixing the locale is therefore not just a matter of order. #### Deduplication `uniq` reduces **adjacent** repeated lines to one. It does not behave as expected on unsorted input: ```sh printf 'a\nb\na\n' | uniq -c printf 'a\nb\na\n' | sort | uniq -c ``` ``` 1 a 1 b 1 a 2 a 1 b ``` Nothing was merged in the first call, because the repeated `a` lines were not adjacent. This is the justification for the `sort | uniq -c` pattern used since the course's first topic. The options answer different questions: | Option | Output | |---|---| | `-c` | repeat count in front of each line | | `-d` | only lines that occur more than once | | `-u` | only lines that occur exactly once | | `-i` | case-insensitive comparison | ```sh printf 'a\nb\nb\nc\n' | uniq -u ``` ``` a c ``` `sort -u` and `sort | uniq` give the same result, but `sort -u` is cheaper: no separate process is created. If `uniq -c` is needed, two steps are required, because `sort -u` does not count. #### Set Operations `comm` compares **two sorted files** and produces three columns: lines only in the first, only in the second, in both. The `-1`, `-2`, `-3` options suppress the corresponding column. The requested paths in the log's first and last fifteen lines are compared below. Both files are prepared deduplicated and sorted with the same locale: ```sh export LC_ALL=C head -15 access.log | cut -d' ' -f7 | sort -u > yesterday.txt tail -15 access.log | cut -d' ' -f7 | sort -u > today.txt ``` ```sh comm -23 yesterday.txt today.txt # only in the first ``` ``` /login ``` ```sh comm -13 yesterday.txt today.txt # only in the second ``` ``` /old-page ``` ```sh comm -12 yesterday.txt today.txt # common to both ``` ``` /api/data /index.html /missing.html /panel /product/12 /product/45 /secret /static/style.css ``` The three calls give, in order, the set difference, the reverse difference, and the intersection. Both files must be sorted, and sorted **with the same locale**; otherwise `comm` warns, or silently produces a wrong result. This tool is the direct way to compare two logs: which paths were requested only in this period, which clients disappeared, which error codes newly appeared. `join` merges two sorted files over a common **key field**; it is the text-based counterpart of the join operation in relational databases. The per-path request count and total bytes are produced in two separate files and sorted by key: ```sh awk '{ count[$7]++ } END { for (y in count) print y, count[y] }' access.log | sort -k1,1 > count.txt awk '{ bytes[$7] += $10 } END { for (y in bytes) print y, bytes[y] }' access.log | sort -k1,1 > size.txt join count.txt size.txt | head -5 ``` ``` /api/data 6 7236 /index.html 4 20480 /login 1 0 /missing.html 4 2048 /old-page 1 0 ``` By default, the first field of both files is taken as the key; the `-1` and `-2` options change the field number, `-t` sets the delimiter. The inputs must be sorted **by the key field** — not by the entire line. This distinction is the source of the most common mistake in using `join`. #### Deterministic Sorting In count output, the order of lines with an equal count depends on the last-resort comparison and the locale. This can prevent the report from coming out the same on every run: ```sh cut -d' ' -f7 access.log | sort | uniq -c | sort -rn | head -5 ``` ``` 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /product/45 ``` When a secondary key is given explicitly, the result becomes deterministic: descending by count first, ascending by name on a tie. ```sh cut -d' ' -f7 access.log \ | LC_ALL=C sort | uniq -c | LC_ALL=C sort -k1,1nr -k2,2 | head -5 ``` ``` 6 /api/data 4 /index.html 4 /missing.html 3 /panel 3 /product/12 ``` Lines with an equal count are now in alphabetical order. This spelling should be used in any script whose output will be tested or compared between two runs. #### Applying to the Script `report.sh` receives two changes. First, the locale of every sort is fixed and a secondary key is given: ```sh export LC_ALL=C top_n() { # top_n FIELD FILE COUNT local field="$1" file="$2" count="$3" cut -d' ' -f"$field" "$file" | sort | uniq -c | sort -k1,1nr -k2,2 | head -"$count" } ``` `export LC_ALL=C` is placed at the top of the script, so every tool it calls uses the same order. This makes the script's output independent of the environment it runs in. Second, an option is added that compares two logs: ```sh compare() { # compare OLD NEW local old="$1" new="$2" cut -d' ' -f7 "$old" | sort -u > "$tmp_dir/old" cut -d' ' -f7 "$new" | sort -u > "$tmp_dir/new" section "Paths seen only in the new log" comm -13 "$tmp_dir/old" "$tmp_dir/new" } ``` This function writes to temporary files and does not clean them up. This gap will be closed with `mktemp` and traps in the Robust Script Writing topic. #### Summary - The default comparison is lexicographic order; `-n` is mandatory on numeric data. - `-k start,end` bounds the key; if the end is not written, the key extends to the end of the line. - Sorting is not stable by default; `-s` turns off the last-resort comparison. - The comparison order depends on the locale; in scripts it must be fixed with `LC_ALL=C`. - `uniq` merges only adjacent repeats; this is why the `sort | uniq -c` pattern is inseparable. - `comm` and `join` require sorted input; the first performs set operations, the second joins over a key. #### Next Step Every tool in this topic worked at the line level. Cutting at the column level, pasting two files side by side, and splitting a large log into pieces requires a separate family of tools. The next lesson takes up that family and closes out the text processing topic with a selection criterion that decides which job belongs to which tool. ### Cutting, Pasting, and Splitting Source: https://academia.sh/en/courses/shell-programming/cutting-pasting-and-splitting Column cutting and its limits, pasting files side by side, character-level transformation, splitting a file into pieces, and the tool-selection criterion. The tools of the previous lessons worked at the line level: they selected lines, transformed lines, sorted lines. Column-level cutting, bringing two files side by side, character-level mapping, and splitting a large file into pieces requires a separate family of tools. This lesson introduces that family and closes out the text processing topic with a criterion that decides which job belongs to which tool. #### Column Cutting `cut` has been used in this course since the first lesson. It has three modes: byte mode (`-b`), character mode (`-c`), and field mode (`-d` and `-f`). ```sh head -2 access.log | cut -d' ' -f1,7,9 ``` ``` 10.0.0.12 /index.html 200 10.0.0.31 /static/style.css 200 ``` ```sh head -2 access.log | cut -c1-11 ``` ``` 10.0.0.12 - 10.0.0.31 - ``` Character mode is used on fixed-width output — aligned tables, fixed-field records. It is useless on data whose field count varies. `cut` is fast and short to write, but it has three firm limits. **The delimiter is a single character.** ```sh printf 'a::b::c\n' | cut -d'::' -f2 ``` ``` cut: the delimiter must be a single character Try 'cut --help' for more information. ``` The field-based tool does the same job with a regular-expression delimiter: ```sh printf 'a::b::c\n' | awk -F'::' '{ print $2 }' ``` ``` b ``` **Consecutive delimiters are not merged.** Every delimiter starts a new field: ```sh printf 'a b c\n' | cut -d' ' -f2 printf 'a b c\n' | awk '{ print $2 }' ``` ``` b ``` `cut` found the second field empty, because there is an empty field between the two spaces. The field-based tool's default delimiter is a run of whitespace, so it returned `b`. In aligned or hand-written text, this difference is decisive. **Field order cannot be changed.** Whatever order the requested field numbers are written in, the output follows the order in the file: ```sh head -1 access.log | cut -d' ' -f9,1 ``` ``` 10.0.0.12 200 ``` Reordering requires the field-based tool: ```sh head -1 access.log | awk '{ print $9, $1 }' ``` ``` 200 10.0.0.12 ``` These three limits leave `cut` a clear domain: **data separated by a single character, with no consecutive delimiters, whose field order will not change.** The log file fit this definition, which is why it was used in the course's earliest topics. #### Pasting Side by Side `paste` merges the corresponding lines of files. It is the reverse of the `cut` operation. ```sh cut -d' ' -f7 access.log | head -3 > path.txt cut -d' ' -f9 access.log | head -3 > code.txt paste path.txt code.txt ``` ``` /index.html 200 /static/style.css 200 /product/12 200 ``` The default separator is a tab; `-d` changes it: ```sh paste -d: path.txt code.txt ``` ``` /index.html:200 /static/style.css:200 /product/12:200 ``` The `-s` option merges each file's lines into a single line; it is the short way to turn a list into comma-separated text: ```sh paste -sd, code.txt ``` ``` 200,200,200 ``` `paste` assumes lines match up **positionally**. If a line is missing in one file, the whole alignment shifts and the result is silently wrong. If matching by key is needed, `join` from the previous lesson is used; `paste` is suited only to data whose order can be trusted. #### Character-Level Transformation `tr` maps input characters one by one. It recognizes no pattern, no field, no line — it works only on character sets, and it only reads from standard input. ```sh printf 'hello\n' | tr 'a-z' 'A-Z' ``` ``` HELLO ``` Three options are the most frequently used: ```sh printf 'aaa bbb\n' | tr -s ' ' # collapse consecutive repeats to one printf 'a1b2c3\n' | tr -d '0-9' # delete characters in the set printf 'a1b2c3\n' | tr -cd '0-9' # delete characters outside the set ``` ``` aaa bbb abc 123 ``` Collapsing consecutive spaces to one with `tr -s ' '` is a common preparation step for making a file usable with `cut`. Its most frequent use is turning a delimiter into a line break to spill every field onto its own line: ```sh tr ' ' '\n' < access.log | grep -c '^"GET$' ``` ``` 27 ``` `tr` is far cheaper than pattern-based tools because it parses nothing. If a character mapping is enough, there is no need to call the stream editor. #### Splitting a File into Pieces `split` divides a large file into pieces of fixed size or a fixed number of lines. ```sh mkdir -p parts split -l 10 access.log parts/log- ls parts ``` ``` log-aa log-ab log-ac ``` ```sh wc -l parts/* ``` ``` 10 parts/log-aa 10 parts/log-ab 10 parts/log-ac 30 total ``` `-l N` splits by line count, `-b N` by byte count. The suffix length is set with `-a`; the default of two letters is enough for up to 676 pieces. There are two uses. First, processing piece by piece a file that does not fit within a memory limit. Second, distributing the pieces to separate processes for parallel processing — if the pieces can be processed independently, the total time is divided by the piece count. It should be kept in mind that splitting by byte boundary can cut a line in half; for line-based data, `-l` is used instead of `-b`. #### Choosing a Tool This topic's six lessons built a family of tools looking at the same data from different angles. The criterion for choosing among them is the shape of the question: | Question | Tool | |---|---| | Which lines fit a pattern | filter (`grep`) | | Change the text inside a line | stream editor (`sed`) | | Compute over fields, accumulate per key | field-based tool (`awk`) | | Take a column from fixed-delimiter data | `cut` | | Merge columns | `paste`, `join` | | Map, delete, compress characters | `tr` | | Sort, deduplicate, do set operations | `sort`, `uniq`, `comm` | Two general rules stand above this table. **Choose the narrowest tool.** The field-based tool can do nearly every job in the table; writing everything with it is still wrong. A narrow tool is shorter to write, runs faster, and tells the reader its intent. The form `awk '/404/ { print }'` does the same job as `grep 404`, but it makes the reader wonder "is there a computation here." **If a problem takes more than three tools to solve, switch tools.** Five-stage pipelines can be written, and were written in this course; but if every line in a script looks like that, the job should be considered to have crossed the shell's boundary. The shell is unrivaled at connecting processes; it is not designed for complex data structures and multi-step business logic. #### Applying to the Script At the end of the text processing topic, `report.sh` produces all three of its sections with their proper tools: ```sh summary() { # summary LOG awk ' { requests++; bytes += $10; class[substr($9, 1, 1)]++ } END { printf "requests: %d\n", requests printf "bytes : %d\n", bytes printf "average : %.1f bytes\n", (requests ? bytes / requests : 0) for (s = 2; s <= 5; s++) printf " %sxx %4d %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0) } ' "$1" } top_paths() { # top_paths LOG COUNT cut -d' ' -f7 "$1" \ | sed -E 's#/[0-9]+#/:id#g' \ | sort | uniq -c | sort -k1,1nr -k2,2 | head -"$2" } error_paths() { # error_paths LOG awk '$9 ~ /^[45]/ { n[$7]++ } END { for (y in n) printf "%3d %s\n", n[y], y }' "$1" \ | sort -k1,1nr -k2,2 } ``` Every function answers a single question, follows a single data path, and its output is deterministic. The script's skeleton, however, is still incomplete: it produces a half report on error, leaves temporary files behind, and expects to be run by hand. #### Summary - `cut` wants a single-character delimiter, does not merge consecutive delimiters, and cannot reorder fields; outside these three limits it is the cheapest column tool. - `paste` brings files together positionally; if key matching is needed, `join` is used. - `tr` works on character sets; it recognizes no pattern and reads only from standard input. - `split` divides a file by line or byte boundary; `-l` is used for line-based data. - The narrowest tool is preferred when choosing; if a problem needs more than three tools, the shell's boundary should be considered crossed. #### Next Step The script produces correct output, but only when everything goes right. When a command fails, the report is left half-finished and the exit code does not report it; an undefined variable silently expands to an empty string. The next topic closes these gaps and starts with strict-mode options: which errors should stop the script, to what extent each option provides that, and the traps the options carry of their own. ## Writing Robust Scripts Strict mode options, catching signals and cleanup, debugging with trace mode, static analysis, and scheduled tasks. ### Error Handling Source: https://academia.sh/en/courses/shell-programming/error-handling Strict mode options, the scope and exceptions of early exit, undefined variable checking, pipeline exit codes, and the error-catching hook. The script produces correct output, but only when everything goes right. When a command fails, the report is left half-finished, and the script's exit code does not report it. An undefined variable silently expands to an empty string. An error in the middle of a pipeline stays invisible. The shell's default behavior is tolerant of failure: even if a command fails, the next one still runs. This is the right choice for interactive use — a mistyped command should not close the session. In a script, it works the other way. **Strict mode** is a set of options that narrows the shell's tolerance. This lesson takes up the options one by one, showing what each catches and — more importantly — what it does not catch. #### Early Exit The `set -e` option terminates the shell after a command whose exit code is nonzero. ```sh #!/usr/bin/env bash # early.sh set -e echo "1: start" false echo "2: THIS LINE IS NOT PRINTED" ``` ``` 1: start ``` ```sh bash early.sh > /dev/null echo $? ``` ``` 1 ``` The script stopped before reaching the second line and carried the failure code outward. The gain here is not just convenience: a script producing half its output and reporting success misleads every system that calls it. #### Early Exit's Exceptions `set -e` does not treat a command's failure as fatal everywhere. The exceptions make sense, but there are many of them, and not knowing them gives a false sense of security. ```sh #!/usr/bin/env bash set -e if false; then echo "a"; fi echo "1: failure in a condition does not stop the script" false || echo "2: failure on the left of || does not stop it" false && echo "not reached" echo "3: failure on the left of && does not stop it either" false | true echo "4: does not stop if the pipeline's last command succeeds" ! false echo "5: a negated command does not stop it" f() { false; echo "f continued"; } if f; then :; fi echo "6: set -e is suspended inside a function called in a condition" ``` ``` 1: failure in a condition does not stop the script 2: failure on the left of || does not stop it 3: failure on the left of && does not stop it either 4: does not stop if the pipeline's last command succeeds 5: a negated command does not stop it f continued 6: set -e is suspended inside a function called in a condition ``` The first five of the six exceptions are by design: the failure of a command used as a condition carries information, it is not an error. The sixth is the most dangerous — early exit is suspended for the **entire body** of a function called within a condition, and real errors inside that body become invisible. For this reason, `set -e` alone is not a guarantee. Critical commands' exit codes must be checked separately, or an error hook must be set up. #### Undefined Variable Checking The `set -u` option treats the expansion of an undefined variable as an error. ```sh #!/usr/bin/env bash # u.sh set -u echo "log: ${LOG}" ``` ``` u.sh: line 4: LOG: unbound variable ``` This option catches the most destructive class of error in shell scripts. The line `rm -rf "$DIR/"` becomes `rm -rf /` when `DIR` is undefined — with the option active, the script never reaches that line at all. The option makes using default-value notation mandatory: ```sh #!/usr/bin/env bash set -u echo "log: ${LOG:-access.log}" echo "first argument: ${1-none}" echo "argument count: $#" ``` ``` log: access.log first argument: none argument count: 0 ``` The `${v-fallback}` and `${v:-fallback}` notations introduced in the Variables and Quoting Rules lesson turn into a requirement here: if an unset positional parameter is going to be read, a default value must be written. This also produces a side effect that documents the script's interface. #### Pipeline Exit Code `set -o pipefail` sets a pipeline's exit code equal to the code of the **rightmost** command that returns a nonzero code. It was introduced in the Pipelines lesson; it is an inseparable part of strict mode. ```sh #!/usr/bin/env bash # pipe.sh set -eo pipefail grep '999' access.log | wc -l echo "THIS LINE IS NOT PRINTED" ``` ``` 0 ``` ```sh bash pipe.sh > /dev/null echo $? ``` ``` 1 ``` Without `pipefail`, this script would print `0` and finish successfully: `wc` ran without a problem, and the pipeline's code was its code. With `pipefail` and `set -e` together, it must be kept in mind that pipelines cut off by `SIGPIPE` will also count as failures. Pipelines in the form `command | head -5` must be suppressed deliberately. #### Narrowing the Field Separator The assignment `IFS=$'\n\t'` removes the space from word splitting: splitting happens only on newline and tab. ```sh IFS=$'\n\t' d="a b:c" printf '<%s>\n' $d ``` ``` ``` Even the unquoted expansion did not split. This is an extra defense against the class of errors caused by file names containing spaces. A warning is needed: this setting is global and affects every piece of code that relies on splitting by space. A loop that wants to split a variable into its words runs for a single round after this setting. While strict mode's other three options are recommended unconditionally, this one has to be decided by looking at what the script does. #### The Arithmetic Trap The Arithmetic and Command Substitution lesson showed that the `(( ))` command's exit code is the **inverse** of the value. Under strict mode this turns into an error that stops the script: ```sh #!/usr/bin/env bash # counter.sh set -e counter=0 (( counter++ )) echo "THIS LINE IS NOT PRINTED (counter=$counter)" ``` ``` ``` ```sh bash counter.sh > /dev/null echo $? ``` ``` 1 ``` The postfix form `counter++` returned the **old** value — that is, zero — and `(( ))` counted this as a failure. The safe form is arithmetic expansion: ```sh #!/usr/bin/env bash set -e counter=0 counter=$(( counter + 1 )) echo "counter=$counter" ``` ``` counter=1 ``` This trap shows why strict mode is applied carefully: the options do not change the script's meaning, but they change which situations count as an "error," and some idioms are incompatible with this new definition. #### Suppressing Expected Failures It is normal for some commands to return a nonzero code; a filter tool's "no match" report is the typical example. Under strict mode, these situations are suppressed explicitly. ```sh #!/usr/bin/env bash set -euo pipefail count=$(grep -c 'NOMATCH' access.log || true) echo "match count: $count" if grep -q 'NOMATCH' access.log; then echo "found"; else echo "not found"; fi ``` ``` match count: 0 not found ``` Two methods were used. `|| true` zeroes the code unconditionally; it suits cases where the number itself is meaningful. Calling within an `if`, on the other hand, already interprets the failure as a condition; this is the correct form for an existence question. `|| true` must be used with restraint. Added to every line, strict mode loses its meaning, and the script reverts to its old form that swallows errors. The rule: **suppression is done knowing why that command's failure is expected, and documented with a comment when necessary.** #### The Error Hook `trap ... ERR` runs a handler for every command that triggers early exit. Reporting where the error occurred makes the silent exits strict mode produces traceable. The `set -E` option makes this hook valid inside functions and subshells too; without it, the hook only runs at the top level. ```sh #!/usr/bin/env bash set -Eeuo pipefail IFS=$'\n\t' handle_error() { local code=$? line=$1 printf 'report.sh: error at line %d, exit code %d\n' "$line" "$code" >&2 exit "$code" } trap 'handle_error "$LINENO"' ERR echo "1: start" grep -q 'NOMATCH' access.log echo "2: THIS LINE IS NOT PRINTED" ``` ``` 1: start report.sh: error at line 13, exit code 1 ``` The first line in the hook's body is written as `local code=$?`; `$?` must be read before any other command runs, or its value changes. `$LINENO` is left inside single quotes so that it expands where the hook is **triggered**, not where it is **set up**. The `trap` command itself and signal handling are the next lesson's subject. #### The Strict Mode Header This lesson's result is a fixed header placed at the top of scripts: ```sh #!/usr/bin/env bash set -Eeuo pipefail ``` Four options close four separate error classes: a failed command, an undefined variable, a code lost in a pipeline, and an error hook lost inside a function. The fifth — narrowing `IFS` — is decided on a per-script basis. The header is not a guarantee, it is a baseline. Keeping `set -e`'s exceptions in mind, critical commands' codes still have to be checked by hand. Strict mode catches what gets overlooked; it does not stand in for what should not be overlooked. #### Summary - `set -e` terminates the shell after a command that returns a nonzero code; it is suspended in condition contexts, on the left of chain operators, and inside the body of functions called in a condition. - `set -u` treats undefined variable expansion as an error and makes default-value notation mandatory. - `set -o pipefail` ties a pipeline's exit code to the code of the rightmost command that exits nonzero. - `(( counter++ ))` reports failure at a value of zero; under strict mode, `counter=$(( counter + 1 ))` is written instead. - Expected failures are suppressed deliberately, with `|| true` or a condition context. - `trap ... ERR`, together with `set -E`, sets up a hook that reports the error's line number. #### Next Step Strict mode stops the script the moment there is an error — but what does it leave behind at the point where it stops? A half-written output file, uncleaned temporary files, a lock another process is waiting on. The next lesson takes up signals and cleanup hooks that run on exit; the safe creation of a temporary file and its deletion on every exit path will also be set up there. ### Traps and Cleanup Source: https://academia.sh/en/courses/shell-programming/traps-and-cleanup The concept of a signal, setting traps, a cleanup hook that runs on exit, the safe creation of temporary files, and a lock that prevents concurrent runs. Strict mode stops the script the moment there is an error. What it leaves behind at the point where it stops is a separate question: a half-written output file, an uncleaned temporary directory, a lock another process is waiting on. A script terminating correctly matters as much as it running correctly. In the How Computers Work course, an interrupt was hardware telling the processor "pause your work." A **signal** is the process-level counterpart of the same idea: an asynchronous notification the kernel, or another process, sends to a running process. This lesson sets up catching signals and the cleanup that runs on exit. #### Signals When a process receives a signal, it applies the default behavior — for most signals, this is termination. Few signals are of interest in scripts: | Signal | Number | Source | |---|---|---| | `SIGHUP` | 1 | the controlling terminal closed | | `SIGINT` | 2 | interrupt from the keyboard | | `SIGTERM` | 15 | a request for clean termination | | `SIGKILL` | 9 | unconditional termination | | `SIGPIPE` | 13 | writing to a pipe whose read end is closed | The Exit Codes lesson showed that a process terminated by a signal returns the code **128 + signal number**; 143 for `SIGTERM`, 130 for `SIGINT`. `SIGKILL` and `SIGSTOP` cannot be caught, ignored, or redefined. This guarantees that the operating system has the last word over a process. Consequently, no cleanup hook runs in a script terminated with `SIGKILL` — it cannot be the **sole** basis for cleanup. #### Setting a Trap The form `trap 'command' signal...` sets the command to run when the signal arrives. The shell also recognizes four **pseudo-signals**: `EXIT`, `DEBUG`, `RETURN`, and `ERR`. `EXIT` is the most important one: it runs no matter how the shell terminates — normal completion, an `exit` call, strict mode's early exit, or after the handler of a caught signal. ```sh #!/usr/bin/env bash set -Eeuo pipefail temp=$(mktemp -d) cleanup() { rm -rf "$temp"; } trap cleanup EXIT echo "data" > "$temp/scratch.txt" echo "temp dir contents: $(ls "$temp")" ``` ``` temp dir contents: scratch.txt ``` When the script finishes, the directory has been deleted. The same holds on the error path: ```sh #!/usr/bin/env bash set -Eeuo pipefail temp=$(mktemp -d) trap 'rm -rf "$temp"' EXIT echo "operation started" false echo "THIS LINE IS NOT PRINTED" ``` ``` operation started ``` Strict mode stopped the script at the `false` line; the `EXIT` trap still ran, and no temporary directory was left behind. A trap is valid from the moment it is set. This is why it is set **as soon as** the resource is created; if an error occurs between the two lines, cleanup does not happen. #### Preserving the Exit Code The `EXIT` trap's body must not change the script's exit code. For the code to be read, the body's **first** command must read `$?`: ```sh #!/usr/bin/env bash # preserved.sh cleanup() { local code=$?; echo "cleanup, preserved code=$code"; } trap cleanup EXIT exit 7 ``` ``` cleanup, preserved code=7 ``` ```sh bash preserved.sh > /dev/null echo $? ``` ``` 7 ``` If the trap's body itself calls `exit`, the script's code becomes that value; if it does not, the original code is preserved. If cleanup failure is not meant to change the script's result, the commands inside the body must have their codes suppressed. #### Re-sending the Signal Catching a signal and doing cleanup erases the information that the process ended **by that signal**: if the script finishes normally, its exit code is 0, and the caller cannot tell it was interrupted. The correct pattern has three steps: clean up, remove the trap, re-send the signal to itself. ```sh #!/usr/bin/env bash # signal.sh cleanup() { echo "cleanup ran (source: $1)"; } trap 'cleanup EXIT' EXIT trap 'cleanup TERM; trap - TERM; kill -TERM $$' TERM echo "ready" kill -TERM $$ ``` In real use, a service manager or the `kill` command sends the signal from outside; here the script sends it to itself so that the experiment can be repeated on its own: ``` ready cleanup ran (source: TERM) cleanup ran (source: EXIT) ``` ```sh bash signal.sh > /dev/null echo $? ``` ``` 143 ``` The code is 143, that is, 128 + 15. A service manager or scheduler managing the process can see this code and tell that the script was **terminated**, not that it failed. The form `trap - SIGNAL` removes the trap and restores the default behavior. If this step is skipped, the `kill` call triggers its own trap again, and an infinite loop results. Notice that the `EXIT` trap also ran: cleanup executed twice. This is why the cleanup function must be written **idempotent** — `rm -rf` and `rmdir` calls do no harm when they try to delete an already-gone path a second time, but a cleanup that increments a counter or sends a message is wrong if it runs twice. #### Creating Temporary Files Safely Producing a temporary file's name by hand — something like `/tmp/report.$$` — is wrong in three ways. **It is predictable.** Process numbers occupy a narrow range and are reused; an attacker can guess the name in advance and plant a symbolic link at that path. When the script opens the file, it writes to wherever the link points. **It contains a race.** The gap between the "does the file exist" check and creating it is open to interference. **It can collide.** Two instances of the same script can receive the same process number at different times. `mktemp` produces the name unpredictably and creates the file **atomically**; if the name collides, it retries. The `-d` option creates a directory and provides a single cleanup point when multiple temporary files are needed. ```sh temp=$(mktemp -d) trap 'rm -rf "$temp"' EXIT ``` `mktemp` opens the files and directories it creates with permissions accessible only to their owner; a file `touch`ed by hand has no such guarantee. The quotes in the line `rm -rf "$temp"` are mandatory. If the variable is left undefined — without `set -u` — the command turns into `rm -rf`; the quotes are a second layer of protection for paths containing spaces. Strict mode's `set -u` option is this line's most important guard. #### Preventing Concurrent Runs A scheduled report script can be restarted before the previous run finishes. If two instances write to the same temporary files, the result is corrupted. The portable way to acquire a lock is creating a directory: `mkdir` is an **atomic** operation that fails if the target already exists. Creating a file does not give the same guarantee, because the "does it exist" check and creation are two separate steps. ```sh #!/usr/bin/env bash # lock.sh set -Eeuo pipefail LOCK="report.lock" if ! mkdir "$LOCK" 2>/dev/null; then echo "another run is in progress, exiting" >&2 exit 75 fi trap 'rmdir "$LOCK"' EXIT echo "lock acquired, working" sleep 1 ``` When a second instance is started while one is running in the background (the `until` loop waits for the first instance to acquire the lock; the `2>&1` on the second call merges the error message into standard output so that it appears in order): ```sh bash lock.sh & until [ -d report.lock ]; do sleep 0.05; done bash lock.sh 2>&1 echo "second run code=$?" wait ``` ``` lock acquired, working another run is in progress, exiting second run code=75 ``` The trap must be set **after** `mkdir` succeeds. Had it been set before, the second instance, unable to acquire the lock, would delete someone else's lock on its way out. The code **75** is a common convention meaning "temporary error, retry later"; a scheduler can distinguish this code from a real error. This pattern's known weakness is that the lock is left behind if the script is terminated with `SIGKILL` or the machine shuts down. In long-lived systems, the process number is written into the lock directory, and when the lock cannot be acquired, whether that process is still alive is tested. #### Applying It to the Script `report.sh` takes on the strict mode header and the cleanup infrastructure: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. set -Eeuo pipefail export LC_ALL=C SCRIPT_NAME="report.sh" temp="" cleanup() { local code=$? [ -n "$temp" ] && rm -rf "$temp" return "$code" } trap cleanup EXIT trap 'cleanup; trap - INT; kill -INT $$' INT trap 'cleanup; trap - TERM; kill -TERM $$' TERM temp=$(mktemp -d) ``` Initializing the `temp` variable with an empty string **before** the trap is set is required under `set -u`: the trap also runs if an error occurs before the `mktemp` call, and at that point the variable would be undefined. The `compare` function from the previous lesson no longer leaves temporary files behind; everything under `$temp` is deleted on exit. #### Summary - A signal is an asynchronous notification sent to a running process; `SIGKILL` and `SIGSTOP` cannot be caught, so cleanup cannot be the sole safeguard. - `trap ... EXIT` runs regardless of how the shell terminates, and is set as soon as the resource is created. - The `EXIT` trap's first command must read `$?`; otherwise the script's exit code is lost. - When a signal is caught, cleanup runs, the trap is removed, and the signal is re-sent to itself; this preserves the exit code as `128 + signal`. - A temporary file is created with `mktemp`: the name is unpredictable, creation is atomic, and permissions are narrow. - Because `mkdir` is atomic, it is used as a portable locking mechanism; the trap is set after the lock is acquired. #### Next Step The script now terminates cleanly and leaves no trace behind. How to find out it is producing the wrong result is a separate question. The next lesson takes up debugging tools: a trace mode that prints every command before it runs, a format variable that makes the trace output readable, a syntax check performed without running the script, and applying the trace to only part of the script. ### Script Debugging Source: https://academia.sh/en/courses/shell-programming/script-debugging Syntax checking, trace mode and prompt formatting, redirecting the trace, and inspecting variables and the call stack. The script terminates cleanly and leaves no trace behind. How to find out it is producing the wrong result is a separate question. Two features make debugging shell scripts hard. First, what actually runs is mostly not the shell itself but the programs it calls; second, expansion rules can make the result different from the text that was written. The second point is decisive: you cannot know what the line `rm "$dir/$name"` deletes without seeing how the shell expanded that line. This lesson's tools make exactly that expanded form visible. #### Checking Without Running `bash -n` parses the script but runs no command. It was introduced in the Script Anatomy lesson; it is the first step for scripts that contain destructive operations. The script under test, `broken.sh`, contains an unclosed condition: ```sh #!/usr/bin/env bash # broken.sh — log summary, contains a syntax error log="$1" if [ -f "$log" ]; then echo "line count: $(wc -l < "$log")" ``` ```sh bash -n broken.sh ``` ``` broken.sh: line 7: syntax error: unexpected end of file ``` The error message (in bash 5.2) gives only the line where the problem was noticed: the parser reached the end of the file and still found an open structure. In bash 5.2 you have to look in the script yourself for where that structure began (bash 5.3 appends something like ``from `if' command on line 5`` to the message, which gives you that line); the last `if` without a matching `fi` is usually the place. `-n` sees only syntax. Nonexistent commands, wrong options, undefined variables, and logic errors pass this check. A script coming out clean with `bash -n` does not mean it will run. #### Trace Mode The `set -x` option writes every command to standard error **after** expansions and **before** execution. It can also be turned on for the whole script with `bash -x script`. The examples run on `extract.sh`, a script that counts the most requested paths in the access log: ```sh # extract.sh — counts the most requested paths log=access.log count=3 top_paths() { cut -d' ' -f7 "$1" | sort | uniq -c | sort -k1,1nr | head -"$2" } top_paths "$log" "$count" ``` The script's own output is discarded to `/dev/null`; only the trace appears on screen. ```sh bash -x extract.sh 2>&1 > /dev/null | head -3 ``` ``` + log=access.log + count=3 + top_paths access.log 3 ``` After these come one trace line for each of the pipeline's five parts; because the parts start at the same time, the order of those lines changes from run to run, so only the first three lines are shown here and in the `PS4` output below. Two pieces of information are visible here. The variables took their expected values. The function was called with the arguments `access.log` and `3`. The pipeline's links are listed separately too; to see a particular link, filter the output: ```sh bash -x extract.sh 2>&1 > /dev/null | grep cut ``` ``` + cut '-d ' -f7 access.log ``` Notice the `'-d '` notation on the `cut` line: the shell shows arguments that require quoting inside quotes in the trace output. This is the most direct evidence for whether an argument containing a space really passed through as a single piece. #### Formatting the Trace Prompt The default `+ ` prefix does not say which line is running in a long script. The `PS4` variable sets this prefix, and it is expanded before every line is printed. ```sh export PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: ' bash -x extract.sh 2>&1 > /dev/null | head -3 ``` ``` + extract.sh:2:main: log=access.log + extract.sh:3:main: count=3 + extract.sh:7:main: top_paths access.log 3 ``` Every line now carries the file name, line number, and function name. The `PS4` assignment is written inside single quotes: the variables must expand not at assignment time, but each time a trace line is written. The function name shows up on the pipeline's links as well: ```sh export PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: ' bash -x extract.sh 2>&1 > /dev/null | grep 'top_paths: cut' ``` ``` + extract.sh:5:top_paths: cut '-d ' -f7 access.log ``` `FUNCNAME[0]` is undefined at the top level; the `:-main` default value makes it work under `set -u` as well. #### Selective Tracing Tracing the whole of a long script leaves the line you are looking for buried inside thousands of lines of output. With the `set -x` / `set +x` pair, only the section of interest is traced. ```sh #!/usr/bin/env bash exec 2>&1 echo "untraced section" set -x code=404 class="${code:0:1}" set +x echo "result: ${class}xx" ``` ``` untraced section + code=404 + class=4 + set +x result: 4xx ``` The closing line falls into the trace output too; the closing itself is a command being executed. #### Separating the Trace Trace output goes to standard error. Since the script's warnings go there too, the two get mixed, and in a scheduled job that logs the script's output, it becomes unreadable. The `BASH_XTRACEFD` variable redirects the trace to a different descriptor: ```sh #!/usr/bin/env bash exec 9> trace.log BASH_XTRACEFD=9 set -x code=404 class="${code:0:1}" set +x echo "report output: ${class}xx" ``` ``` report output: 4xx ``` `trace.log` contents: ``` + code=404 + class=4 + set +x ``` Only the report itself landed on the terminal. The pattern of opening a descriptor with `exec`, introduced in the Redirection lesson, finds its use here: numbers 3 and above are free for this kind of use. #### Verbose Mode `set -v` writes commands **before** expansion, as they appear in the source file. ```sh #!/usr/bin/env bash set -v code=404 echo "class: ${code:0:1}xx" ``` ``` code=404 echo "class: ${code:0:1}xx" class: 4xx ``` `-x` shows the expanded form, `-v` shows the written form. When both are turned on together, an expansion's input and output are seen side by side; this is the most direct diagnostic route when expansion rules are misunderstood. #### The Hook That Runs Before Every Command `trap ... DEBUG` runs before every simple command. The `BASH_COMMAND` variable holds the text of the command about to be executed. ```sh #!/usr/bin/env bash trap 'printf "[%s] %s\n" "$LINENO" "$BASH_COMMAND" >&2' DEBUG code=404 class="${code:0:1}" echo "result: ${class}xx" ``` ``` [3] code=404 [4] class="${code:0:1}" [5] echo "result: ${class}xx" result: 4xx ``` The `DEBUG` hook provides something `set -x` cannot: its body can be **arbitrary code**. Stopping when a specific variable jumps to an unexpected value, or printing a variable's value before every command, is done this way. The cost is cost itself: the hook runs for every command. It is not left in production scripts. #### Inspecting Variables `declare -p` prints a variable's type and content in a reusable form. `echo` is not enough for arrays; `declare -p` shows element boundaries. ```sh codes=(200 404 500) declare -A counts=([404]=4 [500]=3) declare -p codes counts ``` ``` declare -a codes=([0]="200" [1]="404" [2]="500") declare -A counts=([404]="4" [500]="3" ) ``` `-a` shows the indexed array, `-A` the associative one. Only in this output can you see that indices skip in a sparse array, or that an empty element is genuinely empty. #### Reading the Call Stack Bash keeps the call chain in three parallel arrays: `FUNCNAME` holds function names, `BASH_SOURCE` file names, `BASH_LINENO` call lines. Printing these in an error handler shows where the error came from. ```sh # stack.sh where() { local i for (( i = 1; i < ${#FUNCNAME[@]}; i++ )); do printf ' %s() <- %s:%s\n' "${FUNCNAME[i]}" "${BASH_SOURCE[i]}" "${BASH_LINENO[i-1]}" done } inner() { where; } outer() { inner; } outer ``` ``` inner() <- stack.sh:8 outer() <- stack.sh:9 main() <- stack.sh:10 ``` The chain is read from the inside out: the call to `where` came from inside `inner`, the call to `inner` from `outer`, the call to `outer` from the top level. The index shift (`BASH_LINENO[i-1]`) is deliberate; the three arrays hold different facets of the same event with different alignments. When this function is added to the `ERR` hook set up in the previous lesson, the script reports not only the error's line but also its **call path**. #### A Method The order of the tools also gives a method for debugging: 1. Get the syntax past `bash -n`. 2. Add the strict mode header; most errors become visible on their own. 3. Narrow down which section the problem is in — placing a temporary `exit` at the end of a section and bisecting is the fastest way. 4. Trace the narrowed-down section with `set -x` and add location information with `PS4`. 5. If you are looking for a difference between an expansion's input and output, turn it on together with `-v`. 6. Print complex data structures with `declare -p`. This order puts the cheapest tool first. A session that starts with `set -x` can end up spending its time hunting for a syntax error that `bash -n` would have found in a second. #### Summary - `bash -n` checks only syntax; it does not run the script and does not see logic errors. - `set -x` writes every command after expansions; it is direct evidence of whether quoting is working correctly. - `PS4` sets the trace prefix and is re-expanded for every line; it must be assigned inside single quotes. - `BASH_XTRACEFD` sends trace output to a separate descriptor, separating it from the script's own messages. - `set -v` shows the written form, `set -x` the expanded form; used together, an expansion's input and output can be compared. - `declare -p` gives array contents with their boundaries, and `FUNCNAME`/`BASH_SOURCE`/`BASH_LINENO` give the call stack. #### Next Step This lesson's tools find an error after it has occurred. Most errors in shell scripts, though, are determined before they occur: an unquoted expansion, an unchecked exit code, a non-portable option. The next lesson takes up the static analysis tool that catches this class without running the script, and puts `report.sh` through its inspection. ### Static Analysis Source: https://academia.sh/en/courses/shell-programming/static-analysis Finding errors without running the script, common shell error classes, portability checking based on the interpreter declaration, and the correct use of warning suppression. The previous lesson's tools find an error after it has occurred. Most errors in shell scripts, though, are determined before they occur: an unquoted expansion, an unchecked exit code, building a file list from a command's output, using a feature the interpreter declaration does not permit. These errors can be seen by looking at the code itself, without running it. **Static analysis** is the method of parsing source text and searching for known faulty patterns. For the shell, this job is delegated to a tool that parses the script and applies hundreds of pattern rules. The tool's value is that it remembers, all at once, every trap this course has explained one by one. #### An Inspection The script below carries most of the errors covered in previous lessons at once: ```sh #!/bin/sh # flawed.sh log=$1 for f in $(ls logs); do echo "processing: $f" done if [ $log = "access.log" ]; then rows=`wc -l < $log` echo "lines: $rows" fi cd /tmp rm -rf $log_backup ``` The analyzer's long-form output marks the finding beneath the source line and suggests a corrected form. The same findings read more compactly in single-line form: ```sh shellcheck -f gcc flawed.sh ``` ``` flawed.sh:4:10: error: Iterating over ls output is fragile. Use globs. [SC2045] flawed.sh:7:6: note: Double quote to prevent globbing and word splitting. [SC2086] flawed.sh:8:8: note: Use $(...) notation instead of legacy backticks `...`. [SC2006] flawed.sh:8:17: note: Double quote to prevent globbing and word splitting. [SC2086] flawed.sh:11:1: warning: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. [SC2164] flawed.sh:12:8: warning: log_backup is referenced but not assigned. [SC2154] flawed.sh:12:8: note: Double quote to prevent globbing and word splitting. [SC2086] ``` Five separate error classes, in a twelve-line script. The last one's consequence is `rm -rf` running with no argument at all in a script without `set -u`. The default format is more detailed: for findings that carry a registered fix it also writes the corrected line under the line with the heading `Did you mean:`; not every finding gets one. This format suits working with a single finding; for a batch inspection, the single-line format above is appropriate. #### Error Classes Caught Warnings fall into four groups, and each was the subject of a separate lesson in this course. **Expansion and quoting.** Unquoted variable expansion, using `$*` instead of `"$@"`, the wrong form of array expansion. The entire Variables and Quoting Rules lesson belongs to this class. This is the most frequently seen warning. **Exit code.** An unchecked `cd`, unchecked command substitution, assignment combined with `local`, a pipeline code getting lost. The traps from the Error Handling lesson are caught here. **Shell idioms.** Building a file list from a command's output, a missing `-r` in a `read` call, the old backtick notation, the wrong operator inside `[ ]`. Warnings from the Loops and Conditionals lessons. **Undefined name.** Reading a variable that was never assigned, variables that turn into a different name through a typo. This class can go unnoticed even at run time in an interpreted language. Every warning has an identifier (`SC` and four digits). The identifier serves two purposes: reaching the warning's detailed explanation, and — when needed — suppressing that warning in a targeted way. #### Severity Levels Warnings fall into four levels: `error`, `warning`, `info`, `style`. The `-S` option hides everything below a given level: ```sh shellcheck -S warning -f gcc flawed.sh ``` ``` flawed.sh:4:10: error: Iterating over ls output is fragile. Use globs. [SC2045] flawed.sh:11:1: warning: Use 'cd ... || exit' or 'cd ... || return' in case cd fails. [SC2164] flawed.sh:12:8: warning: log_backup is referenced but not assigned. [SC2154] ``` Three findings remain; the quoting and backtick warnings at the `info` and `style` levels were hidden. This is useful when adding analysis to an existing codebase: the `error` level is cleaned up first, then the threshold is lowered in stages. Keeping the threshold permanently high, on the other hand, means giving up most of the tool's value — `info`-level quoting warnings are the ones that most often turn into real errors. #### Portability Checking The analyzer reads the interpreter declaration and applies its rules accordingly. In a script declaring `#!/bin/sh`, bash extensions count as errors: ```sh #!/bin/sh # portable.sh codes=(200 404) if [[ ${#codes[@]} -gt 1 ]]; then echo "multiple codes" fi ``` ```sh shellcheck portable.sh ``` ``` In portable.sh line 3: codes=(200 404) ^-------^ SC3030 (warning): In POSIX sh, arrays are undefined. In portable.sh line 4: if [[ ${#codes[@]} -gt 1 ]]; then ^----------------------^ SC3010 (warning): In POSIX sh, [[ ]] is undefined. For more information: https://www.shellcheck.net/wiki/SC3010 -- In POSIX sh, [[ ]] is undefined. https://www.shellcheck.net/wiki/SC3030 -- In POSIX sh, arrays are undefined. ``` This is the rule from the Script Anatomy lesson enforced by a tool: declaring POSIX while using a bash feature causes the script to break silently on another system. When the same script declares bash, these two warnings are not produced. The limit of this check can be seen from here too: the tool decides based on the **declared** shell. If the declaration is wrong, the check is done wrong. This is why the interpreter declaration must be written to match the features the script actually uses. #### Warning Suppression When a warning needs to be deliberately ignored, a directive is written into the source code: ```sh #!/usr/bin/env bash # shellcheck disable=SC2086 log="a b.log" echo $log ``` The directive applies to the command **after** it; placed at the top of the file, before the first command, it covers the whole file. Suppression is bound by three rules. **An identifier is given.** The `disable` directive can be written without an identifier, but then it turns off every warning and also hides real errors that will occur in the future. **The scope is narrowed.** File-level suppression takes every line in that file out of the check. Writing it directly above the line where the warning arises is preferred. **A justification is written.** Why the suppressed warning is a false alarm is stated with a comment. A suppression with no justification tells the next person reading the code, "I do not know what is going on here." In cases where word splitting is genuinely wanted, there is a better solution: putting the value into an array. The notation `"${array[@]}"` produces no warning and also makes the intent visible in the code. Suppression is reserved for cases where rewriting is not possible. #### Putting the Script Through Inspection `report.sh` has been written throughout this course avoiding the patterns the analyzer would warn about. The small script below follows the same rules: ```sh #!/usr/bin/env bash # clean.sh set -Eeuo pipefail log="${1:-access.log}" [ -r "$log" ] || { echo "cannot read: $log" >&2; exit 3; } count=$(grep -c '' "$log") printf 'lines: %d\n' "$count" ``` ```sh shellcheck clean.sh; echo "code=$?" ``` ``` code=0 ``` The exit code is `0` and there is no output. The tool returns a nonzero code if there is a finding; this makes it possible to turn the check into a validation step: ```sh shellcheck clean.sh && bash -n clean.sh && echo "check passed" ``` ``` check passed ``` The same chain is used for the course's `report.sh` script. Running this command before a submission to version control, or as a continuous integration step, is the cheapest way to keep the quality of shell scripts intact. #### The Limit of Static Analysis The tool sees the code's **form**, not its **meaning**. What it cannot catch: - A wrong field number (writing `$8` instead of `$7`). - A wrong regular expression (the pattern is valid but does not describe the intended set). - Wrong business logic (a percentage divided by the wrong total). - Errors that depend on the runtime environment (a nonexistent command, insufficient permission). These classes require **testing**: running the script with a known input and comparing its output against the expected output. This course's `access.log` file being fixed and reproducible exists precisely to make this kind of testing possible. Static analysis does not substitute for testing; it reduces the number of errors testing has to find. #### Summary - Static analysis parses a script without running it and searches for known faulty patterns. - The main classes caught are quoting, exit codes, shell idioms, and undefined names; all were the subject of separate lessons in this course. - The check is done according to the interpreter declaration; bash extensions produce warnings in a script that declares POSIX. - Suppression is done with an identifier, a narrow scope, and a justification; if word splitting is wanted, the correct solution is using an array. - The tool returns a nonzero code if there is a finding; this turns the check into a validation step. - Static analysis sees form, not meaning; a wrong field number and wrong business logic can only be found through testing. #### Next Step The script is correct, robust, and checked — but it is still run by hand. A report's value lies in its being produced regularly. The final lesson takes up scheduled tasks: reading a cron entry, at which points the scheduler's environment departs from an interactive shell, where output should go, and preventing overlapping runs. ### Scheduled Tasks Source: https://academia.sh/en/courses/shell-programming/scheduled-tasks Reading a cron entry, how the scheduler's environment differs from an interactive shell, managing output, and a wrapper that prevents overlap. The script is correct, robust, and checked — but it is still run by hand. A report's value lies in its being produced regularly: a tool that analyzes the previous day's log every morning and drops the result somewhere is a different thing from the same tool run by hand. This lesson takes up preparing a script for scheduled execution. The subject is not just the scheduling definition; the real issue is that the script will be run by a **non-human caller**, in a **narrow environment**, **without anyone watching**. #### The Cron Entry On Unix systems, scheduled tasks are defined to a scheduler service. The definition table is kept per user, and every line consists of five time fields and a command: ``` minute hour day of month month day of week command ``` | Field | Range | |---|---| | minute | 0–59 | | hour | 0–23 | | day of month | 1–31 | | month | 1–12 | | day of week | 0–7 (0 and 7 are Sunday) | Every field takes one of four forms: a fixed number, `*` (every value), `a-b` (a range), `a,b,c` (a list). Appending `/n` to a range selects every `n`th step. ``` 30 6 * * * every day at 06:30 0 */4 * * * every four hours, on the hour 15 2 * * 1 every Monday at 02:15 0 0 1 * * midnight on the first of every month */10 * * * * every ten minutes ``` Two fields together narrow the constraint, but the **day-of-month and day-of-week** pair is the exception: if both take a value other than `*`, the conditions combine with "or," not "and." The entry `0 0 13 * 5` runs on the 13th of the month **or** on Friday. This is the most commonly misread point in these entries. The definition table is edited with `crontab -e`, listed with `crontab -l`. `crontab -r` deletes the table **without asking for confirmation**; since it is a single letter away from `-e`, the habit of listing should be built around `-l`. Some schedulers recognize shorthands such as `@daily`, `@hourly`, `@reboot`. These are not defined in POSIX; portable entries use the five-field form. Another class of scheduler service is the service manager's **timer units**. They offer the ability to define dependencies, make up for missed runs, and merge run logs with the service infrastructure. The service manager model is the subject of the System Administration course. #### The Scheduler's Environment The most common reason scheduled tasks fail is not the script, it is the environment. The scheduler runs the command **without a login shell**: shell startup files are not read, there are no shell functions or aliases, and most environment variables are undefined. The most visible difference is the search path. The `env.sh` below prints the search path and checks whether a command named `tool` can be found: ```sh #!/usr/bin/env bash # env.sh echo "PATH=$PATH" if command -v tool > /dev/null; then echo "tool found: yes" else echo "tool found: no" fi ``` `tool` is a small script that sits in our own `bin` directory. The same script run first in a session where that directory has been added to the search path (only the last line is shown), then in a narrow environment: ```sh mkdir -p bin printf '#!/bin/sh\necho hello\n' > bin/tool chmod +x bin/tool PATH="$PWD/bin:$PATH" bash env.sh | tail -n 1 env -i PATH=/usr/bin:/bin bash env.sh ``` ``` tool found: yes PATH=/usr/bin:/bin tool found: no ``` A tool present in an interactive session may not be found in a scheduled run. The result shows up as "the script works by hand but not under the scheduler," and since the scheduler is silent, noticing it is delayed. Three precautions are taken. **State the search path in the script.** Writing `export PATH=...` at the top of the script gives independence from the calling environment. If the paths of the tools used are known, this is the most robust solution. **Write file paths as absolute.** The scheduler's starting directory is not guaranteed; relative paths resolve in a different directory. Every path the script reads or writes must be absolute. **Define the environment variables needed inside the script.** Locale, language, and time zone variables are not inherited from the interactive session. This is also why the line `export LC_ALL=C` was placed at the top of the script in this course: sort order would otherwise stay ambiguous in the scheduler's environment. There is one more detail specific to the definition table: the `%` character in the command field means end of line, and cannot be written without escaping. Commands containing `%`, such as date formatting, are not written directly into the table — they go into a wrapper script. #### Where Output Goes The scheduler tries to collect the standard output and standard error the task produces and deliver them to the user. If delivery is not configured, the output is lost or piles up in a local mail queue. This behavior produces two consequences. First, a task that produces output on every run generates constant notifications; the notifications become unread after a while. Second, error messages get lost the same way. The correct arrangement is to manage output explicitly: ``` 30 6 * * * $HOME/report/report-job.sh ``` and doing the redirection **inside** the script. Done inside the script, the paths stay readable, the `%` problem does not arise, and the redirection logic enters version control. The stream separation set up in this topic's first lesson pays off here: the report itself is written to one file, diagnostic messages to a separate one. If the two are mixed, the next step reading the report mistakes the warning text for data. #### The Wrapper Script The scheduled task does not call the working script directly; it wraps it in a **wrapper**. The wrapper sets up the environment, acquires the lock, redirects the output, and calls the working script. ```sh #!/usr/bin/env bash # report-job.sh set -Eeuo pipefail export PATH=/usr/local/bin:/usr/bin:/bin export LC_ALL=C ROOT="$HOME/report" LOCK="$ROOT/lock" LOG_DIR="$ROOT/log" mkdir -p "$LOG_DIR" if ! mkdir "$LOCK" 2>/dev/null; then echo "previous run still in progress" >&2 exit 75 fi trap 'rmdir "$LOCK"' EXIT "$ROOT/report.sh" -n 3 "$ROOT/access.log" \ >> "$LOG_DIR/report.out" 2>> "$LOG_DIR/report.err" ``` Five decisions stand together here. **Lock.** The `mkdir` pattern set up in the Traps and Cleanup lesson finds its real purpose here: a task that runs every ten minutes and sometimes takes longer than ten minutes collides with itself without a lock. The code **75** reports "temporary error." **Environment.** `PATH` and `LC_ALL` are fixed inside the script; the environment the scheduler provides is not trusted. **Absolute paths.** There is no relative path at all; whatever the starting directory is, the script finds the same files. **Separate output.** The report is appended to `report.out`, diagnostic messages to `report.err`. The redirections use `>>`; each run does not overwrite the previous one. **Strict mode.** The wrapper is itself a script and is subject to the same rules. Log files growing forever is a separate problem, solved with log rotation tools; these tools are taken up in the System Administration course. #### Testing the Task A scheduled task must be tested under the scheduler's conditions before it is defined. The testing order is as follows: 1. Run the wrapper by hand; verify the output files are created in the right place. 2. Repeat the same call in a narrow environment: `env -i PATH=/usr/bin:/bin ./report-job.sh`. This is a close approximation of the scheduler's environment. 3. Test the lock: start a second instance while one is running and verify the code 75. 4. Test the error path: give a nonexistent log name and see a meaningful message land in `report.err`. 5. Write the entry into the table a few minutes ahead of the actual time, and observe the first run. The exit code is the last link in this chain. What reports a task's failure to a monitoring system is the exit code; this is why the codes a script produces must be chosen deliberately. The code scheme used in this course was as follows: | Code | Meaning | |---|---| | 0 | report produced | | 2 | usage error (invalid option or value) | | 3 | input error (log missing, unreadable) | | 75 | temporary error, retry later (lock not acquired) | | 128 + n | terminated by signal | #### The Finished Script The final form of the script developed throughout the course: ```sh #!/usr/bin/env bash # report.sh — produces a summary report from the access log. # Usage: report.sh [-n COUNT] [LOG] set -Eeuo pipefail export LC_ALL=C SCRIPT_NAME="report.sh" LINE_PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$' temp="" error() { printf '%s: error: %s\n' "$SCRIPT_NAME" "$*" >&2; } warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; } die() { error "$1"; exit "${2:-1}"; } section() { printf '\n== %s ==\n' "$*"; } cleanup() { local code=$? [ -n "$temp" ] && rm -rf "$temp" return "$code" } trap cleanup EXIT trap 'cleanup; trap - INT; kill -INT $$' INT trap 'cleanup; trap - TERM; kill -TERM $$' TERM trap 'error "$LINENO. line unexpected error"' ERR summary() { awk ' { requests++; bytes += $10; class[substr($9, 1, 1)]++ } END { printf " requests: %d\n", requests printf " bytes : %d\n", bytes printf " average : %.1f bytes\n", (requests ? bytes / requests : 0) for (s = 2; s <= 5; s++) printf " %sxx %4d %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0) } ' "$1" } top_paths() { cut -d' ' -f7 "$1" \ | sed -E 's#/[0-9]+#/:id#g' \ | sort | uniq -c | sort -k1,1nr -k2,2 | head -"$2" } error_paths() { awk '$9 ~ /^[45]/ { n[$7]++ } END { for (y in n) printf "%3d %s\n", n[y], y }' "$1" \ | sort -k1,1nr -k2,2 } count=5 while getopts ':n:' option; do case "$option" in n) count="$OPTARG" ;; :) die "-$OPTARG expects a value" 2 ;; \?) die "unknown option -$OPTARG" 2 ;; esac done shift $((OPTIND - 1)) log="${1:-access.log}" [[ "$count" =~ ^[1-9][0-9]*$ ]] || die "-n expects a positive integer: $count" 2 [ -f "$log" ] || die "not a regular file or does not exist: $log" 3 [ -r "$log" ] || die "no read permission: $log" 3 [ -s "$log" ] || warn "log is empty: $log" temp=$(mktemp -d) valid="$temp/valid.log" grep -E "$LINE_PATTERN" "$log" | awk 'NF == 10' > "$valid" || true invalid=$(( $(grep -c '' "$log") - $(grep -c '' "$valid") )) [ "$invalid" -eq 0 ] || warn "$invalid lines not in the expected format, excluded from analysis" section "Log: $log" summary "$valid" section "Top $count requested paths" top_paths "$valid" "$count" section "Paths returning errors" error_paths "$valid" ``` The script's last addition is the step that separates valid lines before analysis. The line pattern from the Regular Expressions lesson and the field-count check from the Field-Based Processing lesson are applied together: neither was sufficient alone. Discarded lines are counted and reported to standard error, while counts are computed only over the valid lines — so a corrupted input does not silently distort the report. With a clean log: ```sh chmod +x report.sh ./report.sh -n 3 ``` ``` == Log: access.log == requests: 30 bytes : 117780 average : 3926.0 bytes 2xx 18 %60.0 3xx 3 %10.0 4xx 6 %20.0 5xx 3 %10.0 == Top 3 requested paths == 6 /api/data 6 /product/:id 4 /index.html == Paths returning errors == 4 /missing.html 3 /api/data 2 /secret ``` With a log that has two malformed lines added, the first seven lines of the output are shown; `2>&1` folds the warning into the same stream, and `sed -n '1,7p'` reads all of its input, so it does not close the pipe early (`head -7` would not give that guarantee): ```sh cp access.log all.log printf 'malformed line\nthis one is malformed too\n' >> all.log ./report.sh -n 2 all.log 2>&1 | sed -n '1,7p' ``` ``` report.sh: warning: 2 lines not in the expected format, excluded from analysis == Log: all.log == requests: 30 bytes : 117780 average : 3926.0 bytes 2xx 18 %60.0 ``` The warning went to standard error and the report to standard output (here `2>&1` shows them in a single stream), and the counts were unaffected by the malformed lines. The wrapper can be tested together with this script. The commands below put `report.sh` and `access.log` into the `$HOME/report` directory, run the wrapper, and list the output files that result: ```sh mkdir -p "$HOME/report" cp report.sh access.log "$HOME/report/" bash report-job.sh ls "$HOME/report/log" ``` ``` report.err report.out ``` The script has a known behavior: when its output is piped into a command that closes early, like `head`, the `ERR` hook is triggered by `SIGPIPE` because `pipefail` is active. This is a deliberate trade-off of strict mode; shortening the output must be done with the `-n` option. #### Summary - A cron entry consists of five time fields and a command; when day of month and day of week are both given, the conditions combine with "or." - The scheduler runs the command without a login shell; the search path is narrow, startup files are not read, and most environment variables are undefined. - The search path and locale are fixed inside the script, and all file paths are written as absolute. - Output is managed explicitly: the report and diagnostic messages are appended to separate files. - The wrapper script sets up the environment, acquires the lock, and redirects output; the lock prevents overlapping runs. - Exit codes are the monitoring system's only input; usage, input, and temporary-error classes are reported with separate codes. #### Course Wrap-Up This course started with a single question: exactly where does a command write the data it writes? The answer was three streams and file descriptors. Everything that follows from there is an expansion of the same idea. **Streams and Pipelines** established connecting commands together: redirection being applied before the command, the pipeline's processes running concurrently, variables lost in a subshell, and the exit code convention. **Script Structure** turned one-line commands into a program. At the center of this section stood the thing that sets the shell apart from other languages: an expansion's result is **reparsed**. Word splitting, pathname expansion, and quoting rules are this course's academic core; most errors in shell scripts are born here. **Text Processing** handed the work to its dedicated tools: the pattern language, line filtering, the stream editor, field-based processing, and the sort family. The selection criterion that closed the section — choose the narrowest tool, switch tools when a problem needs more than three of them — is this course's practical inheritance. **Writing Robust Scripts** prepared the script for unattended operation: strict mode, cleanup hooks, tracing tools, static analysis, and scheduled runs. The same script was developed throughout the course. `report.sh` began as a fixed pipeline; it gained variables, acquired an argument interface, learned to validate its inputs, was split into functions, reduced its counting to a single pass, and learned to stop on errors and clean up after itself. Its final form is a tool that passes inspection and can be handed to a scheduler. The shell's limit also showed itself on this journey. The shell has no equal at connecting processes together and working with the file system. It was not designed for complex data structures, decimal arithmetic, structured data formats, or extensive error handling. A good shell script is a script that knows its own limit. The next course — **System Administration** — takes up the system this script runs on. `report.sh` was handed to a scheduler; but how is that scheduler itself defined and inspected as a service? The script created a process; how are processes monitored, prioritized, and managed with signals? Log files are growing; how is disk and file system configuration changed? In this course, signals, exit codes, and processes were seen from **inside** a script. The next course looks at them from **outside** the system.