Loops

The report currently produces a single list. When a separate count is wanted for each status-code class, the same pipeline has to be written by hand four times. In the Programming Fundamentals course, this was the classic justification for iteration.

Loops in the shell build the same structures, but where the list comes from depends on the shell’s expansion rules. This lesson takes up the loop forms and — the most commonly miswritten part of shell programming — the pattern for reading a file line by line.

Iterating Over a List

The for loop runs the body for every element of the word list given to it.

for class in 2 3 4 5; do
  count=$(cut -d' ' -f9 access.log | grep -c "^${class}")
  echo "${class}xx -> $count"
done

2xx -> 18
3xx -> 3
4xx -> 6
5xx -> 3

The $( ... ) notation turns a command’s output into a value; the details will be taken up in this topic’s final lesson.

The concept of a “word” in the list is the same word-splitting concept established in the previous lessons. If the list comes from a variable, quoting directly determines the loop’s iteration count:

codes="200 404 500"
for code in $codes;   do printf '<%s> ' "$code"; done; echo
for code in "$codes"; do printf '<%s> ' "$code"; done; echo

<200> <404> <500> 
<200 404 500> 

This is one of the rare cases where word splitting is deliberately wanted. Still, keeping a value in a single string and relying on it being split is fragile; if more than one value needs to be carried, an array is the correct tool. Arrays are the subject of this topic’s seventh lesson.

Iterating Over File Names

The correct way to iterate over the files in a directory is to write the pathname pattern directly into the list:

mkdir -p logs; : > 'logs/a.log'; : > 'logs/b c.log'
for f in logs/*.log; do echo "  [$f]"; done

  [logs/a.log]
  [logs/b c.log]

The name containing a space stayed as a single piece; because the result of pathname expansion is not subject to word splitting. This is the practical counterpart of the rule established in the second lesson.

Writing the same job as for f in $(ls logs) is wrong: ls’s output is text and undergoes word splitting; every name containing a space gets split apart. In the shell, the tool for producing a file list is not ls, it is pathname expansion.

A pattern matching no file at all is a separate trap:

for f in logs/*.missing; do echo "  [$f]"; done

  [logs/*.missing]

If there is no match, the shell leaves the pattern literal, and the loop runs once, with a nonexistent file name. There are two solutions. The portable one is to put an existence test at the start of the body:

for f in logs/*.missing; do
  [ -e "$f" ] || continue
  echo "  [$f]"
done

The bash-specific one is the nullglob option; a pattern with no match expands to an empty list, and the loop does not run at all:

shopt -s nullglob
for f in logs/*.missing; do echo "  [$f]"; done
shopt -u nullglob

The option is a global setting and affects every pattern while it is active; turning it on at the start of the script and leaving it on can lead to unexpected behavior in later lines.

Counted Loops

There are three forms for a fixed number of iterations.

Brace expansion produces a fixed list at parse time:

for i in {1..5}; do printf '%s ' "$i"; done; echo

1 2 3 4 5 

Brace expansion cannot contain a variable — {1..$n} does not work as expected, because brace expansion is applied before variable expansion. This is one of the observable consequences of expansion order.

The C-style loop is a bash extension and can take a variable bound:

for ((i = 1; i <= 5; i++)); do printf '%s ' "$i"; done; echo

1 2 3 4 5 

A command that produces a sequence is the portable third way: for i in $(seq 1 "$n"). Because of the command invocation, it is the most expensive option, but it works in the POSIX shell language.

Condition-Based Loops

while repeats the body as long as the condition command returns 0; until as long as it returns nonzero.

counter=0
until [ "$counter" -ge 3 ]; do
  echo "counter=$counter"
  counter=$((counter + 1))
done

counter=0
counter=1
counter=2

The two can be used interchangeably; until saves writing a negation of the condition, and is preferred in situations where it improves readability.

Reading a File Line by Line

This is the most commonly miswritten pattern in shell programming. The correct form has three parts, and each part has its own justification:

while IFS= read -r line; do
  ...
done < file

The -r option turns off backslash processing. Without this option, read counts the backslash as an escape character and corrupts the data.

The IFS= assignment prevents the leading and trailing whitespace of the line from being trimmed.

Redirection, not a pipe. A loop at the right end of a pipeline runs in a subshell, and the variables it assigns are lost; this was shown in the Pipeline lesson.

The effect of the three parts can be measured. The following file has three lines: the first has two spaces, the second starts with a backslash, the third has whitespace on both ends.

printf 'one  two\n\\escaped\n  spaced edge  \n' > odd.txt
while read line; do printf '<%s>\n' "$line"; done < odd.txt

<one  two>
<escaped>
<spaced edge>

while IFS= read -r line; do printf '<%s>\n' "$line"; done < odd.txt

<one  two>
<\escaped>
<  spaced edge  >

In the first form, the backslash was swallowed and the edge whitespace was trimmed. The second gave the file as it is.

A fourth detail is the last line lacking a newline. If read reaches the end of the file without seeing a newline, it writes what it read into the variable but returns a nonzero code; the loop condition misreads this code, and the last line does not get processed:

printf 'first\nsecond' > missing.txt
while IFS= read -r s; do echo "read: $s"; done < missing.txt

read: first

An addition is made to the condition: if the variable is not empty, the loop runs one more iteration.

while IFS= read -r s || [ -n "$s" ]; do echo "read: $s"; done < missing.txt

read: first
read: second

This pattern is required in every script that reads externally sourced data; files that do not end with a newline are not rare.

Reading a Line Split into Fields

If read takes multiple variables, it splits the line according to IFS and gives the last variable everything that remains. Because the log’s fields are separated by spaces, it can be read directly:

head -3 access.log | while read -r address _ _ _ _ method path _ code bytes; do
  printf '%-12s %-4s %-18s %s %s\n' "$address" "${method#\"}" "$path" "$code" "$bytes"
done

10.0.0.12    GET  /index.html        200 5120
10.0.0.31    GET  /static/style.css  200 2048
10.0.0.12    GET  /product/12        200 8410

The name _ is a convention, not a language element: fields that are not of interest get assigned to the same variable. The ${method#\"} expansion strips the leading quote; string-processing expansions will be gathered together in the Arithmetic and Command Substitution lesson.

This loop is sufficient for data with a fixed field count. When the field count varies or a calculation is needed, a field-based tool is more suitable; in the Text Processing topic, the same job will come down to a single line.

Loop Control

break terminates the loop it is in, continue skips the iteration. Both can take a number argument to affect outer loops.

for code in 200 301 404 500 xyz 200; do
  case "$code" in
    [0-9][0-9][0-9]) : ;;
    *) echo "invalid code, skipping: $code"; continue ;;
  esac
  if [ "$code" -ge 500 ]; then echo "server error found: $code"; break; fi
  echo "processed: $code"
done

processed: 200
processed: 301
processed: 404
server error found: 500

break 2 exits two levels out:

for a in 1 2; do
  for b in x y; do
    [ "$b" = y ] && break 2
    echo "$a$b"
  done
done

1x

A numbered break quickly hurts readability; beyond two levels, putting the loop into a function and using return becomes clearer. Functions are the subject of the next lesson.

Applying It to the Script

report.sh gains a second section: the distribution of status-code classes.

log_file=access.log

echo "== Status code classes =="
for class in 2 3 4 5; do
  count=$(cut -d' ' -f9 "$log_file" | grep -c "^${class}")
  case "$class" in
    2) name="successful"    ;;
    3) name="redirect"      ;;
    4) name="client error"  ;;
    5) name="server error"  ;;
  esac
  printf '  %sxx %-16s %3d\n' "$class" "$name" "$count"
done

== Status code classes ==
  2xx successful        18
  3xx redirect           3
  4xx client error       6
  5xx server error       3

The numbers add up to thirty; equal to the number of lines in the log. This kind of internal consistency check is a cheap way to verify a report script’s correctness.

Summary

• A for list is formed by the shell’s expansion rules; an unquoted variable is split, a quoted one is not.
• A file list is produced from pathname expansion, not from ls output; if there is no match, the pattern stays literal, and [ -e "$f" ] || continue or nullglob is required.
• Brace expansion cannot take a variable, because it is applied before variable expansion.
• The correct form for reading line by line is while IFS= read -r line; do ... done < file; if a pipe is used, the loop falls into a subshell.
• In files that do not end with a newline, the last line is processed only with the || [ -n "$line" ] addition.

Next Step

As the script grows, the same work — reporting errors, printing a section header, counting a field — repeats in more than one place. The next lesson takes up functions: what a function definition is in the shell, how arguments pass, the difference between return and exit, and where the shell’s scope model departs from the Programming Fundamentals course’s.