Line Filtering

The pattern language is built. The first tool that applies it is the filter that selects, from a stream, the lines that fit a pattern. This tool is the most frequently called of shell text processing, and most of its options turn the behavior “write the matching lines” into a different question.

This lesson takes up those questions and the correct option for each.

Selecting, Counting, Locating

The default behavior is to write the matching lines. The -n option adds the line number:

grep -n '500' access.log

8:192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180
13:192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180
27:10.0.2.7 - - [07/Feb/2024:09:35:44 +0000] "POST /api/data HTTP/1.1" 500 180

-c gives only the number. What is counted is the number of matching lines, not the number of matches:

grep -Ec  '\.' access.log
grep -Eo '\.' access.log | wc -l

30
     131

All thirty lines have at least one dot; the total dot count is 131. The distinction separates the questions “how many requests got a 404” from “how many times does the string 404 occur.” If a match count is wanted, -o spills each match onto its own line and it is counted.

The grep -c '' form gives a line count, because the empty pattern matches every line, and it does not produce wc -l’s alignment padding:

grep -c '' access.log

30

Inverting

The -v option selects lines that do not match. It is the most-used option in filtering jobs; it is the direct way to drop noise lines.

grep -vc '" 200 ' access.log

13

Thirteen of thirty requests got a code other than 200. -v can be combined with -c; combining it with -o is meaningless, because a non-matching line has no match to write.

Narrowing the Boundaries of a Match

Three options restrict which part of the line a pattern can match against.

-i removes case sensitivity. -w requires the pattern to start and end at word boundaries. -x requires the pattern to match the entire line.

printf '%s\n' 'GET /a' 'get /b' 'GETTER /c' > case.txt
grep -c  'GET' case.txt
grep -ic 'GET' case.txt
grep -wc 'GET' case.txt

2
3
1

The file contains the lines GET /a, get /b, and GETTER /c. The default search selected two, the case-insensitive search three, the word search only one — GET inside GETTER is not a word.

-x is a shortcut for writing anchors:

printf '%s\n' '404' '4040' > code.txt
grep -c  '404' code.txt
grep -xc '404' code.txt

2
1

The line 4040 in the file fit the unanchored pattern. In checks that want an exact match, forgetting -x silently produces an inflated result.

Fixed String Mode

The -F option interprets the pattern not as a regular expression but as a literal string.

printf '%s\n' 'a.b' 'axb' > dot.txt
grep -c  'a.b' dot.txt
grep -Fc 'a.b' dot.txt

2
1

-F matters for two reasons. First, correctness: if the pattern comes from a variable — user input, a file name, a configuration value — the ., *, [ characters inside it are interpreted as pattern syntax, and the search gives a wider result than expected. Second, using user input as a pattern is a security boundary: a crafted pattern can deliberately blow up the matching cost.

Rule: if you did not write the pattern yourself, use -F.

Patterns can be kept in a file; the -f option counts each line as a separate pattern:

printf '%s\n' '403' '500' > patterns.txt
grep -Fcf patterns.txt access.log

5

Multi-File Search

When more than one file is given, the file name is prefixed to the output:

mkdir -p log-dir
head -10 access.log > log-dir/monday.log
tail -10 access.log > log-dir/tuesday.log
grep -c '404' log-dir/*.log

log-dir/monday.log:1
log-dir/tuesday.log:2

-h suppresses this prefix, -H adds it even for a single file. If the output is going to be parsed in a script, the option must be written explicitly; an output whose prefix changes with the file count is a fragile foundation.

-l gives the names of files where a match was found, -L the names where it was not:

grep -l '500' log-dir/*.log

log-dir/monday.log
log-dir/tuesday.log

-r searches a directory tree recursively. The order files are visited is whatever order the file system gives, and it varies from implementation to implementation; if sorted output is needed, the result must be sorted separately with sort.

-m N stops after the first N matches in each file. For the question “does it exist” on large files, this avoids reading the whole file.

Context Lines

-A N writes the N lines after a match, -B N before, -C N on both sides. A -- separator is placed between blocks.

grep -A 1 '" 500 ' access.log | head -4

192.168.1.5 - - [07/Feb/2024:09:17:22 +0000] "GET /api/data HTTP/1.1" 500 180
10.0.0.12 - - [07/Feb/2024:09:18:00 +0000] "GET /api/data HTTP/1.1" 200 3300
--
192.168.1.5 - - [07/Feb/2024:09:21:38 +0000] "GET /api/data HTTP/1.1" 500 180

This is used in debugging to look at the requests before and after a fault. Because the separator line breaks the output’s format, context options are not used on lines whose output will flow into another tool.

Using the Exit Code as a Condition

The -q option writes nothing; it stops at the first match and reports the result only through its exit code. This is the correct form of an “does it exist” question in a script:

if grep -q '" 500 ' access.log; then
  echo "server error found"
fi

server error found

The form grep -c ... > /dev/null does not do the same job: -c writes 0 even without a match, and the exit code still depends on whether there was a match, but the entire file is read regardless. -q stops at the first match.

The three-code contract built in the Exit Codes lesson applies here: 0 found, 1 not found, 2 file error. In scripts running under strict mode, the “not found” case must be used as a condition, or || true must be added, so it does not stop the script.

Applying to the Script

report.sh gains two new sections. The first counts the log’s broken lines; the second lists the paths that returned an error.

log_file=access.log

LINE_PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$'

malformed=$(grep -Evc "$LINE_PATTERN" "$log_file")
if [ "$malformed" -gt 0 ]; then
  warn "$malformed lines are not in the expected format and are included in the counts"
fi

section "Paths returning errors"
grep -E '" [45][0-9]{2} ' "$log_file" | cut -d' ' -f7 | sort | uniq -c | sort -rn

When run with a log that has a broken line:

./report.sh all.log

The warning falls to standard error, the report continues to standard output. On a clean log, the error-returning-paths section gives this:

   4 /missing.html
   3 /api/data
   2 /secret

The quotes and spaces in the grep -E '" [45][0-9]{2} ' pattern separate the status code field from another number in the log. This is imitating the concept of a field with a pattern; in the following lessons, field-based tools will do the same job by position, and the pattern will no longer be needed.

Filtering’s Place in the Pipeline

Concurrency was established in the pipeline lesson. Its practical consequence is an ordering rule: put the step that shrinks the data at the start of the pipeline.

grep -E '" [45][0-9]{2} ' "$log_file" | cut -d' ' -f7 | sort | uniq -c

In this pipeline, grep brings thirty lines down to nine; sort sorts nine lines. Had sorting come last, thirty lines would have been sorted. That sorting cost depends on input size — as shown in the Algorithms course — is the justification for this rule.

The rule carries one exception: filtering must not drop lines the next step needs. In transformations that require context, filtering is deferred to later.

Summary

• -c counts matching lines, not matches; a match count is spilled into lines with -o and counted.
• -v inverts the selection; -w requires a word match, -x a full-line match.
• -F interprets the pattern literally and is required for patterns coming from outside.
• In multi-file search, the output prefix changes with the file count; in a script, -h or -H is written explicitly.
• -q writes nothing and stops at the first match; it is the correct form of an existence check in a script.
• The filter that shrinks the data is placed at the start of the pipeline.

Next Step

Filtering selects lines but does not change them. Reformatting the timestamp in the log, simplifying field delimiters, or masking a field requires transforming the line itself. The next lesson takes up the stream editor: substitution, address and range selection, rewriting with backreferences, and the mismatch between implementations in the in-place editing option.