Transforming with the Stream Editor

Filtering selects lines but does not change them. Taking only the hour from the log’s timestamp, reducing paths with a numeric identifier to a single pattern, or turning the line into a delimited record requires transforming the line itself.

A stream editor is a filter that reads its input line by line, applies a sequence of commands to each line, and writes the result. Its difference from an interactive editor is that the decisions are made in advance: no file is opened, nothing is displayed, the stream only flows through.

Substitution

The most frequently used command is s: s/pattern/replacement/flags.

head -2 access.log | sed 's/HTTP\/1.1/HTTP/'

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP" 200 5120
10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP" 200 2048

Because the pattern contains a slash, it had to be escaped. The delimiter is free; the first character after the s command is taken as the delimiter:

head -1 access.log | sed 's|HTTP/1.1|HTTP|'

Choosing | or # in patterns that contain a path removes the backslash pileup.

The default behavior is to change only the first match on each line. Flags change this:

echo 'a-b-c-d' | sed 's/-/+/'
echo 'a-b-c-d' | sed 's/-/+/g'
echo 'a-b-c-d' | sed 's/-/+/3'

a+b-c-d
a+b+c+d
a-b-c+d

The g flag changes every match, a number flag changes only the Nth match. The I flag removes case sensitivity. POSIX defined this feature in its latest edition in the lowercase form; the uppercase spelling used here is an implementation extension. It is present in common implementations.

Reusing the Matched Text

Inside the replacement, & stands for the entire match:

head -1 access.log | sed -E 's/ [0-9]{3} [0-9]+$/[&]/'

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1"[ 200 5120]

Groups are recalled with \1, \2 … This is the way to rewrite a line into a desired format:

sed -E 's/^([0-9.]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) [^"]*" ([0-9]{3}) ([0-9]+)$/\1|\3|\4|\5|\6/' access.log | head -3

10.0.0.12|GET|/index.html|200|5120
10.0.0.31|GET|/static/style.css|200|2048
10.0.0.12|GET|/product/12|200|8410

A space-separated log line containing quotes turned into a five-field record separated by pipes. The second group (the timestamp) was captured but not used; capturing it is required, because the pattern has to cover the entire line.

Choosing the Syntax

sed uses basic regular expression syntax by default. The -E option switches to the extended syntax.

In the basic syntax, support for some operators depends on the implementation:

echo 'aaa' | sed 's/a\+/X/'
echo 'aaa' | sed -E 's/a+/X/'

X
X

In the GNU version both commands work; the \+ operator is recognized as an extension in the basic syntax. In the BSD version (macOS) \+ is not recognized: the first command reads the pattern as “an a and a plus sign,” finds no match and writes the line aaa unchanged.

Conclusion: if +, ?, |, and {n,m} are going to be used, write -E. The -E option is present in both the BSD and GNU versions; the -r option seen in older GNU scripts is GNU-specific.

Addresses

An address written before a command determines which lines the command applies to.

Address	Meaning
5	the fifth line
$	the last line
/pattern/	lines matching the pattern
2,4	from the second line to the fourth
/a/,/b/	from the line matching a to the line matching b
address!	lines outside the address

The d command deletes the line. In a file containing the lines one, two, three, four:

printf 'one\ntwo\nthree\nfour\n' > words.txt
sed '2,3d' words.txt

one
four

sed '$d' words.txt

one
two
three

sed '/tw/d' words.txt

one
three
four

sed '/tw/!d' words.txt

two

The ! mark inverts the selection; the last call, by saying “delete the ones that do not match,” left only the one that did.

The p command writes the line. Because the stream editor already writes every line by default, p alone produces a repeat; the -n option turns off the default writing and p becomes the selector:

sed -n '2,3p' words.txt

two
three

This pair — -n and p — is filtering written with the stream editor. For selecting lines alone, the filter tool is more suitable; when transformation and selection are both needed, the stream editor does both in a single pass.

Other Commands

y/abc/xyz/ transliterates characters; it uses literal, character-for-character mapping, not a pattern:

echo 'abc' | sed 'y/abc/xyz/'

xyz

The q command ends the stream once its address is reached: sed '100q' writes the first hundred lines and stops, never reading the rest of the file.

Multiple commands are chained with -e or with semicolons:

head -1 access.log | sed -e 's/HTTP\/1.1//' -e 's/  */ /g'

10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html " 200 5120

Commands are applied to each line in sequence; the second command works on the first one’s output. This means one substitution can feed another, and the result changes if the order changes.

In-Place Editing: Not Portable

The -i option changes the file in place. How this option takes its argument is incompatible between the two major implementations, and this is one of the most common portability bugs in shell scripts.

In the GNU version the suffix is written attached to the option and is optional; if no suffix is given, no backup is made:

sed -i 's/one/ONE/' words.txt
cat words.txt

ONE
two
three
four

In the BSD version (macOS), -i mandatorily wants a separate backup-suffix argument. The same command fails there with sed: -I or -i may not be used with stdin: the option took the following s/one/ONE/ as the backup suffix, words.txt became the script, and no file operand was left, so sed tried to read standard input, which -i does not allow. The correct BSD spelling gives the empty suffix explicitly: sed -i '' 's/one/ONE/' words.txt.

In the GNU version that spelling does not work: if a separate empty argument is given, that argument is taken as the script and the string after it is taken for a file name:

sed -i '' 's/one/ONE/' words.txt

sed: can't read s/one/ONE/: No such file or directory

Neither spelling works on both versions. Not using -i in scripts is the right decision whenever portability is needed. In its place, the pattern built in the Redirection lesson is written:

sed 's/one/ONE/' words.txt > words.new && mv words.new words.txt

This pattern carries three advantages: it works the same way everywhere, the original file is not corrupted if sed fails, and mv is atomic on the same file system — a process reading the file sees either the old or the new content, never a half-written one.

The -i option itself does the same thing internally: it writes to a temporary file and swaps it in. The only thing it offers is brevity of notation, and its cost is portability.

Applying to the Script

Part of the paths in the log contain a numeric identifier: when /product/12 and /product/45 are counted separately, both stay near the bottom of the list. Reducing the identifiers to a single symbol makes visible which endpoint is busy:

log_file=access.log

cut -d' ' -f7 "$log_file" | sed -E 's#/[0-9]+#/:id#g' | sort | uniq -c | sort -rn

      6 /product/:id
      6 /api/data
      4 /missing.html
      4 /index.html
      3 /static/style.css
      3 /panel
      2 /secret
      1 /old-page
      1 /login

/product/:id rose to the top of the list with six requests; counted separately, it sat in fourth and fifth place with three requests each. Normalization changes what the count measures — this is not a formatting fix but an analysis decision, and it should be stated in the report output.

The second application pulls the hour field out of the timestamp:

log_file=access.log
sed -E 's/^.*:([0-9]{2}):[0-9]{2}:[0-9]{2} .*$/\1/' "$log_file" | sort | uniq -c

     30 09

Because the sample log falls entirely within a single hour, the output is a single line; in a real log spread across days, this produces a column giving the load distribution.

The ^.*: part of the pattern is greedy and advances up to the last colon; the following groups therefore match the minute and second. Greediness working here is not a coincidence, but the pattern’s correctness depends on it — patterns that rely on greediness are fragile, and a field-based solution is safer.

Summary

• Substitution has the form s/pattern/replacement/flag; the delimiter is free, and the default behavior is one match per line.
• & stands for the entire match, \1, \2 for groups; this is the tool for rewriting a line.
• In the basic syntax, support for operators such as \+ depends on the implementation; -E is present in both common versions and is preferred.
• An address determines a command’s scope; ! inverts the selection, -n with p gives selective writing.
• The -i option’s argument handling is incompatible between the BSD and GNU versions; the portable form is to write to a temporary file and swap it in.

Next Step

The stream editor sees a line as text; it can only imitate fields with a pattern. In column-shaped data such as a log, this leads to fragile patterns. The next lesson takes up field-based processing: the tool that automatically splits a line into fields, can compute over fields, and can keep an accumulation per key — so the counts written with associative arrays in the previous topic collapse into a single line.