--- title: 'Transforming with the Stream Editor' source: 'https://academia.sh/en/courses/shell-programming/transforming-with-the-stream-editor' course: 'Shell Programming' language: en updated: '2026-08-17T18:10:03+00:00' license: 'CC BY-SA 4.0' --- # Transforming with the Stream Editor The substitution command and its flags, rewriting with backreferences, address and range selection, and the mismatch between implementations in in-place editing. Filtering selects lines but does not change them. Taking only the hour from the log's timestamp, reducing paths with a numeric identifier to a single pattern, or turning the line into a delimited record requires transforming the line itself. A **stream editor** is a filter that reads its input line by line, applies a sequence of commands to each line, and writes the result. Its difference from an interactive editor is that the decisions are made in advance: no file is opened, nothing is displayed, the stream only flows through. ## Substitution The most frequently used command is `s`: `s/pattern/replacement/flags`. ```sh head -2 access.log | sed 's/HTTP\/1.1/HTTP/' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP" 200 5120 10.0.0.31 - - [07/Feb/2024:09:12:51 +0000] "GET /static/style.css HTTP" 200 2048 ``` Because the pattern contains a slash, it had to be escaped. The delimiter is free; the first character after the `s` command is taken as the delimiter: ```sh head -1 access.log | sed 's|HTTP/1.1|HTTP|' ``` Choosing `|` or `#` in patterns that contain a path removes the backslash pileup. The default behavior is to change **only the first match on each line**. Flags change this: ```sh echo 'a-b-c-d' | sed 's/-/+/' echo 'a-b-c-d' | sed 's/-/+/g' echo 'a-b-c-d' | sed 's/-/+/3' ``` ``` a+b-c-d a+b+c+d a-b-c+d ``` The `g` flag changes every match, a number flag changes only the `N`th match. The `I` flag removes case sensitivity; it is not defined in POSIX but is present in common implementations. ## Reusing the Matched Text Inside the replacement, `&` stands for the entire match: ```sh head -1 access.log | sed -E 's/ [0-9]{3} [0-9]+$/[&]/' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html HTTP/1.1"[ 200 5120] ``` Groups are recalled with `\1`, `\2` … This is the way to rewrite a line into a desired format: ```sh sed -E 's/^([0-9.]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) [^"]*" ([0-9]{3}) ([0-9]+)$/\1|\3|\4|\5|\6/' access.log | head -3 ``` ``` 10.0.0.12|GET|/index.html|200|5120 10.0.0.31|GET|/static/style.css|200|2048 10.0.0.12|GET|/product/12|200|8410 ``` A space-separated log line containing quotes turned into a five-field record separated by pipes. The second group (the timestamp) was captured but not used; capturing it is required, because the pattern has to cover the entire line. ## Choosing the Syntax `sed` uses basic regular expression syntax by default. The `-E` option switches to the extended syntax. In the basic syntax, support for some operators depends on the implementation: ```sh echo 'aaa' | sed 's/a\+/X/' echo 'aaa' | sed -E 's/a+/X/' ``` ``` aaa X ``` The first line produced no change: this implementation does not recognize the `\+` operator in the basic syntax, read the pattern as "an `a` and a plus sign," and found no match. The same pattern works in the GNU version. Conclusion: **if `+`, `?`, `|`, and `{n,m}` are going to be used, write `-E`.** The `-E` option is present in both the BSD and GNU versions; the `-r` option seen in older GNU scripts is GNU-specific. ## Addresses An **address** written before a command determines which lines the command applies to. | Address | Meaning | |---|---| | `5` | the fifth line | | `$` | the last line | | `/pattern/` | lines matching the pattern | | `2,4` | from the second line to the fourth | | `/a/,/b/` | from the line matching `a` to the line matching `b` | | `address!` | lines **outside** the address | The `d` command deletes the line. In a file containing the lines `one`, `two`, `three`, `four`: ```sh sed '2,3d' words.txt ``` ``` one four ``` ```sh sed '$d' words.txt ``` ``` one two three ``` ```sh sed '/tw/d' words.txt ``` ``` one three four ``` ```sh sed '/tw/!d' words.txt ``` ``` two ``` The `!` mark inverts the selection; the last call, by saying "delete the ones that do not match," left only the one that did. The `p` command writes the line. Because the stream editor already writes every line by default, `p` alone produces a repeat; the `-n` option turns off the default writing and `p` becomes the selector: ```sh sed -n '2,3p' words.txt ``` ``` two three ``` This pair — `-n` and `p` — is filtering written with the stream editor. For selecting lines alone, the filter tool is more suitable; when transformation and selection are both needed, the stream editor does both in a single pass. ## Other Commands `y/abc/xyz/` transliterates characters; it uses literal, character-for-character mapping, not a pattern: ```sh echo 'abc' | sed 'y/abc/xyz/' ``` ``` xyz ``` The `q` command ends the stream once its address is reached: `sed '100q'` writes the first hundred lines and stops, never reading the rest of the file. Multiple commands are chained with `-e` or with semicolons: ```sh head -1 access.log | sed -e 's/HTTP\/1.1//' -e 's/ */ /g' ``` ``` 10.0.0.12 - - [07/Feb/2024:09:12:44 +0000] "GET /index.html " 200 5120 ``` Commands are applied to each line **in sequence**; the second command works on the first one's output. This means one substitution can feed another, and the result changes if the order changes. ## In-Place Editing: Not Portable The `-i` option changes the file in place. How this option takes its argument is incompatible between the two major implementations, and this is one of the most common portability bugs in shell scripts. In the BSD version, `-i` **mandatorily** wants a backup-suffix argument: ```sh sed -i 's/one/ONE/' words.txt ``` ``` sed: -I or -i may not be used with stdin ``` The option took the following `s/one/ONE/` sequence as the backup suffix and was left with no file operand to work on, so it tried to read from standard input, which `-i` does not allow. The correct form is to give the empty suffix explicitly: ```sh sed -i '' 's/one/ONE/' words.txt ``` In the GNU version, the suffix is written **attached to the option**; if a separate empty argument is given, it is taken for a file name: ```sh sed -i '' 's/one/ONE/' words.txt ``` ``` sed: can't read s/one/ONE/: No such file or directory ``` Neither spelling works on both versions. Not using `-i` in scripts is the right decision whenever portability is needed. In its place, the pattern built in the Redirection lesson is written: ```sh sed 's/one/ONE/' words.txt > words.new && mv words.new words.txt ``` This pattern carries three advantages: it works the same way everywhere, the original file is not corrupted if `sed` fails, and `mv` is atomic on the same file system — a process reading the file sees either the old or the new content, never a half-written one. The `-i` option itself does the same thing internally: it writes to a temporary file and swaps it in. The only thing it offers is brevity of notation, and its cost is portability. ## Applying to the Script Part of the paths in the log contain a numeric identifier: when `/product/12` and `/product/45` are counted separately, both stay near the bottom of the list. Reducing the identifiers to a single symbol makes visible which **endpoint** is busy: ```sh log_file=access.log cut -d' ' -f7 "$log_file" | sed -E 's#/[0-9]+#/:id#g' | sort | uniq -c | sort -rn ``` ``` 6 /product/:id 6 /api/data 4 /missing.html 4 /index.html 3 /static/style.css 3 /panel 2 /secret 1 /old-page 1 /login ``` `/product/:id` rose to the top of the list with six requests; counted separately, it sat in fourth and fifth place with three requests each. Normalization changes what the count measures — this is not a formatting fix but an analysis decision, and it should be stated in the report output. The second application pulls the hour field out of the timestamp: ```sh sed -E 's/^.*:([0-9]{2}):[0-9]{2}:[0-9]{2} .*$/\1/' "$log_file" | sort | uniq -c ``` ``` 30 09 ``` Because the sample log falls entirely within a single hour, the output is a single line; in a real log spread across days, this produces a column giving the load distribution. The `^.*:` part of the pattern is greedy and advances up to the **last** colon; the following groups therefore match the minute and second. Greediness working here is not a coincidence, but the pattern's correctness depends on it — patterns that rely on greediness are fragile, and a field-based solution is safer. ## Summary - Substitution has the form `s/pattern/replacement/flag`; the delimiter is free, and the default behavior is one match per line. - `&` stands for the entire match, `\1`, `\2` for groups; this is the tool for rewriting a line. - In the basic syntax, support for operators such as `\+` depends on the implementation; `-E` is present in both common versions and is preferred. - An address determines a command's scope; `!` inverts the selection, `-n` with `p` gives selective writing. - The `-i` option's argument handling is incompatible between the BSD and GNU versions; the portable form is to write to a temporary file and swap it in. ## Next Step The stream editor sees a line as text; it can only imitate fields with a pattern. In column-shaped data such as a log, this leads to fragile patterns. The next lesson takes up field-based processing: the tool that automatically splits a line into fields, can compute over fields, and can keep an accumulation per key — so the counts written with associative arrays in the previous topic collapse into a single line.