Cutting, Pasting, and Splitting

The tools of the previous lessons worked at the line level: they selected lines, transformed lines, sorted lines. Column-level cutting, bringing two files side by side, character-level mapping, and splitting a large file into pieces requires a separate family of tools.

This lesson introduces that family and closes out the text processing topic with a criterion that decides which job belongs to which tool.

Column Cutting

cut has been used in this course since the first lesson. It has two modes: field mode (-d and -f) and character mode (-c).

head -2 access.log | cut -d' ' -f1,7,9

10.0.0.12 /index.html 200
10.0.0.31 /static/style.css 200

head -2 access.log | cut -c1-11

10.0.0.12 -
10.0.0.31 -

Character mode is used on fixed-width output — aligned tables, fixed-field records. It is useless on data whose field count varies.

cut is fast and short to write, but it has three firm limits.

The delimiter is a single character.

printf 'a::b::c\n' | cut -d'::' -f2

cut: bad delimiter

The field-based tool does the same job with a regular-expression delimiter:

printf 'a::b::c\n' | awk -F'::' '{ print $2 }'

b

Consecutive delimiters are not merged. Every delimiter starts a new field:

printf 'a  b  c\n' | cut -d' ' -f2
printf 'a  b  c\n' | awk '{ print $2 }'

b

cut found the second field empty, because there is an empty field between the two spaces. The field-based tool’s default delimiter is a run of whitespace, so it returned b. In aligned or hand-written text, this difference is decisive.

Field order cannot be changed. Whatever order the requested field numbers are written in, the output follows the order in the file:

head -1 access.log | cut -d' ' -f9,1

10.0.0.12 200

Reordering requires the field-based tool:

head -1 access.log | awk '{ print $9, $1 }'

200 10.0.0.12

These three limits leave cut a clear domain: data separated by a single character, with no consecutive delimiters, whose field order will not change. The log file fit this definition, which is why it was used in the course’s earliest topics.

Pasting Side by Side

paste merges the corresponding lines of files. It is the reverse of the cut operation.

cut -d' ' -f7 access.log | head -3 > path.txt
cut -d' ' -f9 access.log | head -3 > code.txt
paste path.txt code.txt

/index.html	200
/static/style.css	200
/product/12	200

The default separator is a tab; -d changes it:

paste -d: path.txt code.txt

/index.html:200
/static/style.css:200
/product/12:200

The -s option merges each file’s lines into a single line; it is the short way to turn a list into comma-separated text:

paste -sd, code.txt

200,200,200

paste assumes lines match up positionally. If a line is missing in one file, the whole alignment shifts and the result is silently wrong. If matching by key is needed, join from the previous lesson is used; paste is suited only to data whose order can be trusted.

Character-Level Transformation

tr maps input characters one by one. It recognizes no pattern, no field, no line — it works only on character sets, and it only reads from standard input.

printf 'hello\n' | tr 'a-z' 'A-Z'

HELLO

Three options are the most frequently used:

printf 'aaa  bbb\n' | tr -s ' '     # collapse consecutive repeats to one
printf 'a1b2c3\n'   | tr -d '0-9'   # delete characters in the set
printf 'a1b2c3\n'   | tr -cd '0-9'  # delete characters outside the set

aaa bbb
abc
123

Collapsing consecutive spaces to one with tr -s ' ' is a common preparation step for making a file usable with cut.

Its most frequent use is turning a delimiter into a line break to spill every field onto its own line:

tr ' ' '\n' < access.log | grep -c '^"GET$'

27

tr is far cheaper than pattern-based tools because it parses nothing. If a character mapping is enough, there is no need to call the stream editor.

Splitting a File into Pieces

split divides a large file into pieces of fixed size or a fixed number of lines.

mkdir -p parts
split -l 10 access.log parts/log-
ls parts

log-aa
log-ab
log-ac

wc -l parts/*

      10 parts/log-aa
      10 parts/log-ab
      10 parts/log-ac
      30 total

-l N splits by line count, -b N by byte count. The suffix length is set with -a; the default of two letters is enough for up to 676 pieces.

There are two uses. First, processing piece by piece a file that does not fit within a memory limit. Second, distributing the pieces to separate processes for parallel processing — if the pieces can be processed independently, the total time is divided by the piece count.

It should be kept in mind that splitting by byte boundary can cut a line in half; for line-based data, -l is used instead of -b.

Choosing a Tool

This topic’s six lessons built a family of tools looking at the same data from different angles. The criterion for choosing among them is the shape of the question:

Question	Tool
Which lines fit a pattern	filter (grep)
Change the text inside a line	stream editor (sed)
Compute over fields, accumulate per key	field-based tool (awk)
Take a column from fixed-delimiter data	cut
Merge columns	paste, join
Map, delete, compress characters	tr
Sort, deduplicate, do set operations	sort, uniq, comm

Two general rules stand above this table.

Choose the narrowest tool. The field-based tool can do nearly every job in the table; writing everything with it is still wrong. A narrow tool is shorter to write, runs faster, and tells the reader its intent. The form awk '/404/ { print }' does the same job as grep 404, but it makes the reader wonder “is there a computation here.”

If a problem takes more than three tools to solve, switch tools. Five-stage pipelines can be written, and were written in this course; but if every line in a script looks like that, the job should be considered to have crossed the shell’s boundary. The shell is unrivaled at connecting processes; it is not designed for complex data structures and multi-step business logic.

Applying to the Script

At the end of the text processing topic, report.sh produces all three of its sections with their proper tools:

summary() {                     # summary LOG
  awk '
    { requests++; bytes += $10; class[substr($9, 1, 1)]++ }
    END {
      printf "requests: %d\n", requests
      printf "bytes   : %d\n", bytes
      printf "average : %.1f bytes\n", (requests ? bytes / requests : 0)
      for (s = 2; s <= 5; s++)
        printf "  %sxx %4d  %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0)
    }
  ' "$1"
}

top_paths() {                   # top_paths LOG COUNT
  cut -d' ' -f7 "$1" \
    | sed -E 's#/[0-9]+#/:id#g' \
    | sort | uniq -c | sort -k1,1nr -k2,2 | head -"$2"
}

error_paths() {                 # error_paths LOG
  awk '$9 ~ /^[45]/ { n[$7]++ } END { for (y in n) printf "%3d %s\n", n[y], y }' "$1" \
    | sort -k1,1nr -k2,2
}

Every function answers a single question, follows a single data path, and its output is deterministic. The script’s skeleton, however, is still incomplete: it produces a half report on error, leaves temporary files behind, and expects to be run by hand.

Summary

• cut wants a single-character delimiter, does not merge consecutive delimiters, and cannot reorder fields; outside these three limits it is the cheapest column tool.
• paste brings files together positionally; if key matching is needed, join is used.
• tr works on character sets; it recognizes no pattern and reads only from standard input.
• split divides a file by line or byte boundary; -l is used for line-based data.
• The narrowest tool is preferred when choosing; if a problem needs more than three tools, the shell’s boundary should be considered crossed.

Next Step

The script produces correct output, but only when everything goes right. When a command fails, the report is left half-finished and the exit code does not report it; an undefined variable silently expands to an empty string. The next topic closes these gaps and starts with strict-mode options: which errors should stop the script, to what extent each option provides that, and the traps the options carry of their own.