Field-Based Processing

The stream editor sees a line as text; it can only imitate fields with a pattern. Taking the ninth field from the log required writing a pattern that described the entire line, and that pattern breaks the moment the format changes even slightly.

There is a separate class of tool for column-shaped data. The field-based processing tool splits input into records, records into fields, and runs a program made of pattern–action pairs. This lesson builds that model and collapses the counts written with associative arrays in the previous topic into a single line.

The Record and Field Model

Input is split, by default, into lines — records. Each record is separated into fields by runs of whitespace. $1 gives the first field, $0 the entire record, $NF the last field.

head -2 access.log | awk '{ print NR, NF, $1, $9, $NF }'

1 10 10.0.0.12 200 5120
2 10 10.0.0.31 200 2048

Built-in variables complete the model’s surroundings:

Variable	Content
NR	number of records read (accumulates across files)
FNR	record’s position within that file
NF	number of fields in that record
FS	field separator (default: runs of whitespace)
OFS	output field separator (default: a single space)
FILENAME	name of the file being read

That the default separator is “runs of whitespace” matters: consecutive spaces count as a single separator, and leading whitespace on a line is ignored. This is the point where it departs from the cut command — cut counts every separator individually and produces empty fields in aligned text.

Pattern and Action

A program is made of pattern { action } pairs. If the record fits the pattern, the action runs.

awk '$9 >= 500 { print $1, $7, $9 }' access.log

192.168.1.5 /api/data 500
192.168.1.5 /api/data 500
10.0.2.7 /api/data 500

Both parts are optional, and a default fills in whichever is missing:

• If no pattern is written, the action applies to every record.
• If no action is written, the default action is { print }.

awk '$9 == 404' access.log | grep -c ''

4

A pattern can also be a regular expression. /pattern/ tests the whole record, $9 ~ /pattern/ tests a specific field:

awk '$9 ~ /^[45]/ { n++ } END { print n }' access.log

The entire program is written inside single quotes. The $ sign in the program is a field reference; if double quotes are used, the shell mistakes it for variable expansion and the program breaks. This is the place where the rule from the Variables and Quoting Rules lesson is most often needed.

BEGIN and END

The BEGIN block runs before the first record is read, the END block after the last. They are used to write a header, set up configuration, and report accumulations.

awk '
  BEGIN { print "address          bytes" }
  { t += $10 }
  END { printf "total: %d bytes, %d requests, average %.1f\n", t, NR, t / NR }
' access.log

address          bytes
total: 117780 bytes, 30 requests, average 3926.0

Variables are used without being declared; on first use they hold zero in numeric context, the empty string in string context. This is why the line t += $10 needed no separate initialization.

printf formatting uses the same specifiers as the shell’s printf command, and it can print a decimal number — the integer restriction of shell arithmetic does not apply here.

Associative Arrays

Arrays are only associative and need no declaration. A per-key count is the one-line equivalent of the shell loop from the previous topic:

awk '{ count[$7]++ } END { for (y in count) printf "%3d %s\n", count[y], y }' access.log \
  | sort -rn | head -5

  6 /api/data
  4 /missing.html
  4 /index.html
  3 /static/style.css
  3 /product/45

A per-key sum follows the same pattern:

awk '{ bytes[$7] += $10 } END { for (y in bytes) printf "%8d %s\n", bytes[y], y }' access.log \
  | sort -rn | head -5

   36120 /panel
   25230 /product/12
   21990 /product/45
   20480 /index.html
    7236 /api/data

The order produced by a for (key in array) loop is undefined; sorted output needs an external sort. This is a direct consequence of associative arrays being implemented with a hash table, defined in the Data Structures course.

Changing the Field Separator

The -F option sets the field separator. The value can be a single character or a regular expression.

printf 'a:b:c\nd:e:f\n' | awk -F: '{ print $2 }'

b
e

The output separator is a separate variable:

head -2 access.log | awk 'BEGIN { OFS="|" } { print $1, $7, $9 }'

10.0.0.12|/index.html|200
10.0.0.31|/static/style.css|200

In the print command, fields are separated by commas; the comma is what OFS replaces. If a space is written instead of a comma, the fields are concatenated with no separator at all — this is string-concatenation syntax, and it is a silent source of bugs.

Assigning to a field rebuilds the record:

head -1 access.log | awk '{ $1="MASKED"; print }' | cut -c1-46

MASKED - - [07/Feb/2024:09:12:44 +0000] "GET /

When the record is rebuilt, the fields are joined with OFS; the multiple spaces in the original line are not preserved. This is the expected behavior in field masking and redaction operations.

Numeric and String Context

A field can be interpreted both as a number and as a string; which interpretation applies depends on the operator and on where the operands came from. A field coming from input that looks like a number is compared numerically; but when compared against a string constant, it can take on string context.

awk 'BEGIN { x="10"; y="9"; print (x   > y  ) ? "string: 10>9" : "string: 10<9" }'
awk 'BEGIN { x="10"; y="9"; print (x+0 > y+0) ? "number: 10>9"  : "number: 10<9"  }'

string: 10<9
number: 10>9

The way to remove the ambiguity is to force the context explicitly: +0 for a numeric comparison, concatenation with "" for a string comparison. In programs working with externally sourced data, this habit prevents results that are hard to explain.

Passing a Shell Variable

Shell variables are not embedded in the program text; they are passed with the -v option.

threshold=400
awk -v threshold="$threshold" '$9 >= threshold { n++ } END { print n " requests " threshold " and above" }' access.log

9 requests 400 and above

Embedding the value in the program text — by closing and reopening quotes — is wrong for two reasons: quote and backslash characters in the value break the program, and a value coming from outside becomes program code and can run unwanted commands. -v preserves this boundary.

Catching Broken Records

Field count is the cheapest criterion for format validation:

awk 'NF != 10 { print FILENAME ":" FNR ": field count " NF }' all.log

all.log:31: field count 2
all.log:32: field count 9

The Regular Expressions lesson did the same job with a pattern describing the entire line. A field-count check performs a narrower validation but is much shorter to write and is resilient to format changes. The two can be used together: field count for fast elimination, the pattern for detailed validation.

A Multi-Pattern Program

A program can hold more than one pattern–action pair; every record is tested against all of them in sequence.

awk '
  $9 ~ /^2/     { success++ }
  $9 ~ /^[45]/  { error++; error_path[$7]++ }
  END {
    printf "success=%d error=%d\n", success, error
    for (y in error_path) printf "  %-14s %d\n", y, error_path[y]
  }
' access.log | sort

  /api/data      3
  /missing.html  4
  /secret        2
success=18 error=9

If a record fits more than one pattern, every corresponding action runs; the “first matching branch” rule from a case structure does not apply here.

Applying to the Script

Until now, report.sh reread the log from scratch for every count: the line count, the byte total, and six separate passes for four status classes. All of it collapses into one pass:

log_file=access.log

awk -v count="$count" '
  { requests++; bytes += $10; path[$7]++; class[substr($9, 1, 1)]++ }
  END {
    printf "requests: %d\n", requests
    printf "bytes   : %d\n", bytes
    printf "average : %.1f bytes\n", (requests ? bytes / requests : 0)
    print ""
    print "status code classes:"
    for (s = 2; s <= 5; s++)
      printf "  %sxx %4d  %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0)
  }
' "$log_file"

requests: 30
bytes   : 117780
average : 3926.0 bytes

status code classes:
  2xx   18  %60.0
  3xx    3  %10.0
  4xx    6  %20.0
  5xx    3  %10.0

There are three gains. The log is read once. Percentages are decimal and need no scaling trick. The divide-by-zero guard is written the same way as the shell’s ternary operator.

There is also a loss: part of the report logic is no longer in the shell but in an embedded program. The script’s readability depends on this program staying short and single-purpose. Two-hundred-line embedded programs are a sign the shell script’s boundary has been crossed.

Summary

• Input is split into records, records into fields; $0 is the entire record, NF the field count, NR the record position. The default separator is runs of whitespace.
• A program is made of pattern { action } pairs; with no pattern, every record is matched, with no action, print applies, and a record is tested against every pair.
• BEGIN and END blocks run outside the record stream; variables need no declaration and start at zero.
• A per-key count and sum with associative arrays are written in a single line; traversal order is undefined.
• Shell values are not embedded in the program text, they are passed with -v.
• Numeric and string context depends on the operands; it is forced explicitly with +0 and "".

Next Step

The counts have been produced, but sorting them has always been left to an outside command. The next lesson takes up that command and two tools alongside it: sorting by key, deduplicating adjacent repeats, and comparing two sorted streams with set operations — and also why sorting depends on the locale, and why that is a source of bugs in scripts.