---
title: 'Field-Based Processing'
source: 'https://academia.sh/en/courses/shell-programming/field-based-processing'
course: 'Shell Programming'
language: en
updated: '2026-08-17T18:10:03+00:00'
license: 'CC BY-SA 4.0'
---

# Field-Based Processing

The record and field model, the pattern–action structure, built-in variables, accumulation with associative arrays, and producing a report in a single pass.

The stream editor sees a line as text; it can only imitate fields with a pattern.
Taking the ninth field from the log required writing a pattern that described the
entire line, and that pattern breaks the moment the format changes even slightly.

There is a separate class of tool for column-shaped data. The **field-based
processing tool** splits input into records, records into fields, and runs a program
made of pattern–action pairs. This lesson builds that model and collapses the counts
written with associative arrays in the previous topic into a single line.

## The Record and Field Model

Input is split, by default, into lines — **records**. Each record is separated into
**fields** by runs of whitespace. `$1` gives the first field, `$0` the entire record,
`$NF` the last field.

```sh
head -2 access.log | awk '{ print NR, NF, $1, $9, $NF }'
```

```
1 10 10.0.0.12 200 5120
2 10 10.0.0.31 200 2048
```

Built-in variables complete the model's surroundings:

| Variable | Content |
|---|---|
| `NR` | number of records read (accumulates across files) |
| `FNR` | record's position within that file |
| `NF` | number of fields in that record |
| `FS` | field separator (default: runs of whitespace) |
| `OFS` | output field separator (default: a single space) |
| `FILENAME` | name of the file being read |

That the default separator is "runs of whitespace" matters: consecutive spaces count
as a single separator, and leading whitespace on a line is ignored. This is the point
where it departs from the `cut` command — `cut` counts every separator individually and
produces empty fields in aligned text.

## Pattern and Action

A program is made of `pattern { action }` pairs. If the record fits the pattern, the
action runs.

```sh
awk '$9 >= 500 { print $1, $7, $9 }' access.log
```

```
192.168.1.5 /api/data 500
192.168.1.5 /api/data 500
10.0.2.7 /api/data 500
```

Both parts are optional, and a default fills in whichever is missing:

- If no pattern is written, the action applies to **every record**.
- If no action is written, the default action is `{ print }`.

```sh
awk '$9 == 404' access.log | grep -c ''
```

```
4
```

A pattern can also be a regular expression. `/pattern/` tests the whole record, `$9 ~
/pattern/` tests a specific field:

```sh
awk '$9 ~ /^[45]/ { n++ } END { print n }' access.log
```

The entire program is written **inside single quotes**. The `$` sign in the program is
a field reference; if double quotes are used, the shell mistakes it for variable
expansion and the program breaks. This is the place where the rule from the Variables
and Quoting Rules lesson is most often needed.

## `BEGIN` and `END`

The `BEGIN` block runs before the first record is read, the `END` block after the last.
They are used to write a header, set up configuration, and report accumulations.

```sh
awk '
  BEGIN { print "address          bytes" }
  { t += $10 }
  END { printf "total: %d bytes, %d requests, average %.1f\n", t, NR, t / NR }
' access.log
```

```
address          bytes
total: 117780 bytes, 30 requests, average 3926.0
```

Variables are used without being declared; on first use they hold zero in numeric
context, the empty string in string context. This is why the line `t += $10` needed no
separate initialization.

`printf` formatting uses the same specifiers as the shell's `printf` command, and it
can print a decimal number — the integer restriction of shell arithmetic does not apply
here.

## Associative Arrays

Arrays are only associative and need no declaration. A per-key count is the
one-line equivalent of the shell loop from the previous topic:

```sh
awk '{ count[$7]++ } END { for (y in count) printf "%3d %s\n", count[y], y }' access.log \
  | sort -rn | head -5
```

```
  6 /api/data
  4 /missing.html
  4 /index.html
  3 /static/style.css
  3 /product/45
```

A per-key sum follows the same pattern:

```sh
awk '{ bytes[$7] += $10 } END { for (y in bytes) printf "%8d %s\n", bytes[y], y }' access.log \
  | sort -rn | head -5
```

```
   36120 /panel
   25230 /product/12
   21990 /product/45
   20480 /index.html
    7236 /api/data
```

The order produced by a `for (key in array)` loop is undefined; sorted output needs an
external `sort`. This is a direct consequence of associative arrays being implemented
with a hash table, defined in the Data Structures course.

## Changing the Field Separator

The `-F` option sets the field separator. The value can be a single character or a
regular expression.

```sh
printf 'a:b:c\nd:e:f\n' | awk -F: '{ print $2 }'
```

```
b
e
```

The output separator is a separate variable:

```sh
head -2 access.log | awk 'BEGIN { OFS="|" } { print $1, $7, $9 }'
```

```
10.0.0.12|/index.html|200
10.0.0.31|/static/style.css|200
```

In the `print` command, fields are separated by **commas**; the comma is what `OFS`
replaces. If a space is written instead of a comma, the fields are concatenated with no
separator at all — this is string-concatenation syntax, and it is a silent source of
bugs.

Assigning to a field rebuilds the record:

```sh
head -1 access.log | awk '{ $1="MASKED"; print }' | cut -c1-46
```

```
MASKED - - [07/Feb/2024:09:12:44 +0000] "GET /
```

When the record is rebuilt, the fields are joined with `OFS`; the multiple spaces in
the original line are not preserved. This is the expected behavior in field masking and
redaction operations.

## Numeric and String Context

A field can be interpreted both as a number and as a string; which interpretation
applies depends on the operator and on where the operands came from. A field coming
from input that looks like a number is compared numerically; but when compared against
a string constant, it can take on string context.

```sh
awk 'BEGIN { x="10"; y="9"; print (x   > y  ) ? "string: 10>9" : "string: 10<9" }'
awk 'BEGIN { x="10"; y="9"; print (x+0 > y+0) ? "number: 10>9"  : "number: 10<9"  }'
```

```
string: 10<9
number: 10>9
```

The way to remove the ambiguity is to force the context explicitly: `+0` for a numeric
comparison, concatenation with `""` for a string comparison. In programs working with
externally sourced data, this habit prevents results that are hard to explain.

## Passing a Shell Variable

Shell variables are not embedded in the program text; they are passed with the `-v`
option.

```sh
threshold=400
awk -v threshold="$threshold" '$9 >= threshold { n++ } END { print n " requests " threshold " and above" }' access.log
```

```
9 requests 400 and above
```

Embedding the value in the program text — by closing and reopening quotes — is wrong
for two reasons: quote and backslash characters in the value break the program, and a
value coming from outside becomes program code and can run unwanted commands. `-v`
preserves this boundary.

## Catching Broken Records

Field count is the cheapest criterion for format validation:

```sh
awk 'NF != 10 { print FILENAME ":" FNR ": field count " NF }' all.log
```

```
all.log:31: field count 2
all.log:32: field count 9
```

The Regular Expressions lesson did the same job with a pattern describing the entire
line. A field-count check performs a narrower validation but is much shorter to write
and is resilient to format changes. The two can be used together: field count for fast
elimination, the pattern for detailed validation.

## A Multi-Pattern Program

A program can hold more than one pattern–action pair; every record is tested against
all of them in sequence.

```sh
awk '
  $9 ~ /^2/     { success++ }
  $9 ~ /^[45]/  { error++; error_path[$7]++ }
  END {
    printf "success=%d error=%d\n", success, error
    for (y in error_path) printf "  %-14s %d\n", y, error_path[y]
  }
' access.log | sort
```

```
  /api/data      3
  /missing.html  4
  /secret        2
success=18 error=9
```

If a record fits more than one pattern, every corresponding action runs; the "first
matching branch" rule from a `case` structure does not apply here.

## Applying to the Script

Until now, `report.sh` reread the log from scratch for every count: the line count,
the byte total, and six separate passes for four status classes. All of it collapses
into one pass:

```sh
log_file=access.log

awk -v count="$count" '
  { requests++; bytes += $10; path[$7]++; class[substr($9, 1, 1)]++ }
  END {
    printf "requests: %d\n", requests
    printf "bytes   : %d\n", bytes
    printf "average : %.1f bytes\n", (requests ? bytes / requests : 0)
    print ""
    print "status code classes:"
    for (s = 2; s <= 5; s++)
      printf "  %sxx %4d  %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0)
  }
' "$log_file"
```

```
requests: 30
bytes   : 117780
average : 3926.0 bytes

status code classes:
  2xx   18  %60.0
  3xx    3  %10.0
  4xx    6  %20.0
  5xx    3  %10.0
```

There are three gains. The log is read once. Percentages are decimal and need no
scaling trick. The divide-by-zero guard is written the same way as the shell's
ternary operator.

There is also a loss: part of the report logic is no longer in the shell but in an
embedded program. The script's readability depends on this program staying short and
single-purpose. Two-hundred-line embedded programs are a sign the shell script's
boundary has been crossed.

## Summary

- Input is split into records, records into fields; `$0` is the entire record, `NF`
  the field count, `NR` the record position. The default separator is runs of
  whitespace.
- A program is made of `pattern { action }` pairs; with no pattern, every record is
  matched, with no action, `print` applies, and a record is tested against every pair.
- `BEGIN` and `END` blocks run outside the record stream; variables need no
  declaration and start at zero.
- A per-key count and sum with associative arrays are written in a single line;
  traversal order is undefined.
- Shell values are not embedded in the program text, they are passed with `-v`.
- Numeric and string context depends on the operands; it is forced explicitly with
  `+0` and `""`.

## Next Step

The counts have been produced, but sorting them has always been left to an outside
command. The next lesson takes up that command and two tools alongside it: sorting by
key, deduplicating adjacent repeats, and comparing two sorted streams with set
operations — and also why sorting depends on the locale, and why that is a source of
bugs in scripts.