Arrays and Associative Arrays

Every count in the report reads the log from start to end once more: five counts mean five scans. This is not noticeable on small files, and becomes apparent as the log grows. Bringing it down to a single scan requires keeping intermediate results in memory.

In the Data Structures course, two structures were defined for this job: the array, accessed by index, and the hash table, accessed by key. The shell offers a form of both. This lesson takes them up, along with their syntax traps and — unavoidable for associative arrays — version dependency.

Indexed Arrays

An array is defined with a word list in parentheses. Indices start at zero.

classes=(2 3 4 5)
echo "first=${classes[0]} last=${classes[3]} count=${#classes[@]}"

first=2 last=5 count=4

Braces are mandatory: writing $classes[0] appends the text [0] after the value of the classes variable. Without braces, $classes alone gives the first element — not the whole array.

Bulk Expansion

There are two forms for expanding the array as a whole, and the difference is the same as the "$@"–"$*" distinction in argument lists.

show_args() { printf 'count=%d\n' "$#"; printf '  <%s>\n' "$@"; }
paths=("/api/data" "/product/12" "three word path")
show_args "${paths[@]}"
show_args "${paths[*]}"
show_args ${paths[@]}

count=3
  </api/data>
  </product/12>
  <three word path>
count=1
  </api/data /product/12 three word path>
count=5
  </api/data>
  </product/12>
  <three>
  <word>
  <path>

The rule has not changed: "${array[@]}" is written. The unquoted form splits the elements, the [*] form collapses them all into a single piece of text.

Arrays cannot be passed directly to functions; in the shell, function arguments are only a word list. Passing is done through expansion:

show() { printf 'received: %s\n' "$@"; }
show "${paths[@]}"

received: /api/data
received: /product/12
received: three word path

This means the array passes by being copied; changes inside the function do not propagate back to the caller.

Appending, Indices, and Sparseness

The += operator appends to the end of the array. ${!array[@]} gives the indices in use.

paths+=("/panel")
echo "count=${#paths[@]} indices=${!paths[@]}"

count=4 indices=0 1 2 3

Shell arrays can be sparse: when an element is deleted, the remaining ones do not shift, and a gap forms in the indices.

unset 'paths[1]'
echo "count=${#paths[@]} indices=${!paths[@]}"
echo "last element=${paths[-1]}"

count=3 indices=0 2 3
last element=/panel

This shows why iterating over an array with for ((i = 0; i < ${#array[@]}; i++)) is wrong: the element count is 3, but the valid indices are 0, 2, and 3. The correct forms are for e in "${array[@]}" or for i in "${!array[@]}".

The quote in unset 'paths[1]' is required; if left unquoted, [1] can be interpreted as a pathname pattern.

Slicing is done with ${array[@]:start:count}:

full=(a b c d e)
echo "1..3: ${full[@]:1:3}"

1..3: b c d

Building an Array from Command Output

mapfile (a synonym is readarray) reads a stream line by line into an array. The -t option discards line endings.

mapfile -t codes < <(cut -d' ' -f9 access.log | sort -u)
echo "distinct code count=${#codes[@]} -> ${codes[*]}"

distinct code count=8 -> 200 201 301 302 304 403 404 500

Writing array=( $(command) ) is common but wrong: the output undergoes word splitting and pathname expansion, and lines containing spaces get split apart. mapfile does not apply either operation. It is available from bash 4.0 onward.

Associative Arrays

The declare -A declaration creates an array whose keys are strings. This is the shell’s counterpart to a hash table, and it makes single-pass counting possible.

declare -A count
while read -r code; do
  count["$code"]=$(( ${count["$code"]:-0} + 1 ))
done < <(cut -d' ' -f9 access.log)

for key in "${!count[@]}"; do printf '%s -> %s\n' "$key" "${count[$key]}"; done | sort

200 -> 17
201 -> 1
301 -> 1
302 -> 1
304 -> 1
403 -> 2
404 -> 4
500 -> 3

The expression ${count["$code"]:-0} makes a key seen for the first time start from zero; it shows that default-value expansions also work on array elements.

The order of keys is undefined — a natural consequence of a hash table. If sorted output is wanted, sort is required; this is why it is there in the example above.

A key’s presence is tested with the -v test:

[[ -v count[404] ]] && echo "404 key exists"
[[ -v count[999] ]] || echo "999 key does not exist"

404 key exists
999 key does not exist

Taking a total per key is the same pattern. The loop below computes the total number of bytes sent to each path in a single scan:

declare -A bytes
while read -r path b; do
  bytes["$path"]=$(( ${bytes["$path"]:-0} + b ))
done < <(cut -d' ' -f7,10 access.log)

for path in "${!bytes[@]}"; do printf '%8d %s\n' "${bytes[$path]}" "$path"; done | sort -rn | head -5

   36120 /panel
   25230 /product/12
   21990 /product/45
   20480 /index.html
    7236 /api/data

Version Dependency

Associative arrays are available from bash’s 4.0 version onward. On an older bash, declare -A is not recognized — and the result is dangerous because it is silent.

#!/usr/bin/env bash
declare -A count
count[404]=4
echo "404 -> ${count[404]}"

Bash 4 and later:

404 -> 4

Bash 3.2:

assoc.sh: line 2: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
404 -> 4

The script printed an error message but kept running and produced a result that looked correct. The reason is this: once the declaration failed, count became an ordinary indexed array, and the key 404 was interpreted as a numeric index. As long as the keys are integers, the result comes out correct by coincidence.

Once the keys are strings, the concealment ends:

text-key.sh: line 3: /api/data: syntax error: operand expected (error token is "/api/data")

An indexed array treats the text inside the brackets as an arithmetic expression, and cannot evaluate a path name.

The solution is to test the requirement at the start of the script. The BASH_VERSINFO array holds the version components as numbers:

#!/usr/bin/env bash
if ((BASH_VERSINFO[0] < 4)); then
  echo "this script requires bash 4 or later (found: $BASH_VERSION)" >&2
  exit 4
fi
declare -A count

The script stops immediately when called with an interpreter older than version four. The call below explicitly selects a system shell that has stayed at version 3.2:

/bin/bash version-check.sh
echo "exit code: $?"

this script requires bash 4 or later (found: 3.2.57(1)-release)
exit code: 4

This test reports the error at its source, not where it surfaces. It should be present in every script that requires a specific version.

The POSIX Shell Has No Arrays

The POSIX shell language defines neither indexed nor associative arrays. Its only list structure is positional parameters, and they can be filled with set --:

set -- 2 3 4 5
echo "count=$#  first=$1  last=$4"
for s in "$@"; do printf '%s ' "$s"; done; echo

count=4  first=2  last=5
2 3 4 5 

There is only one portable list, and it overwrites the script’s own arguments; this is a serious limitation.

The POSIX solution for per-key counting is not to build a data structure in the shell, but to hand the job off to a tool designed for it. One line of a field-based processing tool takes the place of this entire lesson’s loop, and works the same everywhere:

awk '{ count[$9]++ } END { for (k in count) print k, count[k] }' access.log | sort

200 17
201 1
301 1
302 1
304 1
403 2
404 4
500 3

This tool will be taken up in detail in the Text Processing topic. The lesson here is about choice: building a data structure in the shell makes sense when the data needs to stay in the shell; if only counting is needed, an external tool is both faster and more portable.

Summary

• Array elements are accessed with ${array[i]}; unbraced $array gives only the first element.
• "${array[@]}" expands elements as separate words; [*] collapses them into one piece of text, the unquoted form splits them.
• Shell arrays can be sparse; because the element count and the indices may not line up, "${!array[@]}" is used instead of a counted loop.
• mapfile -t reads a stream into an array without applying splitting or expansion.
• Associative arrays are available from bash 4.0 onward; when missing, the declaration fails but the script keeps running. A version test with BASH_VERSINFO is mandatory.
• The POSIX shell language has no arrays; a field-based tool is used for per-key counting.

Next Step

In this lesson’s counts, the notations $(( ... )) and $( ... ) were used without justification. The next lesson defines these two notations: why arithmetic in the shell is integer-only, how decimal calculation is done, which characters are lost when a command’s output is turned into a value, and the cost of that conversion.