Scheduled Tasks

The script is correct, robust, and checked — but it is still run by hand. A report’s value lies in its being produced regularly: a tool that analyzes the previous day’s log every morning and drops the result somewhere is a different thing from the same tool run by hand.

This lesson takes up preparing a script for scheduled execution. The subject is not just the scheduling definition; the real issue is that the script will be run by a non-human caller, in a narrow environment, without anyone watching.

The Cron Entry

On Unix systems, scheduled tasks are defined to a scheduler service. The definition table is kept per user, and every line consists of five time fields and a command:

minute  hour  day of month  month  day of week  command

Field	Range
minute	0–59
hour	0–23
day of month	1–31
month	1–12
day of week	0–7 (0 and 7 are Sunday)

Every field takes one of four forms: a fixed number, * (every value), a-b (a range), a,b,c (a list). Appending /n to a range selects every nth step.

30 6 * * *          every day at 06:30
0 */4 * * *         every four hours, on the hour
15 2 * * 1          every Monday at 02:15
0 0 1 * *           midnight on the first of every month
*/10 * * * *        every ten minutes

Two fields together narrow the constraint, but the day-of-month and day-of-week pair is the exception: if both take a value other than *, the conditions combine with “or,” not “and.” The entry 0 0 13 * 5 runs on the 13th of the month or on Friday. This is the most commonly misread point in these entries.

The definition table is edited with crontab -e, listed with crontab -l. crontab -r deletes the table without asking for confirmation; since it is a single letter away from -e, the habit of listing should be built around -l.

Some schedulers recognize shorthands such as @daily, @hourly, @reboot. These are not defined in POSIX; portable entries use the five-field form.

Another class of scheduler service is the service manager’s timer units. They offer the ability to define dependencies, make up for missed runs, and merge run logs with the service infrastructure. The service manager model is the subject of the System Administration course.

The Scheduler’s Environment

The most common reason scheduled tasks fail is not the script, it is the environment. The scheduler runs the command without a login shell: shell startup files are not read, there are no shell functions or aliases, and most environment variables are undefined.

The most visible difference is the search path. The same script, run in a narrow environment:

env -i PATH=/usr/bin:/bin bash env.sh

PATH=/usr/bin:/bin
is shellcheck found: no

A tool present in an interactive session may not be found in a scheduled run. The result shows up as “the script works by hand but not under the scheduler,” and since the scheduler is silent, noticing it is delayed.

Three precautions are taken.

State the search path in the script. Writing export PATH=... at the top of the script gives independence from the calling environment. If the paths of the tools used are known, this is the most robust solution.

Write file paths as absolute. The scheduler’s starting directory is not guaranteed; relative paths resolve in a different directory. Every path the script reads or writes must be absolute.

Define the environment variables needed inside the script. Locale, language, and time zone variables are not inherited from the interactive session. This is also why the line export LC_ALL=C was placed at the top of the script in this course: sort order would otherwise stay ambiguous in the scheduler’s environment.

There is one more detail specific to the definition table: the % character in the command field means end of line, and cannot be written without escaping. Commands containing %, such as date formatting, are not written directly into the table — they go into a wrapper script.

Where Output Goes

The scheduler tries to collect the standard output and standard error the task produces and deliver them to the user. If delivery is not configured, the output is lost or piles up in a local mail queue.

This behavior produces two consequences. First, a task that produces output on every run generates constant notifications; the notifications become unread after a while. Second, error messages get lost the same way.

The correct arrangement is to manage output explicitly:

30 6 * * * /opt/report/report-job.sh

and doing the redirection inside the script. Done inside the script, the paths stay readable, the % problem does not arise, and the redirection logic enters version control.

The stream separation set up in this topic’s first lesson pays off here: the report itself is written to one file, diagnostic messages to a separate one. If the two are mixed, the next step reading the report mistakes the warning text for data.

The Wrapper Script

The scheduled task does not call the working script directly; it wraps it in a wrapper. The wrapper sets up the environment, acquires the lock, redirects the output, and calls the working script.

#!/usr/bin/env bash
set -Eeuo pipefail
export PATH=/usr/local/bin:/usr/bin:/bin
export LC_ALL=C

ROOT="/opt/report"
LOCK="$ROOT/lock"
LOG_DIR="$ROOT/log"

mkdir -p "$LOG_DIR"

if ! mkdir "$LOCK" 2>/dev/null; then
  echo "previous run still in progress" >&2
  exit 75
fi
trap 'rmdir "$LOCK"' EXIT

"$ROOT/report.sh" -n 3 "$ROOT/access.log" \
  >> "$LOG_DIR/report.out" 2>> "$LOG_DIR/report.err"

Five decisions stand together here.

Lock. The mkdir pattern set up in the Traps and Cleanup lesson finds its real purpose here: a task that runs every ten minutes and sometimes takes longer than ten minutes collides with itself without a lock. The code 75 reports “temporary error.”

Environment. PATH and LC_ALL are fixed inside the script; the environment the scheduler provides is not trusted.

Absolute paths. There is no relative path at all; whatever the starting directory is, the script finds the same files.

Separate output. The report is appended to report.out, diagnostic messages to report.err. The redirections use >>; each run does not overwrite the previous one.

Strict mode. The wrapper is itself a script and is subject to the same rules.

Log files growing forever is a separate problem, solved with log rotation tools; these tools are taken up in the System Administration course.

Testing the Task

A scheduled task must be tested under the scheduler’s conditions before it is defined. The testing order is as follows:

1. Run the wrapper by hand; verify the output files are created in the right place.
2. Repeat the same call in a narrow environment: env -i PATH=/usr/bin:/bin ./report-job.sh. This is a close approximation of the scheduler’s environment.
3. Test the lock: start a second instance while one is running and verify the code 75.
4. Test the error path: give a nonexistent log name and see a meaningful message land in report.err.
5. Write the entry into the table a few minutes ahead of the actual time, and observe the first run.

The exit code is the last link in this chain. What reports a task’s failure to a monitoring system is the exit code; this is why the codes a script produces must be chosen deliberately. The code scheme used in this course was as follows:

Code	Meaning
0	report produced
2	usage error (invalid option or value)
3	input error (log missing, unreadable)
75	temporary error, retry later (lock not acquired)
128 + n	terminated by signal

The Finished Script

The final form of the script developed throughout the course:

#!/usr/bin/env bash
# report.sh — produces a summary report from the access log.
# Usage: report.sh [-n COUNT] [LOG]
set -Eeuo pipefail
export LC_ALL=C

SCRIPT_NAME="report.sh"
LINE_PATTERN='^[0-9.]+ - - \[[^]]+\] "[A-Z]+ [^ ]+ HTTP/[0-9.]+" [0-9]{3} [0-9]+$'
temp=""

error()  { printf '%s: error: %s\n'  "$SCRIPT_NAME" "$*" >&2; }
warn() { printf '%s: warning: %s\n' "$SCRIPT_NAME" "$*" >&2; }
die() { error "$1"; exit "${2:-1}"; }
section() { printf '\n== %s ==\n' "$*"; }

cleanup() {
  local code=$?
  [ -n "$temp" ] && rm -rf "$temp"
  return "$code"
}
trap cleanup EXIT
trap 'cleanup; trap - INT;  kill -INT  $$' INT
trap 'cleanup; trap - TERM; kill -TERM $$' TERM
trap 'error "$LINENO. line unexpected error"' ERR

summary() {
  awk '
    { requests++; bytes += $10; class[substr($9, 1, 1)]++ }
    END {
      printf "  requests: %d\n", requests
      printf "  bytes   : %d\n", bytes
      printf "  average : %.1f bytes\n", (requests ? bytes / requests : 0)
      for (s = 2; s <= 5; s++)
        printf "  %sxx %4d  %%%.1f\n", s, class[s], (requests ? class[s] * 100 / requests : 0)
    }
  ' "$1"
}

top_paths() {
  cut -d' ' -f7 "$1" \
    | sed -E 's#/[0-9]+#/:id#g' \
    | sort | uniq -c | sort -k1,1nr -k2,2 | head -"$2"
}

error_paths() {
  awk '$9 ~ /^[45]/ { n[$7]++ } END { for (y in n) printf "%3d %s\n", n[y], y }' "$1" \
    | sort -k1,1nr -k2,2
}

count=5
while getopts ':n:' option; do
  case "$option" in
    n)  count="$OPTARG" ;;
    :)  die "-$OPTARG expects a value" 2 ;;
    \?) die "unknown option -$OPTARG" 2 ;;
  esac
done
shift $((OPTIND - 1))

log="${1:-access.log}"
[[ "$count" =~ ^[1-9][0-9]*$ ]] || die "-n expects a positive integer: $count" 2
[ -f "$log" ] || die "not a regular file or does not exist: $log" 3
[ -r "$log" ] || die "no read permission: $log" 3
[ -s "$log" ] || warn "log is empty: $log"

temp=$(mktemp -d)
valid="$temp/valid.log"
grep -E "$LINE_PATTERN" "$log" | awk 'NF == 10' > "$valid" || true
invalid=$(( $(grep -c '' "$log") - $(grep -c '' "$valid") ))
[ "$invalid" -eq 0 ] || warn "$invalid lines not in the expected format, excluded from analysis"

section "Log: $log"
summary "$valid"

section "Top $count requested paths"
top_paths "$valid" "$count"

section "Paths returning errors"
error_paths "$valid"

The script’s last addition is the step that separates valid lines before analysis. The line pattern from the Regular Expressions lesson and the field-count check from the Field-Based Processing lesson are applied together: neither was sufficient alone. Discarded lines are counted and reported to standard error, while counts are computed only over the valid lines — so a corrupted input does not silently distort the report.

With a clean log:

./report.sh -n 3

== Log: access.log ==
  requests: 30
  bytes   : 117780
  average : 3926.0 bytes
  2xx   18  %60.0
  3xx    3  %10.0
  4xx    6  %20.0
  5xx    3  %10.0

== Top 3 requested paths ==
   6 /api/data
   6 /product/:id
   4 /index.html

== Paths returning errors ==
  4 /missing.html
  3 /api/data
  2 /secret

With a log that has two malformed lines added:

./report.sh -n 2 all.log

report.sh: warning: 2 lines not in the expected format, excluded from analysis

== Log: all.log ==
  requests: 30
  bytes   : 117780
  average : 3926.0 bytes
  2xx   18  %60.0
...

The warning went to standard error, the report to standard output, and the counts were unaffected by the malformed lines.

The script has a known behavior: when its output is piped into a command that closes early, like head, the ERR hook is triggered by SIGPIPE because pipefail is active. This is a deliberate trade-off of strict mode; shortening the output must be done with the -n option.

Summary

• A cron entry consists of five time fields and a command; when day of month and day of week are both given, the conditions combine with “or.”
• The scheduler runs the command without a login shell; the search path is narrow, startup files are not read, and most environment variables are undefined.
• The search path and locale are fixed inside the script, and all file paths are written as absolute.
• Output is managed explicitly: the report and diagnostic messages are appended to separate files.
• The wrapper script sets up the environment, acquires the lock, and redirects output; the lock prevents overlapping runs.
• Exit codes are the monitoring system’s only input; usage, input, and temporary-error classes are reported with separate codes.

Course Wrap-Up

This course started with a single question: exactly where does a command write the data it writes? The answer was three streams and file descriptors. Everything that follows from there is an expansion of the same idea.

Streams and Pipelines established connecting commands together: redirection being applied before the command, the pipeline’s processes running concurrently, variables lost in a subshell, and the exit code convention.

Script Structure turned one-line commands into a program. At the center of this section stood the thing that sets the shell apart from other languages: an expansion’s result is reparsed. Word splitting, pathname expansion, and quoting rules are this course’s academic core; most errors in shell scripts are born here.

Text Processing handed the work to its dedicated tools: the pattern language, line filtering, the stream editor, field-based processing, and the sort family. The selection criterion that closed the section — choose the narrowest tool, switch tools when a problem needs more than three of them — is this course’s practical inheritance.

Writing Robust Scripts prepared the script for unattended operation: strict mode, cleanup hooks, tracing tools, static analysis, and scheduled runs.

The same script was developed throughout the course. report.sh began as a fixed pipeline; it gained variables, acquired an argument interface, learned to validate its inputs, was split into functions, reduced its counting to a single pass, and learned to stop on errors and clean up after itself. Its final form is a tool that passes inspection and can be handed to a scheduler.

The shell’s limit also showed itself on this journey. The shell has no equal at connecting processes together and working with the file system. It was not designed for complex data structures, decimal arithmetic, structured data formats, or extensive error handling. A good shell script is a script that knows its own limit.

The next course — System Administration — takes up the system this script runs on. report.sh was handed to a scheduler; but how is that scheduler itself defined and inspected as a service? The script created a process; how are processes monitored, prioritized, and managed with signals? Log files are growing; how is disk and file system configuration changed? In this course, signals, exit codes, and processes were seen from inside a script. The next course looks at them from outside the system.