What Is a Shell

The previous lesson defined the shell as a replaceable program that interprets commands. That definition does not yet explain anything: what exactly does the shell do with a typed line?

The answer also explains the command line’s most common mistakes. Why a filename with a space splits into two separate arguments, who actually interprets the asterisk, how it is known whether a command succeeded — all of these are the result of a single parsing process.

The Prompt

The shell signals that it is waiting for a command with a prompt. The prompt’s content is configurable; the username, machine name, and current directory are commonly shown. In this course’s examples, the prompt is shown as a single character:

$ echo Hello
Hello

Lines starting with $ are the typed command; the lines below them are the command’s output. Documentation carries an established convention: the ordinary user prompt is shown as $, the superuser prompt as #. If a command starting with # is seen in a document, it is understood that the command requires administrative privilege.

The Line Is Split Into Words

The shell first splits the line into words. The split is done on whitespace characters, and consecutive spaces count as a single separator. The first word is the name of the program to run; the remaining words are passed to it as arguments.

$ echo Hello    world
Hello world

The output has one space instead of four. This is not because the echo command compresses spaces: echo received two arguments — Hello and world — and printed them joined by a single space. The spaces between them were gone before echo even ran.

The argument count can be counted directly:

$ file="measurement 01.csv"
$ set -- $file; echo $#
2
$ set -- "$file"; echo $#
1

$# gives the argument count. When the variable’s value is written without quotes, the shell split it into two words; written inside quotes, it left it as a single word. This is why filenames with spaces must be written in quotes; otherwise the command looks for two files that do not exist.

The same difference can be shown with a print format that makes word boundaries visible:

$ printf '[%s]\n' $file
[measurement]
[01.csv]
$ printf '[%s]\n' "$file"
[measurement 01.csv]

Expansions Happen Before the Command

Splitting into words is not the only operation. Before the program runs, the shell applies several expansions to the words. Three are used in this course: variable expansion ($name), filename expansion (*, ?), and home directory expansion (~).

The decisive point is this: the shell performs the expansion, not the program that runs. The program sees only the result of the expansion.

$ echo /usr/bin/zc*
/usr/bin/zcat /usr/bin/zcmp
$ ls /usr/bin/zc*
/usr/bin/zcat  /usr/bin/zcmp

Both commands produced the same two filenames. The * character never reached the ls command; the shell replaced the pattern with matching filenames, and ls received two ready-made arguments. The only difference between echo’s output and ls’s is that ls performs column alignment.

This has two consequences. First, if the pattern matches no file, the shell leaves it unchanged and the program receives a meaningless name containing *. Second, and more important, what a command will do can be seen in advance: putting echo in front of a command shows the argument list the shell will produce, without running it. This will become a safety habit in the deletion lesson.

Quoting also stops expansion:

$ echo "/usr/bin/zc*"
/usr/bin/zc*

Double quotes block filename expansion and word splitting, but not variable expansion. Single quotes block all of them: text inside single quotes is preserved literally.

Options and Arguments

A command line’s arguments are of two kinds. An option changes the command’s behavior and, by convention, starts with a dash; the rest are the objects the command will operate on.

Options have three written forms, and all three give the same result:

$ ls -l -a /home
total 16
drwxr-xr-x 1 root    root    4096 Jul 26 18:53 .
drwxr-xr-x 1 root    root    4096 Jul 26 18:46 ..
drwxr-x--- 3 student student 4096 Jul 26 18:49 student
$ ls -la /home
total 16
drwxr-xr-x 1 root    root    4096 Jul 26 18:53 .
drwxr-xr-x 1 root    root    4096 Jul 26 18:46 ..
drwxr-x--- 3 student student 4096 Jul 26 18:49 student

Single-letter options are written with a single dash and can be combined: -l -a is the same as -la. Long options are written with two dashes (--all) and cannot be combined; each is a separate word. The long-option form is not defined in POSIX; it is a GNU tool family addition, so the single-letter form is preferred wherever portability matters.

If a filename starts with a dash, trouble follows: the command mistakes it for an option. This is what the -- separator is for. Everything after the word -- is not interpreted as an option. Why this separator is a security matter will be seen in the deletion lesson.

Builtins and External Programs

Not every command is a separate program. Some commands are implemented inside the shell itself; these are called builtins. Which one a name is can be asked directly:

$ type -a cd pwd echo
cd is a shell builtin
pwd is a shell builtin
pwd is /usr/bin/pwd
pwd is /bin/pwd
echo is a shell builtin
echo is /usr/bin/echo
echo is /bin/echo

Three different situations appear. cd is only a builtin. pwd and echo are both a builtin and exist as external programs; in that case, the builtin wins.

cd being a builtin is not a preference but a necessity. A program’s working directory belongs to its own process; a cd running as a separate program would change its own directory and exit, leaving the shell’s directory where it was. To change directory, the command must run inside the shell’s own process.

echo being both a builtin and an external program is for portability: it needs to be callable even in environments without a shell. The behavior of the two implementations is not exactly identical; this is why printf is preferred over echo for formatted output.

Exit Status

Every command leaves behind an integer when it ends: the exit status. The convention runs in reverse — zero reports success, and every nonzero value reports a failure. This direction was chosen because there is one way to succeed and countless ways to fail.

The last command’s status is held in the $? variable:

$ true; echo $?
0
$ false; echo $?
1
$ ls /missing; echo $?
ls: cannot access '/missing': No such file or directory
2

ls reports its failure both with an error message and with a nonzero status. The message is for humans; the status code is for the scripts that chain commands together.

Two values come from the shell itself and need to be told apart:

$ lss -l
bash: lss: command not found
$ echo $?
127

127 reports that the command was not found; 126 that it was found but could not be run. The first is a search problem, the second a permission problem. The difference between these two numbers shows exactly what the next lesson and the permissions topic each solve.

Where error messages are written to is a separate matter: standard output and standard error are two separate streams and can be redirected independently. Using this distinction belongs to the Shell Programming course.

Summary

• The shell first splits the line into words; the first word is the program name, the rest are arguments.
• Variable, filename, and home directory expansions happen in the shell before the program runs; the program sees only the result.
• Double quotes stop word splitting and filename expansion but not variable expansion; single quotes stop all of them.
• Single-letter options can be combined, long options cannot; the -- separator ends option interpretation.
• Builtins run in the shell’s own process; this is why cd has to be a builtin.
• In exit status, zero shows success; 127 reports the command was not found, 126 that it could not be run.

Next Step

Commands now need files to operate on. But files are not placed at random: every branch of the hierarchy starting from the root directory has a specific meaning. The next lesson introduces this layout, establishes the distinction between absolute and relative paths, and builds the project tree that will be developed throughout the course.