The Command Path

In the previous lesson, a name’s type was determined with type, and when the answer was file, it meant the name resolved to a file. But when ls is typed, no full path is given; the shell finds the file somewhere.

This lesson unpacks the rule behind that search. The same rule explains, all at once, why a command is not found, why the same name can resolve to two different programs, and why a program in the current directory does not run directly.

Environment Variables

Every process has an environment table mapping names to values. When a process creates a new process, this table is copied over; the child process inherits the environment, but any change it makes never propagates back to the parent.

$ env | grep -E '^(HOME|PATH|PWD|SHELL|LANG|LC_ALL)='
PWD=/home/student
HOME=/home/student
LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Not every variable defined in the shell enters the environment. An ordinary assignment only creates a shell variable; this variable is not passed to programs that are run. For a variable to enter the environment, it must be marked with export. The distinction is this: a shell variable lives in the shell’s own memory, an environment variable is copied to child processes.

Three environment variables are used directly in this course: HOME holds the path to the home directory, PWD the working directory, and PATH the command search list. Localization variables such as LC_ALL affect the language of command output and sort order; they are kept at the standard setting for this course’s output.

Search Order

When the shell sees a command name, it decides in a fixed order:

1. Is the name a keyword? (if, for, while)
2. Is the name an alias?
3. Is the name a shell function?
4. Is the name a builtin?
5. If not, it is searched for in the directories in the PATH list.

The first four live in the shell’s own memory, and the file system is never consulted. The fifth step searches on disk.

The PATH variable consists of directory paths separated by colons. The shell scans these directories left to right and picks the first file that carries the name and has execute permission. It does not look at the remaining directories. If the name is found in none of them, it gives “command not found” with exit status 127.

The order being left to right means position in the search list carries a meaning of priority.

Shadowing

The same name can exist in more than one directory. In that case, whichever comes first in the list wins and shadows the other. The effect can be shown by setting up a separate directory:

$ mkdir -p /tmp/demo
$ printf '#!/bin/sh\necho fake version\n' > /tmp/demo/date
$ chmod +x /tmp/demo/date
$ type -a date
date is /usr/bin/date
date is /bin/date
$ PATH="/tmp/demo:$PATH"
$ type -a date
date is /tmp/demo/date
date is /usr/bin/date
date is /bin/date
$ date
fake version

type -a lists every match carrying the name, in order; the first line is the one that will run. Adding the new directory to the front of the list did not make the system’s date command unreachable — it merely fell later in the search order.

Shadowing is both a tool and a risk. A tool: a command’s own version can be moved ahead without touching the system version. A risk: if a writable directory sits at the front of the search list, anyone who can write a file to that directory can hijack command names. This is why the write permissions of directories added to the search list must be judged by the criteria covered in the permissions topic.

The demo directory is removed once its job is done; leaving such a directory in place permanently causes unexpected behavior in later sessions.

The Current Directory Is Not in the List

The current directory (.) does not appear in PATH. This has a direct consequence: running a program in a directory requires specifying a path.

$ pwd
/home/student/project/scripts
$ summary.sh
bash: summary.sh: command not found
$ echo $?
127
$ ./summary.sh
raw measurement files: 0

The difference between summary.sh and ./summary.sh is a matter of the shell’s perspective. The first is a name with no slash in it; the shell searches for it in the PATH list and does not find it. The second contains a slash; the shell treats it as a path, does no search, and runs that file directly.

Leaving the current directory out of the list is a deliberate security decision. If it were included, leaving a file named ls in a directory everyone can write to would be enough to make everyone entering that directory run that file. This is why the search list consists of fixed, controlled directories.

If the current directory is to be added to the list at all, it belongs at the end, at least; that is the one position from which it cannot shadow system commands.

Execute Permission

It is not enough for a file to sit in a directory on PATH; it must also be marked executable.

$ ls -l summary.sh
-rw-r--r-- 1 student student 142 Jul 26 18:49 summary.sh
$ ./summary.sh
bash: ./summary.sh: Permission denied
$ echo $?
126
$ chmod +x summary.sh
$ ls -l summary.sh
-rwxr-xr-x 1 student student 142 Jul 26 18:49 summary.sh
$ ./summary.sh
raw measurement files: 0

Two exit statuses report separate problems: 127 the file was not found, 126 it was found but could not be run. The chmod +x command adds execute permission; what the permission bits mean and how to read the string -rwxr-xr-x will be established in the first lesson of the permissions topic.

The #!/bin/sh notation on the script’s first line tells the kernel which interpreter to run the file with. The kernel reads this line, calls the named program, and hands it the file as an argument. This is why a shell script can be run directly even though it is not an executable binary.

The 0 seen in the output reports that the raw measurement directory is still empty; the next topic will begin by filling it.

The Hash Table

If PATH were scanned from the start for every command, frequently used commands would cause repeated directory reads. The shell avoids this by storing the paths it finds in a hash table.

$ PATH="$HOME/project/scripts:$PATH"
$ type summary.sh
summary.sh is /home/student/project/scripts/summary.sh
$ summary.sh
raw measurement files: 0
$ type summary.sh
summary.sh is hashed (/home/student/project/scripts/summary.sh)
$ hash
hits	command
   2	/home/student/project/scripts/summary.sh
$ hash -r
$ type summary.sh
summary.sh is /home/student/project/scripts/summary.sh

After the first call, type’s answer changes: the path is now recorded in the table. This has a side effect — when a program is moved or deleted, the shell can keep using the old path and give a “no such file or directory” error. hash -r clears the table and forces a fresh search.

This behavior is a typical use of the hash table from the Data Structures course: the result of a repeated, expensive search is cached, in exchange for accepting the risk that the cached information can go stale.

Permanently Extending the Search List

A PATH assignment made in the shell lasts only for that session. For it to be permanent, it must be written to a configuration file that is read when the session starts:

$ tail -2 ~/.profile
PATH="$HOME/project/scripts:$PATH"
export PATH

~/.profile is the user configuration file read at session start. The effect is visible in a new session:

$ echo "$PATH"
/home/student/project/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ summary.sh
raw measurement files: 0

Preserving the old value with $PATH in the assignment matters. An assignment of the form PATH="$HOME/project/scripts" would overwrite the list, and none of the system commands would be found. This is a mistake that is a hassle to reverse: until it is fixed, most commands, including ls, do not run, and the fix requires typing the full path.

Summary

• The shell searches for a name in order as a keyword, alias, function, and builtin; if it finds none, it scans the PATH directories left to right.
• The first matching file wins; the other matches for the same name are shadowed and can be seen with type -a.
• The current directory is not in PATH; running a program there requires the ./ prefix, and this is a safeguard against command hijacking in writable directories.
• Finding the file is not enough; without execute permission, the exit status is 126.
• The shell caches the paths it finds in a hash table; hash -r is needed when a program is moved.
• A permanent PATH change is written to a configuration file read at session start, and the old value must be preserved.

Next Step

The system has been introduced, the tree has been built, the script has run — but the count of measurements the script reports is still zero. The next topic descends to the files themselves: how to navigate between directories, what each column of a listing’s output reports, and how the project/data/raw directory will be filled.