--- title: 'Archiving and Compression' source: 'https://academia.sh/en/courses/introduction-to-linux/archiving-and-compression' course: 'Introduction to Linux' language: en updated: '2026-08-17T18:09:55+00:00' license: 'CC BY-SA 4.0' --- # Archiving and Compression Separating archiving from compression, the archive format carrying metadata, the ratio–cost trade-off of compression algorithms, and verifying content before extracting. The question of packaging the tree into a single file starts where the previous lesson left off, and immediately splits into three sub-questions: how is a directory structure converted into a flat byte sequence, are permissions and timestamps preserved, are symbolic links stored with their targets' content or with their path strings? This lesson first separates the two operations — they are confused because most tools do both together — and then establishes how archives are verified. ## Two Separate Operations **Archiving** is converting multiple files, together with their metadata, into a single byte sequence. It does not shrink size; it even grows it a little. **Compression** is re-encoding a single byte sequence with a shorter representation. It does not change the number of files. The separation is a direct consequence of the Unix tradition's "small tools, one job" assumption. Every compression algorithm can be paired with every archive format; if the two jobs were merged into a single program, this freedom would be lost. The archive format places a **header block** in front of each file's content. The header holds the name, size, permissions, ownership, timestamp, and file type. Because of this, files extracted from an archive recover not only their content but also their metadata — this is something an ordinary concatenation operation cannot do. ``` $ tar -cf archive/data.tar data $ ls -l archive total 56 drwxr-xr-x 2 student student 4096 Jul 26 19:04 raw-backup -rw-r--r-- 1 student student 51200 Jul 26 19:10 data.tar ``` The options determine the operation: `-c` creates an archive, `-f` gives the archive file's name. The `data` directory given as the argument is taken recursively. The archive size being larger than the sum of the content is an expected result. The format uses fixed-length blocks and adds padding at the end of the archive; for small files, this padding can be larger than the data. ## Looking Inside the Archive Listing an archive's contents before extracting it is a habit that should be applied: ``` $ tar -tvf archive/data.tar drwxr-xr-x student/student 0 2026-07-26 19:08 data/ drwxr-xr-x student/student 0 2026-07-26 19:03 data/processed/ -rw-r--r-- student/student 47 2026-07-26 19:03 data/processed/measurement-01.csv -rw-r--r-- student/student 47 2026-07-26 19:03 data/processed/daily-summary.csv drwxr-xr-x student/student 0 2026-07-26 19:10 data/raw/ -rw-r--r-- student/student 47 2026-07-26 19:02 data/raw/measurement-01.csv -rw-r--r-- student/student 47 2026-07-26 19:03 data/raw/measurement-02.csv -rw-r--r-- student/student 42019 2026-07-26 19:10 data/raw/measurement-04.csv -rw-r--r-- student/student 47 2026-07-26 19:03 data/raw/measurement-03.csv lrwxrwxrwx student/student 0 2026-07-26 19:08 data/current.csv -> raw/measurement-03.csv ``` `-t` lists, `-v` adds detail. The output confirms everything established in the previous lessons. Permissions and ownership are preserved. Timestamps are preserved. The symbolic link in the last line has been stored **as a link, not as a copy** of its target: the type character `l`, size `0`, and the target path string are visible. If following symbolic links is wanted, the `-h` option is given; in that case the target's content, not the link, goes into the archive, and the archive grows. If the tree is being moved as a whole, the default behavior is correct — both the link and its target are in the archive, and the relative target continues to resolve. The reason for the listing habit is that archives **carry their own directory structure**. A well-formed archive contains a single top-level directory and, when extracted, creates only that. An archive that is not like this dumps dozens of files directly into the current directory and overwrites files with the same names. Listing before extracting shows this in advance. ## Removal of Absolute Paths If a file is added to the archive with an absolute path, the tool removes the leading slash and warns: ``` $ tar -cf /tmp/absolute.tar /home/student/project/docs tar: Removing leading `/' from member names $ tar -tf /tmp/absolute.tar home/student/project/docs/ ``` This behavior is a security measure. If absolute paths were preserved, a user extracting an archive could receive one that overwrites system files. When paths are kept relative, the archive can affect at most the directory it is extracted into. The practical consequence of the warning is that archives are created **standing above the tree, with a relative path**: the command `tar -cf archive/data.tar data` produces paths starting with `data/`, and wherever it is extracted, it creates a `data` directory there. ## Extracting an Archive The `-x` option extracts, `-C` specifies the target directory: ``` $ mkdir -p /tmp/restore $ tar -xf archive/data.tar -C /tmp/restore $ find /tmp/restore | sort /tmp/restore /tmp/restore/data /tmp/restore/data/current.csv /tmp/restore/data/raw /tmp/restore/data/raw/measurement-01.csv /tmp/restore/data/raw/measurement-02.csv /tmp/restore/data/raw/measurement-03.csv /tmp/restore/data/raw/measurement-04.csv /tmp/restore/data/processed /tmp/restore/data/processed/daily-summary.csv /tmp/restore/data/processed/measurement-01.csv $ ls -l /tmp/restore/data/current.csv lrwxrwxrwx 1 student student 22 Jul 26 19:08 /tmp/restore/data/current.csv -> raw/measurement-03.csv ``` The tree has come back complete, and the symbolic link has been rebuilt as a link. Thanks to its relative target, it resolves in its new location too — the payoff of the relative-target preference from the links lesson is collected here. The `-C` option removes the risk of dumping the archive into the current directory: extraction is always done into an empty directory or one set aside for this purpose. ## Compression Algorithms Compression tools work on a single file and usually replace the original file with the compressed one; the `-k` option preserves the original file. The result of three common algorithms on the same data: ``` $ gzip -k /tmp/m4.csv $ bzip2 -k /tmp/m4.csv $ xz -k /tmp/m4.csv $ ls -l /tmp/m4.csv* -rw-r--r-- 1 student student 42019 Jul 26 19:10 /tmp/m4.csv -rw-r--r-- 1 student student 5579 Jul 26 19:10 /tmp/m4.csv.bz2 -rw-r--r-- 1 student student 10776 Jul 26 19:10 /tmp/m4.csv.gz -rw-r--r-- 1 student student 2444 Jul 26 19:10 /tmp/m4.csv.xz ``` The same 42019-byte measurement file has dropped to 10776, 5579, and 2444 bytes respectively. The ordering is not a coincidence; it follows from the algorithms' design choices. | Tool | Approach | Trade-off | |---|---|---| | `gzip` | Dictionary-based coding, small window | Fastest, lowest ratio, most widely supported | | `bzip2` | Block-sorting transform | Medium speed, medium ratio | | `xz` | Dictionary-based coding, very large window | Slowest and most memory, highest ratio | The common principle is this: a higher compression ratio requires searching for patterns further back in history, which means more memory and more processing. None of the algorithms is "better" than another; they sit at different points. The selection criterion is how many times the data will be compressed and how many times decompressed. In an archive produced once and downloaded many times, the compression cost is paid once, and a high ratio pays off. In daily backups produced frequently, the compression time is paid every time. There is also data that cannot be compressed: image and audio formats that are already compressed, encrypted files, and random data. Compressing these does not shrink the size, it grows it a little — because the output must carry at least as much information as the input. ## Combining Archiving and Compression The tool can chain archiving and compression in a single command: ``` $ tar -czf archive/data.tar.gz data $ tar -cJf archive/data.tar.xz data $ ls -l archive total 76 drwxr-xr-x 2 student student 4096 Jul 26 19:04 raw-backup -rw-r--r-- 1 student student 51200 Jul 26 19:10 data.tar -rw-r--r-- 1 student student 11330 Jul 26 19:10 data.tar.gz -rw-r--r-- 1 student student 4416 Jul 26 19:10 data.tar.xz ``` `-z` applies gzip, `-j` bzip2, `-J` xz. The extension convention reflects the order: `.tar.gz` means archived first, then compressed. The order matters. Archiving first and compressing second lets repetitions between files be found too; because the measurement files share the same header line and similar number patterns, this gain is real. In exchange, reading a single file requires extracting from the start of the archive. Compressing each file separately preserves random access but misses repetition between files. The padding in archives also misleadingly raises the compression ratio: part of the 51200-byte archive dropping to 4416 bytes is the padding being compressed, not the data. When evaluating a compression ratio, the criterion is the size of the original data, not of the archive. Now that the archives are ready, the directory copy left over from the previous lesson is unnecessary: ``` $ rm -r archive/raw-backup $ ls -l archive total 72 -rw-r--r-- 1 student student 51200 Jul 26 19:10 data.tar -rw-r--r-- 1 student student 11330 Jul 26 19:10 data.tar.gz -rw-r--r-- 1 student student 4416 Jul 26 19:10 data.tar.xz ``` The archive's advantage over a directory copy is not that it is a single file, but that it freezes the metadata and link structure at a point in time. ## A Note on Portability The archiving tool has two common implementations, and their option sets are not identical. The `-c`, `-t`, `-x`, `-f`, `-v`, `-C` options above mean the same thing in both. The compression shortcuts (`-z`, `-j`, `-J`) are common but not standard; in portable scripts, compression is applied as a separate step through a pipeline. In older implementations, options can also be written without a dash (`tar cf`). This form continues to work; the dashed form is preferred in written documentation. ## Summary - Archiving converts many files, together with their metadata, into a single byte sequence; compression shortens a single byte sequence. The two operations are separate and can be combined independently. - Archive headers carry permissions, ownership, timestamp, and file type; symbolic links are stored with their target strings. - An archive is listed before extracting; absolute paths are removed, and extraction is done into a directory set aside with `-C`. - As compression ratio increases, memory and processing cost rise; the choice depends on how many times the data will be compressed and decompressed. - Archiving first and compressing second catches repetition between files but loses random access. - Data that is already compressed or random cannot be compressed; the output grows a little. ## Next Step Now that the archive has become a single file, it can be sent to another machine. But archiving and sending the entire tree every time is wasteful when only a small part of the tree has changed. The next lesson takes up remote copying and synchronization tools: how a connection is established, which tool sends only what changed, and why a trailing slash on a path changes the result.