File Transfer

The previous lesson reduced the project data to a single archive file. This file can now be sent to another machine — but archiving and sending the whole tree every time is wasteful when only a small part has changed.

This lesson compares two approaches: a tool that copies the file as is, and a tool that sends only the difference. Before either, the transport layer both are built on is introduced.

In the examples below, the remote machine is referred to by the name server, and the remote account by the name backup.

Encrypted Shell

The standard way to run a command on a remote machine is the encrypted shell (ssh). It provides three things: the connection is encrypted, the user’s identity is verified, and the server’s identity is verified. The third is often skipped but matters most; encryption means nothing if who is on the other side is unknown.

On first connecting to a server, its identity is asked about:

$ ssh backup@server pwd
The authenticity of host 'server (127.0.0.1)' can't be established.
ED25519 key fingerprint is SHA256:BXO0kZ7GJ+yv/Idqx3h4fOdWRWCgAH6QbY4y0M/YuHk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'server' (ED25519) to the list of known hosts.
/home/backup

The server has its own key pair, and the digest of its public key is shown as a fingerprint. Once approved, this fingerprint is saved to a local file. No question is asked on later connections:

$ ssh backup@server pwd
/home/backup

If the saved fingerprint does not match on a later connection, the tool refuses the connection and warns. This is the mechanism that catches a man-in-the-middle attempt. The warning also appears when the server has been reinstalled; in both cases, the correct response is not to suppress the warning but to verify the fingerprint through an independent channel.

The /home/backup path in the output shows that the command ran on the remote machine, in the home directory of the backup account.

The user’s identity is verified in two ways: a password or a key pair. With a key pair, the private key stays on the local machine and the public key is authorized on the server account; the password never goes onto the network. Key management and server configuration are the subject of the System Administration course.

Remote Copying

scp is the remote counterpart of local copying, and it uses the same transport layer. A remote path is written in the form user@machine:path:

$ scp archive/data.tar.xz backup@server:incoming/
$ ssh backup@server ls -l incoming
total 8
-rw-r--r-- 1 backup backup 4416 Jul 26 19:16 data.tar.xz

The colon separator is decisive: backup@server:incoming/ specifies a remote path, while backup@server alone specifies a local file of that name. When the separator is forgotten, the command gives no error — it creates a local file.

In remote paths, a relative path is relative to the home directory of the logging-in user. The path incoming/ means /home/backup/incoming/.

scp sends the file as a whole; it retransmits even if the same content already exists at the target. Sending a directory requires -r, just as in local copying.

Verifying the Transfer

Transfer completeness is tested with a digest of the content:

$ sha256sum archive/data.tar.xz
6608712cb484f9629dcdf1cbaa3bced3cb87ff00b24d6d32646775ccff3b5443  archive/data.tar.xz
$ ssh backup@server sha256sum incoming/data.tar.xz
6608712cb484f9629dcdf1cbaa3bced3cb87ff00b24d6d32646775ccff3b5443  incoming/data.tar.xz

If the two digests are the same, the two files’ content is the same. A size comparison does not give this guarantee: two different contents can have the same size, but two different contents with the same digest cannot be found in practice.

Digest verification is not only against transfer errors; it also tests whether the archive was changed at the source. This is why digest files are published alongside packages in software distributions — the same principle will be taken up together with the chain of signatures in the repository configuration lesson.

Synchronizing

rsync does not copy, it synchronizes: it compares the source with the target and sends only the difference. The comparison is done, by default, on size and modification time.

It can be asked to show what it would do before actually running:

$ rsync -avn data/ backup@server:incoming/data/
sending incremental file list
created directory incoming/data
./
current.csv -> raw/measurement-03.csv
raw/
raw/measurement-01.csv
raw/measurement-02.csv
raw/measurement-03.csv
raw/measurement-04.csv
processed/
processed/daily-summary.csv
processed/measurement-01.csv

sent 351 bytes  received 81 bytes  864.00 bytes/sec
total size is 42,270  speedup is 97.85 (DRY RUN)

-n performs a dry run: nothing is transferred, only what would be done is listed — the (DRY RUN) in the last line confirms this. -a is an option set that preserves metadata — permissions, timestamps, and symbolic links; the current.csv -> raw/measurement-03.csv line appearing as a link in the list is a result of this. -v prints the files being processed.

After the dry run is approved, the same command is run without -n. The real difference is visible on the second run:

$ rsync -av data/ backup@server:incoming/data/
sending incremental file list
raw/measurement-01.csv

sent 430 bytes  received 43 bytes  946.00 bytes/sec
total size is 42,284  speedup is 89.40

Only one measurement file had changed in the meantime. Only that file is in the list; the data sent is 430 bytes, while the tree’s total size exceeds 42 kilobytes. Doing the same job with scp -r would mean sending the entire tree every time.

Unchanged files not being opened at all is what makes synchronization the standard tool for regular backups: the cost is proportional not to the size of the tree, but to the size of the change.

The Slash Changes the Meaning of the Source

The most often misused detail of the synchronization tool is the slash at the end of the source path.

In the commands above, the source was written as data/: the trailing slash means “the contents of this directory,” and the content is placed directly into the target. When the slash is removed, the meaning changes:

$ rsync -avn data backup@server:incoming/trial/
sending incremental file list
created directory incoming/trial
data/
data/current.csv -> raw/measurement-03.csv
data/raw/
data/raw/measurement-01.csv
data/raw/measurement-02.csv
data/raw/measurement-03.csv
data/raw/measurement-04.csv
data/processed/
data/processed/daily-summary.csv
data/processed/measurement-01.csv

sent 370 bytes  received 84 bytes  908.00 bytes/sec
total size is 42,284  speedup is 93.14 (DRY RUN)

The paths are now listed with a data/ prefix: the directory itself is placed inside the target, and the result becomes incoming/trial/data/.... The two forms of the same command produce two separate trees, one level apart.

A slash on the target path has no such effect. The rule applies only to the source, and a dry run makes this difference visible every time.

Leftovers at the Target

By default, synchronization does not delete extra files at the target. If a file not present in the source is sitting at the target, it stays there:

$ rsync -av data/ backup@server:incoming/data/
sending incremental file list
./

sent 329 bytes  received 21 bytes  700.00 bytes/sec
total size is 42,284  speedup is 120.81
$ ssh backup@server ls incoming/data
old-measurement.csv
current.csv
raw
processed

Making the target an exact mirror of the source requires --delete:

$ rsync -avn --delete data/ backup@server:incoming/data/
sending incremental file list
deleting old-measurement.csv

sent 326 bytes  received 39 bytes  730.00 bytes/sec
total size is 42,284  speedup is 115.85 (DRY RUN)

This option is destructive, and the warning from the deletion lesson applies here exactly: a misspelled source path, or one pointing to an empty directory, deletes everything at the target. Every command using --delete must be run with -n first; the deleting lines in the dry-run list are the exact list of files that will be deleted.

Choosing a Tool

Criterion	scp	rsync
Data sent	The entire file	Only the difference
Repeated run	Same cost	Cost proportional to change
Metadata preservation	Limited	Full, with -a
Dry run	No	Yes, with -n
Deletion at target	No	Yes, with --delete
Requirement on the other side	None	The tool must be installed

Copying is enough for one-time, single-file transfers; synchronization is preferred for repeated tree transfers. If interactive browsing and selective downloading are needed, the sftp tool provides a file-system-like session over the same transport layer.

Summary

• The encrypted shell verifies both the user and the server; the server’s fingerprint is approved on the first connection and compared on later ones.
• Remote paths are written in the form user@machine:path; when the colon is forgotten, the command silently creates a local file.
• A transfer is verified by comparing content digests; equal size gives no guarantee.
• Synchronization sends only the files that changed; the cost is proportional to the size of the change, not of the tree.
• A trailing slash on the source path means “the contents of the directory,” the form without a slash means “the directory itself.”
• --delete makes the target an exact mirror of the source and is destructive; it must be checked with a dry run first.

Next Step

The files on the remote machine belong to the backup account; the local files to another. How this distinction is built — what a user’s identity is stored as in the system, what groups are for, what it means for a file to belong to someone — has not been explained yet. The next topic builds the account model and moves from there to permission bits.