How to Master the rsync Command in Linux

maketecheasier ^ | 1 February 2019 | Alexandru Andrei

[ShadowAce]

As its name suggests, rsync is a synchronization tool. The first time you use it, it simply copies files and directories from one location to another. Subsequent uses, however, synchronize only the differences between these locations. So, if you add one file to the source directory, rsync will copy only the new file to the destination. This saves a lot of time and bandwidth, especially when copying to a remote location.

Imagine you have 50GB of data you have to synchronize to a remote server. You can schedule rsync to run daily. Instead of having to copy the whole 50GB over and over again, the utility will only send the differences. This might mean it only has to upload a few megabytes of data every day, instead of gigabytes.

In the featured picture above, you can see how the first time 104MB were transferred, while during the second use only 31KB were required.

How to Install rsync on Various Linux Distributions

On Debian and Debian-based distributions, like Ubuntu, you can install rsync with:

sudo apt install rsync

On Fedora, install with:

dnf install rsync

And on distributions derived from RedHat (e.g., CentOS):

yum install rsync

To install rsync on Arch Linux, use the command:

pacman -S rsync

On openSUSE, use:

zypper install rsync

If it isn’t included in your default repositories, consult the openSUSE rsync installation page.

How to Use rsync to Synchronize Local Files or Directories

The general syntax of rsync is: rsync [options] /path/to/source /path/to/destination.

[options] stands for command line options which you can find in the manual:

man rsync

Press q to quit the manual.

To copy a file:

rsync /bin/ls .

The command above is equivalent to rsync /bin/ls /home/your_username. This copies the “ls” command to the directory you are currently in. By default, this should be /home/your_username. The dot . stands for current directory. Pathnames can be absolute (/home/your_username/Downloads/chrome.zip) or relative to your current directory (Downloads/chrome.zip).

When you deal with very large files, you can use this command:

rsync -P /path/to/file /path/to/destination

This will show progress and allow you to resume an interrupted transfer. In case you need to resume, use:

rsync -P --append path/to/file /path/to/destination

Without --append, the transfer would start from scratch.

To synchronize directories:

rsync -av /path/to/source /path/to/destination

Example command:

rsync -av /bin .

Effect of Trailing Slash in rsync (Important)

In the previous command note that the path to the source directory doesn’t contain a trailing slash / (/bin vs /bin/). Without a trailing slash, rsync copies the directory itself. So rsync -av /bin . results in what is depicted in the following image.

When a trailing slash is used, rsync copies the contents in that directory. In this case every file in /bin would be scattered around in the destination directory. So rsync -av /bin/ . would result in what is depicted in the following image.

It’s very important to remember this difference, as using TAB to autocomplete directory names automatically adds a trailing slash at the end. So keep that in mind to avoid creating a mess in the destination directory.

How to Use rsync to Synchronize with Remote Locations (e.g. Servers)

What was described in the previous section holds true when working with remote locations. The only difference is that you have to prepend the login username and address of the remote location. To send your /bin directory to a remote computer, use:

rsync -av /bin your_username@203.0.113.10:/home/your_username

rsync connects through OpenSSH. This means you have to set up the OpenSSH server and login credentials on the remote destination beforehand. Note that your_username refers to the username created on the server. This might be different from the username you have on your local machine.

If you own a domain name and point it to your server IP, you can use the domain in the command:

rsync -av /bin your_username@example.com:/home/your_username

On large transfers, it’s useful to also compress data before sending it through the wire by adding the -z parameter:

rsync -avz /bin your_username@example.com:/home/your_username

You can omit the compression parameter when you send files that are already compressed (audio files like MP3, videos, JPEG images). They cannot be compressed further, so you would actually waste time by using -z on these.

To copy files from the remote destination to your local computer, just reverse locations:

rsync -avz your_username@example.com:/bin /home/your_username

Conclusion

This covers the most important rsync commands you need to know. If you ever need more control over how to synchronize directories, consult the online rsync manual and scroll a few pages down until you find “Options Summary.”

[ShadowAce]

[usconservative]

Excellent post and well documented! To steal a line from an old Geico commercial - “so easy a caveman can do it.”

[marktwain]

Thanks for posting Linux threads.

[SteveH]

linux rsync bump

[mostly_lies]

Thank you

[Texas Fossil]

Thank you.

A useful command line function that I have never used before.

[zeugma]

rsync is an extraordinarily powerful command. As is the case with powerful tools, you must be very careful when using it. If you get the to and from directories confused, it can have pretty catastrophic results, so it's important to double-check before you hit return!

One thing this description of the rsync command doesn't really speak to is the efficiency of the program. You can replicate an entire directory structure and all of its files from one location to another, either on your local disk, or to another computer on the other side of the planet. The real beauty of rsync is that once the remote files have been copied to the desired location, subsequent calls to rsync the two locations can be accomplished using minimal time.

In the following, I have a directory that contains a bunch of files in it. I'm going to use rsync to make a copy of the directory and its files to another location. I'm using 3 additional arguments with rsync. '--verbose' will make rsync echo everything it does to the screen. '--archive' tells it to only replicate changes.(we will see why this is important in a second). '--delete' tells it to delete anything in the target directory that doesn't exist in the source directory. This is so the directories will be as identical as possible.

$ rsync --verbose --archive --delete source1/ target1/
sending incremental file list
created directory target1
./
DSCN0693.JPG
DSCN0694.JPG
DSCN0695.JPG
DSCN0696.JPG
DSCN0697.JPG
DSCN0698.JPG
DSCN0699.JPG

sent 52,720,357 bytes  received 182 bytes  105,441,078.00 bytes/sec
total size is 52,707,014  speedup is 1.00
$ ls -l source1 target1
source1:
total 51484
-rw-r--r-- 1 zeugma zeugma  132058 Feb  5 08:29 DSCN0693.JPG
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG

target1:
total 51484
-rw-r--r-- 1 zeugma zeugma  132058 Feb  5 08:29 DSCN0693.JPG
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG

You can see above that both directories contain the same stuff (before I'd executed the previous command, 'target1' didn't even exist). Now I'm going to add a file to the source directory. Then I'll run rsync again.

$ echo "This is a new file with stuff in it" > source1/newfile.txt
$ rsync --verbose --archive --delete source1/ target1/
sending incremental file list
./
newfile.txt

sent 312 bytes  received 38 bytes  700.00 bytes/sec
total size is 52,707,049  speedup is 150,591.57

$ ls -l source1 target1
source1:
total 51356
-rw-r--r-- 1 zeugma zeugma  132058 Feb  5 08:29 DSCN0693.JPG
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG
-rw-r--r-- 1 zeugma zeugma      35 Feb  5 08:32 newfile.txt

target1:
total 51356
-rw-r--r-- 1 zeugma zeugma  132058 Feb  5 08:29 DSCN0693.JPG
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG
-rw-r--r-- 1 zeugma zeugma      35 Feb  5 08:32 newfile.txt

You'll notice that this time, when I executed the rsync, it didn't copy the files that had already existed. It only copied the text file, because that is all that needed to be done to keep the two directories in sync. As you can imagine, if the two directory structures that you're wanting to replicate are really large, this can save you a heck of a lot of time to keep them in sync. This may not be such a big deal when just copying files from disk to disk, but when doing so across the network, the time savings and bandwidth used can be really significant.

It's also worth mentioning that with the '--archive' flag on, one of the things rsync is doing is looking at the time/date stamps of the files, along with the file size to determine if something should be tagged for transfer or not, so if I had edited one of the jpg files, it would have transferred as well. You can also tell rsync to do additional checks for file differences, but they will slow down the program somewhat.

One last thing to show...

$ rm source1/DSCN0693.JPG
$ rsync --verbose --archive --delete source1/ target1/
sending incremental file list
deleting DSCN0693.JPG
./

sent 213 bytes  received 31 bytes  488.00 bytes/sec
total size is 52,574,991  speedup is 215,471.27

$ ls -l source1 target1
source1:
total 51356
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG
-rw-r--r-- 1 zeugma zeugma      35 Feb  5 08:32 newfile.txt

target1:
total 51356
-rw-r--r-- 1 zeugma zeugma 8862121 Feb  5 08:29 DSCN0694.JPG
-rw-r--r-- 1 zeugma zeugma 8326321 Feb  5 08:29 DSCN0695.JPG
-rw-r--r-- 1 zeugma zeugma 8808912 Feb  5 08:29 DSCN0696.JPG
-rw-r--r-- 1 zeugma zeugma 8854626 Feb  5 08:29 DSCN0697.JPG
-rw-r--r-- 1 zeugma zeugma 8807977 Feb  5 08:29 DSCN0698.JPG
-rw-r--r-- 1 zeugma zeugma 8914999 Feb  5 08:29 DSCN0699.JPG
-rw-r--r-- 1 zeugma zeugma      35 Feb  5 08:32 newfile.txt

As you can see above, I deleted one of the pictures in the source directory. When I used rsync, that file was deleted from the target directory.

To sum up, rsync can be a really powerful tool that is fast and efficient. One common use is to make backups of data from one drive to another. One of the ways I use it is to update pages on my webserver. I'll make whatever changes I have locally, modifying as many files as I want, then when I'm ready to push it out to the server, it's just a matter of a single rsync command to perfectly replicate all of the changes I had made.

[ShadowAce]

One of the ways I use it is to update pages on my webserver.

I use rsync to keep my phone's music collection up to date. Whenever I edit a music file (to add tags--artist, album, year, etc), I can just rsync the directory to my phone and allof the files I edited are moved there.

I can re-arrange my collection as well. If I move a file from one directory to another, rsync will delete the first on the target and add the second to the target.

Very efficient. I've made it more so by scripting the entire process. :)

[wally_bert]

I could probably handle it then.

[upchuck]

rsync is similar to the DOS command XCOPY.

[Bloody Sam Roberts]

Imagine you have 50GB of data you have to synchronize to a remote server. You can schedule rsync to run daily. Instead of having to copy the whole 50GB over and over again, the utility will only send the differences.

Deduplication on the command line...built-in!