How to use rsync and scp for data protection

Synchronize files with rsync

Rsync is a synchronization tool that is distinctive for its ability to send only changed files across the network rather than replicating existing unchanged files each time. It's a longtime staple in the Linux world, so learning to use it for disaster recovery is critical.

Rsync uses the typical source-to-destination syntax used by other tools, such as cp and scp. Here are some of the available rsync options:

The dry run option is particularly helpful when testing recovery scripts.

Basic rsync syntax

You can initiate the rsync replication to or from the local system, which is helpful if you're developing scripts to manage data transfers on your local system.

A local-to-remote transfer places files stored on your server onto a remote system. If you have a large project directory to replicate to a remote office nightly, the syntax might look like the following:

rsync /devteam/projects/* devuser@remoteserver:/devprojects

The wildcard asterisk (*) has its typical meaning here, but with some caveats. It doesn't see hidden files or other files with nontypical attributes. Use the single dot character (.) to mean "this directory" to ensure you replicate hidden files. Be sure to use the -a option in tandem with * to grab all files.

You can still use * to include specific files in the transfer, such as copying all files ending with .txt by typing *.txt in the path.

This situation is a great example of why rsync dry runs are essential for testing a disaster recovery configuration.

If you have a remote system from which you're pulling files to your local server, type the following:

rsync devuser@remoteserver:/devteam/projects/* /devprojects

In both cases above, devuser is an existing user account on the remote server with permissions to read and/or write to the directories.

Common use cases for rsync

There are several areas of data protection and disaster recovery where rsync can be helpful. Consider the following use cases:

• You need to replicate incremental documentation updates from the central office to eight remote branch offices nightly.
• You need to deploy a mirror of the production environment in a failover or test situation.
• You need to centralize user files from six remote branch office servers to a single server in the central office nightly before running a backup program on the server.
• You need to maintain redundant copies of marketing multimedia files efficiently between three remote offices weekly without duplicating files.
• You need to maintain current configuration or data files on remote edge devices.

Automating rsync's basic functionality with scripts and cron scheduling enables administrators to manage data replication easily. This can help satisfy disaster recovery requirements as well as improve user access and network efficiency.

Rsync integration with other operating systems

Do you work in a multi-platform environment consisting of Linux, Windows and macOS systems? Several options exist for integrating non-Linux operating systems into your rsync disaster recovery plans.

Windows Servers can use the Windows Subsystem for Linux. Simply add a distribution, such as Ubuntu, and configure rsync as you would for any other Linux device. MacOS already contains rsync, so those systems are ready to participate immediately.

Copy files with scp

Like rsync, scp is another old-timer in the software world, and with good reason. The scp tool is a straightforward and secure way to manage data transfers among multiple systems. It can be the transfer mechanism in scripted backup and recovery plans, especially when managing configuration files for services like Apache, Nginx and MariaDB.

The scp program brings several advantages if configured correctly, including the following:

• Secure transfers via OpenSSH using TCP port 22.
• Cross-platform functionality among Linux, macOS and Windows systems.
• Key-based authentication for greater security and easier automation without exposing passwords.
• Built-in file compression for efficiency.
• Detailed logging for security audits and accountability.

Scp's syntax is also straightforward, using the standard Linux "from here to there" designation, as in the following example:

scp options /path/to/source user@remoteserver:/path/to/destination

Basic scp syntax

As with rsync, scp can copy files from your local system to a remote destination or pull files from a remote device to your system. Either is appropriate, depending on your requirements.

To push files from your system to a remote destination, use the following scp command:

scp /devteam/projects/* devuser@remoteserver:/devprojects

Pulling files from a remote source is similar. Use the following command to accomplish it:

scp devuser@remoteserver:/devprojects/* /devteam/projects

These commands are easy to integrate into a backup script or other automation workflow to move files among disparate systems quickly and securely.

Common use cases for scp

Coming up with disaster recovery scenarios for scp that demonstrate its usefulness in day-to-day situations is easy. Here are a few areas where scp might help with recovery efforts:

• File transfers and synchronization.
• Backup.
• Software deployment.
• Configuration file management.
• Data migration.

Scripts and other forms of automation regularly rely on scp's straightforward syntax to provide a zero-touch workflow.

Scp integration with other operating systems

Like with rsync, scp is included in most Linux distributions and with macOS. Windows 10, Windows 11, Windows Server 2019 and Windows Server 2022 all include the OpenSSH package. This means users no longer need to add third-party software to their company's servers.

Best practices for scp

Consider the following best practices when integrating scp into your disaster recovery plans:

• Use key-based authentication.
• Enable logging and manage log files carefully.
• Enable compression for efficient storage and network transfers.
• Test the file copy functionality regularly.
• Update the file copy scripts after major infrastructure changes.

Following these best practices with scp will help you maintain an efficient and reliable backup and recovery environment.

A well-written Bash script using rsync and scp can help any administrator move files around the network efficiently and securely, for disaster recovery or simple file management.

Damon Garn owns Cogspinner Coaction and provides freelance IT writing and editing services. He has written multiple CompTIA study guides, including the Linux+, Cloud Essentials+ and Server+ guides, and contributes extensively to TechTarget Editorial and CompTIA Blogs.