Command Center

So what is unique about rsync?

How do I install rsync?

If you are using Debian or Ubuntu Linux, type the following command

If you are using Red Hat Enterprise Linux (RHEL), type the following command

If you are using CentOS/Fedora Core Linux, type the following command

Always use rsync over ssh

rsync command common options

• --delete : delete files that don't exist on sender (system)
• -v : Verbose (try -vv for more detailed information)
• -e "ssh options" : specify the ssh as remote shell
• -a : archive mode
• -r : recurse into directories
• -z : compress file data

Task : Copy file from a local computer to a remote server

Password:
sent 19099 bytes  received 36 bytes  1093.43 bytes/sec
total size is 19014  speedup is 0.99

Task : Copy file from a remote server to a local computer

Task: Synchronize a local directory with a remote directory

Task: Synchronize a remote directory with a local directory

Task: Synchronize a local directory with a remote rsync server

Task: Mirror a directory between my "old" and "new" web server/ftp

Read related previous articles

• How do I sync data between two Load balanced Linux/UNIX servers?
• How do I sync data between two Load balanced Windows 2003 servers?

Other options - rdiff and rdiff-backup

rsync for Windows server/XP

1. DeltaCopy
2. NasBackup

Further readings

How Does Keychain Make It Better Than a Key Less Passphrase?

peerbox.nixcraft.net.in => Remote Backup Server. Works in pull only mode. It will backup server1.nixcraft.net.in and server2.nixcraft.net.in.
vivek-desktop.nixcraft.net.in => My desktop computer.
server1.nixcraft.net.in => General purpose remote server.
server2.nixcraft.net.in => General purpose remote web / mail / proxy server.

Install keychain on CentOS / RHEL / Fedora Linux

Install keychain on Debian / Ubuntu Linux

Install keychain on FreeBSD

How Do I Setup SSH Keys With passphrase?

1. Howto Linux / UNIX setup SSH with DSA public key authentication (password less login)
2. Howto use multiple SSH keys for password less login

How Do I Use Keychain?

### START-Keychain ###
# Let  re-use ssh-agent and/or gpg-agent between logins
/usr/bin/keychain $HOME/.ssh/id_dsa
source $HOME/.keychain/$HOSTNAME-sh
### End-Keychain ###

Fig.01 - Keychain in Action

Task: Clear / Delete All Of Ssh-agent's Key

Security Task: Make Sure Intruder Cannot Use Your Existing SSH-Agent's Keys (only allow cron jobs to use password less login)

/usr/bin/keychain --clear $HOME/.ssh/id_dsa

/usr/bin/keychain --clear $HOME/.ssh/id_rsa

Task: Use Keychain With Backup Scripts for Passwordless login via cron

source $HOME/.keychain/$HOSTNAME-sh

#!/bin/bash
# Remote Server Rsync backup Replication Shell Script
# Local dir location
LOCALBAKPOINT=/iscsi
LOCALBAKDIR=/backups/server1.nixcraft.net.in/wwwroot
# Remote ssh server setup
SSHUER=root
SSHSERVER=server1.nixcraft.net.in
SSHBACKUPROOT=/wwwroot
 
# Make sure you can log in to remote server without a password
source $HOME/.keychain/$HOSTNAME-sh 
 
# Make sure local backup dir exists
[ ! -d ${LOCALBAKPOINT}${LOCALBAKDIR} ] && mkdir -p ${LOCALBAKPOINT}${LOCALBAKDIR}
 
# Start backup
/usr/bin/rsync --exclude '*access.log*' --exclude '*error.log*' -avz -e 'ssh ' ${SSHUER}@${SSHSERVER}:${SSHBACKUPROOT} ${LOCALBAKPOINT}${LOCALBAKDIR}
 
# See if backup failed or not to /var/log/messages file
[ $? -eq 0 ] && logger 'RSYNC BACKUP : Done' || logger 'RSYNC BACKUP : FAILED!'

# Get ssh login info via keychain
cmd_preexec    source /root/.keychain/hostname.example.com-sh

Final Note About Keychain and Security

• Cracker with an advanced attacking with deadly coding skills can still get key from memory. However, keychain makes it pretty difficult for normal users and attackers to steal your keys and use it.
• OpenSSH sshd server offers two additional options to protect abuse of keys. First, make sure root login disabled (PermitRootLogin yes). Second, specify which user accounts on the server are allowed to be used for authentication by adding AuthorizedKeysFile %h/.ssh/authorized_keys_FileName. See sshd_config man page for further details.

Suggested Readings:

• man pages sshd, sshd_config, keychain
• rsnapshot MySQL backup script.
• The OpenSSH project official website.
• The Keychain project official website.
• Introducing ssh-agent and keychain by Daniel Robbins
• http://www.cyberciti.biz/faq/ssh-passwordless-login-with-keychain-for-scripts/

sync -av -e 'ssh -p PORT-NUMBER-HERE' /path/to/source user@backup1.example.com

 
rsync -av -e 'ssh -p 5000' /home/vivek backupop@server1.nixcraft.net.in
 

See also:

Sample Setup

eth1:67.1.2.3
                                                +-----------------+
                                                | us.example.com  |
 eth1:202.54.1.2                                +-----------------+
+----------------------+                        |
|  content.example.com |------------------------+
+----------------------+       VPN/intranet     | eth1:87.1.2.3
   /                                            +-----------------+
   /                                            | uk.example.com  |
   |                                            +-----------------+
   +/var/www/corporate_lan/                     |
                          /css/                 | eth1:123.1.2.3
                          /images/              +-----------------+
                          /js/                  | jp.example.com  |
                          /php_cgi/             +-----------------+
                          /perl_cgi/
                          /java_app/
                          /python_app1/

1. All server runs same version of UNIX or Linux and Apache.
2. DocumentRoot is same for all servers.
3. content.example.com - Your main file server. You need to update or upload all static files here only. Do not upload or create files in other servers. You can now push updates or mirror directories from this server to rest of the nodes.
4. us.example.com - Your USA based web server. This server will sync to (or mirror directories from) upstream server called content.example.com.
5. jp.example.com - Your Japan based web server. This server will sync to (or mirror directories from) upstream server called content.example.com.
6. uk.example.com - Your UK web server. This server will sync to (or mirror directories from) upstream server called content.example.com.
7. All offices are connected using secure vpn or an an intranet - a private computer network that uses Internet Protocol technologies to securely share any part of an organization's network operating system.

Solution # 1: Mirroring Using rsync

 
rsync -av /var/www/corporate_lan root@us.example.com:/var/www/
rsync -av /var/www/corporate_lan root@uk.example.com:/var/www/
rsync -av /var/www/corporate_lan root@jp.example.com:/var/www/
 

 
rsync -av /var/www/corporate_lan/css root@us.example.com:/var/www/
rsync -av /var/www/corporate_lan/css root@uk.example.com:/var/www/
rsync -av /var/www/corporate_lan/css root@jp.example.com:/var/www/
 

--delete option

 
rsync -av --delete /var/www/corporate_lan root@us.example.com:/var/www/
rsync -av --delete /var/www/corporate_lan root@uk.example.com:/var/www/
rsync -av --delete /var/www/corporate_lan root@jp.example.com:/var/www/
 

• Recurse into directories
• Copy symlinks as symlinks
• Preserve all file permissions (so make sure you use same usernames on all servers)
• Preserve group file permissions
• Preserve owner file permissions (you need to run rsync as root)
• Preserve times

 
rsync -z -av --delete /var/www/corporate_lan root@us.example.com:/var/www/
 

rsync -z --compress-level=5 -av --delete /var/www/corporate_lan root@us.example.com:/var/www/

Excluding files

rsync -z --compress-level=5 -av --delete --exclude='cache/*' --exclude='*~'  /var/www/corporate_lan root@us.example.com:/var/www/

cache/*
/dev/
/.conf/
*~

rsync -z --compress-level=5 -av --delete --exclude-from=/root/mirror.exclude  /var/www/corporate_lan root@us.example.com:/var/www/

Sample rsync server mirroring shell script

#!/bin/bash
# Usage: Mirror directories and files to our US, UK and Japan based server.
# --------------------------------------------------------------------------
_upstream="/var/www/corporate_lan"
_servers="root@us.example.com:/var/www/ root@uk.example.com:/var/www/ root@jp.example.com:/var/www/"
_rsync="/usr/bin/rsync"
_exclude="/root/mirror.exclude"
_log="/var/log/rsync_mirror.log"
_opts=""
for e in $_servers
do
        [ -f "${_exclude}" ] && _opts="--exclude-from=$_exclude"
        $_rsync -z -a --delete $_opts  "$_upstream" "$e"
done &>$_log
 

@hourly /root/mirror.dirs

How Do I Call /root/mirror.dir As Soon As New Static File Uploaded In /var/www/corporate_lan?

/var/www/corporate_lan/css/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/mirror.dirs
/var/www/corporate_lan/images/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/mirror.dirs
/var/www/corporate_lan/js/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/mirror.dirs

Solution # 2: Mirroring Using unison

Sample unison server mirroring shell script

#!/bin/bash
_paths="/var/www/corporate_lan/css \
/var/www/corporate_lan/images \
/var/www/corporate_lan/js"
_unison=/usr/bin/unison
_rserver="us.example.com uk.example.com jp.example.com"
for p in ${_paths}
do
 ${_unison} -batch "${p}"  "ssh://${_rserver}/${p}"
done

/var/www/corporate_lan/css/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/unison.mirror.sh
/var/www/corporate_lan/images/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/unison.mirror.sh
/var/www/corporate_lan/js/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /root/unison.mirror.sh

Solution #3: Use 3rd Party Cloud Based Solution

Solution # 4: Replicate Content Using 3rd Party Content Delivery Networks (CDN)

Solution #5: Content Replication Using Data Deduplication

Fig.01: Data deduplication using opendedup

Conclusion

1. Almost all replication schema may requires low latency. So I recommend that you test all of the above methods and see what works out for you.
2. Usually, open source tools are good when static files are not updated, uploaded and deleted at rapid rates (e.g. 1000 of files per second).
3. I've also avoided discussion about commercial enterprise grade solution such as NFS over WAN using WAN accelerator such as Riverbed, HP EFS WAN accelerator and others due to cost issues.

#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo DAYNR=${DAYNR}
let "PREVDAYNR = ((DAYNR + 6) % 7)"
echo PREVDAYNR=${PREVDAYNR}
DIRS="home local"

for DIR in ${DIRS}
do
  echo "----- ----- ----- -----"
  echo "Backup:" ${DIR}
  SRCDIR=/backup/cfmc1/$DIR/current/
  DESTDIR=/backup/cfmc1/$DIR/daily.${DAYNR}/
  PREVDIR=/backup/cfmc1/$DIR/daily.${PREVDAYNR}/
  echo SRCDIR=${SRCDIR}
  echo DESTDIR=${DESTDIR}
  echo PREVDIR=${PREVDIR}

  cp -al ${PREVDIR}* ${DESTDIR}
  rsync -a --delete-after ${SRCDIR} ${DESTDIR}

  echo "Done."
done

cygwin1.dll

cygpopt-0.dll

rsync.exe

c:

cd \bin\rsync

rsync --config="c:\bin\rsync\rsyncd.conf" --daemon

1. Download cwRSync, open up the ZIP file, then extract/run cwRsync_x.x.x_Installer.exe.
   
2. Answer "Yes" when asked if you want to continue with the install.
   
3. Answer "Yes" when asked if you want to install cwRSync as a Windows Service.
   
4. Specify the installation folder where you want to install cwRSync. My personal preference is "c:\bin\cwrsync" instead of the default since our servers already have various command line tools installed under c:\bin.
   
5. Enter the account name and password of the local user account that you are going to use for the cwRSync service. It's a good idea to use a seperate account for the cwRSync service, but you may also specify an existing account name.
   
6. The upload area can be set to anything. In fact, you'll probably be removing whatever you set here when you configure your rsyncd.conf file. For now, set it to be a sub-folder under where you installed the cwRSync executables to.
   
7. Click the "Install" button. The installer will then create the folder where cwRSync is being installed to, (optionally) create the user account for the cwRSync service, and it will set restrictive permissions on the install folder so that only the service's user account has rights.
   
8. That takes care of the basics. If you want, view the installation details prior to exiting the install program and cleaning up. Read the instructions on the popup dialog.
   

1. Right-click on My Computer, pick "Manage".
   
2. In the left panel, scroll down and open up the "Services and Applications" tree, then select "Services".
   
3. Locate the "RsyncServer" service and double-click to open up the properties dialog.
   
4. "General" tab: Change the "Startup type" setting to "Automatic".
   
5. "Log On" tab: Re-type the password for the user account that you're using. Click the "Apply" button to save your changes and Windows will popup a notification that the user account has been granted the rights to logon as a service.
   
6. "Recovery" tab: Change these to match your preferences. My personal preference is to restart the service on the first two failures, do nothing on subsequent failures, reseting the fail count after 1 day and restarting the service after a delay of 30 minutes.
   
7. Click "OK" to save and exit.
   
8. Don't start the service yet, the rsyncd.conf file needs to be configured first.
   

[test]

path = /cygdrive/c/cwrsync/data

read only = false

transfer logging = yes

[joes_backup]

path = /cygdrive/e/backup/joe

read only = false

Labels: 2004, RSync

REFERENCES

http://www.tgharold.com/techblog/labels/RSync.shtml

[root@jud sda]# mkdir -p home/ftp

[root@jud ftp]# pwd
/mnt/sda/home/ftp


###source
root@jud vista_temp]# touch derek.test
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:23 derek.test


###vista_temp will be created in destination /mnt/sda/home/ftp/vista_temp
[root@jud sda]# cat rsyncvista_temp.sh
rsync --bwlimit=200 -v -z -r -L -t /home/ftp/vista_temp /mnt/sda/home/ftp >> /mnt/sda/vista_temp.log


###destination
[root@jud ftp]# cd vista_temp/
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19 derek.test



Test 3: add --delete  //delete extraneous files from dest dirs

[root@jud vista_temp]# touch derek.new
[root@jud vista_temp]# rm derek.test
rm: remove regular empty file `derek.test'? y
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19 derek.new



[root@jud sda]# cat rsyncvista_tempdelete.sh
rsync --bwlimit=200 -v -z -r -L -t --delete /home/ftp/vista_temp /mnt/sda/home/ftp >> /mnt/sda/vista_temp.log


###Source intact
[root@jud vista_temp]# pwd
/home/ftp/vista_temp
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19 derek.new
-rw-r--r-- 1 root root 0 Jan 29 11:23 derek.test


###destination updated
[root@jud vista_temp]# pwd
/mnt/sda/home/ftp/vista_temp
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19 derek.new


##check log
[root@jud sda]# cat vista_temp.log
building file list ... done
building file list ... done
vista_temp/
vista_temp/derek.test

sent 112 bytes  received 48 bytes  320.00 bytes/sec
total size is 0  speedup is 0.00
building file list ... done
deleting vista_temp/derek.test
vista_temp/
vista_temp/derek.new

sent 115 bytes  received 48 bytes  326.00 bytes/sec
total size is 0  speedup is 0.00

# man rsync

EXAMPLES
Here are some examples of how I use rsync.

To backup my wifeâs home directory, which consists of large MS Word  files  and  mail
folders, I use a cron job that runs

rsync -Cavz . arvidsjaur:backup

each night over a PPP connection to a duplicate directory on my machine "arvidsjaur".

To synchronize my samba source trees I use the following Makefile targets:

get:
rsync -avuzb --exclude â*~â samba:samba/ .
put:
rsync -Cavuzb . samba:samba/
sync: get put

this allows me to sync with a CVS directory at the other end  of  the  connection.  I
then do CVS operations on the remote machine, which saves a lot of time as the remote
CVS protocol isnât very efficient.

I mirror a directory between my "old" and "new" ftp sites with the command:

rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"

This is launched from cron every few hours.


OPTIONS SUMMARY
Here is a short summary of the options  available  in  rsync.  Please  refer  to  the
detailed description below for a complete description.

-v, --verbose               increase verbosity
-q, --quiet                 suppress non-error messages
--no-motd               suppress daemon-mode MOTD (see caveat)
-c, --checksum              skip based on checksum, not mod-time & size
-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION             turn off an implied OPTION (e.g. --no-D)
-r, --recursive             recurse into directories
-R, --relative              use relative path names
--no-implied-dirs       donât send implied dirs with --relative
-b, --backup                make backups (see --suffix & --backup-dir)
--backup-dir=DIR        make backups into hierarchy based in DIR
--suffix=SUFFIX         backup suffix (default ~ w/o --backup-dir)
-u, --update                skip files that are newer on the receiver
--inplace               update destination files in-place
--append                append data onto shorter files
-d, --dirs                  transfer directories without recursing
-l, --links                 copy symlinks as symlinks
-L, --copy-links            transform symlink into referent file/dir
--copy-unsafe-links     only "unsafe" symlinks are transformed
--safe-links            ignore symlinks that point outside the tree
-k, --copy-dirlinks         transform symlink to dir into referent dir
-K, --keep-dirlinks         treat symlinked dir on receiver as dir
-H, --hard-links            preserve hard links
-p, --perms                 preserve permissions
-E, --executability         preserve executability
--chmod=CHMOD           affect file and/or directory permissions
-A, --acls                  preserve ACLs (implies -p) [non-standard]
-X, --xattrs                preserve extended attrs (implies -p) [n.s.]
-o, --owner                 preserve owner (super-user only)
-g, --group                 preserve group
--devices               preserve device files (super-user only)
--specials              preserve special files
-D                          same as --devices --specials
-t, --times                 preserve times
-O, --omit-dir-times        omit directories when preserving times
--super                 receiver attempts super-user activities
-S, --sparse                handle sparse files efficiently
-n, --dry-run               show what would have been transferred
-W, --whole-file            copy files whole (without rsync algorithm)
-x, --one-file-system       donât cross filesystem boundaries
-B, --block-size=SIZE       force a fixed checksum block-size
-e, --rsh=COMMAND           specify the remote shell to use
--rsync-path=PROGRAM    specify the rsync to run on remote machine
--existing              skip creating new files on receiver
--ignore-existing       skip updating files that exist on receiver
--remove-source-files   sender removes synchronized files (non-dir)
--del                   an alias for --delete-during
--delete                delete extraneous files from dest dirs
--delete-before         receiver deletes before transfer (default)
--delete-during         receiver deletes during xfer, not before
--delete-after          receiver deletes after transfer, not before
--delete-excluded       also delete excluded files from dest dirs
--ignore-errors         delete even if there are I/O errors
--force                 force deletion of dirs even if not empty
--max-delete=NUM        donât delete more than NUM files
--max-size=SIZE         donât transfer any file larger than SIZE
--min-size=SIZE         donât transfer any file smaller than SIZE
--partial               keep partially transferred files
--partial-dir=DIR       put a partially transferred file into DIR
--delay-updates         put all updated files into place at end
-m, --prune-empty-dirs      prune empty directory chains from file-list
--numeric-ids           donât map uid/gid values by user/group name
--timeout=TIME          set I/O timeout in seconds
-I, --ignore-times          donât skip files that match size and time
--size-only             skip files that match in size
--modify-window=NUM     compare mod-times with reduced accuracy
-T, --temp-dir=DIR          create temporary files in directory DIR
-y, --fuzzy                 find similar file for basis if no dest file
--compare-dest=DIR      also compare received files relative to DIR
--copy-dest=DIR         ... and include copies of unchanged files
--link-dest=DIR         hardlink to files in DIR when unchanged
-z, --compress              compress file data during the transfer
--compress-level=NUM    explicitly set compression level
-C, --cvs-exclude           auto-ignore files in the same way CVS does
-f, --filter=RULE           add a file-filtering RULE
-F                          same as --filter=âdir-merge /.rsync-filterâ
repeated: --filter=â- .rsync-filterâ
--exclude=PATTERN       exclude files matching PATTERN
--exclude-from=FILE     read exclude patterns from FILE
--include=PATTERN       donât exclude files matching PATTERN
--include-from=FILE     read include patterns from FILE
--files-from=FILE       read list of source-file names from FILE
-0, --from0                 all *from/filter files are delimited by 0s
--address=ADDRESS       bind address for outgoing socket to daemon
--port=PORT             specify double-colon alternate port number
--sockopts=OPTIONS      specify custom TCP options
--blocking-io           use blocking I/O for the remote shell
--stats                 give some file-transfer stats
-8, --8-bit-output          leave high-bit chars unescaped in output
-h, --human-readable        output numbers in a human-readable format
--progress              show progress during transfer
-P                          same as --partial --progress
-i, --itemize-changes       output a change-summary for all updates
--out-format=FORMAT     output updates using the specified FORMAT
--log-file=FILE         log what weâre doing to the specified FILE
--log-file-format=FMT   log updates using the specified FMT
--password-file=FILE    read password from FILE
--list-only             list the files instead of copying them
--bwlimit=KBPS          limit I/O bandwidth; KBytes per second
--write-batch=FILE      write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating dest
--read-batch=FILE       read a batched update from FILE
--protocol=NUM          force an older protocol version to be used
--checksum-seed=NUM     set block/file checksum seed (advanced)
-4, --ipv4                  prefer IPv4
-6, --ipv6                  prefer IPv6
--version               print version number
(-h) --help                  show this help (see below for -h comment)

Rsync can also be run as a daemon, in which case the following options are accepted:

--daemon                run as an rsync daemon
--address=ADDRESS       bind to the specified address
--bwlimit=KBPS          limit I/O bandwidth; KBytes per second
--config=FILE           specify alternate rsyncd.conf file
--no-detach             do not detach from the parent
--port=PORT             listen on alternate port number
--log-file=FILE         override the "log file" setting
--log-file-format=FMT   override the "log format" setting
--sockopts=OPTIONS      specify custom TCP options
-v, --verbose               increase verbosity
-4, --ipv4                  prefer IPv4
-6, --ipv6                  prefer IPv6
-h, --help                  show this help (if used after --daemon)