Friday, August 21, 2009

rsync test tutorial

SkyHi @ Friday, August 21, 2009
###initialize this script on the destination host
#rsync -u -v -r --bwlimit=2000 root@backupserver:/backup2/ /mnt/sda/

##Archive 2 directories(html and test1) to /mnt/sda
#cd /mnt/sda
#tar cvf /mnt/sda/backup2009.rar /var/www/html /var/www/test1


[root@jud sda]# mkdir -p home/ftp

[root@jud ftp]# pwd

root@jud vista_temp]# touch derek.test
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:23 derek.test

###vista_temp will be created in destination /mnt/sda/home/ftp/vista_temp
[root@jud sda]# cat
rsync --bwlimit=200 -v -z -r -L -t /home/ftp/vista_temp /mnt/sda/home/ftp >> /mnt/sda/vista_temp.log

[root@jud ftp]# cd vista_temp/
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19 derek.test

Test 3: add --delete //delete extraneous files from dest dirs

[root@jud vista_temp]# touch
[root@jud vista_temp]# rm derek.test
rm: remove regular empty file `derek.test'? y
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19

[root@jud sda]# cat
rsync --bwlimit=200 -v -z -r -L -t --delete /home/ftp/vista_temp /mnt/sda/home/ftp >> /mnt/sda/vista_temp.log

###Source intact
[root@jud vista_temp]# pwd
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19
-rw-r--r-- 1 root root 0 Jan 29 11:23 derek.test

###destination updated
[root@jud vista_temp]# pwd
[root@jud vista_temp]# ll
total 0
-rw-r--r-- 1 root root 0 Jan 29 11:19

##check log
[root@jud sda]# cat vista_temp.log
building file list ... done
building file list ... done

sent 112 bytes received 48 bytes 320.00 bytes/sec
total size is 0 speedup is 0.00
building file list ... done
deleting vista_temp/derek.test

sent 115 bytes received 48 bytes 326.00 bytes/sec
total size is 0 speedup is 0.00

# man rsync

Here are some examples of how I use rsync.

To backup my wifeâs home directory, which consists of large MS Word files and mail
folders, I use a cron job that runs

rsync -Cavz . arvidsjaur:backup

each night over a PPP connection to a duplicate directory on my machine "arvidsjaur".

To synchronize my samba source trees I use the following Makefile targets:

rsync -avuzb --exclude â*~â samba:samba/ .
rsync -Cavuzb . samba:samba/
sync: get put

this allows me to sync with a CVS directory at the other end of the connection. I
then do CVS operations on the remote machine, which saves a lot of time as the remote
CVS protocol isnât very efficient.

I mirror a directory between my "old" and "new" ftp sites with the command:

rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"

This is launched from cron every few hours.

Here is a short summary of the options available in rsync. Please refer to the
detailed description below for a complete description.

-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs donât send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place
--append append data onto shorter files
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the tree
-k, --copy-dirlinks transform symlink to dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-E, --executability preserve executability
--chmod=CHMOD affect file and/or directory permissions
-A, --acls preserve ACLs (implies -p) [non-standard]
-X, --xattrs preserve extended attrs (implies -p) [n.s.]
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve times
-O, --omit-dir-times omit directories when preserving times
--super receiver attempts super-user activities
-S, --sparse handle sparse files efficiently
-n, --dry-run show what would have been transferred
-W, --whole-file copy files whole (without rsync algorithm)
-x, --one-file-system donât cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver
--remove-source-files sender removes synchronized files (non-dir)
--del an alias for --delete-during
--delete delete extraneous files from dest dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during xfer, not before
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from dest dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of dirs even if not empty
--max-delete=NUM donât delete more than NUM files
--max-size=SIZE donât transfer any file larger than SIZE
--min-size=SIZE donât transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at end
-m, --prune-empty-dirs prune empty directory chains from file-list
--numeric-ids donât map uid/gid values by user/group name
--timeout=TIME set I/O timeout in seconds
-I, --ignore-times donât skip files that match size and time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare received files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
-C, --cvs-exclude auto-ignore files in the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter=âdir-merge /.rsync-filterâ
repeated: --filter=â- .rsync-filterâ
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN donât exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *from/filter files are delimited by 0s
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what weâre doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating dest
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
--checksum-seed=NUM set block/file checksum seed (advanced)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (see below for -h comment)

Rsync can also be run as a daemon, in which case the following options are accepted:

--daemon run as an rsync daemon
--address=ADDRESS bind to the specified address
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--config=FILE specify alternate rsyncd.conf file
--no-detach do not detach from the parent
--port=PORT listen on alternate port number
--log-file=FILE override the "log file" setting
--log-file-format=FMT override the "log format" setting
--sockopts=OPTIONS specify custom TCP options
-v, --verbose increase verbosity
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
-h, --help show this help (if used after --daemon)