Wednesday, January 07, 2009
Very basic rsync / cp backup rotation with hardlinks
Here's a very basic script that I use with RSync that makes use of hard links to reduce the overall size of the backup folder. The limitations are:
- Every morning, a server copies the current version of all files across SSH (using scp) into a "current" folder. There are two folders on the source server that get backed up daily (/home and /local).
- Later on that day, we run the following script to rsync any new files into a daily folder (daily.0 through daily.6).
- In order to bootstrap those daily.# folders, you have to use "cp -al current/* daily.2/" on each, which fills out the seven daily backup folders with hardlinks. Change the number in "daily.2" to 0-6 and run the command once for each of the seven days. Do this after the "current" folder has been populated with data pushed by the source server.
- Ideally, the source server should be pushing changes to the "current" folder using rsync. But in our case, the current server is an old Solaris 9 server without rsync. Which means that our backups are likely to be about 2x to 3x larger then they should be.
- RDiff-Backup may have been a better solution for this particular problem (and we may switch).
- This shows a good example of how to calculate the current day of week number (0-6) as well as calculating what the previous day number was (using modulus arithmetic).
- I make no guarantees that permissions or ownership will be preserved. But since the source server strips all of that information in the process of sending the files over the wire with scp, it's a moot point for our current situation. (rdiff-backup is probably a better choice for that.)
It's not pretty, but it will work better once the source server starts pushing the daily changes via rsync instead of completely overwriting the "current" directory every day.
The code should be pretty self explanatory but I'll explain the two key lines.
cp -al ${PREVDIR}* ${DESTDIR}
This overwrites all files in ${DESTDIR}, which is today, with the files from yesterday, but does it by creating hard links of all files. Old files which were deleted since last week will be left behind until the rsync step.
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
This then brings today's folder up to date with any changes as compared to the source directory (a.k.a. "current"). It also deletes any file in today's folder that don't exist in the source directory.
References:
Easy Automated Snapshot-Style Backups with Linux and Rsync
Local incremental snap shots with rsync
- Every morning, a server copies the current version of all files across SSH (using scp) into a "current" folder. There are two folders on the source server that get backed up daily (/home and /local).
- Later on that day, we run the following script to rsync any new files into a daily folder (daily.0 through daily.6).
- In order to bootstrap those daily.# folders, you have to use "cp -al current/* daily.2/" on each, which fills out the seven daily backup folders with hardlinks. Change the number in "daily.2" to 0-6 and run the command once for each of the seven days. Do this after the "current" folder has been populated with data pushed by the source server.
- Ideally, the source server should be pushing changes to the "current" folder using rsync. But in our case, the current server is an old Solaris 9 server without rsync. Which means that our backups are likely to be about 2x to 3x larger then they should be.
- RDiff-Backup may have been a better solution for this particular problem (and we may switch).
- This shows a good example of how to calculate the current day of week number (0-6) as well as calculating what the previous day number was (using modulus arithmetic).
- I make no guarantees that permissions or ownership will be preserved. But since the source server strips all of that information in the process of sending the files over the wire with scp, it's a moot point for our current situation. (rdiff-backup is probably a better choice for that.)
#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo DAYNR=${DAYNR}
let "PREVDAYNR = ((DAYNR + 6) % 7)"
echo PREVDAYNR=${PREVDAYNR}
DIRS="home local"
for DIR in ${DIRS}
do
echo "----- ----- ----- -----"
echo "Backup:" ${DIR}
SRCDIR=/backup/cfmc1/$DIR/current/
DESTDIR=/backup/cfmc1/$DIR/daily.${DAYNR}/
PREVDIR=/backup/cfmc1/$DIR/daily.${PREVDAYNR}/
echo SRCDIR=${SRCDIR}
echo DESTDIR=${DESTDIR}
echo PREVDIR=${PREVDIR}
cp -al ${PREVDIR}* ${DESTDIR}
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
echo "Done."
done
It's not pretty, but it will work better once the source server starts pushing the daily changes via rsync instead of completely overwriting the "current" directory every day.
The code should be pretty self explanatory but I'll explain the two key lines.
cp -al ${PREVDIR}* ${DESTDIR}
This overwrites all files in ${DESTDIR}, which is today, with the files from yesterday, but does it by creating hard links of all files. Old files which were deleted since last week will be left behind until the rsync step.
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
This then brings today's folder up to date with any changes as compared to the source directory (a.k.a. "current"). It also deletes any file in today's folder that don't exist in the source directory.
References:
Easy Automated Snapshot-Style Backups with Linux and Rsync
Local incremental snap shots with rsync
Monday, August 21, 2006
Rsync and SSH on Windows 2003 Server
Taking another stab at setting up RSync and SSH on our Windows 2003 servers. The goal is that we can upload web files to a central server and then have it synchronize the other servers in the array. Once again, I'm going to use the cwRsync and copSSH packages (latest version is 2.0.9).
Installation on a Windows 2003 Domain Controller:
So now if you look in "Active Directory Users and Computers", there should be a newly created account called "SvcwRsync". Since we are installing this on a domain controller, you should rename this account to "SvcwRsync_SERVERNAME" so that it doesn't cause problems for other installations. You'll also need to change the login details for the "RsyncServer" and "OpenSSH SSHD" services.
Once you have things configured, make sure to go to the Services control and set the services to start up automatically. I also recommend configuring the Recovery tab so that the services are automatically restarted after 2 or 5 minutes.
...
Now to start locking things down. First, I'm going to restrict what interfaces (IP addresses) that the cwRSync service can listen on by adding an address line to rsyncd.conf.
address = 127.0.0.1
One the machine that you will be using to talk to the rsync daemon on the host server, you'll also need the cwRsync tools installed along with OpenSSH. Because the rsync daemon can only listen on 127.0.0.1 (localhost), we'll need to create an SSH tunnel from the client machine to the host server before we can talk to the rsync daemon.
One the client machine:
1. Create a new folder under "C:\Program Files\cwRsyncServer\home" for the new user. In my particular case, I'm calling my user "backuppull" because I am pulling backup files off of the rsync server and down to my local machine.
2. Create a ".ssh" folder under that new home folder.
3. Open up a command window (Start, Run, "cmd") and change directories to the home folder ("C:\Program Files\cwRsyncServer\home\backuppull")
4. Create ssh keys for this user. Since we want to do this sync in a batch file without user-interaction, they'll need to be created with null passwords. You may wish to use the "-b 2048" option to create stronger keys (recommended for RSA, DSA can only be up to 1024 bits).
mkdir .ssh
..\..\ssh-keygen -t rsa -N "" -b 2048 -f .ssh\id_rsa
..\..\ssh-keygen -t dsa -N "" -b 1024 -f .ssh\id_dsa
5. You will now need to transfer the public key files to the host server. Again, you will create a new home directory for the user in the "C:\Program Files\cwRsyncServer\home" folder tree along with creating a ".ssh" folder under that home folder. The two files that need to be copied are:
id_dsa.pub
id_rsa.pub
6. Now append the contents of these files to the ".ssh/authorized_keys" file on the host server.
type id_dsa.pub >> authorized_keys
type id_rsa.pub >> authorized_keys
7. Now to configure SSHD on the host server. You will need to find and edit the sshd_config file (probably in "C:\Program Files\cwRsyncServer\etc"). The following changes should be made in the current version default settings.
PermitRootLogin no
PasswordAuthentication no
Installation on a Windows 2003 Domain Controller:
- Download cwRSync, open up the ZIP file, then extract/run cwRsync_Server_x.x.x_Installer.exe.
- Click "Next" to move past the splash screen
- Click "I Agree" to move past the license screen
- Select both the "Rsync Server" and "OpenSSH Server" (unless you have already installed and configured SSH) then click "Next"
- Choose your installation location, the default is "C:\Program Files\cwRsyncServer"
- Click "Install" to begin the installation process
- cwRsync will install and create a default service account with a randomly generated password.
- Write down the service account password.
- Click "Close" when the install has finished.
So now if you look in "Active Directory Users and Computers", there should be a newly created account called "SvcwRsync". Since we are installing this on a domain controller, you should rename this account to "SvcwRsync_SERVERNAME" so that it doesn't cause problems for other installations. You'll also need to change the login details for the "RsyncServer" and "OpenSSH SSHD" services.
Once you have things configured, make sure to go to the Services control and set the services to start up automatically. I also recommend configuring the Recovery tab so that the services are automatically restarted after 2 or 5 minutes.
...
Now to start locking things down. First, I'm going to restrict what interfaces (IP addresses) that the cwRSync service can listen on by adding an address line to rsyncd.conf.
address = 127.0.0.1
One the machine that you will be using to talk to the rsync daemon on the host server, you'll also need the cwRsync tools installed along with OpenSSH. Because the rsync daemon can only listen on 127.0.0.1 (localhost), we'll need to create an SSH tunnel from the client machine to the host server before we can talk to the rsync daemon.
One the client machine:
1. Create a new folder under "C:\Program Files\cwRsyncServer\home" for the new user. In my particular case, I'm calling my user "backuppull" because I am pulling backup files off of the rsync server and down to my local machine.
2. Create a ".ssh" folder under that new home folder.
3. Open up a command window (Start, Run, "cmd") and change directories to the home folder ("C:\Program Files\cwRsyncServer\home\backuppull")
4. Create ssh keys for this user. Since we want to do this sync in a batch file without user-interaction, they'll need to be created with null passwords. You may wish to use the "-b 2048" option to create stronger keys (recommended for RSA, DSA can only be up to 1024 bits).
mkdir .ssh
..\..\ssh-keygen -t rsa -N "" -b 2048 -f .ssh\id_rsa
..\..\ssh-keygen -t dsa -N "" -b 1024 -f .ssh\id_dsa
5. You will now need to transfer the public key files to the host server. Again, you will create a new home directory for the user in the "C:\Program Files\cwRsyncServer\home" folder tree along with creating a ".ssh" folder under that home folder. The two files that need to be copied are:
id_dsa.pub
id_rsa.pub
6. Now append the contents of these files to the ".ssh/authorized_keys" file on the host server.
type id_dsa.pub >> authorized_keys
type id_rsa.pub >> authorized_keys
7. Now to configure SSHD on the host server. You will need to find and edit the sshd_config file (probably in "C:\Program Files\cwRsyncServer\etc"). The following changes should be made in the current version default settings.
PermitRootLogin no
PasswordAuthentication no
Tuesday, August 09, 2005
More rsync links for using rsync as a backup tool
Easy Automated Snapshot-Style Backups with Linux and Rsync
I'll need to come back and revisit this link, from a glance, it looks very well laid out and will be exactly what I want to pattern my backup systems after.
I'll need to come back and revisit this link, from a glance, it looks very well laid out and will be exactly what I want to pattern my backup systems after.
Tuesday, May 03, 2005
cwRSync and copSSH
Note: These directions are works-in-progress... in fact, they might not even work at all. I got side-tracked before I could finish this and will re-visit it at some point in the future.
The folks who created cwRSync (www.itefix.no) have now released a package called copSSH which is basically SSH for windows and works with cwRSync. I'll be refering back to my old post about installing cwRSync. The latest version I have is from late April 2005 and includes bug fixes for Windows Server 2003.
Also see the rsyncd.conf file for configuring rsync.
These steps are for installing rsync in a server configuration (meaning that it will be listening on the listed ports). Since the install process needs to (optionally) create an user account and create a new service, you'll need administrative access to the machine that you are using. (I'm not sure whether members of the Power Users group have enough privileges.)
Now we need to install copSSH.
The folks who created cwRSync (www.itefix.no) have now released a package called copSSH which is basically SSH for windows and works with cwRSync. I'll be refering back to my old post about installing cwRSync. The latest version I have is from late April 2005 and includes bug fixes for Windows Server 2003.
Also see the rsyncd.conf file for configuring rsync.
These steps are for installing rsync in a server configuration (meaning that it will be listening on the listed ports). Since the install process needs to (optionally) create an user account and create a new service, you'll need administrative access to the machine that you are using. (I'm not sure whether members of the Power Users group have enough privileges.)
- Download cwRSync, open up the ZIP file, then extract/run cwRsync_x.x.x_Installer.exe.
- Click "Next" to begin the install.
- Read and agree to the licence.
- Make sure that both the client and server components are checked off and click "Next".
- Choose your installation location. I prefer to put mine in a custom location (C:\bin\cwRsync).
- Click "Install" to begin the installation.
- The default user account is "cwrsync" (with a random password) and it will be installed as a service. You will probably want to change the password to something stronger and adjust the properties of the service in Computer Management. Specifically, I changed the Recovery tab to auto-restart the service after 5 minutes if it dies. I've left the "auto-start" setting to "manual" until I've finished configuration and testing.
- By default, the newly created "cwRSync" folder grants permissions to the Administrators group (full control), the CWRSYNC user account (full control) and the Users account (read/execute).
- Now you should configure your rsyncd.conf file.
Now we need to install copSSH.
- Download copSSH, open up the ZIP file, then extract/run copSSH_x.x.x_Installer.exe.
- Click "Next" to begin the install.
- Read and agree to the licence.
- Change the install folder to match where you installed cwRSync (C:\bin\cwRsync). (This is according to the FAQ on the itefix.no web site.)
- This creates a new service called "OpenSSH SSHD" with a default users account of "SvcCOPSSH"
- You will probably want to change the password to something stronger and adjust the properties of the service in Computer Management. Specifically, I changed the Recovery tab to auto-restart the service after 5 minutes if it dies. I've changed the "auto-start" setting to "manual" until I've finished configuration and testing.
- Notice that the copSSH installation blows away existing permissions on the c:\bin\cwRSync folder. This may require fixing (I have to test first).
- Re-start the SSHD service in manual mode (if you stopped it earlier).
Wednesday, July 21, 2004
Minimal Cygwin install for RSync and SSH
Source links:
How to setup the secure shell daemon on a Windows 2000 machine?
Windows Rsync Server Setup
CygwinInstallationGuide (a wiki topic about the cygwin installation)
Note: The following probably doesn't work (probably missing a package, or the fact that I have GNU's unix tools for Win32 installed is problematic), but I might come back and make it work later so I'm leaving it here for now. I ran into trouble when trying to configure SSH. Right now, I've gone back to my original plan of either hacking apart the Cygwin files and manually copying only the DLLs and EXEs that I need or using the OpenSSH for Windows project at SourceForge.
1. Run the Cygwin setup.exe file and start the instllation. I chose to install to "c:\bin\cygwin", but left the rest of the options "as-is". Pick your mirror (use the Cygwin public mirrors page to find one close to you).
2. On the "Select Packages" screen, select the "Curr" option and make sure it says "Category" next to the "View" button at the top. The installation dialog is (finally) re-sizeable, so stretch it out or maximize it so you can see all of the columns.
3. Beside the "+All" category, it will say "Install", "Uninstall", ... click on the word until all of the categories say "Uninstall". (Note: These steps assume that you're doing a new Cygwin install and that you don't already have Cygwin installed.) Now we can start picking the minimum number of packages required to setup SSH and RSync.
4a. Under the "+Admin" category, you'll need to install the "cygrunsrv" package (click once on the "Skip" indicator under the "New" column). This will turn on a few other packages that this package depends on (mostly under the "+Base", "+Libs", and "+Shells" categories).
4b. Open up the "+Net" category and select the "rsync" and "openssh" packages. You'll also end up with "openssl" which is required in order to use "openssh".
5. Click the "Next" button to start downloading and installing the packages. If the download fails, choose another mirror, double-check your package selections (my copy remembered which packages I had already selected), and try again. The base install size required around 7MB of downloads and expanded out to 24MB (34MB actual due to a 4KB cluster size).
6. Fire up the cygwin shell, you should see a command-line window open with a "$" prompt. Try out a few unix commands (pwd, ls, whoami) to see if things are working.
7. Further steps... (I'll cover these in future posts)
a) Setup your rsync.conf file (in the "etc" folder)
b) create a service account for use by the rsync service
c) create a Windows service using the "cygrunsvc" tool
d) setup OpenSSH and then re-configure rsync to use it
How to setup the secure shell daemon on a Windows 2000 machine?
Windows Rsync Server Setup
CygwinInstallationGuide (a wiki topic about the cygwin installation)
Note: The following probably doesn't work (probably missing a package, or the fact that I have GNU's unix tools for Win32 installed is problematic), but I might come back and make it work later so I'm leaving it here for now. I ran into trouble when trying to configure SSH. Right now, I've gone back to my original plan of either hacking apart the Cygwin files and manually copying only the DLLs and EXEs that I need or using the OpenSSH for Windows project at SourceForge.
1. Run the Cygwin setup.exe file and start the instllation. I chose to install to "c:\bin\cygwin", but left the rest of the options "as-is". Pick your mirror (use the Cygwin public mirrors page to find one close to you).
2. On the "Select Packages" screen, select the "Curr" option and make sure it says "Category" next to the "View" button at the top. The installation dialog is (finally) re-sizeable, so stretch it out or maximize it so you can see all of the columns.
3. Beside the "+All" category, it will say "Install", "Uninstall", ... click on the word until all of the categories say "Uninstall". (Note: These steps assume that you're doing a new Cygwin install and that you don't already have Cygwin installed.) Now we can start picking the minimum number of packages required to setup SSH and RSync.
4a. Under the "+Admin" category, you'll need to install the "cygrunsrv" package (click once on the "Skip" indicator under the "New" column). This will turn on a few other packages that this package depends on (mostly under the "+Base", "+Libs", and "+Shells" categories).
4b. Open up the "+Net" category and select the "rsync" and "openssh" packages. You'll also end up with "openssl" which is required in order to use "openssh".
5. Click the "Next" button to start downloading and installing the packages. If the download fails, choose another mirror, double-check your package selections (my copy remembered which packages I had already selected), and try again. The base install size required around 7MB of downloads and expanded out to 24MB (34MB actual due to a 4KB cluster size).
6. Fire up the cygwin shell, you should see a command-line window open with a "$" prompt. Try out a few unix commands (pwd, ls, whoami) to see if things are working.
7. Further steps... (I'll cover these in future posts)
a) Setup your rsync.conf file (in the "etc" folder)
b) create a service account for use by the rsync service
c) create a Windows service using the "cygrunsvc" tool
d) setup OpenSSH and then re-configure rsync to use it
Hacking together a minimal rsync for windows installation
Based on what I've read elsewhere (links in my previous posting), I think I can pull the relevant pieces out of the Cygwin package. I'll try to keep good notes as to what worked and what didn't, but let me know if you find any errors. Rsync wrapper for Win32 seems to be a good starting point for which DLLs and files I'll need to pull out of the standard Cygwin release.
You can download the files off of any of the Cygwin public mirrors. Grab the following archives and extract them to a temporary directory on your machine.
release/cygwin/cygwin-1.5.10-3.tar.bz2
- contains the DLL file (usr/bin/cygwin1.dll) and a lot of base utilities
release/popt/libpopt0/libpopt0-1.6.4-4.tar.bz2
- contains the usb/bin/cygpopt-0.dll file
release/rsync/rsync-2.6.2-1.tar.bz2
- RSync (rsync executable)
Create a folder where you're going to store the rsync files (I use C:\bin\rsync).
Copy the following files to your rsync folder:
Create your rsync.conf file and put it in your rsync folder.
Test out whether you've gotten rsync working (thanks to "Aaron Johnson's page about rsync" for showing me what command line options to use). To do this, type the following commands:
If you have a log file, there should now be an entry indicating that rsync has started up and is listening on the default port (tcp/873). Looking at the processes in Windows Task Manager, you should see the "rsync.exe" process. You should also now test out some rsync transfers from another workstation to verify that your security settings and module settings are correct.
To do:
- create the user account to use for the rsync service
- setup rsync to run as a service (need the SRVANY.EXE file, I think)
- figure out how to get rsync talking through an SSHD server
You can download the files off of any of the Cygwin public mirrors. Grab the following archives and extract them to a temporary directory on your machine.
release/cygwin/cygwin-1.5.10-3.tar.bz2
- contains the DLL file (usr/bin/cygwin1.dll) and a lot of base utilities
release/popt/libpopt0/libpopt0-1.6.4-4.tar.bz2
- contains the usb/bin/cygpopt-0.dll file
release/rsync/rsync-2.6.2-1.tar.bz2
- RSync (rsync executable)
Create a folder where you're going to store the rsync files (I use C:\bin\rsync).
Copy the following files to your rsync folder:
cygwin1.dll
cygpopt-0.dll
rsync.exe
Create your rsync.conf file and put it in your rsync folder.
Test out whether you've gotten rsync working (thanks to "Aaron Johnson's page about rsync" for showing me what command line options to use). To do this, type the following commands:
c:
cd \bin\rsync
rsync --config="c:\bin\rsync\rsyncd.conf" --daemon
If you have a log file, there should now be an entry indicating that rsync has started up and is listening on the default port (tcp/873). Looking at the processes in Windows Task Manager, you should see the "rsync.exe" process. You should also now test out some rsync transfers from another workstation to verify that your security settings and module settings are correct.
To do:
- create the user account to use for the rsync service
- setup rsync to run as a service (need the SRVANY.EXE file, I think)
- figure out how to get rsync talking through an SSHD server
RSync and Windows
This is a follow-up to my previous post about Securing cwRSync. We were using the "cwRSync package", but when running in server mode it doesn't know how to talk to clients over an SSH-encrypted connection. Which isn't a big deal if you're only talking to other servers on the local network, but is problematic in cases where you have to be wary of eavesdropping (across WiFi links or untrusted networks like the internet). So I've been looking off-and-on over the past month at figuring out how to get an rsync service running using SSH on a Windows server.
One option is to install the full Cygwin package. Which is a bit much for a server (or rather, I'm not comfortable installing Cygwin on a server... yet).
Another option seems to be the OpenSSH for Windows project at SourceForge. That doesn't include rsync though, just scp. So I might look at "Installing ssh and rsync on a Windows machine: minimalist approach" which requires an absolute bare minimum of files to be installed. However, the files at that location are from Jan 2002, which is a bit old and the latest version as of July 2004 for the Cygwin DLL is cygwin-1.5.10-2.
One option is to install the full Cygwin package. Which is a bit much for a server (or rather, I'm not comfortable installing Cygwin on a server... yet).
Another option seems to be the OpenSSH for Windows project at SourceForge. That doesn't include rsync though, just scp. So I might look at "Installing ssh and rsync on a Windows machine: minimalist approach" which requires an absolute bare minimum of files to be installed. However, the files at that location are from Jan 2002, which is a bit old and the latest version as of July 2004 for the Cygwin DLL is cygwin-1.5.10-2.
Friday, June 18, 2004
Securing cwRSync
At the office we're working on setting up cwRSync on the web server array to push the daily web/ftp/smtp log files back to a central point for archiving. Right now, since all of the web servers are on the same LAN segment at the hosting facility, we're just sending the plain text data across the wire to the rsync port (tcp/873). Since the previous solution was to use FTP to move the log files around, it's no worse then the old solution from a security standpoint. (It is, however, much faster and more efficient.) Security is handled solely thorugh the rsyncd.conf "hosts allow" setting (only the internal IP addresses are allowed to be used to transfer the data) with no passwords or shared keys.
However, since the next step is that we want to setup pulling those log files automatically back to the main office, we need to look into locking it down further and putting encryption in place (e.g. routing rsync traffic over an ssh tunnel).
After digging around a bit here's what I've found:
The cwRSync Service does not support SSH, so there's no way to connect securely to a rsync server that is using cwRSync as its daemon. Future releases are expected to add ssh support for cwRSync servers. Locking down through IP address and username/password is the limit of what you can do for security, all traffic is in the clear (unless you have IPSec between the two machines).
However, you can use cwRSync in a client-configuration and route the traffic over SSH to a SSH-capable rsync server.
That being said, I'm going to explore some other packages. All of which will either require that cygwin be installed, or at least that certain cygwin DLLs be installed.
Links:
Rsync wrapper for Win32 - Uses the cygwin DLLs, but doesn't require a full cygwin install, includes SSH.
However, since the next step is that we want to setup pulling those log files automatically back to the main office, we need to look into locking it down further and putting encryption in place (e.g. routing rsync traffic over an ssh tunnel).
After digging around a bit here's what I've found:
The cwRSync Service does not support SSH, so there's no way to connect securely to a rsync server that is using cwRSync as its daemon. Future releases are expected to add ssh support for cwRSync servers. Locking down through IP address and username/password is the limit of what you can do for security, all traffic is in the clear (unless you have IPSec between the two machines).
However, you can use cwRSync in a client-configuration and route the traffic over SSH to a SSH-capable rsync server.
That being said, I'm going to explore some other packages. All of which will either require that cygwin be installed, or at least that certain cygwin DLLs be installed.
Links:
Rsync wrapper for Win32 - Uses the cygwin DLLs, but doesn't require a full cygwin install, includes SSH.
Thursday, June 10, 2004
Installing cwRSync on Windows 2000
The instructions over at cwRSync's install page are a bit vague, so I'm going to jot down the steps that I use. These steps are for installing rsync in a server configuration. Since the install process needs to (optionally) create an user account and create a new service, you'll need administrative access to the machine that you are using. (I'm not sure whether members of the Power Users group have enough privileges.)- Download cwRSync, open up the ZIP file, then extract/run cwRsync_x.x.x_Installer.exe.
- Answer "Yes" when asked if you want to continue with the install.
- Answer "Yes" when asked if you want to install cwRSync as a Windows Service.
- Specify the installation folder where you want to install cwRSync. My personal preference is "c:\bin\cwrsync" instead of the default since our servers already have various command line tools installed under c:\bin.
- Enter the account name and password of the local user account that you are going to use for the cwRSync service. It's a good idea to use a seperate account for the cwRSync service, but you may also specify an existing account name.
- The upload area can be set to anything. In fact, you'll probably be removing whatever you set here when you configure your rsyncd.conf file. For now, set it to be a sub-folder under where you installed the cwRSync executables to.
- Click the "Install" button. The installer will then create the folder where cwRSync is being installed to, (optionally) create the user account for the cwRSync service, and it will set restrictive permissions on the install folder so that only the service's user account has rights.
- That takes care of the basics. If you want, view the installation details prior to exiting the install program and cleaning up. Read the instructions on the popup dialog.
Next, we need to finish setting up the RSync service in Windows.
- Right-click on My Computer, pick "Manage".
- In the left panel, scroll down and open up the "Services and Applications" tree, then select "Services".
- Locate the "RsyncServer" service and double-click to open up the properties dialog.
- "General" tab: Change the "Startup type" setting to "Automatic".
- "Log On" tab: Re-type the password for the user account that you're using. Click the "Apply" button to save your changes and Windows will popup a notification that the user account has been granted the rights to logon as a service.
- "Recovery" tab: Change these to match your preferences. My personal preference is to restart the service on the first two failures, do nothing on subsequent failures, reseting the fail count after 1 day and restarting the service after a delay of 30 minutes.
- Click "OK" to save and exit.
- Don't start the service yet, the rsyncd.conf file needs to be configured first.
You need to configure the rsyncd.conf file and set up your first "module" (a.k.a. a share path). Find your rsyncd.conf file (it's in the folder where you installed cwRSync to) and open it up in a text editor (NotePad works). Now, go read the official rsyncd.conf help page. Read it twice if it's your first time, because it's possible to put a very large gaping security hole into your setup if you're not careful. The default settings at the top of the file are fine, but you may wish to change the "hosts allow = *" to "hosts allow = (your client machine IPs)" as a preventative first step. Then, even if you screw up the other security mechanisms, you've at least limited which IP addresses an attacker can base an attack from. (You can test this by telnet'ing to port 873 and seeing whether the rsync service drops your connection.)
Next, we need to start setting up "modules" in the rsyncd.conf file. "Modules" are basically the same concept as a Windows share, except that you have to use rsync to access the files within the "module". Ignore what it says on the cwRSync install page about rsync modules having to be sub-directories under the cwrsync folder. If you grant correct directory permissions to the cwRSync service account, then the service daemon will be able to read or read/write to the target folders without problems.
The default module installed is called "test". Go ahead and comment it out with '#' symbols and save the file. From my (limited) testing, it does not appear to be necessary to restart the rsync service in order for it to see changes in the rsyncd.conf file.
[test]
path = /cygdrive/c/cwrsync/data
read only = false
transfer logging = yes
There are two basic ways to use rsync and this will affect how you grant permissions to the rsync service account.
The first is a read-only ("pull") setup, where the clients can only pull files from the rsync server. The rsync service account should only have Read & Execute / List Folder Contents / Read permissions for the folder tree that you are going to publish. In addition, when you setup your module in the configuration file, you should specify "read only = true" as a setting.
The second is a "push" setup where clients are writing changes to the rsync server. The rsync service account will require "modify" permissions for the shared directory tree. Under your module configuration section in the rsyncd.conf file, a "push" setup must have "read only = false".
Now, for every directory tree on the rsync server that you wish to share, create a new module section (e.g. "[logs]" or "[web]" or "[joes_backup]"). Verify that the cwRSync service account has proper permissions to the file system tree. Then add the following options (at a minimum) below the module section name:
[joes_backup]
path = /cygdrive/e/backup/joe
read only = false
That allows any client who manages to authenticate with the rsync service to write the E:\Backup\Joe on the rsync server. That is not exactly secure and you should take additional steps to lock it down through the use of "hosts allow", "auth users", "secrets file" and perhaps ssh. Securing your box is a bit beyond the scope of this post. It's also a bit beyond my experience level since I'm just getting started with rsync.
(Update: See Securing cwRSync.)
