Imagine this scenario: your company is growing rapidly and you’re hiring tons of engineers. The passwords to many of your servers are stored in plaintext in configuration files — everyone has access to them. Your main database – the one with all the user data – is available to anyone who has that file.
You fire an engineer. Now what? He knows the passwords to everything. Do you trust him? What if you’re firing him because he’s just a bad egg? What can you do?
Well, you could change the passwords on every single server… but that’s a huge pain in the ass for everyone involved. Luckily, there’s a solution for this problem, one that comes with quite a few other benefits as well: you set up a VPN.
Each computer that has access to the VPN is a “client”, and they each have their own certificate. If you want to remove someone’s access to your network, you just revoke their certificate. It only takes a few seconds, and boom – problem solved. The engineer you fired can’t access the MySQL server even if he knows the password because it’s only accessible via the VPN.
Servers hosted at Rackspace come with two interfaces by default: the Public interface (eth0), which is accessible from the internet, and the Private interface (eth1, also known as Rackspace ServiceNet) that is used from intercommunication between your own servers. Bandwidth on the private interface is unmetered (read: free) and actually uses a whole different ethernet port than the public interface.
After accomplishing (2) our infrastructure should resemble this diagram:
Basically, your whole setup will be walled off from the internet except for the servers that really need to be able to communicate with the outside world, namely the vpn server and your actual web servers. All communication between servers should happen over the private network, since it’s both cheaper and more secure.
http://openvpn.net/index.php/open-source/documentation/howto.html
2. Copy the examples folder to /etc/openvpn/:
First we need to build the key signing infrastructure and the initial server keys. (Tip: do this as root, otherwise too many sudo)
Just run the following command once for each client, replacing clientname with the name of your choice
Before we go any further, let’s set up the server.conf – create a server.conf file at /etc/openvpn/server.conf and paste in the following code:
To do that, you need to set up IP forwarding – this is required to route traffic to other nodes:
The important one is:
You fire an engineer. Now what? He knows the passwords to everything. Do you trust him? What if you’re firing him because he’s just a bad egg? What can you do?
Well, you could change the passwords on every single server… but that’s a huge pain in the ass for everyone involved. Luckily, there’s a solution for this problem, one that comes with quite a few other benefits as well: you set up a VPN.
What is OpenVPN?
OpenVPN is open-source software that allows you to set up a Virtual Private Network. Basically, it allows you to treat remote servers as if they were on your internal network. This gives you fine-grained control over everyone who has access to your systems.Each computer that has access to the VPN is a “client”, and they each have their own certificate. If you want to remove someone’s access to your network, you just revoke their certificate. It only takes a few seconds, and boom – problem solved. The engineer you fired can’t access the MySQL server even if he knows the password because it’s only accessible via the VPN.
The Situation
At Mixpanel we use the Rackspace Cloud for hosting, so this guide (as you may have guessed from the title) will be written from that perspective.Servers hosted at Rackspace come with two interfaces by default: the Public interface (eth0), which is accessible from the internet, and the Private interface (eth1, also known as Rackspace ServiceNet) that is used from intercommunication between your own servers. Bandwidth on the private interface is unmetered (read: free) and actually uses a whole different ethernet port than the public interface.
The Plan
Our first goal in setting up the server is to secure the codebase. To do this we need to do two things:- Set up the OpenVPN server and generate clients
- Update the firewall on your other servers to block external connections and only allow connections via OpenVPN or the local network
Setting up the server
Helpful links:
The OpenVPN HOWTO is quite comprehensive and useful. It should be the first thing you read:http://openvpn.net/index.php/open-source/documentation/howto.html
If you get stuck, this wiki is helpful – particularly these pages:
Setup
1. Install OpenVPN:sudo aptitude install openvpnThis will put a bunch of files in /usr/share/doc/openvpn/, but you should make a copy so that your modifications don’t get lost in a software update (see next step)
2. Copy the examples folder to /etc/openvpn/:
cd /usr/share/doc/openvpn/examples && sudo cp * /etc/openvpn/3. Follow the HOWTO. It will walk you through the rest of the steps, which I will go over briefly.
First we need to build the key signing infrastructure and the initial server keys. (Tip: do this as root, otherwise too many sudo)
cd /etc/openvpn/easy-rsa/2.0 vim vars # edit vars at the bottom to say whatever . ./vars # Source those vars you just edited ./clean-all # Remove any existing keys (none if this is fresh, but do it anyway) ./build-ca # Build the certificate authority key/cert ./build-key-server server # Build server.key and server.crt ./build-dh # Still not sure what this is, Diffie Hellman parameters. Apparently required.Then we can build client keys – basically one for each computer that will connect to the VPN is a client. I plan on using employee emails as client names, but you can use any unique naming scheme.
Just run the following command once for each client, replacing clientname with the name of your choice
./build-key clientname # This will save files in /etc/openvpn/easy-rsa/2.0/keysYour server needs to retain clientname.crt, and the client needs both clientname.key and clientname.crt to be able to connect.
Before we go any further, let’s set up the server.conf – create a server.conf file at /etc/openvpn/server.conf and paste in the following code:
port 1194 proto udp dev tun # Keys we generated earlier ca /etc/openvpn/easy-rsa/2.0/keys/ca.crt cert /etc/openvpn/easy-rsa/2.0/keys/server.crt key /etc/openvpn/easy-rsa/2.0/keys/server.key # This file should be kept secret dh /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem # This will be the internal tun0 connection IP - choose whatever you want server 10.37.73.0 255.255.255.0 ifconfig-pool-persist ipp.txt # This will send all of a client's 10.x.x.x traffic through the VPN push "route 10.0.0.0 255.0.0.0" keepalive 10 120 # Compression - MUST be turned on at both ends. Should be an option on client side as well comp-lzo persist-key persist-tun # Prevent revoked certificates from accessing vpn crl-verify easy-rsa/2.0/keys/crl.pem status log/openvpn-status.log # Verbose, good for testing. Switch to 3 in production. verb 6Next, try running the server – sudo openvpn server.conf. It should run, and you should be able to connect from a client, but it won’t forward traffic across the network yet.
To do that, you need to set up IP forwarding – this is required to route traffic to other nodes:
# temporary solution echo 1 > /proc/sys/net/ipv4/ip_forward # permanent solution vim /etc/sysctl.conf # uncomment net.ipv4.ip_forward = 1The final step is to set up your iptables rules for the vpn server. The most critical rule (and a curiously underdocumented one) applies to the NAT table – it will rewrite the packets that come through the vpn with the vpn’s ip address before passing them along. If you don’t do this, the origin address on the packets is 10.37.73.XX, which the other servers on your network don’t know how to handle.
The important one is:
# Masquerade local subnet - run this on your VPN server. iptables -t nat -A POSTROUTING -s $PRIVATE -o eth1 -j MASQUERADEHere’s a sample iptables ruleset for your VPN server:
* filter # Set default policies -P OUTPUT ACCEPT -P INPUT DROP -P FORWARD DROP # Prevent external packets from using loopback addr -A INPUT -i eth0 -s 127.0.0.1 -j DROP -A FORWARD -i eth0 -s 127.0.0.1 -j DROP -A INPUT -i eth0 -d 127.0.0.1 -j DROP -A FORWARD -i eth0 -d 127.0.0.1 -j DROP # Check source address validity on packets going out to internet -A FORWARD -s ! 10.0.0.0/8 -i eth1 -j DROP # Allow local loopback -A INPUT -s 127.0.0.1 -j ACCEPT -A INPUT -d 127.0.0.1 -j ACCEPT # Allow incoming pings (can be disabled) -A INPUT -p icmp --icmp-type echo-request -j ACCEPT # Allow services such as www and ssh (can be disabled) -A INPUT -p tcp --dport ssh -j ACCEPT # Allow incoming OpenVPN packets -A INPUT -p udp --dport 1194 -j ACCEPT -A INPUT -i tun+ -j ACCEPT -A FORWARD -i tun+ -j ACCEPT # Allow packets from private subnets -A INPUT -i eth1 -j ACCEPT -A FORWARD -i eth1 -j ACCEPT # Keep state of connections from local machine and private subnets -A OUTPUT -m state --state NEW -o eth0 -j ACCEPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A FORWARD -m state --state NEW -o eth0 -j ACCEPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT # log iptables denied calls -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7 # Reject all other inbound - default deny unless explicitly allowed policy -A INPUT -j REJECT -A FORWARD -j REJECT COMMIT * nat # Masquerade local subnet -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE COMMITIf you save this file as /etc/iptables.up.rules on the vpn server, and call
sudo iptables-restore < /etc/iptables.up.rules then you should have a fully functional vpn server. If you connect to it, you should be able to ping your other servers by their internal ip addresses from your local computer.At this point we don’t have the rest of the network blocked off, but that’s the next step. Broadly, you will want to update the iptables firewall on all of your servers to restrict to a set of IP addresses that you own. The rules will look something like this:
# Only allow eth1 from these IPs. -A INPUT -i eth1 -s 10.177.160.190 -j ACCEPT -A INPUT -i eth1 -s 10.176.143.1 -j ACCEPT # ... continue, 1 line per IP # Reject all other ips -A INPUT -i eth1 -j REJECTI’ll go into more detail on this part of the process in my next post. For now, you should enjoy your fully functioning OpenVPN server!
REFERENCES
http://code.mixpanel.com/openvpn-in-the-rackspace-cloud/
