Showing posts with label Reverse Proxy. Show all posts
Showing posts with label Reverse Proxy. Show all posts

Monday, March 7, 2011

An Nginx load balancing, caching, reverse proxy

— SkyHi @ Monday, March 07, 2011
Continuing the evaluation of clustering our main website on Linux KVM virtual machines, below is our test nginx reverse proxy cache config.

The back end is 2 Apache servers, one on the local host and one on a remote host.

The main site files live on NFS. We set up the cache file system in tmpfs. Our site is dynamic using PHP for most pages. The caching appears to eliminate most of the PHP overhead and keeps the number of Apache processes at bay, reducing overall RAM needed on the VM's. We think this configuration will allow us to use 2 VM's and still end up with fewer resources needed than our old dual CPU 3GB RAM physical web server.

Our nginx disk cache file system was set to 50 Meg for testing.

/etc/fstab:

tmpfs /var/lib/nginx tmpfs size=50M,uid=33 0 0

We used the main nginx config file to define default caching parameters.

/etc/nginx/nginx.conf:

# Two processes work well for a single CPU
  user www-data;
  worker_processes  2;

  error_log  /var/log/nginx/error.log;
  pid        /var/run/nginx.pid;

  events {
     worker_connections  1024;
     use epoll;
  }

  http {
    include       /etc/nginx/mime.types;

    # Nginx does the logging
    access_log /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    server_names_hash_bucket_size 64;

    # Have nginx do the compression, turn off Apache's mod_deflate
    gzip                on;
    gzip_comp_level     1;
    gzip_disable        msie6;
    gzip_proxied        any;

    # text/html mime type is automatically included for gzip, have to add the rest
    gzip_types          text/plain text/css application/x-javascript text/xml application/xml application/rss+xml text/javascript;

    # Default cache parameters for use by virtual hosts
    # Set the cache path to tmpfs mounted disk, and the zone name
    # Set the maximum size of the on disk cache to less than the tmpfs file system size
    proxy_cache_path  /var/lib/nginx/cache  levels=1:2  keys_zone=adams:10m max_size=45m;
    proxy_temp_path   /var/lib/nginx/proxy;

    # Putting the host name in the cache key allows different virtual hosts to share the same cache zone
    proxy_cache_key "$scheme://$host$request_uri";
    proxy_redirect off;

    # Pass some client identification headers back to Apache  
    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

    # Cache different return codes for different lengths of time 
    # We cached normal pages for 10 minutes
    proxy_cache_valid 200 302  10m;
    proxy_cache_valid 404      1m;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
  }


Following is an nginx virtual host config with the reverse proxy cache enabled. The host name of the server is megalon-test, the website virtual host name is adams-bal. The proxy also handles SSL, and can selectively decide what to cache and for how long.

/etc/nginx/sites-enabled/adams-bal:
# The Apache load balancing proxy targets for port 80 traffic
  upstream adams-bal {
    server 127.0.0.1;
    server 192.156.134.101;
  }
  # Proxies for port 443 traffic
  upstream sadams-bal {
    server 127.0.0.1:443;
    server 192.156.134.101:443;
  }

  # Virtual host definition 
  server {
    listen megalon-test.adams.edu:80;
    server_name adams-bal.adams.edu;
    error_page 404 = /about/searchasc/notfound.php;

    # The default location definition, we do some rewrites via nginx as well
    location / {
      # We do some rewrites via nginx as well
      include adams-rewrite;

      # Do caching using the adams zone, with the settings defined in /etc/nginx.conf   
      proxy_cache adams;

      # If it's not in the cache pass back to the adams-bal load balanced targets defined above 
      proxy_pass  http://adams-bal$request_uri;
    }

    # Serve static files directly via nginx, set an expires header for the browser 
    location ~* \.(pdf|css|js|png|gif|jpg|ico|swf|mov|doc|pdf|xls|ppt|docx|pptx|xlsx)$ {
      root /home/www/adams;
      expires max;
    }
  }

  # The SSL virtual host definition, setting up the SSL proxy end to middle to end.
  # client -> SSL -> nginx -> SSL -> Apache
  server {
    listen megalon-test.adams.edu:443;
    server_name adams-bal.adams.edu;

    # We use a domain wild card cert, under nginx the intermediate lives
    # in the same file as the domain cert
    ssl  on;
    ssl_certificate  adams.edu_wildcard_chain.crt;
    ssl_certificate_key  adams.edu_wildcard.key;
    ssl_session_timeout  5m;
    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
    ssl_prefer_server_ciphers   on;

    # Don't cache ssl pages, just proxy them
    location / {
      proxy_pass  https://sadams-bal$request_uri;
    }
  }


The Apache back end configuration on Ubuntu uses a couple of files, here are configs for the localhost server, running on the same VM as nginx.

/etc/apache2/ports.conf:
# Set up name virtual host for localhost port 80
  NameVirtualHost 127.0.0.1:80
  Listen 127.0.0.1:80

  # Set up server for SSL
  
  # SSL name based virtual hosts are not yet supported, therefore no
  # NameVirtualHost statement here

    # SSL for adams-bal
    Listen 127.0.0.1:443

    # SSL for another virtual host through nginx
    Listen 127.0.0.2:443
  


The Apache virtual host definition.

/etc/apache2/sites-enabled/adams-bal:
# For port 80, server name the same as nginx
  
    DocumentRoot /home/www/adams
    ServerName adams-bal.adams.edu
  

  
    Options -Indexes FollowSymLinks
    AllowOverride AuthConfig Limit 
    
      Order allow,deny
      Allow from all
     
  

  # For SSL, notice Apache requires the intermediate cert in a separate file 
  
    DocumentRoot /home/www/adams
    ServerName adams-bal.adams.edu
    SSLEngine on
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
    SSLCertificateFile /etc/nginx/adams.edu_wildcard.crt
    SSLCertificateKeyFile /etc/nginx/adams.edu_wildcard.key
    SSLCertificateChainFile /etc/nginx/GoDaddy_Intermediate.crt
 


The Apache configs on the remote host are almost identical, using the actual server IP address rather than localhost. One can envision a swarm of VM's talking to the proxy through a virtual network on private IP's...

The performance and resource usage is very good due to the ram disk caching. We did notice a slow down when the nginx cache filled during a site crawl from a search engine, so make sure there is enough cache to cover the site. Some sites may need a mix of on disk and ram disk caches.


REFERENCES:
http://staff.adams.edu/~cdmiller/posts/nginx-reverse-proxy-cache/

CentOS / Redhat Linux: Install Keepalived To Provide IP Failover For Web Cluster

— SkyHi @ Monday, March 07, 2011
Keepalived provides a strong and robust health checking for LVS clusters. It implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover. How do I install and configure Keepalived for reverse proxy server such as nginx or lighttpd?

If your are using a LVS director to loadbalance a server pool in a production environment, you may want to have a robust solution for healthcheck & failover. This will also work with reverse proxy server such as nginx.

Our Sample Setup

Internet--
         |
    =============
    | ISP Router|
    =============
         |
         |
         |      |eth0 -> 192.168.1.11 (connected to lan)
         |-lb0==|
         |      |eth1 -> 202.54.1.1 (vip master)
         |
         |      |eth0 -> 192.168.1.10 (connected to lan)
         |-lb1==|
                |eth1 -> 202.54.1.1 (vip backup)
Where,
  • lb0 - Linux box directly connected to the Internet via eth1. This is master load balancer.
  • lb1 - Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
  • 202.54.1.1 - This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
  • eth0 is connected to LAN and all other backend software such as Apache, MySQL and so on.
You need to install the following softwares on both lb0 and lb1:
  • keepalived for IP failover.
  • iptables to filter traffic
  • nginx or lighttpd revers proxy server.
DNS settings should be as follows:
  1. nixcraft.in - Our sample domain name.
  2. lb0.nixcraft.in - 202.54.1.11 (real ip assigned to eth1)
  3. lb1.nixcraft.in - 202.54.1.12 (real ip assigned to eth1)
  4. www.nixcraft.in - 202.54.1.1 (VIP for web server) do not assign this IP to any interface.

Install Keepalived

Visit keepalived.org to grab latest source code. You can use the wget command to download the same (you need to install keepalived on both lb0 and lb1):
# cd /opt
# wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
# tar -zxvf keepalived-1.1.19.tar.gz
# cd keepalived-1.1.19

Install Kernel Headers

You need to install the following packages:
  1. Kernel-headers - includes the C header files that specify the interface between the Linux kernel and userspace libraries and programs. The header files define structures and constants that are needed for building most standard programs and are also needed for rebuilding the glibc package.
  2. kernel-devel - this package provides kernel headers and makefiles sufficient to build modules against the kernel package.
Make sure kernel-headers and kernel-devel packages are installed. If not type the following install the same:
# yum -y install kernel-headers kernel-devel

Compile keepalived

Type the following command:
# ./configure --with-kernel-dir=/lib/modules/$(uname -r)/build
Sample outputs:
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
...
.....
..
config.status: creating keepalived/check/Makefile
config.status: creating keepalived/libipvs-2.6/Makefile

Keepalived configuration
------------------------
Keepalived version       : 1.1.19
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
Use VRRP Framework       : Yes
Use Debug flags          : No
Compile and install the same:
# make && make install

Create Required Softlinks

Type the following commands to create service and run it at RHEL / CentOS run level #3 :
# cd /etc/sysconfig
# ln -s /usr/local/etc/sysconfig/keepalived .
# cd /etc/rc3.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived S100keepalived
# cd /etc/init.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived .

Configuration

Your main configuration directory is located at /usr/local/etc/keepalived and configuration file name is keepalived.conf. First, make backup of existing configuration:
# cd /usr/local/etc/keepalived
# cp keepalived.conf keepalived.conf.bak

Edit keepalived.conf as follows on lb0:
vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}
Edit keepalived.conf as follows on lb1 (note priority set to 100 i.e. backup load balancer):
vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 100
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}
Save and close the file. Finally start keepalived on both lb0 and lb1 as follows:
# /etc/init.d/keepalived start

Verify: Keepalived Working Or Not

/var/log/messages will keep track of VIP:
# tail -f /var/log/messages
Sample outputs:
Feb 21 04:06:15 lb0 Keepalived_vrrp: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:06:20 lb0 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1
Verify that VIP assigned to eth1:
# ip addr show eth1
Sample outputs:
3: eth1:  mtu 1500 qdisc pfifo_fast qlen 10000
    link/ether 00:30:48:30:30:a3 brd ff:ff:ff:ff:ff:ff
    inet 202.54.1.11/29 brd 202.54.1.254 scope global eth1
    inet 202.54.1.1/29 scope global secondary eth1

ping failover test

Open UNIX / Linux / OS X desktop terminal and type the following command to ping to VIP:
# ping 202.54.1.1
Login to lb0 and halt the server or take down networking:
# halt
Within seconds VIP should move from lb0 to lb1 and you should not see any drops in ping. On lb1 you should get the following in /var/log/messages:
Feb 21 04:10:07 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) forcing a new MASTER election
Feb 21 04:10:08 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 21 04:10:09 lb1 Keepalived_healthcheckers: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Conclusion

Your server is now configured with IP failover. However, you need to install and configure the following software in order to configure webserver and security:
  1. nginx or lighttpd
  2. iptables
Stay tuned, for more information on above configuration.
This FAQ entry is 1 of 7 in the "CentOS / RHEL nginx Reverse Proxy Tutorial" series. Keep reading the rest of the series:

REFERENCES
http://www.cyberciti.biz/faq/rhel-centos-fedora-keepalived-lvs-cluster-configuration/