High Availability: Web Server Cluster using Apache + Pound + Unison

To achieve high availability, what we really need is to eliminate single point of failure as many point as possible but, it comes with expensive way to do this. What if we just have 2 servers, and we want to have highest web service availability possible with lowest cost?

Most important part in this high availability is data should be sync between these servers. So we need several tools to help us achieve our target:

  • Apache – Web server
  • Pound – HTTP load balancer/failover
  • Keepalived – IP failover
  • Unison – 2 way file synchronization
  • Fsniper – Monitor file and trigger the file synchronization

Following images will give some clear explanation on the architecture that we will setup:

In this setup, SELINUX and iptables has been turning OFF and root privileges is required. I am using following variables:

OS: CentOS 6.2 64bit
Web server #1 IP: 210.48.111.21
Web server #2 IP: 210.48.111.22
Domain: icreateweb.net
Web site public IP: 210.48.111.20
Web directory: /home/icreate/public_html

The steps are similar on both servers, unless specified. To make things easier, we need to enable RPMforge repository because almost all applications that we need is available there:

$ cd /usr/local/src
$ rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
$ rpm -Uhv rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

Both servers has been added following lines into /etc/hosts to ease up SSH communication:

210.48.111.21 webserver1
210.48.111.22 webserver2

We also need to allow SSH between node without password for auto file synchronization. Execute following command:

$ ssh-keygen -t dsa

Press ‘Enter’ until finish. Then, copy the public key to another server (webserver2):

$ ssh-copy-id -i ~/.ssh/id_dsa root@webserver2

Do this on another server as well, but copy the public key to webserver1 in ssh-copy-id command above. This step is critical and should not be skipped.

Web servers

1. Install all required applications using yum. We will skip RPMforge for the moment because I just need simple package of Apache and PHP:

$ yum install httpd* php* -y --disablerepo=rpmforge

2. Create web and logs directory for user icreate:

$ useradd -m icreate
$ chmod 755 /home/icreate
$ mkdir /home/icreate/public_html
$ mkdir /home/icreate/logs
$ touch /home/icreate/logs/access_log
$ touch /home/icreate/logs/error_log

3. We will use Pound as reverse-proxy and load balancer of port 80. So Apache need to run on different port. We will use port 88. Open /etc/httpd/conf/httpd.conf via text editor and make sure following line value as below:

Listen 88

4. Create vhosts.conf under /etc/httpd/conf.d directory and paste following line:

Web server #1:

NameVirtualHost 210.48.111.21:88
 
# Default host
<VirtualHost 210.48.111.21:88>
    ServerName localhost
    ServerAdmin admin@localhost
    DocumentRoot /var/www/html
</VirtualHost>
 
# Virtual host for domain icreateweb.net
<VirtualHost 210.48.111.21:88>
    ServerName icreateweb.net
    ServerAlias www.icreateweb.net
    ServerAdmin webmaster@icreateweb.net
    DocumentRoot /home/icreate/public_html
    ErrorLog /home/icreate/logs/error_log
    CustomLog /home/icreate/logs/access_log combined
</VirtualHost>

Web server #2:

NameVirtualHost 210.48.111.22:88
 
# Default host
<VirtualHost 210.48.111.22:88>
    ServerName localhost
    ServerAdmin admin@localhost
    DocumentRoot /var/www/html
</VirtualHost>
 
# Virtual host for domain icreateweb.net
<VirtualHost 210.48.111.22:88>
    ServerName icreateweb.net
    ServerAlias www.icreateweb.net
    ServerAdmin webmaster@icreateweb.net
    DocumentRoot /home/icreate/public_html
    ErrorLog /home/icreate/logs/error_log
    CustomLog /home/icreate/logs/access_log combined
</VirtualHost>

5. Restart and enable Apache service:

$ chkconfig httpd on
$ service httpd restart

6. Lets just create a html test file to differentiate between 2 web servers:

Web server #1:

$ echo "web server 1" > /home/icreate/public_html/server.html

Web server #2:

$ echo "web server 2" > /home/icreate/public_html/server.html

Website should run in local IP for both servers. Now we need to install and configure other applications that help us achieve high availability.

Unison & Fsniper

1. To keep all files in both servers are sync correctly, we will use Unison to execute file synchronization. Install Unison via yum:

$ yum install unison -y

2. Type following command to initialize Unison profile:

$ unison

3. Lets create Unison configuration file. Using text editor, open /root/.unison/default.prf and add following line. We also ignore server.html which we will use to determine HTTP connection from Pound whether to web #1 or web #2:

Web server #1:

root=/home/icreate/public_html
root=ssh://webserver2//home/icreate/public_html
batch=true
ignore=Name{server.html}

Web server #2:

root=/home/icreate/public_html
root=ssh://webserver1//home/icreate/public_html
batch=true
ignore=Name{server.html}

4. Now lets start first synchronization which is important. Run following command in either one server. In this case, I will run on web server #1:

$ unison default

5. Once completed, your files should be synced between both servers. Next, download and install Fsniper:

$ cd /usr/local/src
$ yum install pcre* file-libs file-devel -y
$ wget http://projects.l3ib.org/fsniper/files/fsniper-1.3.1.tar.gz
$ tar -xzf fsniper-1.3.1.tar.gz
$ cd fsniper-*
$ ./configure
$ make
$ make install

6. Create Fsniper configuration files to watch the directory and trigger the synchronization script. Open /root/.config/fsniper/config and add following line:

watch {
      /home/user/public_html {
      recurse = true
      * {
        handler = echo %%; /root/scripts/file_sync
        }
      }
}

7. Lets create the file_sync script to trigger Unison and check the process. Unison is 2 way replication so only need to have 1 process running in both servers in a same time:

$ mkdir -p /root/scripts
$ vim /root/scripts/file_sync

Web server #1:

#!/bin/bash
# Trigger Unison to do 2 way synchronization
 
# Check if Unison is running on both servers
if [ "$(pidof unison)" ] || [ "$(ssh [email protected] pidof unison)" ]
then
    echo "Unison is running. Exiting"
    exit 0
else
    /usr/bin/unison default
fi

Web server #2:

#!/bin/bash
# Trigger Unison to do 2 way synchronization
 
# Check if Unison is running on both servers
if [ "$(pidof unison)" ] || [ "$(ssh [email protected] pidof unison)" ]
then
    echo "Unison is running. Exiting"
    exit 0
else
    /usr/bin/unison default
fi

8. Now lets start the Fsniper process and allow it to start on boot:

$ /usr/local/bin/fsniper --daemon
$ echo "/usr/local/bin/fsniper --daemon" >> /etc/rc.local

KeepAlived

1. Download and install KeepAlived. This application will allow web server #1 and web server #2 to share the public IP (210.48.111.20) between them:

$ yum install -y openssl openssl-devel popt*
$ cd /usr/local/src
$ wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
$ tar -xzf keepalived-1.2.2.tar.gz
$ cd keepalived-*
$ ./configure
$ make
$ make install

2. Since we have virtual IP which shared between these 2 servers, we need to tell kernel that we have a non-local IP to be bind to Pound later. Add following line into /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind = 1

Run following command to apply the changes:

$ sysctl -p

3. By default, keepalived configuration file will be setup under /usr/local/etc/keepalived/keepalived.conf. We will make things easier by symlink it into /etc directory. We will also need to clear the configuration example inside it:

$ ln -s /usr/local/etc/keepalived/keepalived.conf /etc/keepalived.conf
$ cat /dev/null > /etc/keepalived.conf

4. Lets configure Keepalived:

For web server #1, add following line into /etc/keepalived.conf:

vrrp_script chk_pound {
        script "killall -0 pound"       # verify the pid is exist or not
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
        interface eth0			# interface to monitor
        state MASTER
        virtual_router_id 51		# Assign one ID for this route
        priority 101                    # 101 on master, 100 on backup
        virtual_ipaddress {
            210.48.111.20		# the virtual IP
        }
        track_script {
            chk_pound
        }
}

For web server #2, add following line into /etc/keepalived.conf:

vrrp_script chk_pound {
        script "killall -0 pound"       # verify the pid is exist or not
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
        interface eth0			# interface to monitor
        state MASTER
        virtual_router_id 51		# Assign one ID for this route
        priority 100                    # 101 on master, 100 on backup
        virtual_ipaddress {
            210.48.111.20		# the virtual IP
        }
        track_script {
            chk_pound
        }
}

5. Start Keepalived and make it auto start after boot:

$ keepalived -f /etc/keepalived.conf
$ echo "/usr/local/sbin/keepalived -f /etc/keepalived.conf" >> /etc/rc.local

Pound

1. Install Pound:

$ rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/pound-2.4.3-1.el5.rf.x86_64.rpm

2. Configure Pound by editing the configuration file located under /etc/pound.cfg and paste following line:

User            "nobody"
Group           "nobody"
 
LogLevel        1
Alive           2
 
ListenHTTP
        Address 0.0.0.0
        Port    80
End
 
Service
        HeadRequire "Host: .*icreateweb.net.*"
 
        BackEnd
                Address 210.48.111.21
                Port    88
		TimeOut 300
        End
 
        BackEnd
                Address 210.48.111.22
                Port 88
		TimeOut 300
        End
 
        Session
                Type Cookie
                ID   "JSESSIONID"
                TTL  300
        End
End

3. Allow the service to auto start after boot and start Pound:

$ chkconfig pound on
$ service pound start

4. Check whether Pound is correctly running and listen to port 80:

$ netstat -tulpn | grep pound
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      6175/pound

 

Done! Now you may try to check your website availability by bringing down the HTTP service or the server itself. By using only 2 servers, it is possible to increase the service uptime to the highest possible. Cheers!

7 thoughts on “High Availability: Web Server Cluster using Apache + Pound + Unison

  1. Very useful post, but many commands and a little info what which program does. I don’t know how KeepAllived work. Why one Virtual IP directs users to two different servers? I wanted to install pound on VPS to divide traffic to two servers, but is only one point of failure. Your solution seems better to me, but I don’t understand how it works.

    Reply

    1. The image in the post is for logical view. Physical will only need 2 servers. Pound(web load balancer+failover) and Keepalived(IP failover) will be run on both servers and communicate each other.

      If one server down, Keepalived will make sure the virtual IP 210.48.111.20 will be run on another server, where Pound and Apache will be ready to serve. If only one Apache down, Pound will redirect traffic to another Apache. If Pound down, Keepalived will transfer virtual IP to be hold by another server which having Pound running.

      Reply

  2. Thanks for this post.

    But why did you use pound, instead of haproxy for instance?
    For me, the only addition of pound is his hability to handle SSL and act as an SSL wrapper. Usually for web load-balancing haproxy is the best candidate.

    At the end, I also think that using unison is not the best HA solution. I will prefer using pacemaker + drbd but I have to admit that your setup is easier.

    Cheers!

    Reply

    1. My point here is the easiest way to setup HA in web service, and achieve that using the uncommon/new way. Normal implementation will surely using HAproxy + DRBD + Heartbeat which mostly available on the Internet.

      Reply

  3. Really helpful and important post for deploying a high availability cluster. Thanks alot

    Reply

  4. Unison + Fsniper would be fine for shared directory 2 way sync if the Fsniper handled also the file deletion.
    Unison supports that, but the file deletion is not triggered by Fsniper, so the deleted file is synchronized on nearest file adition/modification.

    Do you have any ideas how to achieve full 2 way sync including deleting files? E.g. Unison with combination with inotifywait? What do you think?

    Thanks a lot.

    Reply

    1. Good idea man.. I am going to cover that on my latest post. inotifywait + Unison for full 2 way sync. Thanks Michal

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *