High Availability: Web Server Cluster using Apache + Pound + Unison

To achieve high availability, what we really need is to eliminate single point of failure as many point as possible but, it comes with expensive way to do this. What if we just have 2 servers, and we want to have highest web service availability possible with lowest cost?

Most important part in this high availability is data should be sync between these servers. So we need several tools to help us achieve our target:

• Apache – Web server
• Pound – HTTP load balancer/failover
• Keepalived – IP failover
• Unison – 2 way file synchronization
• Fsniper – Monitor file and trigger the file synchronization

Following images will give some clear explanation on the architecture that we will setup:

In this setup, SELINUX and iptables has been turning OFF and root privileges is required. I am using following variables:

OS: CentOS 6.2 64bit
Web server #1 IP: 210.48.111.21
Web server #2 IP: 210.48.111.22
Domain: icreateweb.net
Web site public IP: 210.48.111.20
Web directory: /home/icreate/public_html

The steps are similar on both servers, unless specified. To make things easier, we need to enable RPMforge repository because almost all applications that we need is available there:

$ cd /usr/local/src
$ rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
$ rpm -Uhv rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

Both servers has been added following lines into /etc/hosts to ease up SSH communication:

210.48.111.21 webserver1
210.48.111.22 webserver2

We also need to allow SSH between node without password for auto file synchronization. Execute following command:

$ ssh-keygen -t dsa

Press ‘Enter’ until finish. Then, copy the public key to another server (webserver2):

$ ssh-copy-id -i ~/.ssh/id_dsa root@webserver2

Do this on another server as well, but copy the public key to webserver1 in ssh-copy-id command above. This step is critical and should not be skipped.

Web servers

1. Install all required applications using yum. We will skip RPMforge for the moment because I just need simple package of Apache and PHP:

$ yum install httpd* php* -y --disablerepo=rpmforge

2. Create web and logs directory for user icreate:

$ useradd -m icreate
$ chmod 755 /home/icreate
$ mkdir /home/icreate/public_html
$ mkdir /home/icreate/logs
$ touch /home/icreate/logs/access_log
$ touch /home/icreate/logs/error_log

3. We will use Pound as reverse-proxy and load balancer of port 80. So Apache need to run on different port. We will use port 88. Open /etc/httpd/conf/httpd.conf via text editor and make sure following line value as below:

Listen 88

4. Create vhosts.conf under /etc/httpd/conf.d directory and paste following line:

Web server #1:

NameVirtualHost 210.48.111.21:88
 
# Default host
<VirtualHost 210.48.111.21:88>
    ServerName localhost
    ServerAdmin admin@localhost
    DocumentRoot /var/www/html
</VirtualHost>
 
# Virtual host for domain icreateweb.net
<VirtualHost 210.48.111.21:88>
    ServerName icreateweb.net
    ServerAlias www.icreateweb.net
    ServerAdmin webmaster@icreateweb.net
    DocumentRoot /home/icreate/public_html
    ErrorLog /home/icreate/logs/error_log
    CustomLog /home/icreate/logs/access_log combined
</VirtualHost>

Web server #2:

NameVirtualHost 210.48.111.22:88
 
# Default host
<VirtualHost 210.48.111.22:88>
    ServerName localhost
    ServerAdmin admin@localhost
    DocumentRoot /var/www/html
</VirtualHost>
 
# Virtual host for domain icreateweb.net
<VirtualHost 210.48.111.22:88>
    ServerName icreateweb.net
    ServerAlias www.icreateweb.net
    ServerAdmin webmaster@icreateweb.net
    DocumentRoot /home/icreate/public_html
    ErrorLog /home/icreate/logs/error_log
    CustomLog /home/icreate/logs/access_log combined
</VirtualHost>

5. Restart and enable Apache service:

$ chkconfig httpd on
$ service httpd restart

6. Lets just create a html test file to differentiate between 2 web servers:

Web server #1:

$ echo "web server 1" > /home/icreate/public_html/server.html

Web server #2:

$ echo "web server 2" > /home/icreate/public_html/server.html

Website should run in local IP for both servers. Now we need to install and configure other applications that help us achieve high availability.

Unison & Fsniper

1. To keep all files in both servers are sync correctly, we will use Unison to execute file synchronization. Install Unison via yum:

$ yum install unison -y

2. Type following command to initialize Unison profile:

$ unison

3. Lets create Unison configuration file. Using text editor, open /root/.unison/default.prf and add following line. We also ignore server.html which we will use to determine HTTP connection from Pound whether to web #1 or web #2:

Web server #1:

root=/home/icreate/public_html
root=ssh://webserver2//home/icreate/public_html
batch=true
ignore=Name{server.html}

Web server #2:

root=/home/icreate/public_html
root=ssh://webserver1//home/icreate/public_html
batch=true
ignore=Name{server.html}

4. Now lets start first synchronization which is important. Run following command in either one server. In this case, I will run on web server #1:

$ unison default

5. Once completed, your files should be synced between both servers. Next, download and install Fsniper:

$ cd /usr/local/src
$ yum install pcre* file-libs file-devel -y
$ wget http://projects.l3ib.org/fsniper/files/fsniper-1.3.1.tar.gz
$ tar -xzf fsniper-1.3.1.tar.gz
$ cd fsniper-*
$ ./configure
$ make
$ make install

6. Create Fsniper configuration files to watch the directory and trigger the synchronization script. Open /root/.config/fsniper/config and add following line:

watch {
      /home/user/public_html {
      recurse = true
      * {
        handler = echo %%; /root/scripts/file_sync
        }
      }
}

7. Lets create the file_sync script to trigger Unison and check the process. Unison is 2 way replication so only need to have 1 process running in both servers in a same time:

$ mkdir -p /root/scripts
$ vim /root/scripts/file_sync

Web server #1:

#!/bin/bash
# Trigger Unison to do 2 way synchronization
 
# Check if Unison is running on both servers
if [ "$(pidof unison)" ] || [ "$(ssh root@webserver2 pidof unison)" ]
then
    echo "Unison is running. Exiting"
    exit 0
else
    /usr/bin/unison default
fi

Web server #2:

#!/bin/bash
# Trigger Unison to do 2 way synchronization
 
# Check if Unison is running on both servers
if [ "$(pidof unison)" ] || [ "$(ssh root@webserver1 pidof unison)" ]
then
    echo "Unison is running. Exiting"
    exit 0
else
    /usr/bin/unison default
fi

8. Now lets start the Fsniper process and allow it to start on boot:

$ /usr/local/bin/fsniper --daemon
$ echo "/usr/local/bin/fsniper --daemon" >> /etc/rc.local

KeepAlived

1. Download and install KeepAlived. This application will allow web server #1 and web server #2 to share the public IP (210.48.111.20) between them:

$ yum install -y openssl openssl-devel popt*
$ cd /usr/local/src
$ wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
$ tar -xzf keepalived-1.2.2.tar.gz
$ cd keepalived-*
$ ./configure
$ make
$ make install

2. Since we have virtual IP which shared between these 2 servers, we need to tell kernel that we have a non-local IP to be bind to Pound later. Add following line into /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind = 1

Run following command to apply the changes:

$ sysctl -p

3. By default, keepalived configuration file will be setup under /usr/local/etc/keepalived/keepalived.conf. We will make things easier by symlink it into /etc directory. We will also need to clear the configuration example inside it:

$ ln -s /usr/local/etc/keepalived/keepalived.conf /etc/keepalived.conf
$ cat /dev/null > /etc/keepalived.conf

4. Lets configure Keepalived:

For web server #1, add following line into /etc/keepalived.conf:

vrrp_script chk_pound {
        script "killall -0 pound"       # verify the pid is exist or not
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
        interface eth0			# interface to monitor
        state MASTER
        virtual_router_id 51		# Assign one ID for this route
        priority 101                    # 101 on master, 100 on backup
        virtual_ipaddress {
            210.48.111.20		# the virtual IP
        }
        track_script {
            chk_pound
        }
}

For web server #2, add following line into /etc/keepalived.conf:

vrrp_script chk_pound {
        script "killall -0 pound"       # verify the pid is exist or not
        interval 2                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
        interface eth0			# interface to monitor
        state MASTER
        virtual_router_id 51		# Assign one ID for this route
        priority 100                    # 101 on master, 100 on backup
        virtual_ipaddress {
            210.48.111.20		# the virtual IP
        }
        track_script {
            chk_pound
        }
}

5. Start Keepalived and make it auto start after boot:

$ keepalived -f /etc/keepalived.conf
$ echo "/usr/local/sbin/keepalived -f /etc/keepalived.conf" >> /etc/rc.local

Pound

1. Install Pound:

$ rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/pound-2.4.3-1.el5.rf.x86_64.rpm

2. Configure Pound by editing the configuration file located under /etc/pound.cfg and paste following line:

User            "nobody"
Group           "nobody"
 
LogLevel        1
Alive           2
 
ListenHTTP
        Address 0.0.0.0
        Port    80
End
 
Service
        HeadRequire "Host: .*icreateweb.net.*"
 
        BackEnd
                Address 210.48.111.21
                Port    88
		TimeOut 300
        End
 
        BackEnd
                Address 210.48.111.22
                Port 88
		TimeOut 300
        End
 
        Session
                Type Cookie
                ID   "JSESSIONID"
                TTL  300
        End
End

3. Allow the service to auto start after boot and start Pound:

$ chkconfig pound on
$ service pound start

4. Check whether Pound is correctly running and listen to port 80:

$ netstat -tulpn | grep pound
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      6175/pound

Done! Now you may try to check your website availability by bringing down the HTTP service or the server itself. By using only 2 servers, it is possible to increase the service uptime to the highest possible. Cheers!

Related Posts

• High Availability: Configure Piranha for HTTP, HTTPS and MySQL
• PHP Session using Sharedance in Apache Web Cluster
• CentOS: Configure Piranha as Load Balancer (Direct Routing Method)
• Linux: 2 Way File Synchronization and Replication using Unison + FTP
• Manage Multiple MySQL Servers using PHPmyAdmin
• Install MySQL Cluster in Debian
• High Availability: cPanel with MySQL Cluster, Keepalived and HAProxy
• Monitor MySQL Galera Cluster from Split-Brain
• CentOS 6: Install MySQL Cluster – The Simple Way
• Linux: Add New User and Group into .htpasswd