I am currently working on a web cluster project using CentOS. In this project, I have 2 web servers running on Apache and mounted the same document root to serve the HTTP content. I also have 2 servers in front of it to become the load balancer and failover to increase high availability of the two-node web server cluster. The virtual IP will be hold by load balancer #1 with auto failover to load balancer #2.

You may refer to diagram below to get clearer picture:

I am using following variables:

All servers’ OS: CentOS 6.2 64bit
Web server #1: 192.168.0.221
Web server #2: 192.168.0.222
Load balancer #1: 192.168.0.231
Load balancer #2: 192.168.0.232
Virtual IP: 192.168.0.220

Load Balancer Server

1. All steps should be done in both servers unless specified. We will install Piranha and other required packages using yum:

$ yum install piranha ipvsadm -y

2. Open firewall ports as below:

  • Piranha: 3636
  • HTTP: 80
  • Hearbeat: 539

3. Start all required services and make sure they will auto start if server reboot:

$ service piranha-gui start
$ chkconfig piranha-gui on
$ chkconfig pulse on

4. Run following command to set password for user piranha. This will be used when accessing the web-based configuration tools:

$ piranha-passwd

5. Turn on IP forwarding. Open /etc/sysctl.conf and make sure following line has value 1:

net.ipv4.ip_forward = 1

And run following command to activate it:

$ sysctl -p

Load Balancer #1

1. Open Piranha web-based configuration tools at http://192.168.0.231:3636 and login as piranha with respective password. We start with configuring Global Settings as below:

2. Then, go to the Redundancy tab and enter the secondary server IP. In this case, we will put load balancer #2 IP as the redundant server in case load balancer #1 is down:

3. Under Virtual Servers tab, click Add and enter required information as below:

4. Now we need to configure the virtual IP and virtual HTTP server to map into the real HTTP server. Go to Virtual Servers > Real Server and add into the list as below:

Make sure you activate the real server once the adding completed by clicking the (DE)ACTIVATE button.

5.  Now copy the configuration file to load balancer #2 to as below:

$ scp /etc/sysconfig/ha/lvs.cf 192.168.0.232:/etc/sysconfig/ha/

6. Restart Pulse service to apply the new configuration:

$ service pulse restart

You can monitor what is happening with Pulse by tailing the /var/log/message output as below:

$ tail -f /var/log/message

Load Balancer #2

No need to configure anything in this server. We just need to restart Pulse service to get affected with the new configuration changes which being copied over from LB1.

$ service pulse restart

If you see the /var/log/message, pulse in this server will report that it will run on BACKUP mode.

Web Servers

1. Since we are using direct-routing method, regards to your Apache installation, we also need to install another package called arptables_jf. Here is some quote from RedHat documentation page:

Using the arptables_jf method, applications may bind to each individual VIP or port that the real server is servicing. For example, the arptables_jf method allows multiple instances of Apache HTTP Server to be running bound explicitly to different VIPs on the system. There are also significant performance advantages to usingarptables_jf over the IPTables option.

However, using the arptables_jf method, VIPs can not be configured to start on boot using standard Red Hat Enterprise Linux system configuration tools.

We will instsall using yum:

$ yum install arptables_jf -y

2. Configure arptables_jf by executing following command:

In web server #1:

$ arptables -A IN -d 192.168.0.220 -j DROP
$ arptables -A OUT -d 192.168.0.220 -j mangle --mangle-ip-s 192.168.0.221

In web server #2:

$ arptables -A IN -d 192.168.0.220 -j DROP
$ arptables -A OUT -d 192.168.0.220 -j mangle --mangle-ip-s 192.168.0.222

3.  Save the arptables rules and make sure the service is started on boot:

$ service arptables_jf save
$ chkconfig arptables_jf on

4.  Add the virtual IP address in the servers:

$ ip addr add 192.168.0.220 dev eth0

5. Since the IP cannot be started during sysinit (boot time), we can automatically start the IP after sysinit complete. Open /etc/rc.local using text editor:

$ vim /etc/rc.local

And add following line:

/sbin/ip addr add 192.168.0.220 dev eth0

Warning: Every time you restart your network service, please make sure to run step #4 to bring up the virtual IP in real server.

Done. You can now point your website to the virtual IP and you will see that the load balancer #1 will report as below:

$ ipvsadm -L
 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port       Forward Weight  ActiveConn  InActConn
TCP 192.168.0.220:http lblc
-> 192.168.0.221:http       Route   1       0           34
-> 192.168.0.222:http       Route   1       0           19

26 Responses to CentOS: Configure Piranha as Load Balancer (Direct Routing Method)

  1. SebastianNo Gravatar says:

    Hi,

    “I have 2 web servers running on Apache and mounted the same document root to serve the HTTP content.”

    What kind of storage solution have you implemented?

    Regards

  2. AtaaNo Gravatar says:

    There’s a typo in number 5.

  3. GilbertoNo Gravatar says:

    first of all thanks for the post, very helpfull =p
    Have you ever used persistence? I set some servers equal than in your post but I’m facing a problem.
    I set the persistence time to 120s and this is working fine. If a real server turn down the director remove it from the pool of ipvs and doesn’t send new connections to this server BUT if there are some open connections active by persistence time they are mantained in the pool and the user keeps trying to connect on a dead server and every time he tries the persistence time is refreshed, so if he keep refreshing the page he will never get out of this loop…This is normal? I forgot something?
    Thanks very much and congrats for the post

    • SecaGuyNo Gravatar says:

      I have no experience using persistence, but from my understanding LVS remembers the last connection for a specified period of time (120s). If that same client IP address connects again within that period, it will be sent to the same server it connected to previously — bypassing the load-balancing mechanisms.

      Since it says, BYPASSING the load-balancing mechanism, what you were facing is an expected behaviour.

  4. NareshNo Gravatar says:

    great dear, Thanks for sharing

  5. CarlosNo Gravatar says:

    Hello,

    How many real dedicated servers can be used in this system Piranha? Or are only 2 possible?

    Thank you!

  6. Vo Duong HoaNo Gravatar says:

    Hi,

    I see in step 4 Add the virtual IP address in the servers:

    I want to know add this virtual IP in what server?

    Loadbalance server or Webserver

    Thanks

    Hoavn

    • SecaGuyNo Gravatar says:

      U need to add the VIP into the web servers as well. Every packet should have source and destination address. If the Web Server do not have that IP, the packet will never get ready because the system cannot bind the VIP (source address) into that packet. This will create invalid packet and your packet will never get delivered to the recipient.

  7. SebastianNo Gravatar says:

    Hi,

    Thanks for this article.
    I’ve configured ipvsadm with direct routing, arptables and firewall marks (80,443). I’ve noticed rejected packets on the firewall on the active load balancer (INPUT table) any idea why? I can access the web server/pages via floating IP but I wonder why some packets are drooped.
    IPtables rules:


    # iptables -nL
    Chain INPUT (policy ACCEPT)
    target prot opt source destination
    ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
    ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3636
    ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:539
    REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination
    REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    I’m facing also another problem during failover – the virtual IP is added to the passive LB but it seems like it’s still pointing to the second LB for some time and I can’t access the web servers – is it possible that “send_arp” needs some time to broadcast the new MAC (this IPtable rule solve this problem but I’m not sure if it’s safe: iptables -A FORWARD -d FLOATING_IP -p tcp -m multiport –dports 80,443 -j ACCEPT)?

    • SecaGuyNo Gravatar says:

      May I know what is the packet being rejected? Do you have some logs on that? Depending on firewall rules, you have ‘state NEW’ in your ACCEPT rules, so it will reject any packet which is INVALID.

      Every router/switch should have ARP cache. Try check and disable this feature, or you can try to follow this method to clear ARP cache. Depending on caching, it will need to follow his timeout before these devices refresh their ARP table.

      Is your server really down when the virtual IP is added to passive LB? Or you just turn off pulse service? If you put that rule (-A FORWARD) and it is working, it means that your 1st LB is still up (network and iptables) and do the forwarding to multiport to floating IP which located on the 2nd LB. This method should work but not recommended.

      • SebastianNo Gravatar says:

        Thanks for reply.

        With regards to ARP cache it must be a side effect of intensive testing as I was stooping/starting pulse on both servers alternately.

        I’ve also added a firewall rule to accept all traffic from a subnet that the servers are running on and I still can see some dropped packets so it maybe be related to a connection states?


        Feb 19 12:37:07 lb1 kernel: IPTables-Dropped: IN=eth0 OUT= MAC=52:54:00:96:b2:97:00:10:db:ff:10:03:08:00 SRC=CLIENT_IP DST=VIRTUAL_IP LEN=40 TOS=0x00 PREC=0x00 TTL=121 ID=31304 DF PROTO=TCP SPT=23548 DPT=80 WINDOW=65050 RES=0x00 ACK FIN URGP=0 MARK=0x50

  8. AymenNo Gravatar says:

    Hi,
    thank you for the great tutorial.
    i am trying to apply this for a Darwin streaming server cluster.
    i got machine 1 : 4 Darwin streaming servers constituting my cluster with static addresses 192.168.111.6 to 9
    machine 2 : a load balancer and its backup with addresses 192.168.111.30 and 31….the virtual address is 192.168.111.50
    machine 3 : user request testing.

    usually, i simply use rtsp://192.168.111.X/file.sdp to stream….when i try rtsp://192.168.111.50/file.sdp i get nothing while i can directly stream the video directly from the servers…the problem is at the LB level…but i cant find it :((

    • SecaGuyNo Gravatar says:

      Are you forwarding to correct rtsp port on real server? How do you do back-end verification on the monitoring script section? Kindly take note that this tutorial is focusing on using LB on HTTP protocol (tcp port 80).

  9. Sarmed RahmanNo Gravatar says:

    Worked like a charm. Thank you :)

  10. Sarmed RahmanNo Gravatar says:

    running perfectly on with LB1, SRV1 and SRV2. But when I add LB2, pulse does not run on it. It says-

    [root@localhost ~]# service pulse restart
    Shutting down pulse: [FAILED]
    Starting pulse: pulse: cannot create heartbeat socket. running as root?
    [FAILED]

    syslog says something like this-
    May 8 13:50:50 localhost pulse: failed to bind to heartbeat address: Address already in use

    Even when all LB1, SRV1 and SRV2 are shut down, pulse keeps saying the same message. I don’t know what I am doing wrong.

    Any suggestions?

    • SecaGuyNo Gravatar says:

      Hi Sarmed,

      Can you verify if /etc/sysconfig/ha/lvs.cf is exists in LB2? I have made some typo on the post under “Load Balancer #1″ section at step 5.

      Previously the value is /etc/sysconfig/ha/lvs.conf

  11. SpixXxelNo Gravatar says:

    hi,

    i’m trying to implement the same architecture using the direct routing method. In case i have an additionnal layer under the 2 apache real servers and this layer is composed of some tomcat and jboss instances : Will the tomcat instance send the response directly to http user ? or it must move throught the reel server before arriving to http user ?

    Thank you in advance for your help.

    • SecaGuyNo Gravatar says:

      It depends on Apache (real server) role. If it acts like a reverse-proxy, the Tomcat instances will send the response back to Apache to be delivered to HTTP user.

  12. KurtNo Gravatar says:

    Hi,

    I have a question about sharing sessions between the real servers.
    I don’t know if it’s managed by piranha .. ? For example if a real server crashes, does the second keeps dealing with its sessions ? if not, how can we do that ?

    Thanks very much and congrats for the post.
    Regards.

  13. kurtNo Gravatar says:

    re
    Hi,

    I have a question about sharing sessions between the real servers.
    I don’t know if it’s managed by piranha .. ? For example if a real server crashes, does the second keeps dealing with its sessions ? if not, how can we do that ?

    Thanks very much and congrats for the post.
    Regards.

  14. YuriyNo Gravatar says:

    Hello. First of all, thank you very much for tutorial! When I start pulse in step 6, it starts but then crashes in several seconds.
    $ tail -f /var/log/message says the following:
    Dec 21 02:30:38 server3 pulse[4425]: STARTING PULSE AS MASTER
    Dec 21 02:30:56 server3 pulse[4425]: partner dead: activating lvs
    Dec 21 02:30:56 server3 pulse[4425]: Failed to open semaphore: Permission denied

    What kind of permissions pulse refers to?
    It’s interesting to note that “partner dead: activating lvs” message persists when I disasble redundency in Piranha.

    • SecaGuyNo Gravatar says:

      Is SELinux running? Are you running VM? Make sure that you have no connection issue with the backup node and both servers has an identical config file.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>