High Availability: cPanel with MySQL Cluster, Keepalived and HAProxy

I have successfully installed and integrate MySQL Cluster with HAproxy and Keepalived to provide scalable MySQL service with cPanel server run on CentOS 6.3 64bit. As you guys know that cPanel has a function called “Setup Remote MySQL server” which we can use to remotely access and control MySQL server from cPanel.

This will bring a big advantage, because the cPanel server load will be reduced tremendously due to mysqld service and resource will be serve from a cluster of servers. Following picture shows my architecture:

cpanel_cluster

I will be using following variables:

OS: CentOS 6.3 64bit
WHM/cPanel version: 11.34.0 (build 11)
MySQL root password: MhGGs4wYs

The Tricks

  • We will need to use same MySQL root password in all servers including the cPanel server
  • cPanel server’s SSH key need to be installed in all database servers to allow passwordless SSH login
  • All servers must have same /etc/hosts value
  • At least 4 servers for MySQL Cluster ( 2 SQL/Management/LB and 2 Data nodes)
  • mysql1 (active) and mysql2 (passive) will share a virtual IP which run on Keepalived
  • mysql1 and mysql2 will be the load balancer as well, which redirect MySQL traffic from cPanel server to mysql1 and mysql2
  • MySQL Cluster only support ndbcluster storage engine. Databases will be created in ndbcluster by default
  • For mysql1 and mysql2, MySQL will serve using port 3307 because 3306 will be used by HAProxy for load balancing

All Servers

1. In this post, I am going to turn off firewall and SELINUX for all servers:

$ service iptables stop
$ chkconfig iptables off
$ setenforce 0sed -i.bak 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

2. Install ntp using yum to make sure all servers’ time is in sync:

$ yum install ntp -y
$ ntpdate -u my.pool.ntp.org

Prepare the cPanel Server

1. Lets start declaring all hosts naming in /etc/hosts:

192.168.10.100     cpanel      cpanel.mydomain.com
192.168.10.101     mysql1      mysql1.mydomain.com
192.168.10.102     mysql2      mysql2.mydomain.com
192.168.10.103     mysql-data1 mysql-data1.mydomain.com
192.168.10.104     mysql-data2 mysql-data2.mydomain.com
192.168.10.110     mysql       mysql.mydomain.com    #Virtual IP for mysql service

2. Copy /etc/hosts file to other servers:

$ scp /etc/hosts mysql1:/etc
$ scp /etc/hosts mysql2:/etc
$ scp /etc/hosts mysql-data1:/etc
$ scp /etc/hosts mysql-data2:/etc

3. Setup SSH key. This will allow passwordless SSH between cPanel server and MySQL servers:

$ ssh-keygen -t dsa

Just press ‘Enter’ for all prompts.

4. Copy the SSH key to other servers:

$ ssh-copy-id -i ~/.ssh/id_dsa root@mysql1
$ ssh-copy-id -i ~/.ssh/id_dsa root@mysql2
$ ssh-copy-id -i ~/.ssh/id_dsa root@mysql-data1
$ ssh-copy-id -i ~/.ssh/id_dsa root@mysql-data2

5. Setup MySQL root password in WHM. Login to WHM > SQL Services > MySQL Root Password. Enter the MySQL root password and click “Change Password”.

6. Add additional host in WHM > SQL Services > Additional MySQL Access Hosts and add required host to be allowed to access the MySQL cluster as below:

add_host

 

 

Data Nodes (mysql-data1 and mysql-data2)

1. Download and install MySQL storage package from this page:

$ cd /usr/local/src
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-storage-7.1.25-1.el6.x86_64.rpm
$ rpm -Uhv MySQL-Cluster-gpl-storage-7.1.25-1.el6.x86_64.rpm

2. Create a mysql configuration file at /etc/my.cnf and add following line. This configuration will tell the storage to communicate with mysql1 and mysql2 as the management nodes:

[mysqld]
ndbcluster
ndb-connectstring=mysql1,mysql2
 
[mysql_cluster]
ndb-connectstring=mysql1,mysql2

 

SQL Nodes (mysql1 and mysql2)

1. Install required package using yum:

$ yum install perl libaio* pcre* popt* openssl openssl-devel gcc make -y

2. Download all required packages for Keepalived, HAProxy and MySQL Cluster package from this site (management, tools, shared, client, server):

$ cd /usr/local/src
$ wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
$ wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gz
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-management-7.1.25-1.el6.x86_64.rpm
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-tools-7.1.25-1.el6.x86_64.rpm
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-shared-7.1.25-1.el6.x86_64.rpm
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-client-7.1.25-1.el6.x86_64.rpm
$ wget http://mirror.services.wisc.edu/mysql/Downloads/MySQL-Cluster-7.1/MySQL-Cluster-gpl-server-7.1.25-1.el6.x86_64.rpm

3. Extract and compile Keepalived:

$ tar -xzf keepalived-1.2.7.tar.gz
$ cd keepalived-*
$ ./configure
$ make
$ make install

4. Extract and compile HAProxy:

$ tar -xzf haproxy-1.4.22.tar.gz
$ cd haproxy-*
$ make TARGET=linux26 ARCH=x86_64 USE_PCRE=1
$ make install

5. Install mysql packages with following order (management > tools > shared > client > server):

$ cd /usr/local/src
$ rpm -Uhv MySQL-Cluster-gpl-management-7.1.25-1.el6.x86_64.rpm
$ rpm -Uhv MySQL-Cluster-gpl-tools-7.1.25-1.el6.x86_64.rpm
$ rpm -Uhv MySQL-Cluster-gpl-shared-7.1.25-1.el6.x86_64.rpm
$ rpm -Uhv MySQL-Cluster-gpl-client-7.1.25-1.el6.x86_64.rpm
$ rpm -Uhv MySQL-Cluster-gpl-server-7.1.25-1.el6.x86_64.rpm

6. Create new directory for MySQL cluster. We also need to create cluster configuration file config.ini underneath it:

$ mkdir /var/lib/mysql-cluster
$ vim /var/lib/mysql-cluster/config.ini

And add following line:

[ndb_mgmd default]
DataDir=/var/lib/mysql-cluster
 
[ndb_mgmd]
NodeId=1
HostName=mysql1
 
[ndb_mgmd]
NodeId=2
HostName=mysql2
 
[ndbd default]
NoOfReplicas=2
DataMemory=256M
IndexMemory=128M
DataDir=/var/lib/mysql-cluster
 
[ndbd]
NodeId=3
HostName=mysql-data1
 
[ndbd]
NodeId=4
HostName=mysql-data2
 
[mysqld]
NodeId=5
HostName=mysql1
 
[mysqld]
NodeId=6
HostName=mysql2

7. Create the mysql configuration file at /etc/my.cnf and add following line:

[mysqld]
ndbcluster
port=3307
ndb-connectstring=mysql1,mysql2
default_storage_engine=ndbcluster
 
[mysql_cluster]
ndb-connectstring=mysql1,mysql2

Starting the Cluster

1. Start mysql cluster management service:

For mysql1:

$ ndb_mgmd -f /var/lib/mysql-cluster/config.ini --ndb-nodeid=1

For mysql2:

$ ndb_mgmd -f /var/lib/mysql-cluster/config.ini --ndb-nodeid=2

2. Start the mysql cluster storage service in both data nodes (mysql-data1 & mysql-data2):

$ ndbd

3. Start the mysql service (mysql1 & mysql2):

$ service mysql start

4. Login to mysql console and run following command  (mysql1 & mysql2):

mysql> use mysql;
mysql> alter table user engine=ndbcluster;
mysql> alter table db engine=ndbcluster;

5. Check the output of table db and user in mysql database and make sure it should appear as below:

mysql> SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE table_schema=DATABASE();
+---------------------------+------------+
| table_name                | engine     |
+---------------------------+------------+
| user                      | ndbcluster |
| columns_priv              | MyISAM     |
| db                        | ndbcluster |
| event                     | MyISAM     |
| func                      | MyISAM     |
| general_log               | CSV        |
| help_category             | MyISAM     |
| help_keyword              | MyISAM     |
| help_relation             | MyISAM     |
| help_topic                | MyISAM     |
| host                      | MyISAM     |
| ndb_apply_status          | ndbcluster |
| ndb_binlog_index          | MyISAM     |
| plugin                    | MyISAM     |
| proc                      | MyISAM     |
| procs_priv                | MyISAM     |
| servers                   | MyISAM     |
| slow_log                  | CSV        |
| tables_priv               | MyISAM     |
| time_zone                 | MyISAM     |
| time_zone_leap_second     | MyISAM     |
| time_zone_name            | MyISAM     |
| time_zone_transition      | MyISAM     |
| time_zone_transition_type | MyISAM     |
+---------------------------+------------+
24 rows in set (0.00 sec)

6. Check the management status in mysql1. You should see output similar to below:

$ ndb_mgm -e show
Connected to Management Server at: mysql1:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @192.168.10.103 (mysql-5.1.66 ndb-7.1.25, Nodegroup: 0, Master)
id=4 @192.168.10.104 (mysql-5.1.66 ndb-7.1.25, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=1 @192.168.10.101 (mysql-5.1.66 ndb-7.1.25)
id=2 @192.168.10.102 (mysql-5.1.66 ndb-7.1.25)
 
[mysqld(API)] 2 node(s)
id=5 @192.168.10.101 (mysql-5.1.66 ndb-7.1.25)
id=6 @192.168.10.102 (mysql-5.1.66 ndb-7.1.25)

7. Change MySQL root password to follow the MySQL root password in cPanel server (mysql1):

$ mysqladmin -u root password 'MhGGs4wYs'

8. Add MySQL root password into root environment so we do not need to specify password to access mysql console (mysql1 & mysql2):

$ vim /root/.my.cnf

And add following line:

[client]
user="root"
password="MhGGs4wYs"

9. Add haproxy user without password to be used by HAProxy to check the availability of real server (mysql1):

mysql> GRANT USAGE ON *.* TO haproxy@'%';

10. Add root user from any host so cPanel servers can access and control the MySQL cluster (mysql1):

mysql> GRANT USAGE ON *.* TO root@'%' IDENTIFIED BY 'MhGGs4wYs';
mysql> GRANT USAGE ON *.* TO root@'mysql1' IDENTIFIED BY 'MhGGs4wYs';
mysql> GRANT USAGE ON *.* TO root@'mysql2' IDENTIFIED BY 'MhGGs4wYs';
mysql> GRANT ALL PRIVILEGES ON *.* TO root@'%';
mysql> GRANT ALL PRIVILEGES ON *.* TO root@'mysql1';
mysql> GRANT ALL PRIVILEGES ON *.* TO root@'mysql2';

11. The last step, we need to allow GRANT privileges to root@’%’ by running following command in mysql console (mysql1):

mysql> UPDATE mysql.user SET `Grant_priv` = 'Y' WHERE `User` = 'root';

 

Configuring Virtual IP and Load Balancer (mysql1 & mysql2)

1. Configure HAProxy by creating a configuration /etc/haproxy.cfg:

$ vim /etc/haproxy.cfg

And add following line:

defaults
    log global
    mode http
    retries 2
    option redispatch
    maxconn 4096
    contimeout 50000
    clitimeout 50000
    srvtimeout 50000
 
listen mysql_proxy 0.0.0.0:3306
    mode tcp
    balance roundrobin
    option tcpka
    option httpchk
    option mysql-check user haproxy
    server mysql1 192.168.10.101:3307 weight 1
    server mysql2 192.168.10.102:3307 weight 1

2. Next we need to configure virtual IP. Open /etc/sysctl.conf and add following line to allow non-local IP to bind:

net.ipv4.ip_nonlocal_bind = 1

And run following command to apply the changes:

$ sysctl -p

3. Create Keepalived configuration file at /etc/keepalived.conf and add following line:

For mysql1:

vrrp_script chk_haproxy {
      script "killall -0 haproxy"    # verify the pid is exist or not
      interval 2                     # check every 2 seconds
      weight 2                       # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
      interface eth0                 # interface to monitor
      state MASTER
      virtual_router_id 51           # Assign one ID for this route
      priority 101                   # 101 on master, 100 on backup
      virtual_ipaddress {
            192.168.10.110           # the virtual IP
      }
      track_script {
            chk_haproxy
      }
}

For mysql2:

vrrp_script chk_haproxy {
      script "killall -0 haproxy"    # verify the pid is exist or not
      interval 2                     # check every 2 seconds
      weight 2                       # add 2 points of prio if OK
}
 
vrrp_instance VI_1 {
      interface eth0                 # interface to monitor
      state MASTER
      virtual_router_id 51           # Assign one ID for this route
      priority 100                   # 101 on master, 100 on backup
      virtual_ipaddress {
            192.168.10.110 # the virtual IP
      }
      track_script {
            chk_haproxy
      }
}

4. Start HAProxy:

$ haproxy -D -f /etc/haproxy.cfg

5. Start Keepalived:

$ keepalived -f /etc/keepalived.conf

6. Add following line into /etc/rc.local to make sure Keepalived and HAProxy start on boot:

$ vim /etc/rc.local

And add following line:

/usr/local/sbin/haproxy -D -f /etc/haproxy.cfg
/usr/local/sbin/keepalived -f /etc/keepalived.conf

7. Check the virtual IP should be up in mysql1:

$ ip a | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 192.168.10.101/24 brd 192.168.10.255 scope global eth0
inet 192.168.10.110/32 scope global eth0

8. Verify in mysql2 whether Keepalived is running in backup mode:

$ tail /var/log/messages
Dec 14 12:08:56 mysql2 Keepalived_vrrp[3707]: VRRP_Instance(VI_1) Entering BACKUP STATE

9. Check that HAProxy is run on port 3306 and mysqld is run on port 3307:

$ netstat -tulpn | grep -e mysql -e haproxy
tcp   0   0   0.0.0.0:3306     0.0.0.0:*       LISTEN      3587/haproxy
tcp   0   0   0.0.0.0:3307     0.0.0.0:*       LISTEN      3215/mysqld

 

Setup Remote MySQL Server in cPanel Server

1. Go to WHM > SQL Services > Setup Remote MySQL server and enter following details. Make sure the Remote server address is the virtual IP address setup in Keepalived in mysql1:

Remote server address (IP address or FQDN): mysql.mydomain.org
Remote SSH Port                           : 22
Select authentication method              : Public Key (default)
Select an installed SSH Key               : id_dsa

2. Wait for a while and you will see following output:

remote_success

3. Now MySQL Cluster is integrated within WHM/cPanel. You may verify this by accessing into PHPmyAdmin in WHM at WHM > SQL Services > PHPmyAdmin and you should see that you are connected into the MySQL Cluster as screenshot below:

my_cluster

Testing

We can test our MySQL high availability architecture by turning off the power completely for mysql1 or mysql2 and mysql-data1 or mysql-data2 in the same time. You will notice that the MySQL service will still available in cPanel point-of-view.

Here is my PHPmyAdmin for my test blog running on WordPress. You can notice that the database created is under ndbcluster engine:

blog_cluster

I never test this architecture in any production server yet and I cannot assure that all WHM/cPanel SQL functionalities are working as expected. Following features in cPanel has been tried and working well:

  • PHPMyAdmin
  • cPanel MySQL features (MySQL Database and MySQL Database Wizard)

cPanel with CentOS 6 as Internet Gateway

I am going to install a web server running on cPanel with several database servers connected only from the internal network (192.168.10.0/24). Since I need to run some yum installation in every box, I need to have internet access on each of the backend server.

My problem is I do have only 1 public IP provided by my ISP. I have no choice and must add another role to my cPanel box running on CentOS 6.3 to be an internet gateway so my database servers can have internet connection for this deployment phase.

Following picture simply explain the architecture that I am going to use:

Web Server (cPanel)

1. Since this server will going to be a gateway, we must allow the IP forwarding inside kernel. Open /etc/sysctl.conf and change following value:

net.ipv4.ip_forward = 1

2. Save the file and run following command to apply the changes:

$ sysctl -p

3. Lets clear the iptables rules first as we are going to add different rules later:

$ iptables -F

4. We need to allow IP masquerading in interface that facing internet connection, in my case is eth0. We also need to accept all connections from/to the internal network (192.168.10.0/24):

$ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ iptables -A FORWARD -d 192.168.10.0/24 -j ACCEPT 
$ iptables -A FORWARD -s 192.168.10.0/24 -j ACCEPT

5. Save the rules:

$ service iptables save

 

Database Servers

1. In every server, add the internal IP address into /etc/sysconfig/network-script/ifcfg-eth0 as below:

Database Server #1:

DEVICE="eth0"
ONBOOT="yes"
IPADDR=192.168.10.101
NETMASK=255.255.255.0
NETWORK=192.168.10.0

Database Server #2:

DEVICE="eth0"
ONBOOT="yes"
IPADDR=192.168.10.102
NETMASK=255.255.255.0
NETWORK=192.168.10.0

Database Server #3:

DEVICE="eth0"
ONBOOT="yes"
IPADDR=192.168.10.103
NETMASK=255.255.255.0
NETWORK=192.168.10.0

2. Change the gateway to point to the web server (cPanel) by adding following line into /etc/sysconfig/network :

GATEWAY=192.168.10.100

3. Add DNS resolver into /etc/resolv.conf as below:

nameserver 8.8.8.8
nameserver 8.8.4.4

4. Restart network service:

$ service network restart

 

Done! All the database servers should be able to have internet connectivity after the network service restarted. One public IP to be shared among servers?? Not a problem!

 

cPanel: Auto Backup and Remote Transfer using API + PHP

The good thing about cPanel is you can generate your own backup automatically using cPanel API. In this case, we will use PHP to run on schedule to automatically generate backup and transfer via FTP or SCP to another server. This implementation can be done on user level without need to login into cPanel login page.

To integrate with cPanel API, we need a file xmlapi.php which we can get from cPanel GitHub repository at https://github.com/CpanelInc/xmlapi-php. This post is about creating full backup and transfer the backup to a remote location using FTP automatically with cPanel under user privileges, not root privileges. Variable as below:

cPanel: WHM 11.30.6 (build 6)
cPanel user: mycp123
cPanel password: Pas$r12cP
Domain: mycp123.org
Home directory: /home/mycp123

1. Download the PHP API at here https://github.com/CpanelInc/xmlapi-php/downloads. For me I will download the zip format into my local desktop. Unzip it. Inside the folder got several files and folders. We just need to upload xmlapi.php into public_html folder using FTP client.

2. I am login into cPanel to create PHP script by using File Manager. Go to cPanel > File Manager > Web Root > Go > New File > File Name: cpbackup.php > Create New File.

3. Open back the file in text editor mode by right click on the file (cpbackup.php) and select Code Edit > Edit. It should open cPanel code editor. Copy following lines and save:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
<?php
// Must include cPanel API
include "xmlapi.php";
 
// Credentials for cPanel account
$source_server_ip = ""; // Server IP or domain name eg: 212.122.3.77 or cpanel.domain.tld
$cpanel_account = ""; // cPanel username
$cpanel_password = ""; // cPanel password
 
// Credentials for FTP remote site
$ftphost = ""; // FTP host IP or domain name
$ftpacct = ""; // FTP account
$ftppass = ""; // FTP password
$email_notify = ''; // Email address for backup notification
 
$xmlapi = new xmlapi($source_server_ip);
$xmlapi->password_auth($cpanel_account,$cpanel_password);
$xmlapi->set_port('2083');
 
// Delete any other backup before create new backup
$conn_id = ftp_connect($ftphost);
$login_result = ftp_login($conn_id, $ftpacct, $ftppass);
$logs_dir = "/";
 
ftp_chdir($conn_id, $logs_dir);
$files = ftp_nlist($conn_id, ".");
foreach ($files as $file){
    ftp_delete($conn_id, $file);
}
 
ftp_close($conn_id);
 
$api_args = array(
                           'passiveftp',
                           $ftphost,
                           $ftpacct,
                           $ftppass,
                           $email_notify,
                            21,
                            '/'
                         );
 
$xmlapi->set_output('json');
print $xmlapi->api1_query($cpanel_account,'Fileman','fullbackup',$api_args);
 
?>

4. Update the credentials details between line 5 to 15 in the script and save it.

Notes:
The script will check whether any backup exists in destination server and will delete all of them (line 21 to 31). Then, using cPanel API we can create an argument as refer to here http://docs.cpanel.net/twiki/bin/view/ApiDocs/Api1/ApiFileman#Fileman::fullbackup to use passive FTP as transfer mode at line 34 when the backup is ready.

5. We can execute this task manually by accessing the PHP file via browser at http://www.mycp123.org/cpbackup.php. You should see JSON output will be return as below if successful:

{"apiversion":"1","type":"event","module":"Fileman","func":"fullbackup","source":"module","data":{"result":""},"event":{"result":1}}

6. To automate this task, we can simply create cron job to run it weekly. Go to cPanel > Advanced > Cron jobs and use following command:

php -q /home/mycp123/public_html/cpbackup.php

Screen shot as below:

Done! You can create many scripts using cPanel API to automate your repeated task.

cPanel: Trigger Alert from Munin

For cPanel servers, Munin is one of important plugin that need to be enabled in the server. cPanel has provide this plugin to be install with just one click under WHM > cPanel > Manage Plugins > Munin. We can monitor and understand what is going on with our server resources while predicting future problems like hard disk temperature, available disk space, mysql throughput and many more.

As for me, we can take advantage of this monitoring tools to send me alert via email on certain critical services like Apache processes, Exim mail queue and load average of the server. I will use following information to configure Munin:

Email to: [email protected]
Server hostname: myserver.supercone.com
Service need to monitor: Apache processes, mail queue, load average
Alerting level: warning and critical

Apache processes

Usually when the server being attacked via port 80 or 443, it will fully utilize the whole Apache slot. Default cPanel Apache configuration will allow 256 process to be run and the normal usage for this server is below 50 processes. This is represented in Munin under plugin apache_processes under /etc/munin/plugins/apache_processes and can be access in WHM > Plugins > Munin Service Monitor > ::Apache processes as refer to following screen shot:

If you see the picture above, I will need to monitor busy80 by specifying 50 processes as warning and 180 processes as critical:

apache_processes.busy80.warning 50
apache_processes.busy80.critical 180

Exim mail queue

Mail queue monitor is to help us prevent from getting spammed or become spammer. We surely not aware how our mail users use their mailbox and some of them might have their account compromised and being used to spam mails by spammer. If your mail queue is higher than usual, something not good is leading towards you. You might be accused to be hosting spammer, server IP being blacklisted by anti-spam organization and server connection’s might have been blocked to connect to others.

Exim mail queue is being monitor by Munin by a plugin called exim_mailqueue under /etc/munin/plugins/exim_mailqueue. You can access it in browser at WHM > Plugins > Munin Service Monitor > ::Exim Mailqueue as refer to following screenshot:

As you can see that my mail queue average is 480 mails. I will need to monitor mails by specifying 500 as warning and 1000 as critical:

exim_mailqueue.mails.warning 500
exim_mailqueue.mails.critical 1000

Load average

This is the most important part on checking up the stability of a server. Load average is a value that represent your server work load over a period of time. From the Munin monthly load average, I can see my server is running on 0.82 average:

My warning value should be 4 and critical is 8 (usually no of CPU x 2).

load.load.warning 4
load.load.critical 8

Munin Alert

Munin configuration file in cPanel is located under /etc/munin/munin.conf. Open this file via text editor and add following line:

contact.me.command mail -s "Munin notification ${var:host}" webmaster_99@live.com
contact.me.always_send warning critical

This will enable email notification with warning and critical status. Now, we need to configure which monitoring plugins that we want to trigger alert as what we have describe above. In the same file, you should notice following line:

[myserver.supercone.com]
     address 127.0.0.1
     use_node_name yes

Add all the warning and critical triggers (as describe above) under this directive as below:

[myserver.supercone.com]
     address 127.0.0.1
     use_node_name yes
     apache_processes.busy80.warning 50
     apache_processes.busy80.critical 180
     exim_mailqueue.mails.warning 500
     exim_mailqueue.mails.critical 1000
     load.load.warning 4
     load.load.critical 8

Save and restart munin-node service:

$ service munin-node restart

From now on, you will receive email like below if warning or critical alerts triggered:

supercone.com :: myserver.supercone.com :: Apache processes
     WARNINGs: busy servers 80 is 112.83 (outside range[:50]).

cPanel: Install PHP SSH2 Module

One of my developer required PHP SSH2 module to be loaded into the cPanel server. Since this module is not available inside EasyApache, I need to install it separately and integrate to the current configuration that we have build using EasyApache.

1. Download and install libssh2 from this website, http://www.libssh2.org/snapshots/

$ cd /usr/local/src
$ wget http://www.libssh2.org/snapshots/libssh2-1.4.0-20120319.tar.gz
$ tar -xzf libssh2-1.4.0-20120319.tar.gz
$ cd libssh2-*
$ ./configure
$ make all install

2.  Before we install ssh2 module, we need to know where the PHP extension_dir location:

$ php -i | grep extension_dir
/usr/local/lib/php/extensions/no-debug-non-zts-20090626

3. Then, download PECL ssh2 module from here, http://pecl.php.net/package/ssh2 and install the module:

$ cd /usr/local/lib/php/extensions/no-debug-non-zts-20090626
$ wget http://pecl.php.net/get/ssh2/
$ tar -xzf ssh2-0.11.3.tgz
$ mv ssh2-0.11.3 php-ssh2
$ cd php-ssh2
$ phpize
$ ./configure --with-ssh2
$ make
$ make install

4. Now we need to enable the module in php.ini. Retrieve the php.ini location:

$ php -i | grep "Loaded Configuration File"
 Loaded Configuration File => /usr/local/lib/php.ini

And run following command to map the extension into PHP:

$ echo "extension=ssh2.so" >> /usr/local/lib/php.ini

5. Restart Apache web server (if you are using DSO):

$ service httpd restart

Done! You can check if SSH2 module is loaded or not by using following command:

$ php -i | grep ssh2
Registered PHP Streams => compress.zlib, compress.bzip2, php, file, glob, data, http, ftp, phar, zip, ssh2.shell, ssh2.exec, ssh2.tunnel, ssh2.scp, ssh2.sftp
ssh2
libssh2 version => 1.4.0-20120319
banner => SSH-2.0-libssh2_1.4.0-20120319
PWD => /usr/local/lib/php/extensions/no-debug-non-zts-20090626/php-ssh2
_SERVER["PWD"] => /usr/local/lib/php/extensions/no-debug-non-zts-20090626/php-ssh2
_ENV["PWD"] => /usr/local/lib/php/extensions/no-debug-non-zts-20090626/php-ssh2

Upgrade DELL Open Manage Server Administrator (OMSA)

Some of our servers are running on Dell which include Open Manage Server Administrator (OMSA) to manage the physical server  remotely. OMSA can be access via port 1311 in HTTPS by using web browser. Most of the time, I only use OMSA to manage and monitor our physical disk which run on RAID.

The version that I currently use is 6.3.0, which is the old one (because I have not update anything on OMSA since first usage). Following steps show on how to update Dell OMSA with the easiest way. Before upgrade, lets take a look on how OMSA’s current look:

Variable as below:

OS: Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Server IP: 192.168.100.30
OMSA URL: https://192.168.100.30:1311

1. We will use yum to update current OMSA installation. Lets add the Dell Open Manage repository into it:

$ cd /usr/local/src
$ wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

2. The safest way is to stop and remove the old version:

/opt/dell/srvadmin/sbin/srvadmin-services.sh stop
$ yum remove -y srvadmin*

3. Lets install new OMSA:

$ yum install -y srvadmin-all

4. Log out from SSH and relogin back to reset OMSA path.

5. Start OMSA service to load new update:

$ /opt/dell/srvadmin/sbin/srvadmin-services.sh start

And check whether it runs on the correct port:

$ netstat -tulpn | grep 1311
tcp     0       0 0.0.0.0:1311       0.0.0.0:*       LISTEN       29960/dsm_om_connsv

Done! Now lets see how our OMSA new look:

It seems Dell has hired great web developer for this new OMSA interface (version 6.5.0). As for me, it is not really matter as long as all required functionalities are still there. Cheers!

Linux: Log Rotation Customization

Problem Description

Our server has facing problem due to heavy development on web applications project. As the result, it cause our server hard disk turn full within a week. It all cause by the error log, which actually being rotated on weekly basis and has grow to 103GB within a week after rotation!

In the same time, it will consume CPU processing to the peak because the logrotate service will automatically compress the huge log file as what has been setup in /etc/logrotate.conf by default.

Symptom

Following process has been captured when I encounter this problem:

root     13324 38.2  0.0   4028   620 ?        RN   18:59  11:10      \_ gzip -f -

By checking the PID environment, we can see that this process is creating .gz (Gunzip) file for error log in Apache archive directory:

root@server [~] ll /proc/13324/fd
total 0
dr-x------ 2 root root  0 Jan 11 19:00 ./
dr-xr-xr-x 5 root root  0 Jan 11 19:00 ../
lr-x------ 1 root root 64 Jan 11 19:29 0 -> pipe:[1815813470]
l-wx------ 1 root root 64 Jan 11 19:29 1 -> /usr/local/apache/logs/archive/error_log-01-2012.gz
l-wx------ 1 root root 64 Jan 11 19:29 2 -> /usr/local/cpanel/logs/stats_log

Analysis

Following is what has been setup by default on my /etc/logrotate.d/httpd :

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

The log rotation will find any files which ended with ‘log’ (eg: access.log or error.log) under directory /var/log/httpd:

  • missingok – if the log file is missing, go on to the next log file without issuing error
  • notifempty – disable log rotation for empty file
  • sharedscripts – once all log files has been rotated, it will execute postrotate script once. If you are not specifying this option, logrotate will reload httpd for every log file which has been rotated
  • postrotate – bash script to what logrotate will do once the log file rotated. Apache need to be reloaded in order to write to the new files after log rotation complete
  • endscript – means the end line of the postrotate script

 

Solution

So I need to change my log rotation configuration for httpd by adding following line into /etc/logrotate.d/httpd:

/etc/httpd/logs/error_log {
    rotate 5
    nocompress
    notifempty
    size 1024M
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Since I am using cPanel, the Apache log is located under /etc/httpd/logs/error_log:

  • rotate 5 – log rotation will kept the last 5 log files. The rest will be deleted -> For backup purpose
  • nocompress – the last log files will not be compress -> Save CPU consumption
  • notifempty – disable log rotation for empty file -> Do not waste time to process empty file
  • size 1024M – rotate when the file size reach 1GB -> Do not waste disk space
  • postrotate – bash script to what logrotate will do once the log file rotated. Httpd need to be reloaded in order to write to the new files after rotated -> Restart httpd so the new log will take place in new file
  • endscript – means the end line of the postrotate script

cPanel: Create Backup and Transfer to Another Server

If you familiar in administrating cPanel, you should know a script/tool called pkgacct. This is the backup tools being used by cPanel in order to create and manage cPanel user’s account backup. By using this tool, we can take advantage by create a centralized backup server where cPanel account backup will be sent over to another server via FTP on weekly basis.

Following picture shows the architecture of the centralized backup I made:

 

FTP Server

1. I will store the cPanel backup in a Windows 2008 R2 server and I will be using FileZilla as the FTP server. Download the installer from here and follow the installation wizard. Just accept all defaults value during the installation process.

2. Add FTP required port into Windows Firewall:

Start > Administrative Tools > Windows Firewall with Advanced Security > Inbound Rules > New Rule > Port > Next > under Specific local ports enter this value: 20, 21 > Next > Allow the connection > Next > tick all for Domain, Private, Public > Next > put a name like FileZilla FTP > Finish.

3. Create FTP user and assign a directory called centralized_backup under C:\ partition:

FileZilla > Users > Add > enter username and password for respective user. Then go to Shared folders > Add Shared folders > C:\centralized_backup and tick all permissions on files and directories.

4. Make sure you can telnet port 21 from the cPanel servers:

$ telnet cpbackup.mypeng.org 21
Trying 115.10.221.108...
Connected to cpbackup.mypeng.org (115.10.221.108).
Escape character is '^]'.
220-FileZilla FTP server
220 version 0.9.40 beta

5. Setup a schedule task so it will delete files longer than 30 days in the centralized backup directory:

Start > All Programs > Accessories > System Tools > Task Scheduler > Create Task. Enter following information:

General > Name: Delete_old_backup
General > Security options: SYSTEM
Triggers: Weekly  every Sunday of every week
Actions: Start a program
Actions > Program/scripts: forfiles
Actions > Add arguments (optional):

/p "c:\central_backup" /s /d -30 /M *.tar.gz /c "cmd /c del @file : date >= 30 days >NUL"

Click OK to complete the setup. Screenshot of the Actions setup is as below:

cPanel Servers

1. I will use BASH script to automate the process. In each cPanel server, create a file under /root/scripts directory:

$ mkdir -p /root/scripts
$ touch /root/scripts/centralbackup

Copy and paste following contents:

#!/bin/bash
# Generate backups using cpbackup and then transfer them to a central server
# Author: SecaGuy @ blog.secaserver.com
 
# Local server configuration
LHOSTNAME=`hostname`
TEMPPATH='/root/tmp'
 
# FTP server configuration
CHOST='cpbackup.mypeng.org'
CUSERNAME='centralftp'
CPASSWORD='ByyPo3$d'
 
# Dont change line below
FTP=`which ftp`
 
if [ ! -d /var/cpanel/users ]
then
        echo "cPanel users not found. Aborted!"
        exit 1
else
        eof=`ls /var/cpanel/users | egrep -v '^\..$' | egrep -v '^\...$' | wc -l`
        echo "$eof cPanel user(s) found in this server"
 
	[ ! -d $TEMPPATH ] && mkdir -p $TEMPPATH || :
 
        for (( i=1; i<=$eof; i++ ))
        do
                CPUSER=`ls /var/cpanel/users | egrep -v '^\..$' | egrep -v '^\...$' | head -$i | tail -1`
                echo "Creating backup for user $CPUSER.."
                /usr/local/cpanel/scripts/pkgacct $CPUSER $TEMPPATH userbackup
 
                echo "Backup done. Transferring backup to FTP server.."
                FILENAME=`ls $TEMPPATH | grep tar.gz`
                $FTP -n $CHOST <<END_SCRIPT
                quote USER $CUSERNAME
                quote PASS $CPASSWORD
                binary
                passive
                mkdir $LHOSTNAME
		cd $LHOSTNAME
                lcd $TEMPPATH
                put $FILENAME
                quit
END_SCRIPT
 
        echo "Removing temporary files.."
        rm -Rf $TEMPPATH/backup-*
        USERDIR=`cat /var/cpanel/users/$CPUSER | grep HOMEDIRPATHS | sed 's/HOMEDIRPATHS=//g'`
        rm -Rf $USERDIR/backup-*
        echo "Backup for $CPUSER complete!"
 
        done
 
	echo "Process complete!"
        exit 0
fi

2. Change the permission to executable:

$ chmod 755 /root/scripts/centralbackup

Schedule the Backup

Once all cPanel servers have been setup, I can schedule them using cron job on weekly basis (every Sunday at 12:00 AM). So I will add following line into cron job list:

$ crontab -e

Add following line:

0 0 * * 0 /root/scripts/centralbackup

Save the files and restart cron daemon:

$ service crond restart

Notes: You can use cpbackup-exclude.conf as refer to cPanel documentation page, to exclude certain files or directories from being included in this backup.

cPanel: Exclude Directory during Backup

Backup is the first thing-to-do and should not be forgotten by a good system administrator. Since cPanel has their built-in backup creator as well as backup management system, we can take advantage of this tool and use them to suit our needs. In my situation, we have many cPanel accounts and some of them is higher than 10GB of disk usage, mostly due to website uploaded contents.

Creating backup will be a hard thing if you have too many inodes or too much disk consumption. It is a good thing if we can exclude some of the directory for example user_uploaded when creating cPanel backup, and the rest can be backup manually by downloading them to a local server.

In this tutorial I will create a full backup what some directories being excluded. Variables as below:

OS: RHEL 4 32bit
cPanel account: premen
Home directory for user: /home/premen

1. Identify the directory that we want to exclude. In this case, I will exclude the high disk storage directory. Using following command might help:

$ cd /home/premen
$ du -h | grep G

The command should list all directories which more than 1GB of size. Example as below:

6.4G    ./public_html/portal/tmp
8.7G    ./public_html/portal/user_uploaded
15.1G   ./public_html

2. Then we need to generate a file called cpbackup-exclude.conf under respective home directory of the user as refer to cPanel documentation at here:

$ cd /home/premen
$ touch cpbackup-exclude.conf
$ vi  cpbackup-exclude.conf

Paste following line:

public_html/portal/tmp
public_html/portal/user_uploaded

3. Now we will create the backup either via pkgacct script:

/usr/local/cpanel/scripts/pkgacct premen /home userbackup

or you can click “Download or Generate a Full Website Backup” under cPanel as screenshot below:

Once the backup ready, you will notice that both directories have been excluded from the cPanel full backup and your backup size should be smaller and faster to be compressed. Cheers!

cPanel: Berkeley DB error

I found out this error in /var/log/exim_mainlog. When trying to fix the Exim database using /scripts/exim_tidydb, the same error occurred:

$ tail -f /var/log/exim_mainlog
2011-09-08 10:08:13 1R1cV2-0003Yq-J2 Berkeley DB error: page 40: illegal page type or format
2011-09-08 10:08:13 1R1cV2-0003Yq-J2 Berkeley DB error: PANIC: Invalid argument
2011-09-08 10:08:13 1R1cV2-0003Yq-J2 Berkeley DB error: fatal region error detected; run recovery

This can be fixed by the following steps by re-updating Exim and clear up the exim

1. Backup /etc/exim.conf and /var/spool/exim/db :

$ cp /etc/exim.conf /etc/exim.conf.bak
$ cp /var/spool/exim/db /var/spool/exim/db.bak

2. Stop Exim:

$ service exim stop

3. Remove all files under /var/spool/exim/db to make sure we get the new Exim database :

$ rm -Rfv /var/spool/exim/db/*

4.Update Exim:

$ /scripts/eximup --force

5. Restore back the configuration of Exim:

$ cp /etc/exim.conf.bak /etc/exim.conf

6. Restart Exim to load our configuration:

$ service exim restart

Protect Apache Against Slowloris Attack

Slowloris allows a single machine to take down another machine’s web server with minimal bandwidth and side effects on unrelated services and ports. The tools used to launch Slowloris attack can be downloaded at http://ha.ckers.org/slowloris/

Slowloris tries to keep many connections to the target web server open and hold them open as long as possible. It accomplishes this by opening connections to the target web server and sending a partial request. Periodically, it will send subsequent HTTP headers, adding to—but never completing—the request. Affected servers will keep these connections open, filling their maximum concurrent connection pool, eventually denying additional connection attempts from clients.

Following web server has been tested and NOT affected by this kind of attack:

  • IIS6.0
  • IIS7.0
  • lighttpd
  • Squid
  • nginx
  • Cherokee
  • Netscaler
  • Cisco CSS

Since Apache is vulnerable to this attack, we should do some prevention. We need to install one Apache module called mod_antiloris. The module limits the number of threads in READ state on a per IP basis and protecting Apache against the Slowloris attack. Installation instruction as below:
1. Download the installer and install from Sourceforge.net:

$ cd /usr/local/src
$ wget http://sourceforge.net/projects/mod-antiloris/files/mod_antiloris-0.4.tar.bz2/download
$ tar -xvjf mod_antiloris-0.4.tar.bz2
$ cd mod_antiloris-*
$ apxs -a -i -c mod_antiloris.c

2. Restart Apache:

$ service httpd restart

3. Check whether mod_antiloris is loaded:

$ httpd -M | grep antiloris
   antiloris_module (shared)

or you can check using httpd fullstatus command:

$ service httpd fullstatus | grep antiloris
   mod_antiloris/0.4

For cPanel servers, don’t forget to run following command to make sure the new modifications be checked into the configuration system by running:

$ /usr/local/cpanel/bin/apache_conf_distiller --update

We have protect our web server from Slowloris attack. Try by launch the Slowloris attack to your web server and check the Apache status page to see whether it affected or not. Cheers!

 

UPDATE! Slowloris can be used to attack any port. Refer to comment section for more details. (Thanks to Luka Paunović for the highlight!)