Install and Configure HAProxy for MySQL Galera Cluster

In this post, I’m going to cover on how to add load balancer for the MariaDB Galera Cluster. The major steps are:

  1. Ensure all Galera nodes are running as a single cluster (all nodes are primary and synced).
  2. Install HAproxy (you can install it on separate node or on the application server).
  3. Configure clustercheck script. This script performs health check on each backend server.
  4. Configure HAproxy statistic page.
  5. Point the application to load balancer.

Our architecture looks like below:

galera_haproxy_secaserver

Configure clustercheck script

* Steps described in this section should be performed on all DB nodes unless specified otherwise.

1. Firstly, we need to configure the backend health check reporting. We will use Percona’s clustercheck script available here. Get the script and put it under /usr/local/bin by running following commands:

$ git clone https://github.com/olafz/percona-clustercheck
$ cp percona-clustercheck/clustercheck /usr/local/bin

2. The clustercheck script performs regular check on the Galera node by monitoring several MySQL variables/status. It yields a simple HTML output with corresponding HTTP return code (either 503 – Service Unavailable or 200 – OK). To make things easier for HAproxy to trigger the script and get the latest status of the backend, we have to make it listens to a port. We can use xinetd to turn the script into a service daemon and make it listen to a custom port, in this case, I’m going to use 9200.

Create a new file called /etc/xinet.d/mysqlchk, and add following lines:

# default: on
# description: mysqlchk
service mysqlchk
{
  disable = no
  flags = REUSE
  socket_type = stream
  port = 9200
  wait = no
  user = nobody
  server = /usr/local/bin/clustercheck
  log_on_failure += USERID
  only_from = 0.0.0.0/0
  per_source = UNLIMITED
}

3. Then, we need to add the mysqlchk service into xinetd:

$ echo 'mysqlchk      9200/tcp    # MySQL check' >> /etc/services

4. By default, the script will use a MySQL user called “clustercheckuser” with password “clustercheckpassword!”. We need to ensure this MySQL user is exist with the corresponding password before the script would be able to perform health checks. Run following DDL statements on one of the DB node (Galera should replicate the statement to the other nodes):

mysql> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!';
mysql> FLUSH PRIVILEGES;

You can change this value inside the clustercheck script in line 32,33. In this post, I’m going to use the default username and password.

5. Verify if the script returns a correct value:

$ /usr/local/bin/clustercheck > /dev/null
$ echo $?
0

If the DB node is in synced, you should get 0. Otherwise 1 should be the output. The backend health check is configured.

Install HAproxy

* Steps described in this section should be performed on HAproxy or application server.

1. The easy way is using package manager (yum/apt). However, it’s highly recommended to use the latest version available on HAproxy site. Either way, I’ll show the the installation steps here.

a) If you choose to install HAproxy via package manager:

$ yum install haproxy # Redhat/CentOS
$ sudo apt-get install haproxy # Debian/Ubuntu

b) Via source from HAproxy download site:

$ yum install php-curl gcc make # Redhat/CentOS
$ apt-get install php5-curl gcc make # Debian/Ubuntu
$ wget http://www.haproxy.org/download/1.5/src/
$ tar xvzfz
$ cd
$ make TARGET=linux26
$ cp -f haproxy /usr/sbin/haproxy

Installing from source (option b) comes with no init script. So you have to create it manually or simply provision the process via command line (which is non-standard way). I’m not going to cover this unconventional way in this post.

 

Configure HAproxy

Now we have HAproxy installed. We need to configure it to listen on port 3307 for MySQL service and perform back-end health checks. On /etc/haproxy/haproxy.cfg, ensure following lines exist:

global
        pidfile /var/run/haproxy.pid
        daemon
        user haproxy
        group haproxy
        stats socket /var/run/haproxy.socket user haproxy group haproxy mode 600 level admin
 
        maxconn 8192
        spread-checks 3
        quiet
defaults
        mode    tcp
        option  dontlognull
        option tcp-smart-accept
        option tcp-smart-connect
        retries 3
        option redispatch
        maxconn 8192
        timeout check   3500ms
        timeout queue   3500ms
        timeout connect 3500ms
        timeout client  10800s
        timeout server  10800s
 
userlist STATSUSERS
        group admin users admin
        user admin insecure-password admin
        user stats insecure-password yourpassword
 
listen admin_page 0.0.0.0:9600
        mode http
        stats enable
        stats refresh 60s
        stats uri /
        acl AuthOkay_ReadOnly http_auth(STATSUSERS)
        acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
        stats http-request auth realm admin_page unless AuthOkay_ReadOnly
 
listen  mysql_3307
        bind *:3307
        mode tcp
        timeout client  10800s
        timeout server  10800s
        balance leastconn
        option httpchk
        option allbackups
        default-server port 9200 inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 64 maxqueue 128 weight 100
        server db1 10.0.0.187:3306 check
        server db2 10.0.0.188:3306 check
        server db3 10.0.0.189:3306 check

Now enable the service on boot and fire it up:

RHEL/CentOS 6:

$ chkconfig haproxy on # RHEL6
$ service haproxy start # RHEL6

Ubuntu 14.04 and lower, Debian 7 and lower:

$ update-rc.d haproxy defaults
$ sudo service haproxy start

RHEL/CentOS 7, Debian 8, Ubuntu 15.04:

$ systemctl enable haproxy
$ systemctl start haproxy

Verify if HAproxy is listening to the correct ports:

$ sudo netstat -tulpn | grep haproxy
tcp        0      0 0.0.0.0:9600            0.0.0.0:*               LISTEN      370/haproxy
tcp        0      0 0.0.0.0:3307            0.0.0.0:*               LISTEN      370/haproxy

3307 is the MySQL load-balanced port, while 9600 is the HAproxy statistic page. You can login to check the status by going to http://haproxy_ip_address:9600/ and login with username ‘admin’ and password ‘yourpassword’ as configured inside haproxy.cfg. You should see something like below:

Screen Shot 2015-09-17 at 3.00.03 PM

Now, you can redirect the application or MySQL client to HAproxy host on port 3307, for a load balanaced  MySQL connections with auto failover.

Various Ways to Determine Public IP on Linux CLI

Always when you are working a lot with CLI environment, Linux particularly, you would like to know the public IP address especially when you were running on NAT environment. Here is a list of command that you can use to determine the public IP of your host via command line.

Using curl

cURL is mostly available on all Linux distributions, and is the most popular way to determine public IP address of the host. You just need to know the URL or host that will response with the correct public IP as per below:

$ curl ifconfig.me
52.74.127.152
 
$ curl ipecho.net/plain
52.74.127.152
 
$ curl ident.me
52.74.127.152
 
$ curl icanhazip.com
52.74.127.152
 
$ curl bot.whatismyipaddress.com
52.74.127.152

Using wget

Basically, command executed on curl can be replace with wget -qO- option, as per below:

$ wget -qO- http://ipecho.net/plain
52.74.127.152

Using Lynx

Lynx is a text-based browser which runs like a browser for your CLI

$ lynx checkip.dyndns.org # you will be redirected to a text-based browser

If you have the simplest method apart from what being mentioned here, please share it. I can’t imagine how simple it would be!

 

MySQL Encryption using SSH and Supervisor

If MySQL security is one of your concerns, you should use encryption when connecting to the server. Setting up SSL in MySQL is not really straightforward as you have to generate key, certificate and GRANT for specific user with REQUIRE SSL statement. This would bring additional maintenance task for DBA.

The easiest way to achieve this is by using SSH encryption. Instead directing the application to connect to standard port 3306 with plain connection, why don’t you connect it to a ‘forwarding’ port which map to the MySQL port via SSH? SSH is secure and almost everyone with Linux basic knowledge knows how to manage it.

Setting up SSH access and port forwarding

Let’s say we have a application/web server and a MySQL server listening on standard port 3306:

  • 10.0.0.20 – web server
  • 10.0.0.21 – mysql server

Following steps should be performed on the application/web server.

1. As root user, generate a SSH key:

$ whoami 
root
$ ssh-keygen -t rsa # press enter on all prompts

2. Copy the SSH key on web server to MySQL node:

$ ssh-copy-id 10.0.0.21 # enter the root password for 10.0.0.21 if prompted

3. Verify that you can run following command without the MySQL node prompting a password:

$ ssh 10.0.0.21 "ls /usr"

4. Run SSH command to listen to port 10001 on localhost IP and forward it to port 3306 on 10.0.0.21 as root user:

$ ssh -fNg -L 10001:127.0.0.1:3306 10.0.0.21

5. Verify that you got following port listed:

$ netstat -tulpn | grep 10001
$ ps aux | grep ssh

6. Finally, connect to MySQL server using MySQL client securely:

$ mysql -uroot -p -h127.0.0.1 -P10001

At this point, you can redirect your application to communicate through 127.0.0.1 port 10001 as a secured MySQL connection via SSH.

Dedicated SSH user

Above method works fine if you are running as user root, but this is not the safest method. Since running the SSH command does not require super user privilege, we should create a specific user other than root specifically for this process. In this example, I created a user called ‘myuser’.

1. On both servers, create the user and assign a password.:

$ useradd myuser
$ passwd myuser

2. On application/web server, generate a SSH key for myuser:

$ su - myuser
$ whoami 
myuser
$ ssh-keygen -t rsa # press enter on all prompts

3. Then, copy the SSH key to MySQL server, 10.0.0.21:

$ ssh-copy-id 10.0.0.21 # enter the myuser password for 10.0.0.21 if prompted

4. Start the SSH port forwarding using following command:

$ ssh -fNg -L 10001:127.0.0.1:3306 10.0.0.21

5. To auto execute the command after a reboot, just add following line under /etc/rc.local:

runuser -l myuser -c 'ssh -fNg -L 10001:127.0.0.1:3306 10.0.0.21'

Now, it’s safer to have a dedicated user to perform the port forwarding process.

Setting up Supervisor

Now we have secured our MySQL connection. We need to ensure the SSH process is monitored so when it fails (or if the server rebooted) it will be automatically restarted. You can basically put the command directly inside /etc/rc.local (as shown above), which will be executed automatically upon startup, but this does not cover the worst-case scenario where the process crashes, stops or being killed. This is where Supervisor comes in handy.

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

1. Install Supervisor via python easy_install:

$ yum install -y python-setuptools
$ easy_install supervisor

2. Create a configuration file, /etc/supervisord.conf:

$ vim /etc/supervisord.conf

And add following lines:

[supervisord]
nodaemon=false
logfile=/var/log/supervisord.log
pidfile=/var/run/supervisord.pid
 
[program:mysql_secure]
command=ssh -Ng -L 10001:127.0.0.1:3306 10.0.0.21
user=myuser

3. To start Supervisor, just run following command:

$ supervisord -c /etc/supervisord.conf

4. To ensure it starts on boot, we add following line into /etc/rc.local:

/usr/bin/supervisord -c /etc/supervisord.conf

Here is some excerpt from /var/log/supervisord.log indicating it monitors the process correctly:

$ less /var/log/supervisord.log
2015-05-19 20:22:14,093 CRIT Supervisor running as root (no user in config file)
2015-05-19 20:22:14,100 INFO daemonizing the supervisord process
2015-05-19 20:22:14,101 INFO supervisord started with pid 1944
2015-05-19 20:22:15,105 INFO spawned: 'mysql_secure' with pid 1945
2015-05-19 20:22:16,108 INFO success: mysql_secure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-05-19 20:22:24,581 CRIT Supervisor running as root (no user in config file)
2015-05-19 20:22:24,585 INFO daemonizing the supervisord process
2015-05-19 20:22:24,585 INFO supervisord started with pid 1952
2015-05-19 20:22:25,591 INFO spawned: 'mysql_secure' with pid 1953
2015-05-19 20:22:26,801 INFO success: mysql_secure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

That’s all folks!

 

CentOS 7: Installing and Managing MySQL

Starting from CentOS/RHEL 7, standard MySQL (Oracle) package is no longer available and has been replaced by MariaDB. There will be almost no difference when managing MariaDB since it is basically a drop-in replacement for MySQL. Certainly, MariaDB has attracted huge attention and many of existing MySQL users have been switching to MariaDB, this includes Google and Tumblr.

To install MySQL/MariaDB on CentOS 7 box, just use following command:

$ yum install mariadb mariadb-server

In RHEL, when you run yum install mysql, it will automatically install mariadb 5.5. The MySQL configuration still located in the familiar location: /etc/my.cnf, the MySQL error log is located at /var/log/mariadb/mariadb.log while the data directory is still located at /var/lib/mysql.

CentOS 7 runs on systemd, thus to start the service (similar to service mysqld start):

$ systemctl start mariadb.service

** Other options are: restart, stop, status

To enable the service to start on boot (similar to chkconfig mysqld on):

$ systemctl enable mariadb.service

Above are the only differences when managing MySQL running on CentOS/RHEL 7. To retrieve the list of services with the status, use following command:

$ systemctl list-units

Or:

$ systemctl list-unit-files

That’s it. Having MariaDB as replacement for MySQL is definitely a great choice. You would have no regret using it!