CentOS 7: Installing and Managing MySQL

Starting from CentOS/RHEL 7, standard MySQL (Oracle) package is no longer available and has been replaced by MariaDB. There will be almost no difference when managing MariaDB since it is basically a drop-in replacement for MySQL. Certainly, MariaDB has attracted huge attention and many of existing MySQL users have been switching to MariaDB, this includes Google and Tumblr.

To install MySQL/MariaDB on CentOS 7 box, just use following command:

$ yum install mariadb mariadb-server

In RHEL, when you run yum install mysql, it will automatically install mariadb 5.5. The MySQL configuration still located in the familiar location: /etc/my.cnf, the MySQL error log is located at /var/log/mariadb/mariadb.log while the data directory is still located at /var/lib/mysql.

CentOS 7 runs on systemd, thus to start the service (similar to service mysqld start):

$ systemctl start mariadb.service

** Other options are: restart, stop, status

To enable the service to start on boot (similar to chkconfig mysqld on):

$ systemctl enable mariadb.service

Above are the only differences when managing MySQL running on CentOS/RHEL 7. To retrieve the list of services with the status, use following command:

$ systemctl list-units

Or:

$ systemctl list-unit-files

That’s it. Having MariaDB as replacement for MySQL is definitely a great choice. You would have no regret using it!

Install Puppet on CentOS, Redhat, Ubuntu, Debian

Lately, I have been spending my time doing research on developing a Puppet module. As you might know, Puppet is an orchestration tool used by many sysadmins to deploy and configure servers without hassle on repeating the same installation commands over and over again.

With Puppet you just simply:

  1. Install Puppet master
  2. Define node configuration in Puppet master
  3. Install Puppet agent
  4. Let the Puppet agent deploy what you have defined

Developing Puppet module requires me to test the module’s manifest on many popular OS distributions out there. I am going to stick with the subject of this post on installing Puppet on following OS distributions:

  • RHEL 6/CentOS 6
  • RHEL 5/CentOS 5
  • Ubuntu 12.04
  • Ubuntu 14.04
  • Debian 6
  • Debian 7

 

Requirement

Ensure the host’s date and time is synced through ntp and /etc/hosts is configured correctly. Following is the example of /etc/hosts definition that I used:

192.168.10.100    puppetmaster.local
192.168.10.101    mysql1.local   # puppet-agent
192.168.10.102    mysql2.local   # puppet-agent
192.168.10.103    mysql3.local   # puppet-agent

** The /etc/hosts must be same on all nodes so hostname can be resolved to an IP. This is required later during the certificate signing stage by puppet master.

RHEL-based:

yum install -y ntpdate
ntpdate -u my.pool.ntp.org

Debian-based:

sudo apt-get install -y ntpdate
ntpdate -u my.pool.ntp.org

Installing Puppetlabs Repository

We’ll need to install official Puppetlabs repository on all nodes regardless of their role in puppet for the installation purposes. Install the repository definition on respective OS distribution:

RHEL 6/CentOS 6:

rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

RHEL 5/CentOS 5:

rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-5.noarch.rpm

Ubuntu 12.04 (Precise):

wget https://apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb

Ubuntu 14.04 (Trusty):

wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
sudo dpkg -i puppetlabs-release-trusty.deb

Debian 6 (Squeeze):

wget https://apt.puppetlabs.com/puppetlabs-release-squeeze.deb
dpkg -i puppetlabs-release-squeeze.deb

Debian 7 (Wheezy):

wget https://apt.puppetlabs.com/puppetlabs-release-wheezy.deb
dpkg -i puppetlabs-release-wheezy.deb

 

Installing Puppet Master

On puppetmaster.local node, run following command to install Puppet master:

Redhat-based:

yum install -y puppet-server openssl

Debian-based:

sudo apt-get update
sudo apt-get install -y puppetmaster openssl

 

Installing Puppet Agent

On all puppet agent nodes (mysql1.local, mysql2.local, mysql3.local), install puppet agent and its dependencies:

Redhat-based:

yum install -y puppet facter openssl

Debian-based:

sudo apt-get update
sudo apt-get install -y puppet facter openssl

 

Signing the Certificate Authority

Puppet communicates through a secured channel with SSL. When puppet agent runs for the first time, it will auto-generate a new SSL and puppet master must sign it before all the communications begin. Run following command on each agent node:

$ puppet agent --server=puppetmaser.local --no-daemonize --verbose
Info: Creating a new SSL key for mysql1.local
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for ccpuppet.local
Info: Certificate Request fingerprint (SHA256): 6F:8B:92:46:B0:3F:04:0A:4F:8D:BD:56:77:24:77:50:1C:E9:F4:EE:C6:00:5E:82:4F:B0:85:B5:26:72:43:E0
Info: Caching certificate for ca

This will generate a certificate to be signed by the puppet master. Now in the puppet master, list the certificate authority (CA):

$ puppet ca list
mysql1.local  (SHA256) 6F:8B:92:46:B0:3F:04:0A:4F:8D:BD:56:77:24:77:50:1C:E9:F4:EE:C6:00:5E:82:4F:B0:85:B5:26:72:43:E0

Sign the CA for this agent:

$ puppet ca sign mysql1.local
Notice: Signed certificate request for mysql1.local
Notice: Removing file Puppet::SSL::CertificateRequest ccpuppet.local at '/var/lib/puppet/ssl/ca/requests/mysql1.local.pem'
"-----BEGIN CERTIFICATE-----\n
...the key..."

Repeat the above steps on the other nodes, mysql2.local and mysql3.local. Now the puppet master should able to communicate with its agents securely.

 

Configure Puppet Agent

The last step is to update /etc/puppet/puppet.conf and add following line under [main] directive:

server=puppetmaster.local

** If you do not configure as above, you will need to add –server=puppetmaster.local on each of the puppet agent command below.

Now you can test from the agent node:

puppet agent --test

 

Deploy MySQL through Puppet

Puppet is now ready. Let’s deploy a mysql server with the simplest way. Go to Puppet Forge and look for a puppet module called puppetlabs-mysql. To install this module, run following command on to the puppet master node:

puppet module install puppetlabs-mysql

On the puppetmaster.local, create a puppet manifest to define how the agent should deploy at /etc/puppet/manifests/site.pp:

# /etc/puppet/manifests/site.pp
 
# Default node - this is compulsory
node "default" {
}
 
# Define the agent nodes
node "mysql1.local", "mysql2.local", "mysql3.local" {
 class { '::mysql::server':
 root_password => 'strongpassword'
 }
 class { '::mysql::client':
 package_ensure => 'present'
 }
}

Above definition will tell Puppet to install a MySQL server and client package on mysql1.local, mysql2.local and mysql3.local. To immediately start the deployment, go to the agent node and run:

puppet agent --test

Or, wait for the Puppet agent service to apply the catalog automatically (depending on the runinterval value, default is 30 minutes).

Once done, you will have three MySQL servers ready to serve! Imagine how much time you would save if you have many servers, applications, softwares and configurations to maintain.

Sysbench 0.5 + Ubuntu 14.04 (Trusty) + Percona Server or XtraDB Cluster

I need to perform benchmarks on Percona XtraDB Cluster and Percona Server directly from my Ubuntu 14.04 box. Sysbench 0.5 is the latest stable to date and it supports benchmarking Galera cluster. If you are running on Sysbench 0.4, you would most likely encounter one of following errors if you are running it against a Galera cluster with high value of threads:

ALERT: failed to execute mysql_stmt_execute(): Err1317 Query execution was interrupted
ALERT: failed to execute mysql_stmt_execute(): Err1062 Duplicate entry '2199902' for key 'PRIMARY'

This post presumes that you have already installed standard Percona related packages including the apt-get repository to run a MySQL server (as instructed on Percona’s  documentation page).

 

Installing Sysbench 0.5

1. Install compiler tools and required packages:

$ sudo apt-get install -y gcc autoconf automake make libtool libssl-dev libcrypto++9

2. Install mysql_config (available from Percona repository):

$ sudo apt-get install -y libperconaserverclient18-dev

3. Get Sysbench from Launchpad:

$ bzr branch lp:sysbench

4. Compile and install sysbench:

$ cd sysbench
$ ./autogen.sh
$ ./configure --prefix=/usr --mandir=/usr/share/man
$ make
$ sudo make install

5. Create LUA template directory:

$ sudo mkdir /usr/share/sysbench/tests/db -p
$ sudo cp sysbench/tests/db/* /usr/share/sysbench/tests/db

6. Verify if sysbench is running in the correct version:

$ sysbench --version
sysbench 0.5

Running Sysbench

1. Before starting the benchmark, we need to prepare the test data. Following command will generate a table consists of 20 million rows on a Galera Cluster through HAproxy which running on 127.0.0.1 port 33306:

$ sysbench \
--db-driver=mysql \
--mysql-table-engine=innodb \
--oltp-table-size=20000000 \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
--test=/usr/share/sysbench/tests/db/oltp.lua \
prepare

Or we can generate 8 tables consists of 10M rows in parallel with 8 threads (total of 80M rows of data set):

$ sysbench \
--db-driver=mysql \
--mysql-table-engine=innodb \
--oltp-table-size=10000000 \
--oltp-tables-count=8 \
--num-threads=8 \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
--test=/usr/share/sysbench/tests/db/parallel_prepare.lua \
run

2. Start the benchmark against Galera cluster:

$ sysbench \
--db-driver=mysql \
--num-threads=8 \
--max-requests=5000000 \
--oltp-table-size=20000000 \
--oltp-test-mode=complex \
--test=/usr/share/sysbench/tests/db/oltp.lua \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
run

You can perform other tests based on the LUA templates exist under /usr/share/sysbench/tests/db. Just change the –test parameter to the fullpath of the template file and you are good to go.

Details on this can be found at http://www.percona.com/docs/wiki/benchmark:sysbench:olpt.lua

 

Replacing OpenJDK with Oracle JDK on CentOS 6

I encountered some issues with an application running on Java. I thought it would be fine if running on standard OpenJDK release available at CentOS repository since it is really easy to install with yum, but it wasn’t. The application does required JDK from Oracle release (which is lame).

So here what you should do when you want to replace OpenJDK with Java SE JDK:

1. Check the current OpenJDK version. I am going to retain the same version if possible:

java -version
java version "1.6.0_30"
OpenJDK Runtime Environment (IcedTea6 1.13.3) (rhel-5.1.13.3.el6_5-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Check the current RPM package name:

$ rpm -qa | grep openjdk
java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.x86_64

2. Download Java SE Development Kit 6 update 30, which is equivalent to OpenJDK 1.6.0_30. There is no way (AFAIK) that we can download it directly using curl or wget. You need to have an account with Oracle. Sign up using your web browser and you will get a temporary link similar to below. Copy it from  your web browser and use wget to download it directly to the box:

$ wget -O jdk-6u30-linux-x64-rpm.bin http://download.oracle.com/otn/java/jdk/6u30-b12/jdk-6u30-linux-x64-rpm.bin?AuthParam=1397834288_32d1181063a10cd15522d23abbc5e4f7

**You should do this quick. The link will be expired within minutes.

3. Remove the installed OpenJDK package:

$ yum remove -y java-1.6.0-openjdk

4. Install Oracle JDK:

$ chmod 755 jdk-6u30-linux-x64-rpm.bin
$ ./jdk-6u30-linux-x64-rpm.bin

5. Verify the new Java version:

java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Now my application works like a charm! The new path to JRE now is /usr/java/jdk1.6.0_30/jre/

Importing Big mysqldump with Progress Bar

I have been facing hard time to import a big dump file (25 GB in size to be exact). It could take hours to load data in so it could be worth to watch the progress of this import. Fortunately, Linux has always had a convenient way to do this. I just need to install pv,  which can monitor the progress of piping data.

Enough with talking, let’s start installing!

I am using CentOS 6.4 box, so it requires me to install EPEL repo at the first place:

$ rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Install pv using yum:

$ yum install -y pv

Installation done. Let the importing begin!

$ pv /home/user/my_big_dump.sql | mysql -uroot -p
928MB 0:07:41 [ 2.3MB/s] [==>                                                      ]  4% ETA 2:35:52

 

So now I can sit back and relax while watching up the import progress. If no error occurred, the import could finish within two and a half hours. Cheers!

Debian: Converting Apache + PHP to FastCGI – The Simple Way

I have a server running on Debian 6 64bit installed with Apache and PHP5 using apt-get package manager. By default, it will configure DSO  as the PHP handler (more details here). I need to convert it to serve PHP through FastCGI (mod_fcgid) to save memory usage. It turns to be that converting them is easy and require simple steps.

 

Here is my pre-configured Apache + PHP settings (installed using apt-get install apache2 php5 command):

$ dpkg --get-selections | grep apache
apache2             install
apache2-mpm-prefork install
apache2-utils       install
apache2.2-bin       install
apache2.2-common    install
libapache2-mod-php5 install
 
$ dpkg --get-selections | grep php
libapache2-mod-php5 install
php5                install
php5-cli            install
php5-common         install
php5-suhosin        install

 

1. Update repository:

$ apt-get update

2. Install required packages for fcgid:

$ apt-get install apache2-mpm-worker libapache2-mod-fcgid php5-cgi

3. Enable cgi.fix_pathinfo in /etc/php5/apache2/php.ini:

cgi.fix_pathinfo=1

4. Stop Apache:

$ /etc/init.d/apache2 stop

5. Disable php5, since we will be using php-cgi:

$ a2dismod php5

6. Setup the virtual host for the website under /etc/apache2/sites-available/default:

 <VirtualHost *:80>
        ServerName www.example.com
        ServerAdmin admin@example.com
        DocumentRoot /var/www
 
        <Directory /var/www>
                Options +ExecCGI
                AllowOverride AuthConfig FileInfo Limit
                AddHandler fcgid-script .php
                FCGIWrapper /usr/lib/cgi-bin/php .php
                Order Deny,Allow
                Allow from All
        </Directory>
 
        ErrorLog /var/log/apache2/error.log
        LogLevel warn
 
        CustomLog /var/log/apache2/access.log combined
 
</VirtualHost>

7. Start Apache:

$ /etc/init.d/apache2 start

 

Done! You are now running on FastCGI as the PHP handler. You can verify this with phpinfo:

phpinfo

 

Install grsecurity with Yum

Easiest way to install grsecurity. The good thing about it is only grsecurity provides protection against zero-day and other advanced threats that buys administrators valuable time while vulnerability fixes make their way out to distributions and production testing.

1. Go into repository directory:

$ cd /etc/yum.repos.d/
$ wget http://repos.coredumb.net/grsecurity/grsecurity.repo

2. Install grsecurity kernel and administrator tools:

$ yum clean all
$ yum install kernel gradm

3. Reboot so it could load the grkernel:

$ reboot

 

Importing IMDb Sample Data Set to MySQL

Here in this post, I am going to show you on how to import IMDb plain text data files available at this page into your MySQL database server. I was using this data set to perform various benchmark tests around a moderately-large database size. If you want to have a small MySQL database sample data, you can try to use Sakila or World database available at MySQL Documentation Page.

I will be using CentOS 6.3 64bit as the OS platform and presume that MySQL has already installed and running. Make sure that you are having at least 4 GB of free space in the partition that MySQL @@datadir reside to allow this long-running process (3 to 5 hours – depending on your hardware specs + server workload) to successfully complete.

We will be using a Python-based package called IMDbPy. This package requires you to have Python with some development libraries installed as well as SQLObject. Then, we will need to download all data files from IMDb mirror site: ftp://ftp.fu-berlin.de/pub/misc/movies/database/, create the database and start the import process.

 

Installing IMDbPy and SQLObject

1. Install required packages using yum:

$ yum install -y gcc python python-devel openssl-devel libxml2-devel libxslt-devel zlib-devel MySQL-python python-setuptools python-pip

2. Install SQLObject using Python EasyInstall:

$ easy_install -U SQLObject

3. Download IMDbPy from this page into the MySQL server, extract it and start the installation process:

$ wget http://prdownloads.sourceforge.net/imdbpy/IMDbPY-4.9.tar.gz
$ tar -xzf IMDbPY-4.9.tar.gz
$ cd IMDbPY-*
$ python setup.py install

 

Importing Data

1. Create a directory to dump all the data files that we will download:

$ mkdir /root/data
$ cd /root/data

2. Download only .gz file from the IMDb mirror site to /root/data :

$ wget -r --accept="*.gz" --no-directories --no-host-directories --level 1 ftp://ftp.fu-berlin.de/pub/misc/movies/database/

3. Create a database in MySQL called ‘imdb’, with user ‘imdb’ and password ‘imdb’. We will then GRANT the user to the designated database:

mysql> CREATE DATABASE imdb;
mysql> GRANT ALL PRIVILEGES ON imdb.* TO 'imdb'@'localhost' IDENTIFIED BY 'imdb';
mysql> FLUSH PRIVILEGES;

3. Start the import process with -u and -d flag:

$ imdbpy2sql.py -d /root/data/ -u 'mysql://imdb:[email protected]/imdb'

Take note that -d is the directory of the .gz dump files are located and -u is the connection string for our MySQL database server. You can change the connection string to any of SQLObject’s supported database such as PostgreSQL, SQLite, Firebird and MAX DB. Please refer to this documentation for details.

You will see similar output as below which indicates the importing process has started:

SCANNING movies: Last Sunset (2006) (movieID: 2130001)
SCANNING movies: Legend of Hell (2012) (movieID: 2140001)
SCANNING movies: Lifestyles of Squirrels (2011) (movieID: 2150001)
SCANNING movies: Los signos del tiempo (1983) (movieID: 2160001)
SCANNING movies: Madame T (2012) (movieID: 2170001)
SCANNING movies: Marijji ringu (2007) (movieID: 2180001)
SCANNING movies: Menculik miyabi (2010) (movieID: 2190001)
* FLUSHING MoviesCache...

Wait up until it finish and you will have large sample data to play around in your MySQL server!

 

Install MariaDB Galera Cluster in Ubuntu

I am going to show you on how to install MariaDB Cluster (with Galera) in Ubuntu Precise (12.04) LTS 64bit. You need at least 3 hosts running together to form a reliable cluster.

Hosts list:

maria1 – 192.168.10.101
maria2 – 192.168.10.102
maria3 – 192.168.10.103

 

Prepare Hosts

In all hosts, this is my /etc/hosts definition:

192.168.10.101        maria1 maria1.mycluster.com
192.168.10.102        maria2 maria2.mycluster.com
192.168.10.103        maria3 maria3.mycluster.com

 

Install MariaDB Cluster

1. Install the GPG key required by apt:

$ sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 1BB943DB

2. Add the apt repository:

$ sudo add-apt-repository 'deb http://mirror.stshosting.co.uk/mariadb/repo/5.5/ubuntu precise main'

3. Get the latest repo:

$ sudo apt-get update

4. Install MariaDB Cluster and related packages:

$ sudo apt-get install -y galera  mariadb-galera-server-5.5 mariadb-client-5.5 libmariadbclient18 mariadb-client-core-5.5 rsync netcat-openbsd

** Make sure you put the same MySQL root password on all hosts

 

Install Percona Xtrabackup

1. We are going to use Xtrabackup to perform State Snapshot Transfer (SST). Install GPG key for Percona:

$ sudo apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A

2. Add the apt repository for Percona:

$ sudo add-apt-repository 'deb http://repo.percona.com/apt precise  main'

3. Update and install Percona Xtrabackup:

$ sudo apt-get update && sudo apt-get -y install percona-toolkit percona-xtrabackup

 

Configure MariaDB Cluster

1. Once the installation completed, you need to comment following lines in /etc/mysql/my.cnf:

#bind-address
#default_storage_engine
#query_cache_limit
#query_cache_size

2. Add following lines for wsrep configuration options in /etc/mysql/my.cnf under [mysqld] directive:

binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_doublewrite=1

3. Add following lines into specific node under [mysqld] directive:

maria1:

# wsrep provider configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=256M; gcache.page_size=128M"
wsrep_cluster_address=gcomm://
wsrep_cluster_name="MariaDB_Cluster"
wsrep_node_address="maria1"
wsrep_node_name="maria1"
wsrep_sst_method=xtrabackup
wsrep_sst_auth="root:MyR00tPasswd"
wsrep_node_incoming_address=192.168.10.101
wsrep_sst_receive_address=192.168.10.101
wsrep_slave_threads=16

maria2:

# wsrep provider configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=256M; gcache.page_size=128M"
wsrep_cluster_address=gcomm://maria1
wsrep_cluster_name="MariaDB_Cluster"
wsrep_node_address="maria2"
wsrep_node_name="maria2"
wsrep_sst_method=xtrabackup
wsrep_sst_auth="root:MyR00tPasswd"
wsrep_node_incoming_address=192.168.10.102
wsrep_sst_receive_address=192.168.10.102
wsrep_slave_threads=16

maria3:

# wsrep provider configuration
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=256M; gcache.page_size=128M"
wsrep_cluster_address=gcomm://maria1
wsrep_cluster_name="MariaDB_Cluster"
wsrep_node_address="maria3"
wsrep_node_name="maria3"
wsrep_sst_method=xtrabackup
wsrep_sst_auth="root:MyR00tPasswd"
wsrep_node_incoming_address=192.168.10.103
wsrep_sst_receive_address=192.168.10.103
wsrep_slave_threads=16

 

Initialize the Cluster

1. Login to maria1 to initialize the cluster by restarting the service:

$ sudo service mysql restart

2. Login to maria2 and maria3 and restart MySQL. This will allow maria2 and maria3 to perform SST from maria1:

$ sudo service mysql restart

3. Login to MariaDB console and check for following values:

MariaDB [(none)]> show status like 'wsrep_%';

And you should see following output:

| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_incoming_addresses | 192.168.10.101:3306,192.168.10.102:3306,192.168.10.103:3306 |
| wsrep_cluster_size | 3 |
| wsrep_ready | ON |

You could now point your web server or application server to one of MariaDB cluster node to query.

 

Notes

Once the cluster successfully initialized and connected, you should change the wsrep_cluster_address to other nodes in case if the server rebooted. So, login to respective server and change the wsrep_cluster_address to following value:

wsrep_cluster_address=gcomm://maria1,maria2,maria3

CentOS: Install and Configure MongoDB Sharded Cluster

In this post I am going to deploy a MongoDB sharded cluster. MongoDB is an open-source NoSQL, document-oriented database designed for ease of development and scaling.  I am going to use 3 servers, and all the /etc/hosts definition would be as below:

192.168.0.41        mongo1 mongo1.cluster.local
192.168.0.42        mongo2 mongo2.cluster.local
192.168.0.43        mongo3 mongo3.cluster.local

All servers running CentOS 6.3 64bit with firewall and SElinux turned off. All steps must be executed in all servers unless specified.

Install MongoDB

1. Install EPEL repo:

$ rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

2. Install MongoDB and all required components:

$ yum install mongodb* -y --enablerepo=epel

 

Config Servers

1. Create config database directory. By default, MongoDB will use /data/configdb:

$ mkdir -p /data/configdb

2. Default port for config server is 27019. Start config servers:

$ mongod --configsvr --fork --logpath /var/log/mongodb.log --logappend

You should see following output:

forked process: 5464
all output going to: /var/log/mongodb.log
child process started successfully, parent exiting

 

Routing Servers

1. By default, mongos will listen on port 27017. Start mongos as below:

$ mongos --configdb mongo1,mongo2,mongo3 --fork --logpath /var/log/mongodb.log --logappend

You should see following output:

forked process: 5534
all output going to: /var/log/mongodb.log
child process started successfully, parent exiting

Shard Servers

1. Create default data directory. By default, MongoDB will use /data/db:

$ mkdir -p /data/db

2. By default, mongod with –shardsvr option will listen on port 27018. Start mongod as below:

$ mongod --shardsvr --fork --logpath /var/log/mongodb.log --logappend

You should see following output:

forked process: 5675
all output going to: /var/log/mongodb.log
child process started successfully, parent exiting

 

MongoDB Sharding

1. Verify that MongoDB services are listening to correct ports:

$ netstat -tulpn | grep mongo
 
tcp     0    0      0.0.0.0:27017     0.0.0.0:*     LISTEN    5534/mongos
tcp     0    0      0.0.0.0:27018     0.0.0.0:*     LISTEN    5675/mongod
tcp     0    0      0.0.0.0:27019     0.0.0.0:*     LISTEN    5464/mongod
tcp     0    0      0.0.0.0:28017     0.0.0.0:*     LISTEN    5534/mongos
tcp     0    0      0.0.0.0:28018     0.0.0.0:*     LISTEN    5675/mongod
tcp     0    0      0.0.0.0:28019     0.0.0.0:*     LISTEN    5464/mongod

2. SSH into mongo1 and type mongo to access the mongos console:

$ mongo

3. Use admin database to list the sharding status:

mongos> use admin
mongos> db.runCommand( { listshards : 1 } );

You should get this reply:

{ "shards" : [ ], "ok" : 1 }

4. Add the sharded servers by specifying the hostname and MongoDB shard service port:

mongos> sh.addShard( "mongo1:27018");
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard( "mongo2:27018");
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> sh.addShard( "mongo3:27018");
{ "shardAdded" : "shard0002", "ok" : 1 }

5. Download this JSON example file and import into database mydb:

$ wget http://media.mongodb.org/zips.json
$ mongoimport --db mydb --collection zip --file zips.json
connected to: 127.0.0.1
Mon Mar 25 06:22:35 imported 29470 objects

6. Enable sharding for mydb:

mongos> sh.enableSharding ("mydb");
{ "ok" : 1 }

7. Check sharding status:

mongos> sh.status()
--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{ "_id" : "shard0000", "host" : "mongo1:27018" }
{ "_id" : "shard0001", "host" : "mongo2:27018" }
{ "_id" : "shard0002", "host" : "mongo3:27018" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "mydb", "partitioned" : true, "primary" : "shard0000" }
{ "_id" : "test", "partitioned" : false, "primary" : "shard0001" }

You can see database mydb has been partitioned by MongoDB with value true.

CentOS: Install MongoDB – The Simple Way

I am in phase of learning a NoSQL database called MongoDB. I will be using a CentOS 6.3 64bit box with minimal ISO installation disc with several package installed like perl, vim, wget, screen, sudo and cronie using yum.

We will use EPEL repo, which includes MongoDB installation package to simplify the deployment.

1. Install EPEL repo for CentOS 6. You can get the link from here, http://dl.fedoraproject.org/pub/epel/6/x86_64/:

$ rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

2. Install MongoDB using yum:

$ yum install mongodb* -y

3. Configure mongod to start on boot and start the service:

$ chkconfig mongod on
$ service mongod start

4. MongoDB will be using ports 27017-27019 and 28017. We will add it into the iptables rules:

$ iptables -A INPUT -m tcp -p tcp --dport 27017:27019 -j ACCEPT
$ iptables -A INPUT -m tcp -p tcp --dport 28017 -j ACCEPT

5. Check whether MongoDB is listening to the correct port:

$ netstat -tulpn | grep mongod
tcp        0      0 127.0.0.1:27017             0.0.0.0:*                   LISTEN      26575/mongod

6. Login into MongoDB console by using this command:

$ mongo

7. In the console, you can use help command to see the list of supported command as below:

> help
db.help()         help on db methods
db.mycoll.help()  help on collection methods
sh.help()         sharding helpers
rs.help()         replica set helpers
help admin        administrative help
help connect      connecting to a db help
help keys         key shortcuts
help misc         misc things to know
help mr           mapreduce
 
show dbs                    show database names
show collections            show collections in current database
show users                  show users in current database
show profile                show most recent system.profile entries with time >= 1ms
show logs                   show the accessible logger names
show log [name]             prints out the last segment of log in memory, 'global' is default
use <db_name>               set current database
db.foo.find()               list objects in collection foo
db.foo.find( { a : 1 } )    list objects in foo where a == 1
it                          result of the last line evaluated; use to further iterate
DBQuery.shellBatchSize = x  set default number of items to display on shell
exit                        quit the mongo shell

So now I have required stuffs installed for MongoDB. Lets learn MongoDB by starting at this page: http://docs.mongodb.org/manual/tutorial/getting-started/#create-a-collection-and-insert-documents

CentOS: Install OpenLDAP with Webmin – The Simple Way

Installing OpenLDAP with Webmin will require a lot of steps. I have created a BASH script to install OpenLDAP with Webmin in CentOS 6 servers. To install, simply download the installer script at here:

Installation example will be as below. I am using a freshly installed CentOS 6.3 64bit installed with minimal ISO, with wget and perl installed.

1. Download and extract the installer script:

$ cd /usr/local/src
$ wget http://blog.secaserver.com/files/openldap_installer.sh

2. Change the permission to 755:

$ chmod 755 openldap_installer.sh

3. Execute the script and follow the wizard as example below:

$ ./openldap_installer.sh
===========================================================
           This script will install OpenLDAP
It assumes that there is no OpenLDAP installed in this host
   SElinux will be disabled and firewall will be stopped
===========================================================
 
What is the root domain? [eg mydomain.com]: majimbu.net
What is the administrator domain? [eg ldap.majimbu.net or manager.majimbu.net]: ldap.majimbu.net
What is the administrator password that you want to use?: MyN23pQ
Do you want to install Webmin/Do you want me to configure your Webmin LDAP modules? [Y/n]: Y

You should see the installation process output as below:

=================================================================
Kindly review following details before proceed with installation:
=================================================================
Hostname: ldap.majimbu.net
Root DN: dc=majimbu,dc=net
Administrator DN: cn=ldap,dc=majimbu,dc=net
Administrator Password: MyN23pQ
Webmin installation: Y
=================================================================
 
Can I proceed with the installation? [Y/n]: Y
Checking whether openldap-servers has been installed..
openldap-servers package not found. Proceed with installation
Disabling SElinux and stopping firewall..
iptables: Flushing firewall rules:                                 [ OK ]
iptables: Setting chains to policy ACCEPT: filter                  [ OK ]
iptables: Unloading modules:                                       [ OK ]
Installing OpenLDAP using yum..
Package cronie-1.4.4-7.el6.x86_64 already installed and latest version
Package sudo-1.7.4p5-13.el6_3.x86_64 already installed and latest version
OpenLDAP installed
Configuring OpenLDAP database..
Configuring monitoring privileges..
Configuring database cache..
Generating SSL..
Generating a 2048 bit RSA private key
..........................+++
......................+++
writing new private key to '/etc/openldap/certs/majimbu_key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:MY
State or Province Name (full name) []:Kuala Lumpur
Locality Name (eg, city) [Default City]:Bukit Bintang
Organization Name (eg, company) [Default Company Ltd]:Majimbu Net Corp
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:ldap.majimbu.net
Email Address []:[email protected]
Configuring LDAP service..
Checking OpenLDAP configuration..
config file testing succeeded
OpenLDAP installation done. Starting SLAPD..
Starting slapd:                                                    [ OK ]
Configuring LDAP client inside this host..
Checking the Webmin installation..
Webmin package not found in this host. Installing Webmin..
Retrieving http://www.webmin.com/download/rpm/webmin-current.rpm
warning: /var/tmp/rpm-tmp.XmXunn: Header V3 DSA/SHA1 Signature, key ID 11f63c51: NOKEY
Preparing... ########################################### [100%]
Operating system is CentOS Linux
    1:webmin ########################################### [100%]
Webmin install complete. You can now login to http://ldap.majimbu.net:10000/
as root with your root password.
Webmin installed.
Configuring webmin LDAP server module..
Configuring webmin LDAP client module..
Installation completed! [ OK ]
============================================================================
    You may need to open following port in firewall: 389, 636, 10000
Dont forget to refresh your Webmin module! Login to Webmin > Refresh Modules
============================================================================

 

4. Installation done. We need to refresh the Webmin module from the Webmin page. Login into Webmin > Refresh Modules:

webmin_refresh

 

5. You need to refresh again the Webmin page so the activated module will be listed in the side menu as screen shot below:

webmin_ldap

You can now start to create your LDAP object using your Webmin modules Webmin > Servers > LDAP Server To add port exception into firewall rules, you can use following command:

$ iptables -I INPUT -m tcp -p tcp --dport 389 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 636 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 10000 -j ACCEPT