Various Ways to Determine Public IP on Linux CLI

Always when you are working a lot with CLI environment, Linux particularly, you would like to know the public IP address especially when you were running on NAT environment. Here is a list of command that you can use to determine the public IP of your host via command line.

Using curl

cURL is mostly available on all Linux distributions, and is the most popular way to determine public IP address of the host. You just need to know the URL or host that will response with the correct public IP as per below:

$ curl ifconfig.me
52.74.127.152
 
$ curl ipecho.net/plain
52.74.127.152
 
$ curl ident.me
52.74.127.152
 
$ curl icanhazip.com
52.74.127.152
 
$ curl bot.whatismyipaddress.com
52.74.127.152

Using wget

Basically, command executed on curl can be replace with wget -qO- option, as per below:

$ wget -qO- http://ipecho.net/plain
52.74.127.152

Using Lynx

Lynx is a text-based browser which runs like a browser for your CLI

$ lynx checkip.dyndns.org # you will be redirected to a text-based browser

If you have the simplest method apart from what being mentioned here, please share it. I can’t imagine how simple it would be!

 

CentOS 7: Installing and Managing MySQL

Starting from CentOS/RHEL 7, standard MySQL (Oracle) package is no longer available and has been replaced by MariaDB. There will be almost no difference when managing MariaDB since it is basically a drop-in replacement for MySQL. Certainly, MariaDB has attracted huge attention and many of existing MySQL users have been switching to MariaDB, this includes Google and Tumblr.

To install MySQL/MariaDB on CentOS 7 box, just use following command:

$ yum install mariadb mariadb-server

In RHEL, when you run yum install mysql, it will automatically install mariadb 5.5. The MySQL configuration still located in the familiar location: /etc/my.cnf, the MySQL error log is located at /var/log/mariadb/mariadb.log while the data directory is still located at /var/lib/mysql.

CentOS 7 runs on systemd, thus to start the service (similar to service mysqld start):

$ systemctl start mariadb.service

** Other options are: restart, stop, status

To enable the service to start on boot (similar to chkconfig mysqld on):

$ systemctl enable mariadb.service

Above are the only differences when managing MySQL running on CentOS/RHEL 7. To retrieve the list of services with the status, use following command:

$ systemctl list-units

Or:

$ systemctl list-unit-files

That’s it. Having MariaDB as replacement for MySQL is definitely a great choice. You would have no regret using it!

CentOS/Redhat: How to Install MySQL Sandbox

When you have started to work heavily with MySQL servers, and you need to create, install, configure and destroy MySQL instances, MySQL Sandbox is a great tool to automate those tasks. MySQL Sandbox allows us to deploy a MySQL test environment within seconds with simple commands.

So following is what I would do to install MySQL Sandbox on my CentOS box and create 3 mysql instances for my test environment:

1. Install Perl CPAN package:

$ yum install perl-CPAN

2. Connect to Perl MCPAN shell :

$ perl -MCPAN -e shell

**Answer ‘yes’ for the prompt. Perl will auto-configure most of the stuff for you

3. Install required Perl modules:

install Test::More
install MySQL::Sandbox

4. MySQL Sandbox is recommended to run as a non-root user. Create a user called sandbox and switch to the user’s environment:

$ useradd sandbox
$ su - sandbox

5. Download the MySQL Sandox package from Launchpad:

$ wget https://launchpadlibrarian.net/187745286/MySQL-Sandbox-3.0.47.tar.gz

6. Add MySQL sandbox into environment path of user sandbox’s bashrc:

$ vim ~/.bashrc

And append following line:

export PATH=$PATH:/home/sandbox/MySQL-Sandbox-3.0.47/bin

7. Re-login as the sandbox user to activate environment:

$ exit
$ su - sandbox

8. Download MySQL generic TAR file from MySQL Server archive page:

$ wget http://downloads.mysql.com/archives/get/file/mysql-5.6.20-linux-glibc2.5-x86_64.tar.gz

** I used MySQL 5.6.20 for this installation

Installation completed. To create a single MySQL instances, use following command:

$ make_sandbox mysql-5.6.20-linux-glibc2.5-x86_64.tar.gz

I use following command to create 3 MySQL instances for my test environment:

$ make_multiple_sandbox mysql-5.6.20-linux-glibc2.5-x86_64.tar.gz

Your instances should be ready and running within minutes. They will be created under /home/sandbox/sandboxes.

Install Puppet on CentOS, Redhat, Ubuntu, Debian

Lately, I have been spending my time doing research on developing a Puppet module. As you might know, Puppet is an orchestration tool used by many sysadmins to deploy and configure servers without hassle on repeating the same installation commands over and over again.

With Puppet you just simply:

  1. Install Puppet master
  2. Define node configuration in Puppet master
  3. Install Puppet agent
  4. Let the Puppet agent deploy what you have defined

Developing Puppet module requires me to test the module’s manifest on many popular OS distributions out there. I am going to stick with the subject of this post on installing Puppet on following OS distributions:

  • RHEL 6/CentOS 6
  • RHEL 5/CentOS 5
  • Ubuntu 12.04
  • Ubuntu 14.04
  • Debian 6
  • Debian 7

 

Requirement

Ensure the host’s date and time is synced through ntp and /etc/hosts is configured correctly. Following is the example of /etc/hosts definition that I used:

192.168.10.100    puppetmaster.local
192.168.10.101    mysql1.local   # puppet-agent
192.168.10.102    mysql2.local   # puppet-agent
192.168.10.103    mysql3.local   # puppet-agent

** The /etc/hosts must be same on all nodes so hostname can be resolved to an IP. This is required later during the certificate signing stage by puppet master.

RHEL-based:

yum install -y ntpdate
ntpdate -u my.pool.ntp.org

Debian-based:

sudo apt-get install -y ntpdate
ntpdate -u my.pool.ntp.org

Installing Puppetlabs Repository

We’ll need to install official Puppetlabs repository on all nodes regardless of their role in puppet for the installation purposes. Install the repository definition on respective OS distribution:

RHEL 6/CentOS 6:

rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

RHEL 5/CentOS 5:

rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-5.noarch.rpm

Ubuntu 12.04 (Precise):

wget https://apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb

Ubuntu 14.04 (Trusty):

wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
sudo dpkg -i puppetlabs-release-trusty.deb

Debian 6 (Squeeze):

wget https://apt.puppetlabs.com/puppetlabs-release-squeeze.deb
dpkg -i puppetlabs-release-squeeze.deb

Debian 7 (Wheezy):

wget https://apt.puppetlabs.com/puppetlabs-release-wheezy.deb
dpkg -i puppetlabs-release-wheezy.deb

 

Installing Puppet Master

On puppetmaster.local node, run following command to install Puppet master:

Redhat-based:

yum install -y puppet-server openssl

Debian-based:

sudo apt-get update
sudo apt-get install -y puppetmaster openssl

 

Installing Puppet Agent

On all puppet agent nodes (mysql1.local, mysql2.local, mysql3.local), install puppet agent and its dependencies:

Redhat-based:

yum install -y puppet facter openssl

Debian-based:

sudo apt-get update
sudo apt-get install -y puppet facter openssl

 

Signing the Certificate Authority

Puppet communicates through a secured channel with SSL. When puppet agent runs for the first time, it will auto-generate a new SSL and puppet master must sign it before all the communications begin. Run following command on each agent node:

$ puppet agent --server=puppetmaser.local --no-daemonize --verbose
Info: Creating a new SSL key for mysql1.local
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for ccpuppet.local
Info: Certificate Request fingerprint (SHA256): 6F:8B:92:46:B0:3F:04:0A:4F:8D:BD:56:77:24:77:50:1C:E9:F4:EE:C6:00:5E:82:4F:B0:85:B5:26:72:43:E0
Info: Caching certificate for ca

This will generate a certificate to be signed by the puppet master. Now in the puppet master, list the certificate authority (CA):

$ puppet ca list
mysql1.local  (SHA256) 6F:8B:92:46:B0:3F:04:0A:4F:8D:BD:56:77:24:77:50:1C:E9:F4:EE:C6:00:5E:82:4F:B0:85:B5:26:72:43:E0

Sign the CA for this agent:

$ puppet ca sign mysql1.local
Notice: Signed certificate request for mysql1.local
Notice: Removing file Puppet::SSL::CertificateRequest ccpuppet.local at '/var/lib/puppet/ssl/ca/requests/mysql1.local.pem'
"-----BEGIN CERTIFICATE-----\n
...the key..."

Repeat the above steps on the other nodes, mysql2.local and mysql3.local. Now the puppet master should able to communicate with its agents securely.

 

Configure Puppet Agent

The last step is to update /etc/puppet/puppet.conf and add following line under [main] directive:

server=puppetmaster.local

** If you do not configure as above, you will need to add –server=puppetmaster.local on each of the puppet agent command below.

Now you can test from the agent node:

puppet agent --test

 

Deploy MySQL through Puppet

Puppet is now ready. Let’s deploy a mysql server with the simplest way. Go to Puppet Forge and look for a puppet module called puppetlabs-mysql. To install this module, run following command on to the puppet master node:

puppet module install puppetlabs-mysql

On the puppetmaster.local, create a puppet manifest to define how the agent should deploy at /etc/puppet/manifests/site.pp:

# /etc/puppet/manifests/site.pp
 
# Default node - this is compulsory
node "default" {
}
 
# Define the agent nodes
node "mysql1.local", "mysql2.local", "mysql3.local" {
 class { '::mysql::server':
 root_password => 'strongpassword'
 }
 class { '::mysql::client':
 package_ensure => 'present'
 }
}

Above definition will tell Puppet to install a MySQL server and client package on mysql1.local, mysql2.local and mysql3.local. To immediately start the deployment, go to the agent node and run:

puppet agent --test

Or, wait for the Puppet agent service to apply the catalog automatically (depending on the runinterval value, default is 30 minutes).

Once done, you will have three MySQL servers ready to serve! Imagine how much time you would save if you have many servers, applications, softwares and configurations to maintain.

Percona Server Installation Error – libssl.so.10 and libcrypto.so.10

I stumbled upon one error when installing Percona Server and socat via yum repository with following error:

--> Processing Dependency: libssl.so.10(libssl.so.10)(64bit) for package: socat-1.7.2.3-1.el6.x86_64
--> Processing Dependency: libcrypto.so.10(libcrypto.so.10)(64bit) for package: socat-1.7.2.3-1.el6.x86_64

It turns out that:

“Red Hat upgraded the version of OpenSSL in EL6 from 1.0.0 to 1.0.1 during the 6.4-6.5 cycle, in order to resolve a years-old feature request. This package is no longer binary compatible, and programs that were built against OpenSSL 1.0.0 must be rebuilt from source against 1.0.1.”

What do we need to do then?

Luckily the package is available at IUS repository:

rpm -Uhv http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/epel-release-6-5.noarch.rpm
rpm -Uhv http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/ius-release-1.0-13.ius.el6.noarch.rpm
yum install yum-plugin-replace
yum replace --enablerepo=ius-archive openssl --replace-with openssl10

(Press y for any question)

Verify the dependent libraries are exist (libcrypto.so.10 and libssl.so.10) :

ls /usr/lib64/ | grep -e ssl.so -e crypto.so
libcrypto.so.10
libcrypto.so.1.0.1e
libssl3.so
libssl.so.10
libssl.so.1.0.1e

Then, try again with the Percona Server installation command. In some cases, you might need to remove the installed epel-release package since it’s a little bit outdated with the current release 6.8.

Hope the solution will help you guys out there!

 

 

Sysbench 0.5 + Ubuntu 14.04 (Trusty) + Percona Server or XtraDB Cluster

I need to perform benchmarks on Percona XtraDB Cluster and Percona Server directly from my Ubuntu 14.04 box. Sysbench 0.5 is the latest stable to date and it supports benchmarking Galera cluster. If you are running on Sysbench 0.4, you would most likely encounter one of following errors if you are running it against a Galera cluster with high value of threads:

ALERT: failed to execute mysql_stmt_execute(): Err1317 Query execution was interrupted
ALERT: failed to execute mysql_stmt_execute(): Err1062 Duplicate entry '2199902' for key 'PRIMARY'

This post presumes that you have already installed standard Percona related packages including the apt-get repository to run a MySQL server (as instructed on Percona’s  documentation page).

 

Installing Sysbench 0.5

1. Install compiler tools and required packages:

$ sudo apt-get install -y gcc autoconf automake make libtool libssl-dev libcrypto++9

2. Install mysql_config (available from Percona repository):

$ sudo apt-get install -y libperconaserverclient18-dev

3. Get Sysbench from Launchpad:

$ bzr branch lp:sysbench

4. Compile and install sysbench:

$ cd sysbench
$ ./autogen.sh
$ ./configure --prefix=/usr --mandir=/usr/share/man
$ make
$ sudo make install

5. Create LUA template directory:

$ sudo mkdir /usr/share/sysbench/tests/db -p
$ sudo cp sysbench/tests/db/* /usr/share/sysbench/tests/db

6. Verify if sysbench is running in the correct version:

$ sysbench --version
sysbench 0.5

Running Sysbench

1. Before starting the benchmark, we need to prepare the test data. Following command will generate a table consists of 20 million rows on a Galera Cluster through HAproxy which running on 127.0.0.1 port 33306:

$ sysbench \
--db-driver=mysql \
--mysql-table-engine=innodb \
--oltp-table-size=20000000 \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
--test=/usr/share/sysbench/tests/db/oltp.lua \
prepare

Or we can generate 8 tables consists of 10M rows in parallel with 8 threads (total of 80M rows of data set):

$ sysbench \
--db-driver=mysql \
--mysql-table-engine=innodb \
--oltp-table-size=10000000 \
--oltp-tables-count=8 \
--num-threads=8 \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
--test=/usr/share/sysbench/tests/db/parallel_prepare.lua \
run

2. Start the benchmark against Galera cluster:

$ sysbench \
--db-driver=mysql \
--num-threads=8 \
--max-requests=5000000 \
--oltp-table-size=20000000 \
--oltp-test-mode=complex \
--test=/usr/share/sysbench/tests/db/oltp.lua \
--mysql-host=127.0.0.1 \
--mysql-port=33306 \
--mysql-user=sbtest \
--mysql-password=password \
run

You can perform other tests based on the LUA templates exist under /usr/share/sysbench/tests/db. Just change the –test parameter to the fullpath of the template file and you are good to go.

Details on this can be found at http://www.percona.com/docs/wiki/benchmark:sysbench:olpt.lua

 

Replacing OpenJDK with Oracle JDK on CentOS 6

I encountered some issues with an application running on Java. I thought it would be fine if running on standard OpenJDK release available at CentOS repository since it is really easy to install with yum, but it wasn’t. The application does required JDK from Oracle release (which is lame).

So here what you should do when you want to replace OpenJDK with Java SE JDK:

1. Check the current OpenJDK version. I am going to retain the same version if possible:

java -version
java version "1.6.0_30"
OpenJDK Runtime Environment (IcedTea6 1.13.3) (rhel-5.1.13.3.el6_5-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Check the current RPM package name:

$ rpm -qa | grep openjdk
java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.x86_64

2. Download Java SE Development Kit 6 update 30, which is equivalent to OpenJDK 1.6.0_30. There is no way (AFAIK) that we can download it directly using curl or wget. You need to have an account with Oracle. Sign up using your web browser and you will get a temporary link similar to below. Copy it from  your web browser and use wget to download it directly to the box:

$ wget -O jdk-6u30-linux-x64-rpm.bin http://download.oracle.com/otn/java/jdk/6u30-b12/jdk-6u30-linux-x64-rpm.bin?AuthParam=1397834288_32d1181063a10cd15522d23abbc5e4f7

**You should do this quick. The link will be expired within minutes.

3. Remove the installed OpenJDK package:

$ yum remove -y java-1.6.0-openjdk

4. Install Oracle JDK:

$ chmod 755 jdk-6u30-linux-x64-rpm.bin
$ ./jdk-6u30-linux-x64-rpm.bin

5. Verify the new Java version:

java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Now my application works like a charm! The new path to JRE now is /usr/java/jdk1.6.0_30/jre/

Install grsecurity with Yum

Easiest way to install grsecurity. The good thing about it is only grsecurity provides protection against zero-day and other advanced threats that buys administrators valuable time while vulnerability fixes make their way out to distributions and production testing.

1. Go into repository directory:

$ cd /etc/yum.repos.d/
$ wget http://repos.coredumb.net/grsecurity/grsecurity.repo

2. Install grsecurity kernel and administrator tools:

$ yum clean all
$ yum install kernel gradm

3. Reboot so it could load the grkernel:

$ reboot

 

Importing IMDb Sample Data Set to MySQL

Here in this post, I am going to show you on how to import IMDb plain text data files available at this page into your MySQL database server. I was using this data set to perform various benchmark tests around a moderately-large database size. If you want to have a small MySQL database sample data, you can try to use Sakila or World database available at MySQL Documentation Page.

I will be using CentOS 6.3 64bit as the OS platform and presume that MySQL has already installed and running. Make sure that you are having at least 4 GB of free space in the partition that MySQL @@datadir reside to allow this long-running process (3 to 5 hours – depending on your hardware specs + server workload) to successfully complete.

We will be using a Python-based package called IMDbPy. This package requires you to have Python with some development libraries installed as well as SQLObject. Then, we will need to download all data files from IMDb mirror site: ftp://ftp.fu-berlin.de/pub/misc/movies/database/, create the database and start the import process.

 

Installing IMDbPy and SQLObject

1. Install required packages using yum:

$ yum install -y gcc python python-devel openssl-devel libxml2-devel libxslt-devel zlib-devel MySQL-python python-setuptools python-pip

2. Install SQLObject using Python EasyInstall:

$ easy_install -U SQLObject

3. Download IMDbPy from this page into the MySQL server, extract it and start the installation process:

$ wget http://prdownloads.sourceforge.net/imdbpy/IMDbPY-4.9.tar.gz
$ tar -xzf IMDbPY-4.9.tar.gz
$ cd IMDbPY-*
$ python setup.py install

 

Importing Data

1. Create a directory to dump all the data files that we will download:

$ mkdir /root/data
$ cd /root/data

2. Download only .gz file from the IMDb mirror site to /root/data :

$ wget -r --accept="*.gz" --no-directories --no-host-directories --level 1 ftp://ftp.fu-berlin.de/pub/misc/movies/database/

3. Create a database in MySQL called ‘imdb’, with user ‘imdb’ and password ‘imdb’. We will then GRANT the user to the designated database:

mysql> CREATE DATABASE imdb;
mysql> GRANT ALL PRIVILEGES ON imdb.* TO 'imdb'@'localhost' IDENTIFIED BY 'imdb';
mysql> FLUSH PRIVILEGES;

3. Start the import process with -u and -d flag:

$ imdbpy2sql.py -d /root/data/ -u 'mysql://imdb:[email protected]/imdb'

Take note that -d is the directory of the .gz dump files are located and -u is the connection string for our MySQL database server. You can change the connection string to any of SQLObject’s supported database such as PostgreSQL, SQLite, Firebird and MAX DB. Please refer to this documentation for details.

You will see similar output as below which indicates the importing process has started:

SCANNING movies: Last Sunset (2006) (movieID: 2130001)
SCANNING movies: Legend of Hell (2012) (movieID: 2140001)
SCANNING movies: Lifestyles of Squirrels (2011) (movieID: 2150001)
SCANNING movies: Los signos del tiempo (1983) (movieID: 2160001)
SCANNING movies: Madame T (2012) (movieID: 2170001)
SCANNING movies: Marijji ringu (2007) (movieID: 2180001)
SCANNING movies: Menculik miyabi (2010) (movieID: 2190001)
* FLUSHING MoviesCache...

Wait up until it finish and you will have large sample data to play around in your MySQL server!

 

Linux: Follow and Download using wget

Your Linux box is incomplete if you do not have wget installed. It is a simple CLI based application used to download files from network.  It support HTTP, HTTPS and FTP protocols as well as retrieval through HTTP proxies.

 

 

Download latest version of PHPmyAdmin from SourceForge.

 

You can just simply use the latest link of SourceForge software and wget will automatically follow it. You also need to specify the ‘content-disposition’ flag. This option is useful for some file-downloading CGI programs that use “Content-Disposition” header to describe what the name of a downloaded file should be:

$ wget --content-disposition http://sourceforge.net/projects/phpmyadmin/files/latest/download

Download TokuMX  with cookies and redirection from Tokutek download site.

 

To download from this page, I was required to have cookies from download.php=tokumx-1.0-3-linux-x86_64.tgz page and use the same cookies to download from download.php?df=1. You can navigate to this page if you want to try that with a web browser, http://www.tokutek.com/download.php?download_file=tokumx-1.0.3-linux-x86_64.tgz.

$ wget --keep-session-cookies \
--save-cookies cookies.txt \
http://www.tokutek.com/download.php?download_file=tokumx-1.0.3-linux-x86_64.tgz && \
wget --load-cookies cookies.txt \
--content-disposition http://www.tokutek.com/download.php?df=1 && \
rm -f download.php* cookies.txt

 

Download Certain Files from Web Directory

 

You can download certain files (*.gz) under same directory using wget with -r flag. You need to specify –no-directories and –no-host-directories to output the files in the current directory while –level is the depth of the directory it will go:

$ wget -r --accept="*.gz" \
--no-directories \
--no-host-directories \
--level 1 \
ftp://ftp.fu-berlin.de/pub/misc/movies/database/

 

I will keep updating this page to share my collection of wget commands.

 

Ubuntu: Error Installing MySQL Server

I encountered following error when trying to upgrade MySQL server in Ubuntu 12.04:

dpkg: error processing mysql-server-5.5 (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because MaxReports is reached already
                                                              Setting up mysql-client (5.5.22-0ubuntu1) ...
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.5; however:
  Package mysql-server-5.5 is not configured yet.
dpkg: error processing mysql-server (--configure):
 dependency problems - leaving unconfigured
No apport report written because MaxReports is reached already
                                                              Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Errors were encountered while processing:
 mysql-server-5.5
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install.  Trying to recover:
Setting up mysql-server-5.5 (5.5.22-0ubuntu1) ...
start: Job failed to start
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing mysql-server-5.5 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.5; however:
  Package mysql-server-5.5 is not configured yet.
dpkg: error processing mysql-server (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mysql-server-5.5
 mysql-server

The easiest way to overcome this is by removing all mysql related packages in the server:

$ sudo apt-get purge mysql*

Then, try again to reinstall the mysql-server package as follow:

$ sudo apt-get install -y  mysql-client mysql-server

Make sure to backup your data files before performing the upgrade! Hope this workaround will help others!

How to Fix ‘Too many open files’ Problem

I have been facing following problem when executing Percona Xtrabackup in my CentOS 6.3 box:

xtrabackup_55 version 2.1.3 for Percona Server 5.5.16 Linux (x86_64) 
(revision id: 608) 
xtrabackup: uses posix_fadvise(). 
xtrabackup: cd to /var/lib/mysql 
xtrabackup: Target instance is assumed as followings. 
xtrabackup: innodb_data_home_dir = ./ 
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend 
xtrabackup: innodb_log_group_home_dir = ./ 
xtrabackup: innodb_log_files_in_group = 2 
xtrabackup: innodb_log_file_size = 67108864 
xtrabackup: using O_DIRECT 
130619 12:57:36 InnoDB: Warning: allocated tablespace 2405, old maximum 
was 9 
130619 12:57:37 InnoDB: Operating system error number 24 in a file 
operation. 
InnoDB: Error number 24 means 'Too many open files'. 
InnoDB: Some operating system error numbers are described at 
InnoDB: 
http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html 
InnoDB: Error: could not open single-table tablespace file 
InnoDB: We do not continue the crash recovery, because the table may become 
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. 
InnoDB: To fix the problem and start mysqld:

Linux / UNIX sets soft and hard limit for the number of file handles and open files. By default the value is too low as you can check using following command:

$ ulimit -n
1024

To increase open files limitation, you can use several ways:

1. Set the limit using ulimit command

$ ulimit -n 8192

This is temporary solution as it will increase the limit accordingly per login session.  Once you logged out and login again, this value will back to default.

2. Permanently define in /etc/security/limits.conf

To make it permanent, you can define the values (soft and hard limit) at /etc/security/limits.conf by adding following lines:

* soft nofile 8192
* hard nofile 8192

The soft limit is the value that the kernel enforces for the corresponding resource. The hard limit acts as a ceiling for the soft limit. Reboot the server to apply the changes. Or, if you do not want to reboot, add following line into the respective user’s .bashrc file, as in my case is root:

$ echo "ulimit -n 8192" >> ~/.bashrc

You will then need to relogin into the session to see the changes.

If the problem still persists, you might need to increase the limit higher and retry again the failed process.

Warning

Do not set the value to unlimited as it can caused PAM to fail and you will not able to SSH or console into the box with following error:

Apr 19 09:22:15 rh02 sshd[5679]: error: PAM: pam_open_session(): Permission denied

This issue has been reported in this bug report.

Further reading:

http://ss64.com/bash/ulimit.html
http://ss64.com/bash/limits.conf.html