Install VPN PPTP Server on CentOS 6

At this moment, my designer encounter problem to access target.com, an online shopping website to see some of the stuff for their design work. Target.com only allowed connections from USA and Canada at this moment due to their website crash issue last couple of weeks. Since this is quite urgent, I need to setup a VPN server so they can use it as a jump point to access websites in USA and Canada. I will use my MySQL server to serve as VPN server as well.

In this tutorial, I will use pptp as protocol to connect to VPN server using a username and password, with 128 bit MPPE encryption. Variable as below:

OS: CentOS 6 64bit
VPN server:  209.85.227.26
VPN client IP: 209.85.227.27 – 209.85.227.30
VPN username: vpnuser
Password: myVPN$99

1. Install ppp via yum:

$ yum install ppp -y

2. Download and install pptpd (the daemon for point-to-point tunneling). You can find the correct package at this website http://poptop.sourceforge.net/yum/stable/packages/ :

$ cd /usr/local/src
$ wget http://poptop.sourceforge.net/yum/stable/packages/pptpd-1.3.4-2.el6.x86_64.rpm
$ rpm -Uhv pptpd-1.3.4-2.el6.x86_64.rpm

3. Once installed, open /etc/pptpd.conf using text editor and add following line:

localip 209.85.227.26
remoteip 209.85.227.27-30

4. Open /etc/ppp/options.pptpd and add  authenticate method, encryption and DNS resolver value:

require-mschap-v2
require-mppe-128
ms-dns 8.8.8.8

5. Lets create user to access the VPN server. Open /etc/ppp/chap-secrets and add the user as below:

vpnuser pptpd myVPN$99 *

The format is: [username] [space] [server] [space] [password] [space][IP addresses]

6. We need to allow IP packet forwarding for this server. Open /etc/sysctl.conf via text editor and change line below:

net.ipv4.ip_forward = 1

7. Run following command to take effect on the changes:

$ sysctl -p

8. Allow IP masquerading in IPtables by executing following line:

$ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ service iptables save
$ service iptables restart

Update: Once you have done with step 8, check the rules at /etc/sysconfig/iptables. Make sure that the POSTROUTING rules is above any REJECT rules.

9. Turn on the pptpd service at startup and reboot the server:

$ chkconfig pptpd on
$ init 6

Once the server is online after reboot, you should now able to access the PPTP server from the VPN client. You can monitor /var/log/messages for ppp and pptpd related log. Cheers!

Manage Multiple MySQL Servers using PHPmyAdmin

In my environment, I have 5 different MySQL database servers running separately under different geographical location. Since it run standalone and not in cluster mode, I need to have one platform to manage these database servers altogether.

PHPmyAdmin is able to do this, with some changes on the configuration files. You just need to allow the MySQL user and host on every database server to be connected to. The setup that I am going to do will be as below:

Variables that I used as below:

Web Server: Apache 2.2.19
PHP: 5.3.2
Web Server IP: 212.77.103.146
PHPmyAdmin directory: /var/www/html/pma
User: pmaroot
Password: passdb432^^

1.  Download PHPmyAdmin PHP source at http://www.phpmyadmin.net/home_page/downloads.php . In this case, I will download this version phpMyAdmin-3.4.7-english.tar.gz. I assume you download the installer under /usr/local/src directory. We will need to rename the folder and paste it into directory that has been setup inside Apache to put PHPmyAdmin files:

$ cd /usr/local/src
$ tar -xzf phpMyAdmin-3.4.7-english.tar.gz
$ mv phpMyAdmin-3.4.7-english /var/www/html/pma

2.  Now lets create another root user just to manage databases using PHPmyAdmin. Execute following command in all MySQL database servers:

mysql> CREATE USER 'pmaroot'@'%' IDENTIFIED BY 'passdb432^^';
mysql> GRANT ALL PRIVILEGES ON *.* TO [email protected]'%';

3. Make sure all database servers are listening to all IP which accessible from outside. To simplify this, just remove or comment if you find following lines in your my.cnf file (usually located under /etc) :

#bind-address=127.0.0.1
#bind-address=localhost

4. To differentiate our MySQL servers easily, better we add the servers’ hostname into PHPmyAdmin server /etc/hosts file. Based on diagram above, I will add following line into the web server /etc/hosts:

china.mysql		118.144.76.16
usa.mysql		209.85.227.26
spain.mysql		84.236.148.11
singapore.mysql		202.156.14.10
colombia.mysql		190.0.39.34

5. So now we need to create PHPmyAdmin configuration files to include all databases server as above. Copy the configuration example as below to the active configuration file:

$ cd /var/www/html/pma
$ cp config.sample.inc.php config.inc.php

6. Open config.inc.php using text editor and add following value (actually you can put anything for the blowfish secret):

$cfg['blowfish_secret'] = 'youcanputanyphraseinsidethisquote'; /* YOU MUST FILL IN THIS FOR COOKIE AUTH! */

7. Inside this file you will also see following line:

/*
 * First server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'localhost';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;

Copy those whole line for another 4 times and change the appropriate host value. Example as below:

/*
 * First server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'china.mysql';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;
 
/*
 * Second server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'usa.mysql';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;
 
/*
 * Third server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'spain.mysql';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;
 
/*
 * Fourth server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'singapore.mysql';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;
 
/*
 * Fifth server
 */
$i++;
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
$cfg['Servers'][$i]['host'] = 'colombia.mysql';
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysql';
$cfg['Servers'][$i]['AllowNoPassword'] = false;

Done. Now I should be able to open the PHPmyAdmin via web browser at http://212.77.103.146/pma . I can select my MySQL servers I want to connect and access it using pmaroot user as created above. Screenshot as below:

Install ModSecurity in Apache2 – The Easiest Way

ModSecurity is a module for Apache to act as a web application firewall, which bring another security layer to your website. Nowadays, it is very important to have this protection so your website will be protected from Internet threats. Based on my experience as system administrator, our intrusion detection system averagely detected 10 – 15 internet threats per server per day. These include brute-force attack, blind SQL injection, XSS attack and many more.

Apache is the most popular web server in the world. For those who use Apache, I strongly advise you to have ModSecurity enabled in your production web server. You will never know when your website being target, or why it being target. Protection is the best cure!

I will use standard CentOS 6 distribution with Apache installed using yum run as DSO. Variables as below:

OS: CentOS 6 64bit
Apache directory: /etc/httpd
Apache configuration: /etc/httpd/conf/httpd.conf
ModSecurity configuration: /etc/httpd/conf.d/modsecurity.conf

1. Install Apache via yum and make sure it running properly:

$ yum install -y httpd*
$ chkconfig httpd on
$ service httpd start

2. Install all the needed packages via yum:

$ yum install pcre* libxml2* libcurl* lua* libtool openssl -y

3. Download mod_security source file at http://www.modsecurity.org/download/. In this case I will download modsecurity-apache_2.6.2.tar.gz :

$ cd /usr/local/src
$ tar -xzf  modsecurity-apache_2.6.2.tar.gz

4. Extract the downloaded files, navigate to the folder, configure and install:

$ cd modsecurity-apache*
$ ./configure
$ make
$ make install

5. Copy the ModSecurity configuration file into Apache configuration directory:

$ cp modsecurity.conf-recommended /etc/httpd/conf.d/modsecurity.conf

6. Activate the mod_security and unique_id modules in Apache configuration file. Open /etc/httpd/conf/httpd.conf via text editor and add following line:

LoadModule security2_module modules/mod_security2.so
LoadModule unique_id_module modules/mod_unique_id.so

7. Now we need to turn on the protection in ModSecurity configuration file. Open /etc/httpd/conf.d/modsecurity.conf via text editor and change following line:

SecRuleEngine DetectionOnly

To:

SecRuleEngine On

8. Restart Apache so mod_security can be loaded into Apache environment:

$ service httpd restart

Done! Your website now has been protected with Apache ModSecurity. You can tweak the rules inside modsecurity.conf files to suit your website requirement. You can check what is happening by reviewing the log file located under /var/log/modsec_audit.log.

Show Real IP in Apache Access Log (Akamai)

Since our company is using Akamai as the Content Delivery provider (CDN), we need to see the real IP to make sure we can do some log processing accurately.

The Akamai CDN basically working like example below:

Akamai Edge Server will work as the reverse-proxy to cache the website contents from requests they received on their edge servers anywhere around the world. This will speed up the website loading page and decrease the server load of the web server. The only problem of this implementation is that you can only seen Akamai’s IP connecting to your web server.

What will happen to my AWstats results then?
How to block suspected IP in firewall based on mod_security rules since they only can see Akamai IP?

Even though Akamai has provide summary of bandwidth usage and some other related informations, we still need to see the real IP, at least in our Apache access log. Akamai is adding another header called True-Client-IP, which used to return back the request to the real client after web server return back the result to Akamai edge server. What we need to do is just replace and add some line to our httpd.conf and restart the Apache web server. Following example is tested in a hosting server which run on cPanel. Variable as below:

Domain: mydomain.com
Log location: /usr/local/apache/domlogs (default for cPanel)

1. Open /etc/httpd/conf/httpd.conf using your favourite text editor and find your domain virtual host. You will see something like below:

<VirtualHost 12.34.123.115:80>
    ServerName mydomain.com
    ServerAlias www.mydomain.com
    DocumentRoot /home/mydomain/public_html
    ServerAdmin webmaster@mydomain.com
    UseCanonicalName Off
    CustomLog /usr/local/apache/domlogs/mydomain.com combined
    CustomLog /usr/local/apache/domlogs/mydomain.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
    ## User crazyd # Needed for Cpanel::ApacheConf
    <IfModule !mod_disable_suexec.c>
        SuexecUserGroup mydomain mydomain
    </IfModule>
    ScriptAlias /cgi-bin/ /home/mydomain/public_html/cgi-bin/
 
    # To customize this VirtualHost use an include file at the following location
    # Include "/usr/local/apache/conf/userdata/std/2/mydomain/mydomain.com/*.conf"

</VirtualHost>

2. Now, we need to replace and add the LogFormat and CustomLog variable as below:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
SetEnvIf True-Client-IP "^.*\..*\..*\..*" forwarded
CustomLog "/usr/local/apache/domlogs/mydomain.com_real_ip" combined env=!forwarded
CustomLog "/usr/local/apache/domlogs/mydomain.com_real_ip" proxy env=forwarded

3. So it will be like below:

<VirtualHost 12.34.123.115:80>
    ServerName mydomain.com
    ServerAlias www.mydomain.com
    DocumentRoot /home/mydomain/public_html
    ServerAdmin webmaster@mydomain.com
    UseCanonicalName Off
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
    SetEnvIf True-Client-IP "^.*\..*\..*\..*" forwarded
    CustomLog "/usr/local/apache/domlogs/mydomain.com" combined env=!forwarded
    CustomLog "/usr/local/apache/domlogs/mydomain.com" proxy env=forwarded
    CustomLog /usr/local/apache/domlogs/mydomain.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
 
    ## User crazyd # Needed for Cpanel::ApacheConf
    <IfModule !mod_disable_suexec.c>
        SuexecUserGroup mydomain mydomain
    </IfModule>
    ScriptAlias /cgi-bin/ /home/mydomain/public_html/cgi-bin/
 
    # To customize this VirtualHost use an include file at the following location
    # Include "/usr/local/apache/conf/userdata/std/2/mydomain/mydomain.com/*.conf"

</VirtualHost>

4. Now we need to run some command in cPanel and restart Apache. I prefer to do graceful restart:

$ /usr/local/cpanel/bin/apache_conf_distiller --update --main
$ apachectl -k graceful

Done! Now the AWstats will deliver correctly and you also able to monitor the real IP in access log using command:

$ tail -f /usr/local/apache/domlogs/mydomain.com

Create iSCSI Target in OpenFiler

If you have a SAN storage, or a dedicated server to serve as file and storage service to other server, I am suggesting you to use Openfiler. This operating system is specifically built to manage and deliver file-based Network Attached Storage and block-based Storage Area Networking in a single framework.

In this tutorial, I will not showing you on how to install Openfiler. I am just showing you on how to setup iSCSI target to be mounted in another server. Variables as follow:

OS: Openfiler 2.99 64bit
Openfiler IP: 10.1.1.1
Disk device: /dev/sdb
Disk size: 50 GB
Server that mount the iSCSI: 10.1.1.100

1. We start by reviewing the block drive layout which detected in the system. Login to the Openfiler web adminitration portal with default credentials as below:

Username: openfiler
Password: password

2. Make sure we turn on the iSCSI services and make it run. Go to Openfiler > Services and make sure it appear as below:

3.  Lets specify which host can connect to this storage server. So in this case, I want to allow 10.1.1.100 to access iSCSI target which we will create later. Go to Openfiler > System > Network Access Configuration and specify which host you want to allow:

4. We need to create physical volume for /dev/sdb. Go to Openfiler > Volumes > Block Devices, select information as screenshot below and click Create:

5. Create volume group for /dev/sdb1 by go to Openfiler > Volumes > Volumes Group.  I will put server1_vg as the name because I want to mount this in server1 once ready.

You should see something like this:

6. Create volume as ‘data‘ inside server1_vg volume group by go to Openfiler > Volumes > Add Volume. Make sure you select ‘block (iSCSI, FC, etc)‘ as the volume type:

7. Now we can do iSCSI mapping. Go to Openfiler > iSCSI Targets > LUN Mapping, and click Map.

8. Make sure we allow host access to this target. Go to Openfiler > iSCSI Targets > Network ACL, and allow which host you want to access to the target:

9. iSCSI target ready. Now you can connect them to any host you want and make sure you install the iSCSI initiator at the remote server.

Process summary will be as below:

  1. Create physical volume
  2. Create volume group
  3. Create volume
  4. Map volume with LUN
  5. Allow hosts define in step 3
  6. Mount into the destination server

Installing Java 1.6 in CentOS 6 – The Simplest Way

Default repository in CentOS 6 will give you Java 1.5 JRE and SDK packages. I will show you on how to install version 1.6 using yum.  You just need to enable RPMforge repository and another simple steps required after that.

Variables as follow:

OS: CentOS 6 64bit
Current Java version: 1.5
Upgraded Java version: 1.6

1. Install RPMforge into yum repository:

$ cd /usr/local/src
$ rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
$ rpm -Uhv rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

2.  Remove previous Java version 1.5, if you have it installed:

$ yum remove java-1.5-*

3. Lets install Java 1.6. The package name will be java-1.6.0-openjdk.x86_64:

$ yum install java-1.6.0-openjdk.x86_64 -y

4. We need to export the JAVA_HOME environment. This steps is optional:

$ export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre

5. Lets check our latest Java version:

$ java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.9) (rhel-1.36.b17.el6_0-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

As simple as ABC!

Eliminate Web Site Bad Traffic

Nowadays, web hosting has introduced many kind of internet traffic, which we can classified into 3 categories:

  1. Clean/good traffic
  2. Bot/crawler traffic
  3. Threat/bad traffic

Threat or bad traffic can harm the website and also can bring serious effect to the server, if you not have security in mind. Following example explained how bad traffic can bring consequences to your website:

  • You receive so many spam comments in your blog post with different IPs
  • Your website is being targeted by DOS or DDOS attack
  • Your website being injected with malicious code. This will usually happened if you have Javascript embedded in your HTML code.
  • Your website being tagged as ‘The site may harm your computer’ by Google Safe Browsing
  • You being accused by the web browser to be hosting malware

What we really want is to accept only clean traffic to our website. The most easiest way to achieve this objective is to use Cloudflare service. Cloudflare will convert your ‘naked’ and ‘exposed’ website into a protected website. The concept is they will route every single web traffic into their cloud network to filter out bad and good traffic, then just forward the good traffic to your website. This service is FREE for life!

What you need to do is:

  1. Go to cloudflare.com and register
  2. Follow the installation wizard online
  3. Change your domain name server to their name server at the domain registrar
  4. Wait for the propagation complete
  5. Done. You are protected!

Since the connections is routed to their network (because we will using their name server), they can log almost full information of our website traffic, not like Google Analytics or Quantcast, where they do tracking using Javascript which embedded into your website. Their reporting is also informative and we can see daily report on what is going on to our web traffic. Example as below:

 

From the screenshot above, you can see that I have report on how many good, bot and bad traffic to my website, how many bandwidth has been saved, how many processed request can be saved (by eliminating bad request) and so on.

I am not doing this for their behalf as promotion or what. It is worth to try. I just want to share with you on how to achieve best result with simplest and most effective way!

Bash Script – Delete Comments from a C program

I wrote a bash script to delete comments from a C program. C language will required /* and */ between the contents of the comment. Example as below:

MQLONG  Reason;      /* Qualifying reason      */
MQOD    ObjDesc = {MQOD_DEFAULT}; /* Object descriptor      */
MQLONG  OpenOptions; /* Options control MQOPEN *//*----------------------------------------- */
   /* Initialize the Object Descriptor (MQOD)  */
   /* control block.  (The remaining fields    */
   /* are already initialized.)                */
   /*------------------------------------------*/
   strncpy( ObjDesc.ObjectName,
            SYSTEM_REPLY_MODEL,
            MQ_Q_NAME_LENGTH );

This bash script will help to clear out whatever character contains between these 2 comment characters. I am using sed, which is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline) and also with some help from Regex. Script as below:

#!/bin/bash
# Bash scripts to delete comments from a C program
 
name=${0##*/}
 
Usage="Usage: {script name} {target file}"
 
if [ $# -eq 0 ]; then                   # if no argument specified
        echo $Usage                     # print Usage string value
        exit 1
fi
 
until [ $# -eq 0 ]
do
        case $1 in
                -h) echo $Usage         # print Usage string value if argument is -h
                        exit 0;;
                *) FILE=$1              # declare the next argument as `FILE`
                        shift;;
        esac
done
 
if [ ! -f $FILE ]; then                         # if the file is not exist
        echo "'$FILE' is not a exist"           # print the file name is not a exist
        exit 1                                  # terminate the program
else
        sed -i '/\*/s/\/\*.*\*\/$//' $FILE              # remove the comment using sed
        echo "Comments for $FILE has been removed"      # print the status
fi

If the script name saved as comment_remover under root directory and the target file is /home/user1/program.c , you can execute the script as follow (make sure the script is executable):

$ /root/comment_remover /home/user1/program.c
Comments for /home/user1/program.c has been removed

It will turn the example I use above to:

MQLONG  Reason;
MQOD    ObjDesc = {MQOD_DEFAULT};
MQLONG  OpenOptions;
⋮
 
   strncpy( ObjDesc.ObjectName,
            SYSTEM_REPLY_MODEL,
            MQ_Q_NAME_LENGTH );

I hope this will help some other people out there. Happy scripting!

NginX: Gateway Time-Out

One of the web server that run on Nginx is having 504 Gateway Time-out error. This can be fixed by increasing the send_timeout directive in nginx.conf.

Steps as below:

1. Open the nginx configuration files via your favourite text editor:

$ vi /usr/local/nginx/nginx.conf

2. Find send_timeout directive and increase the value higher. In this case, I increase the value from 3m to 10m:

send_timeout 10m;

3. Save the file and reload nginx using following command:

$ kill -HUP `ps -ef | grep nginx | grep master | awk {'print $2'}`

Notes: Some user might still get this error after applying the solution. It also depending on how your application works. Other directives that need to be consider for this error are:

client_header_timeout
client_body_timeout
send_timeout
fastcgi_read_timeout

Export MySQL Query Result to CSV

One of my developer team wants me to export some SQL query result and export it to CSV so they can import it to the local MySQL server. Since the database is quite big, around 2 million rows, dumping the whole table can really impact the database performance because he request this during peak hours.

After browse around in the Internet, I found the solution in StackOverflow forum. I share it here for the knowledge base. Variables as below:

Database server: MySQL 5.0
Database name: product_system
Table: tblproduct
Fields to be exported: ProductID and Name
Export to: tblproduct.csv
Username: root
Password:  dbserverpass123$

Following command should be run in SSH environment:

$ mysql -u root -p"dbserverpass123$"  product_system -B -e "select ProductID,Name from tblproduct;" | sed 's/\t/","/g;s/^/"/;s/$/"/;s/\n//g' > tblproduct.csv

After that, I copied over the csv into public_html folder so it can be downloaded by my developer team.

$ cp tblproduct.csv /home/user1/public_html

Done. Simple and easy!

Linux: /tmp: Read-only file system Error

One of the server that I manage has problem as below when I want to edit some files in crontab:

$ crontab -e
/tmp/crontab.XXXX1ibTLU: Read-only file system

It shows that the /tmp partition is unwriteable. The read-only has been mounted as read-only because file-system facing some error. To fix this, we need to do file system check (fsck) for /tmp partition. Before we do fsck, we need to unmount the directory but following error occurred:

$ umount /tmp
/tmp: Device or resource busy

It seems like /tmp directory is locked to be unmounted due to some files are already in process/being opened/being executed by some other processes. Using lsof, we can list out all the open files:

$ lsof | grep /tmp
mysqld  2599 mysql    5u   REG    7,0    0 6098 /tmp/ibaqFhew (deleted)
mysqld  2599 mysql    6u   REG    7,0    0 6099 /tmp/ibC7Yfbn (deleted)
mysqld  2599 mysql    7u   REG    7,0    0 6100 /tmp/ibJ8AFbe (deleted)
mysqld  2599 mysql   11u   REG    7,0    0 6101 /tmp/ibrLO9t5 (deleted)

As we can see that mysqld is locking some temporary files in /tmp directory. The 2nd column shows PID of the locking process. We need to stop this process using kill command:

$ kill -9 2599

Only then we are able to unmount the /tmp:

$ umount /tmp

Make sure that there is no error being prompt during the unmounting process. Now we can proceed to do fsck with -f (force) and -y (always accept prompt as Yes) to automate the file system check process:

$ fsck -f -y /tmp
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/usr/tmpDSK: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 6097 has zero dtime.  Fix? yes
 
Inodes that were part of a corrupted orphan linked list found.  Fix? yes
 
Inode 6098 was part of the orphaned inode list.  FIXED.
Inode 6099 was part of the orphaned inode list.  FIXED.
Inode 6100 was part of the orphaned inode list.  FIXED.
Inode 6101 was part of the orphaned inode list.  FIXED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences:  -(6097--6101)
Fix? yes
 
Free inodes count wrong for group #3 (2025, counted=2030).
Fix? yes
 
Free inodes count wrong (127695, counted=127700).
Fix? yes
 
/usr/tmpDSK: ***** FILE SYSTEM WAS MODIFIED *****
/usr/tmpDSK: 316/128016 files (3.2% non-contiguous), 66394/512000 blocks

Now the file system has been modified and fixed. We can remount back the partition using following command:

$ mount -a

You should able to use back the /tmp partition at this time, as well as I can do some changes on the crontab!