Solaris: Setting Up Package Manager and Installer

Oracle Solaris 11 is one of the  most popular UNIX-based operating system which widely download to be used by enterprise user, after FreeBSD. If you have FreeBSD basic knowledge, you should able to play around with Solaris as well.

If FreeBSD we have ports, the package manager and application installer, in Solaris, we can use pkgutil. It is rather simple to use and similar to apt-get and yum on how to use it. We will use OpenCSW, which aims to produce an easy to use open source software distribution installable on top of Solaris and Solaris-based systems, developed by a transparent and democratic community.

My Oracle Solaris version:

$ cat /etc/release
Oracle Solaris 11 11/11 X86
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Assembled 18 October 2011

1. Run following command as root to install the pkgutil application:

$ pkgadd -d http://get.opencsw.org/now

2. We need to set some environment for CSW. Open ~/.profile using text editor:

$ nano ~/.profile

And append :/opt/csw/bin at the end of PATH line and add another line for XFILESEARCHPATH so it will be shown like below:

export PATH=/usr/bin:/usr/sbin:/opt/csw/bin
export XFILESEARCHPATH=/opt/csw/lib/X11/%T/%N%C:/usr/openwin/lib/X11/%T/%N%C

To check whether it is correctly inserted, use following command:

$ echo $PATH
/usr/bin:/usr/sbin:/opt/csw/bin
$ echo $XFILESEARCHPATH
/opt/csw/lib/X11/%T/%N%C:/usr/openwin/lib/X11/%T/%N%C

3. Now we should able to use pkgutil. Lets update our catalog:

$ pkgutil -U

Note: If you unable to execute the command with error similar to ‘command not found’, you need to relogin your session or type su – command

4. It will download the list from this mirror: http://mirror.opencsw.org/opencsw/unstable/i386/5.11/ . To find available application in the catalog, we can use following command:

$ pkgutil -a mysql
libmysqlclient15   CSWlibmysqlclient15 5.0.92,REV=2011.10.05     1.4 MB
libmysqlclient_r15 CSWlibmysqlclient-r15 5.0.92,REV=2011.10.05   1.4 MB
mysql4             CSWmysql4 4.1.22,REV=2008.01.20              13.0 MB
mysql4bench        CSWmysql4bench 4.1.22,REV=2006.11.28        487.4 KB
mysql4client       CSWmysql4client 4.1.22,REV=2007.12.01         2.4 MB
mysql4devel        CSWmysql4devel 4.1.22,REV=2008.01.20          2.1 MB
mysql4rt           CSWmysql4rt 4.1.22,REV=2008.01.20             3.6 MB
mysql4test         CSWmysql4test 4.1.22,REV=2006.11.28           1.3 MB
mysql5             CSWmysql5 5.0.92,REV=2011.10.05              11.0 MB
mysql5client       CSWmysql5client 5.0.92,REV=2011.10.05         3.0 MB
mysql5devel_stub   CSWmysql5devel 5.0.92,REV=2011.10.05          7.9 KB
mysql5rt_stub      CSWmysql5rt 5.0.92,REV=2011.10.05             7.9 KB
mysql_dev          CSWmysql-dev 5.0.92,REV=2011.10.05          115.8 KB
php4_mysql         CSWphp4mysql 4.4.9,REV=2009.05.28            17.3 KB
php5_mysql         CSWphp5-mysql 5.3.8,REV=2011.09.04           22.6 KB
php5_mysqli        CSWphp5-mysqli 5.3.8,REV=2011.09.04          49.8 KB
php5_pdomysql      CSWphp5-pdomysql 5.3.8,REV=2011.09.04        17.3 KB
php5mysql_stub     CSWphp5mysql 5.3.8,REV=2011.09.04             2.5 KB
php5mysqli_stub    CSWphp5mysqli 5.3.8,REV=2011.09.04            2.5 KB
php5pdomysql_stub  CSWphp5pdomysql 5.3.8,REV=2011.09.04          2.5 KB
pm_dbdmysql        CSWpmdbdmysql 4.013,REV=2010.02.11          169.8 KB
py_mysql           CSWpy-mysql 1.2.3,REV=2011.03.13             41.0 KB
pymysql            CSWpymysql 1.2.3,REV=2011.03.13               1.8 KB
rb18_mysql_2_8_1   CSWrb18-mysql-2-8-1 2.8.1,REV=2011.08.14    217.0 KB

5. Now lets install application using pkgutil:

$ pkgutil -i -y mysql5

To update application:

$ pkgutil -u mysql5

To remove application:

$ pkgutil -r mysql5

To update all applications:

$ pkgutil -u

From here you may start to manage your applications and let your Solaris working!

Ubuntu: Install Windows Application using Winetricks

Ubuntu is surely good for end-user operating system but sometimes we still need to have Windows application more. Using Wine, we can run Windows applications in UNIX/LINUX environment. If you are using Ubuntu, installing Wine is just simple if you are using Ubuntu Software Center. Just search for ‘Wine’ and then install.

In order to run Windows application correctly, we need to install some of the Windows-only requirement like .NET Framework, windows-based fonts and some of Windows libraries (dll) file. We can achieve this by using Winetricks, an extension built for Wine to help installing Windows application in a better way (other than manually download and execute).

Variable that I used is:

OS: Ubuntu 10.04 (Lucid)
User: mikey
Wine location: /home/mikey/.wine or ~/.wine

1. Download and install Winetricks:

$ cd ~/Downloads
$ wget http://winetricks.org/winetricks
$ chmod +x winetricks
$ sudo cp winetricks /usr/local/bin

2. We can browse around on categories that Winetricks could do for us:

$ winetricks apps list
$ winetricks benchmarks list
$ winetricks dlls list
$ winetricks fonts list
$ winetricks games list
$ winetricks settings list

3. In this case, I will start by installing .NET Framework 2.0:

$ winetricks dotnet20

4. To install many applications, we can run following command:

$ winetricks corefonts vcrun6 mfc40 vb6run msxml6

5. To remove all applications which has been installed by Winetricks and re-initialize Wine:

$ rm -Rf ~/.wine
$ wineboot

 

Linux: Install JAWStats – Beautiful Statistic using AWStats Core

If you are a webmaster, then you should know the well-known log analyzer software called AWStats (Advanced Web Statistics). AWStats will convert your Apache access logs into a statistic page with lot of graphs and infographics.

One thing lack of it is the way AWStats represent the data in HTML. So, we will use JAWStats, another extension created in conjunction with AWStats and produces clear and informative charts, graphs and tables about your website visitors.

To do this, we need to have AWStats installed. JAWStats will use AWStats’s Data File (for example: awstats092011.domainname.com.txt) as the database to generate a beautiful statistic page. I will use variable as below:

OS: CentOS 6.2 64bit
AWStats path: /var/www/html/awstats
AWStats CGI path: /var/www/cgi-bin
JAWStats path: /var/www/html/jawstats
Domain: myseri.com.my
Log location: /var/log/httpd/myseri.com.my

AWStats

1. Download AWStats from the main website:

$ cd /usr/local/src
$ wget http://prdownloads.sourceforge.net/awstats/awstats-7.0.tar.gz
$ tar -xzf awstats-7.0.tar.gz

2. Install AWStats:

$ mv /usr/local/src/awstats-7.0/wwwroot/cgi-bin/* /var/www/cgi-bin
$ mv /usr/local/src/awstats-7.0/wwwroot/* /var/www/html/awstats

3. Configure AWStats to process our domain’s log file in cgi-bin folder:

$ cd /var/www/cgi-bin
$ cp awstats.model.conf awstats.myseri.com.my.conf

Open the domain’s configuration file and edit following line via text editor:

LogFile="/var/log/httpd/mylog.log"
SiteDomain="myseri.com.my"
HostAliases="www.myseri.com.my REGEX[myseri\.com.my$]"
DirCgi="/var/www/cgi-bin"
DirIcons="../awstats/icon"

4. Let AWStats run to generate the statistic data:

/var/www/cgi-bin/awstats.pl -config=myseri.com.my -update

Note: It might take some times for this process to complete depending on your log size

5. Once done, we need to allow AWStats to update the statistic automatically every 2 hours via cron job:

$ crontab -e

Add following line:

0 */2 * * * /var/www/cgi-bin/awstats.pl -config=myseri.com.my -update

JAWStats

1. Download JAWStats:

$ mkdir -p cd /usr/local/src/jawstats
$ wget http://static.jawstats.com/src/jawstats-0.7beta.tar.gz
$ tar -xzf jawstats-0.7beta.tar.gz

UPDATE: I received a lot of feedback that PHP 5.3 will cause a blank page to JAWStats. If you are using PHP 5.3 and above, you might need to download the latest version in GitHub here: https://github.com/mimesis/jawstats/downloads

2. Copy the JAWStats folder to website directory and copy the configuration file as well:

$ mv /usr/local/src/jawstats /var/www/html
$ cd /var/www/html/jawstats
$ cp config.dist.php config.php

3. Configure JAWStats by editing config.php via text editor and add following line:

$aConfig["myseri.com.my"] = array(
"statspath" => "/var/www/cgi-bin/",
"updatepath" => "/var/www/cgi-bin/",
"siteurl" => "http://www.myseri.com.my",
"sitename" => "myseri.com.my",
"theme" => "default",
"fadespeed" => 250,
"password" => "mypassword",
"includes" => "",
"language" => "en-gb"
);

 

Result

To see the result, just go to the browser at http://192.168.0.55/jawstats/ (this is my web server main IP) and you should see something like below:

The data in this page is updated once every 2 hours depending on how you setup the cron job for AWStats to run. You might need to increate PHP memory_limit to higher depending on how big your data generated by AWstats.

CentOS: Upgrading CentOS Release 6.0 to 6.2

The best server maintenance practice is to have all software run up-to-date by following the latest stable release. Most of our servers are has been upgraded to CentOS 6 from CentOS 5 (major release), but also need to upgrade from CentOS 6.0 to CentOS 6.2 (minor release) which usually comes by every about 4 to 8 weeks after upstream release (RedHat).

It just a simple step by the way and variables as below:

OS: CentOS 6.0 64bit
Current release:  CentOS 6.2

1. Check our current kernel and release version:

$ uname -a
Linux centos.local 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/centos-release
CentOS Linux release 6.0 (Final)

2. Before upgrade, it is recommended to clean all cached files from any enabled repository:

$ yum clean all

3. Lets start upgrading. It takes some time depending on your connectivity to CentOS repository:

$ yum update

4. Once completed, proceed to reboot:

$ init 6

5. Check our latest kernel and release version:

$uname -a
Linux centos.local 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/centos-release
 CentOS release 6.2 (Final)

To automate this process, we can repeat this process bi-monthly to make sure our operating system is up-to-date with the current release. Run crontab -e and add following line:

0 0 */14 * * yum clean all; yum update

Dont forget to restart crond:

$ service crond restart

Linux: Mount FTP as File System

My developer team requesting me to mount an external FTP account into our web server so they can do file manipulation process between them. To achieve this, I need to mount FTP as file system, so they transparently not realized that they are browsing to an FTP account which has been localized into the server.

I will be using CurlFTPFS, a FTP filesystem based on cURL and FUSE. Variable as below:

OS: CentOS 6 64bit
FTP host: ftp.mydear.org
FTP directory: public_html
FTP username: [email protected]
FTP password: By55k#ds
Mount directory: /mnt/ftp/ftpuser

1. Lets install all requirements via yum:

$ yum install fuse* libcurl* glib* glibc.i686 file-libs file-devel file-static curl -y

2. Download and install CurlFTPFS:

$ cd /usr/local/src
$ wget http://cdnetworks-kr-2.dl.sourceforge.net/project/curlftpfs/curlftpfs/0.9.1/curlftpfs-0.9.1.tar.gz
$ tar -xzf curlftpfs-0.9.1.tar.gz
$ cd curlftpfs-*
$ ./configure
$ make
$ make install

3. We will use .netrc capabilities in storing FTP credentials. Use text editor and create one file called /root/.netrc (if still not exist) and enter following information:

machine ftp.mydear.org
login [email protected]
password By55k#ds

4. Change the permission so it will not accessible by others and prepare the mount directory:

$ chmod 600 /root/.netrc
$ mkdir -p /mnt/ftp/ftpuser

5. Since the developer team need to browse the mounted directory, I need to create a specific user and assign correct permission and ownership to the directory:

$ useradd -m developer -p 'develPASS'
$ chown developer.developer /mnt/ftp/ftpuser -Rf

6. Lets get the UID and GID of user/group developer to be used when mounting the FTP account:

$ id -u developer
501
$ id -g developer
502

7. Mount the FTP account into directory with ownership and allowing other (since we mount it as root) option:

$ curlftpfs ftp.mydear.org /mnt/ftp/ftpuser -o uid=501 -o gid=502 -o allow_other

Done! My developer team now can browse /mnt/ftp/ftpuser in the server and do their file manipulation work. To allow this FTP to be mounted automatically after reboot, you can put following line into /etc/fstab:

curlftpfs#ftp.mydear.org /mnt/ftp/ftpuser fuse rw,uid=501,gid=502,user,noauto,allow_other 0 0

And also, if you want to unmount it, simply run following command:

$ fusermount -uz /mnt/ftp/ftpuser

Notes: You might encounter unmount error if using above command. Depending on kernel and fuse version, upgrading the them might solve the problem.

Linux: Install LiteSpeed Web Server

Lets open our mind with another web server, which is 6x faster than Apache in handling static files and 50% faster in processing PHP, called LiteSpeed Web Server (LSWS). Variable as below:

OS: CentOS 6 64bit
LiteSpeed version: Standard Edition – Version 4.1.9
Domain/Website: mydear.org
IP: 192.168.0.222
Root path: /usr/local/lsws/mydear
Web path: /usr/local/lsws/mydear/public_html

1. Download the installer at http://litespeedtech.com/litespeed-web-server-downloads.html . As for this case, I will use Linux (x86) version:

$ cd /usr/local/src
$ wget http://litespeedtech.com/packages/4.0/lsws-4.0.20-std-i386-linux.tar.gz

2. Install the requirement to be run in 64 bit environment:

yum install glibc -y

3. Extract the installer and install:

$ tar -xzf lsws-4.0.20-std-i386-linux.tar.gz
$ cd lsws-4*
$ ./install.sh

The installation wizard will guide us on how to install as below:

Please specify the destination directory. You must have permissions to
create and manage the directory. It is recommended to install the web server
at /opt/lsws, /usr/local/lsws or in your home directory like '~/lsws'.
 
ATTENTION: The user 'nobody' must be able to access the destination
directory.
 
Destination [/usr/local/lsws]:
 
Please specify the user name of the administrator.
This is the user name required to log into the administration web interface.
 
User name [admin]: admin
 
Please specify the administrator's password.
This is the password required to log into the administration web interface.
 
Password:
Retype password:
 
Please specify administrators' email addresses.
It is recommended to specify a real email address,
Multiple email addresses can be set by a comma
delimited list of email addresses. Whenever something
abnormal happened, a notificiation will be sent to
emails listed here.
 
Email addresses [[email protected]]: [email protected]
 
As you are the root user, you must choose the user and group
whom the web server will be running as. For security reason, you should choose
a non-system user who does not have login shell and home directory such as
'nobody'.
 
User [nobody]:
Please choose the group that the web server running as.
 
User 'nobody' is the member of following group(s): nobody
Group [nobody]:
 
Please specify the port for normal HTTP service.
Port 80 is the standard HTTP port, only 'root' user is allowed to use
port 80, if you have another web server running on port 80, you need to
specify another port or stop the other web server before starting LiteSpeed
Web Server.
You can access the normal web page at http://:/
 
HTTP port [8088]: 80
 
Please specify the HTTP port for the administration web interface,
which can be accessed through http://:/
 
Admin HTTP port [7080]:
 
Cannot find RUBY installation, remember to fix up the ruby path configuration
before you can use our easy RubyOnRails setup.
 
You can setup a global script handler for PHP with the pre-built PHP engine
shipped with this package now. The PHP engine runs as Fast CGI which
outperforms Apache's mod_php.
You can always replace the pre-built PHP engine with your customized PHP
engine.
 
Setup up PHP [Y/n]: Y
Suffix for PHP script(comma separated list) [php]:
 
Installing, please wait...
 
Generating key pair for web console login page, please wait ...
Generating RSA private key, 512 bit long modulus
...............++++++++++++
...............................++++++++++++
e is 65537 (0x10001)
 
AWStats Integration
 
AWStats is a popular log analyzer that generates advanced web server
statistics. LiteSpeed web server seamlessly integrates AWStats into
its Web Admin Interface. AWStats configuration and statistics update
have been taken care of by LiteSpeed web server.
 
Note: If AWStats has been installed already, you do not need to
install again unless a new version of AWStats is available.
 
Would you like to install AWStats Add-on module [y/N]? y
 
[INFO] Creating a symbolic link from './awstats-7.0' to './awstats'
[OK] AWStats 7.0 has been successfully installed as a litespeed
add-on module.
 
Congratulations! The LiteSpeed Web Server has been successfully installed.
Command line script - "/usr/local/lsws/bin/lswsctrl"
can be used to start or stop the server.
 
It is recommended to limit access to the web administration interface.
Right now the interface can be accessed from anywhere where this
machine can be reached over the network.
 
Three options are available:
1. If the interface needs to be accessed only from this machine, just
change the listener for the interface to only listen on the loopback
interface - localhost(127.0.0.1).
2. If the interface needs to be accessible from limited IP addresses or sub
networks, then set up access control rules for the interface accordingly.
3. If the interface has to be accessible via internet, SSL (Secure Sockets
Layer) should be used. Please read respective HOW-TOs on SSL configuration.
 
To change configurations of the interface, login and click
"Interface Configuration" button on the main page.
The administration interface is located at http://localhost:/
or http://:/
 
Would you like to have LiteSpeed Web Server started automatically
when the server restarts [Y/n]? Y
 
[OK] The startup script has been successfully installed!
Would you like to start it right now [Y/n]? Y
 
[OK] litespeed: pid=3314.
 
LiteSpeed Web Server started successfully! Have fun!

The sequence I press on the keyboard is:

Enter > Enter > mypassword > mypassword > [email protected] > Enter > Enter > 80 > Enter > Y > y > Y > Y

4. Lets check whether LSWS is running:

$ netstat -tulpn | grep lite
tcp   0   0   0.0.0.0:7080   0.0.0.0:*   LISTEN   3314/litespeed (lsh
tcp   0   0   0.0.0.0:80     0.0.0.0:*   LISTEN   3314/litespeed (lsh

5. Allow the LSWS ports in IPtables:

$ iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 7080 -j ACCEPT
$ service iptables save
$ service iptables restart

6. Open web browser and go to http://192.168.0.222:7080 to access the administration console. Before we configure the virtual host, we need to configure the listeners so it learn to the correct IP:

Web administration > Configuration > Listeners > Default > General > Edit > Select the IP Address of the pointed website, in this case I will use 192.168.0.222 > Save

Restart the LSWS:

Web administration > Action > Graceful Restart > OK

7. Lets configure our website to run. Create the directory via SSH and then configure the virtual host:

$ mkdir -p /usr/local/lsws/mydear/public_html

Web administration > Configuration > Virtual Host Templates > PHP_SuEXEC > Add and insert the website details as screenshot below:

8. Next we need to instantiate the domain from the template. Refer to screenshot below:

Click Instantiate > Yes.

9. Restart LSWS:

Web administration > Action > Graceful Restart > OK

Done! You can access the website via http://mydear.org in the browser with PHP support. It is also easy to recompile PHP, Ruby and Python by using the web administration console of LSWS.

CentOS: Increase MySQL Uptime with MySQL Proxy

How can we achieve 100% MySQL uptime? For me, I would answer as follow:

  • Cluster all MySQL servers together
  • Load balance and failover between each of the cluster members
  • Make them run in redundant network line

I have tried to use Galera for active-active multi master replication in  previous post. This setup will surely help on scaling up the MySQL infrastructure that we have without headache. What I need the most is a loadbalancer and auto-failover in front of them. We can use a lot of reverse-proxy provider out there like HAProxy, Heartbeat and others but here I will use MySQL Proxy. Why? Because it require only 3 steps to make it run! I repeat, 3 steps!

Architecture that I will implement as follow:

Variables as below:

OS: CentOS 6 64bit
Server IP: 192.168.0.88

1. Download MySQL Proxy at http://dev.mysql.com/downloads/mysql-proxy/. I will download the “Red Hat & Oracle Linux 5 (x86, 64-bit), Compressed TAR Archive” version. I will run the MySQL proxy under /usr/local directory as below:

$ cd /usr/local
$ wget http://mysql.oss.eznetsols.org/Downloads/MySQL-Proxy/mysql-proxy-0.8.2-linux-rhel5-x86-64bit.tar.gz
$ tar -xzf mysql-proxy-0.8.2-linux-rhel5-x86-64bit.tar.gz
$ mv mysql-proxy-0.8.2-linux-rhel5-x86-64bit mysql-proxy

2. Open MySQL port in iptables:

$ service iptables start
$ iptables -A INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
$ service iptables save
$ service iptables restart

2. Lets start it! Run following command as root:

$ /usr/local/mysql-proxy/bin/mysql-proxy -P 0.0.0.0:3306 -b 192.168.0.91:3306 -b 192.168.0.92:3306 &

3. We just need to put above command (in step 2) to /etc/rc.local via text editor to make sure it started automatically after reboot.

To stop the mysql-proxy service, just kill the process by run following command:

$ kill -9 `pidof mysql-proxy`

Done! You can change your database information in your web server to the new IP, 192.168.0.88 (the mysql proxy server). Try to reboot MySQL DB1 and let web server make a SQL request and you will notice that your MySQL query is returnable by another server, MySQL DB2.

Windows: Reboot Server on Specific Time

Windows Update is surely an important application and need to be enabled when using Windows server. The problem with Windows Update is it usually need to be rebooted after any update applied. This will add another job to system administrator because rebooting a server will cause downtime, and any cause of downtime will affect on system availability.

By using AT command in command prompt, we can schedule a server reboot on any specific time we wants. The time must be in 24-hours format and follow with shutdown command as below:

C:\Users\Administrator> at 22:00 shutdown /r /f /t 0
Added a new job with job ID = 1

It just a simple command but it can automate server administrator job!

cPanel: Create Backup and Transfer to Another Server

If you familiar in administrating cPanel, you should know a script/tool called pkgacct. This is the backup tools being used by cPanel in order to create and manage cPanel user’s account backup. By using this tool, we can take advantage by create a centralized backup server where cPanel account backup will be sent over to another server via FTP on weekly basis.

Following picture shows the architecture of the centralized backup I made:

 

FTP Server

1. I will store the cPanel backup in a Windows 2008 R2 server and I will be using FileZilla as the FTP server. Download the installer from here and follow the installation wizard. Just accept all defaults value during the installation process.

2. Add FTP required port into Windows Firewall:

Start > Administrative Tools > Windows Firewall with Advanced Security > Inbound Rules > New Rule > Port > Next > under Specific local ports enter this value: 20, 21 > Next > Allow the connection > Next > tick all for Domain, Private, Public > Next > put a name like FileZilla FTP > Finish.

3. Create FTP user and assign a directory called centralized_backup under C:\ partition:

FileZilla > Users > Add > enter username and password for respective user. Then go to Shared folders > Add Shared folders > C:\centralized_backup and tick all permissions on files and directories.

4. Make sure you can telnet port 21 from the cPanel servers:

$ telnet cpbackup.mypeng.org 21
Trying 115.10.221.108...
Connected to cpbackup.mypeng.org (115.10.221.108).
Escape character is '^]'.
220-FileZilla FTP server
220 version 0.9.40 beta

5. Setup a schedule task so it will delete files longer than 30 days in the centralized backup directory:

Start > All Programs > Accessories > System Tools > Task Scheduler > Create Task. Enter following information:

General > Name: Delete_old_backup
General > Security options: SYSTEM
Triggers: Weekly  every Sunday of every week
Actions: Start a program
Actions > Program/scripts: forfiles
Actions > Add arguments (optional):

/p "c:\central_backup" /s /d -30 /M *.tar.gz /c "cmd /c del @file : date >= 30 days >NUL"

Click OK to complete the setup. Screenshot of the Actions setup is as below:

cPanel Servers

1. I will use BASH script to automate the process. In each cPanel server, create a file under /root/scripts directory:

$ mkdir -p /root/scripts
$ touch /root/scripts/centralbackup

Copy and paste following contents:

#!/bin/bash
# Generate backups using cpbackup and then transfer them to a central server
# Author: SecaGuy @ blog.secaserver.com
 
# Local server configuration
LHOSTNAME=`hostname`
TEMPPATH='/root/tmp'
 
# FTP server configuration
CHOST='cpbackup.mypeng.org'
CUSERNAME='centralftp'
CPASSWORD='ByyPo3$d'
 
# Dont change line below
FTP=`which ftp`
 
if [ ! -d /var/cpanel/users ]
then
        echo "cPanel users not found. Aborted!"
        exit 1
else
        eof=`ls /var/cpanel/users | egrep -v '^\..$' | egrep -v '^\...$' | wc -l`
        echo "$eof cPanel user(s) found in this server"
 
	[ ! -d $TEMPPATH ] && mkdir -p $TEMPPATH || :
 
        for (( i=1; i<=$eof; i++ ))
        do
                CPUSER=`ls /var/cpanel/users | egrep -v '^\..$' | egrep -v '^\...$' | head -$i | tail -1`
                echo "Creating backup for user $CPUSER.."
                /usr/local/cpanel/scripts/pkgacct $CPUSER $TEMPPATH userbackup
 
                echo "Backup done. Transferring backup to FTP server.."
                FILENAME=`ls $TEMPPATH | grep tar.gz`
                $FTP -n $CHOST <<END_SCRIPT
                quote USER $CUSERNAME
                quote PASS $CPASSWORD
                binary
                passive
                mkdir $LHOSTNAME
		cd $LHOSTNAME
                lcd $TEMPPATH
                put $FILENAME
                quit
END_SCRIPT
 
        echo "Removing temporary files.."
        rm -Rf $TEMPPATH/backup-*
        USERDIR=`cat /var/cpanel/users/$CPUSER | grep HOMEDIRPATHS | sed 's/HOMEDIRPATHS=//g'`
        rm -Rf $USERDIR/backup-*
        echo "Backup for $CPUSER complete!"
 
        done
 
	echo "Process complete!"
        exit 0
fi

2. Change the permission to executable:

$ chmod 755 /root/scripts/centralbackup

Schedule the Backup

Once all cPanel servers have been setup, I can schedule them using cron job on weekly basis (every Sunday at 12:00 AM). So I will add following line into cron job list:

$ crontab -e

Add following line:

0 0 * * 0 /root/scripts/centralbackup

Save the files and restart cron daemon:

$ service crond restart

Notes: You can use cpbackup-exclude.conf as refer to cPanel documentation page, to exclude certain files or directories from being included in this backup.

cPanel: Exclude Directory during Backup

Backup is the first thing-to-do and should not be forgotten by a good system administrator. Since cPanel has their built-in backup creator as well as backup management system, we can take advantage of this tool and use them to suit our needs. In my situation, we have many cPanel accounts and some of them is higher than 10GB of disk usage, mostly due to website uploaded contents.

Creating backup will be a hard thing if you have too many inodes or too much disk consumption. It is a good thing if we can exclude some of the directory for example user_uploaded when creating cPanel backup, and the rest can be backup manually by downloading them to a local server.

In this tutorial I will create a full backup what some directories being excluded. Variables as below:

OS: RHEL 4 32bit
cPanel account: premen
Home directory for user: /home/premen

1. Identify the directory that we want to exclude. In this case, I will exclude the high disk storage directory. Using following command might help:

$ cd /home/premen
$ du -h | grep G

The command should list all directories which more than 1GB of size. Example as below:

6.4G    ./public_html/portal/tmp
8.7G    ./public_html/portal/user_uploaded
15.1G   ./public_html

2. Then we need to generate a file called cpbackup-exclude.conf under respective home directory of the user as refer to cPanel documentation at here:

$ cd /home/premen
$ touch cpbackup-exclude.conf
$ vi  cpbackup-exclude.conf

Paste following line:

public_html/portal/tmp
public_html/portal/user_uploaded

3. Now we will create the backup either via pkgacct script:

/usr/local/cpanel/scripts/pkgacct premen /home userbackup

or you can click “Download or Generate a Full Website Backup” under cPanel as screenshot below:

Once the backup ready, you will notice that both directories have been excluded from the cPanel full backup and your backup size should be smaller and faster to be compressed. Cheers!

cPanel: PHPList Subscriber Mailing List Maintenance

In my company, I am also responsible to handle and manage the mailing list server. We are using PHPList, a popular mailing list program to blast out mails and we use it to send our latest news, promotions, announcements and notification to our subscribers. We have a lot of subscribers, which include some of them are problematic mail recipients with such following error:

  • mailbox unavailable
  • no such user
  • user rejected
  • domain already expired

In order to do maintenance on the subscriber list, I need to make sure only active subscriber (which have active mailbox) exist in our mailing list. By referring to exim_mainlog and looking for recipient bounce error, we can remove unwanted recipient from our subscriber list. Variable as below:

OS: CentOS 4 64bit
PHPlist version: 2.10.5
Mail server log: /var/log/exim_mainlog
PHPList database: plist_db
PHPList Username: plist_userdb
PHPList password: p412#Yf

1. Lets generate error log from exim_mainlog so we can easily extract the error information. We will use eximstats command and generate the output to html files, same as what you will see under WHM > Email > View Mail Statistics:

$ /usr/sbin/eximstats -html=/root/mailstats.html -nr -nt -nvr /var/log/exim_mainlog

2. If we go through the generated html file, you can see the list of error captured by exim_mainlog. Example error as below:

<li>1 - [email protected] R=fail_remote_domains: The mail server could not deliver mail to [email protected] The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.

The error above describing that the remote domain is no longer exist and unreachable. We do not want this email account anymore in our subscriber list so we need to extract the email address and remove from PHPlist database.

3. Lets extract all emails which related to this error to a file called domain_error.txt:

$ cat /root/mailstats.html | grep "The account or domain may not exist, they may be blacklisted, or missing the proper dns entries" > domain_error.txt

4. From the domain_error.txt list, we can extract email address only so we can pass this value to PHPlist database via SQL statement:

perl -wne'while(/[\w\.\-][email protected][\w\.\-]+\w+/g){print "$&\n"}' domain_error.txt | sort -u > delete_list.txt

5. We should now have a list of email address that we want. We will use a BASH script to automate the deletion process. Create a file called phplist_delete under /root/scripts folder:

mkdir -p /root/scripts
touch /root/scripts/phplist_delete

Copy and paste following scripts and change the configuration value to the one that suits you. You can retrieve the database information inside PHPlist config.php file (usually under config directory):

#!/bin/bash
# Delete PHPlist subscriber who listed in delete_list.txt
 
# Configuration value
HOST="localhost"
DATABASE="plist_db"
USERNAME="plist_userdb"
PASSWORD='p412#Yf'
DELETE_LIST="/root/delete_list.txt"
 
###
MYSQL="$(which mysql)"
 
if [ ! -f $DELETE_LIST ]
then
        echo "List file not found!"
        exit 1
else
        echo "Deleting email addresses in PHPlist database.."
        cat $DELETE_LIST | while read emailadd; do
                $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE FROM $DATABASE.phplist_user_user WHERE email like '$emailadd';"
                echo "$emailadd: deleted!"
        done
        $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE from $DATABASE.phplist_user_user_attribute WHERE userid NOT IN (SELECT  id FROM $DATABASE.phplist_user_user);"
        $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE from $DATABASE.phplist_user_user_history WHERE userid NOT IN (SELECT id FROM $DATABASE.phplist_user_user);"
        echo "Process completed!"
fi

6. Change the permission:

chmod 755 /root/scripts/phplist_delete

7. Lets run the script and you are done!

/root/scripts/phplist_delete

Sort and Count IP in Apache Access Logs

My boss has asked me to generate a simple report on list of IP addresses which access to our website yesterday with a “ref=1” keyword. What I need most is the Apache access logs which located under /etc/httpd/domlogs/mydomain.net (default Apache logs for cPanel servers).

From the log files, I will need to extract yesterday’s log which is 22 November 2011 and output them into another file called 20111122.txt. Command as below:

$ cat /etc/httpd/logs/domlogs/mydomain.net | grep "22/11/2011" > 20111122.txt

Next, I extract the logs to match the keyword (ref=1) and output to another file:

$ cat /etc/httpd/logs/domlogs/20111122.txt | grep '\bref=1\b' > 20111122_keyword.txt

Once the keyword extract, I will do the counting and generate a report in http root folder so I can view it via web browser:

$ cat /etc/httpd/logs/domlogs/20111122_keyword.txt | awk '{print $1}' | cut -d: -f1 | sort | uniq -c | sort -n > /home/mydomain/public_html/ip_report.txt

After that, I view the generate report via web browser at http://mydomain.net/ip_report.txt and following output should appear:

----------------
    818 118.100.150.10
    821 175.142.245.213
    824 124.43.99.74
    829 137.186.89.211
    835 92.101.85.191
    855 41.78.17.172
    855 85.65.48.168
    855 87.69.176.55
    858 201.241.218.180
    880 14.99.3.152
    889 190.197.28.142
    889 41.72.10.60
    902 201.248.128.162
    905 123.176.15.119
    911 115.132.145.225
    911 115.66.95.153
    918 151.27.94.55
    939 189.115.158.192
    947 65.49.71.172
    949 91.185.109.175
    952 186.146.220.235
    980 85.230.95.192
   1019 41.236.216.132
   1030 38.111.147.83
   1040 99.66.113.53
   1062 41.70.178.190
   1071 175.144.133.164
   1107 41.78.17.186
----------------

The left column is number of IP counted while the right column is the IP address respectively. This simple report should be enough to suit what my boss wants!