Email Migration without Downtime

I have been assigned a task by the big boss to do our company’s email migration without any downtime. Downtime in this case is refer to the time when MX record pointing propagates around the world to the new server. It could cause your emails to be sent to the old server, instead of the new server until the client DNS server clear their DNS cache of the host record.

To achieve this, we need to have a server in between to be the store-and-forward mail server aka backup MX server. Point the MX record to this server and then we change the forwarding IP from old server to the new server. So every new email will be sent directly to the new server without any propagation delay. Following picture shows what will be happened when migrating email server:

I am using following variables:

Backup MX Server: 211.44.22.100
Backup MX OS: CentOS 6.2 64bit
Old mail server: 182.22.60.20
New mail server: 60.1.222.100
Domain: mygreek.biz

Backup MX server

1. To eliminate DNS propagation during migration, we will need to setup a backup MX server at least 3 days before the migration been scheduled. We will use Postfix, a mail transfer agent to serve as store-and-forward mail server. Install Postfix via yum, turn off sendmail and SELINUX:

$ setenforce 0
$ service sendmail stop
$ chkconfig sendmail off
$ yum install -y postfix

2. Edit /etc/postfix/main.cf using text editor as below:

myhostname = forwarder.mygreek.biz   #server hostname
mydomain = localhost
myorigin = $mydomain
inet_interfaces = all                #listen to all available IPs
mydestination = $myhostname, localhost.$mydomain, $mydomain
mynetworks_style = host
relay_domains = mygreek.biz
transport_maps = hash:/etc/postfix/transport

3. Open /etc/postfix/transport to setup the forwarding for relayed domain. Point it to the old server:

mygreek.biz smtp:[182.22.60.20]

4. Activate the Postfix map, start the Postfix service and enable the service on boot:

$ postmap /etc/postfix/transport
$ chkconfig postfix on
$ service postfix start

Backup MX should able to receive and forward emails to the old mail server now. Dont forget to allow port 25 via IPtables so Postfix can run smoothly.

MX record in NS

Once the backup MX server ready, we then need to update the MX records in the name server. Login into the name server and update the MX records and point it to this backup MX:

mygreek.biz.    MX    10    mail.mygreek.biz.
mail             A          211.44.22.100

Make sure you are doing this at least 3 days before the migration start to avoid DNS propagation affect the migration.

Migrating Email

1. Before we start the migration, we will need to stop old server mail service. Since the old server is using cPanel, I will need to stop and disable Exim service so backup MX can take over to store the email in queue until the new mail server up:

WHM > Service Configuration > Service Manager > Unchecked “Enabled” and “Monitor” for Exim > Save

2. I will use cPanel migration utility called “Copy an account from another server”. Just follow the wizard and wait until the account copy complete.

3. Once the email migration completed, we then need to update the forwarding IP in the backup MX to the new server. Login into the backup MX server and edit /etc/postfix/transport:

mygreek.biz smtp:[60.1.222.100]

Then, update the Postfix mapping and restart the service to push the emails in queue to the new mail server:

$ postmap /etc/postfix/transport
$ service postfix restart

In new server, you will notice that you will start to receive emails which already in queue from backup MX server. This will guarantee no lost email and eliminate email being sent to the old server.

Completing the Migration

Monitor the email flow for a while and make sure all emails are delivered to the recipient respectively. If everything run smoothly, it is time to update the MX record and point it to the new mail server so we can remove the MX backup server later:

mygreek.biz.    MX    10    mail.mygreek.biz.
mail             A          60.1.222.100

Please allow the MX backup server to run at least 3 days to make sure new MX has propagated around the world. To double confirm on this, you can use this website to check whether your mail host has been propagated or not:

http://www.whatsmydns.net/

If everything run smoothly, you can shut down and remove the backup MX server and migration consider completed.

cPanel: PHPList Subscriber Mailing List Maintenance

In my company, I am also responsible to handle and manage the mailing list server. We are using PHPList, a popular mailing list program to blast out mails and we use it to send our latest news, promotions, announcements and notification to our subscribers. We have a lot of subscribers, which include some of them are problematic mail recipients with such following error:

  • mailbox unavailable
  • no such user
  • user rejected
  • domain already expired

In order to do maintenance on the subscriber list, I need to make sure only active subscriber (which have active mailbox) exist in our mailing list. By referring to exim_mainlog and looking for recipient bounce error, we can remove unwanted recipient from our subscriber list. Variable as below:

OS: CentOS 4 64bit
PHPlist version: 2.10.5
Mail server log: /var/log/exim_mainlog
PHPList database: plist_db
PHPList Username: plist_userdb
PHPList password: p412#Yf

1. Lets generate error log from exim_mainlog so we can easily extract the error information. We will use eximstats command and generate the output to html files, same as what you will see under WHM > Email > View Mail Statistics:

$ /usr/sbin/eximstats -html=/root/mailstats.html -nr -nt -nvr /var/log/exim_mainlog

2. If we go through the generated html file, you can see the list of error captured by exim_mainlog. Example error as below:

<li>1 - [email protected] R=fail_remote_domains: The mail server could not deliver mail to [email protected] The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.

The error above describing that the remote domain is no longer exist and unreachable. We do not want this email account anymore in our subscriber list so we need to extract the email address and remove from PHPlist database.

3. Lets extract all emails which related to this error to a file called domain_error.txt:

$ cat /root/mailstats.html | grep "The account or domain may not exist, they may be blacklisted, or missing the proper dns entries" > domain_error.txt

4. From the domain_error.txt list, we can extract email address only so we can pass this value to PHPlist database via SQL statement:

perl -wne'while(/[\w\.\-][email protected][\w\.\-]+\w+/g){print "$&\n"}' domain_error.txt | sort -u > delete_list.txt

5. We should now have a list of email address that we want. We will use a BASH script to automate the deletion process. Create a file called phplist_delete under /root/scripts folder:

mkdir -p /root/scripts
touch /root/scripts/phplist_delete

Copy and paste following scripts and change the configuration value to the one that suits you. You can retrieve the database information inside PHPlist config.php file (usually under config directory):

#!/bin/bash
# Delete PHPlist subscriber who listed in delete_list.txt
 
# Configuration value
HOST="localhost"
DATABASE="plist_db"
USERNAME="plist_userdb"
PASSWORD='p412#Yf'
DELETE_LIST="/root/delete_list.txt"
 
###
MYSQL="$(which mysql)"
 
if [ ! -f $DELETE_LIST ]
then
        echo "List file not found!"
        exit 1
else
        echo "Deleting email addresses in PHPlist database.."
        cat $DELETE_LIST | while read emailadd; do
                $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE FROM $DATABASE.phplist_user_user WHERE email like '$emailadd';"
                echo "$emailadd: deleted!"
        done
        $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE from $DATABASE.phplist_user_user_attribute WHERE userid NOT IN (SELECT  id FROM $DATABASE.phplist_user_user);"
        $MYSQL -h $HOST -u $USERNAME -p$PASSWORD <<< "DELETE from $DATABASE.phplist_user_user_history WHERE userid NOT IN (SELECT id FROM $DATABASE.phplist_user_user);"
        echo "Process completed!"
fi

6. Change the permission:

chmod 755 /root/scripts/phplist_delete

7. Lets run the script and you are done!

/root/scripts/phplist_delete

Linux: Email Alert on MySQL Replication Failure

In my environment, MySQL replication is really important because we are splitting different web servers with different database server to balance the load between MySQL servers. It quite traditional ways because this is kind of old database servers which sustain until today.

There is simple way to setup a monitoring script to alert us via email on any replication error. This script need to be run on slave server of MySQL replication.

Variable as below:

Database server: MySQL 5.0.77
Hostname: mysql.mydomain.org
Database host: localhost
Database name: mymarket_data
Database user: root
Database password: [email protected]

1. Create directory and file for the script:

$ mkdir -p /root/scripts
$ touch /root/scripts/check_slave.php

2. Copy and paste following PHP scripts and put it into a file called check_slave.php:

<?php
/**
 * Description: This script checks the slave status on the configured host
 *              printing "BAD <description>" or "OK <description>" for failure
 *              or success respectively.
 *              The possible failures could include:
 *              1) connection failure
 *              2) query failure (permissions, network, etc.)
 *              3) fetch failure (???)
 *              4) slave or io thread is not running
 *              5) Unknown master state (seconds_behind_master is null)
 *              6) seconds_behind_master has exceeded the configured threshold
 *
 *              If none of these condition occur, we asssume success and return
 *              an "OK" response, otherwise we include the error we can find
 *              (mysqli_connect_error() or $mysqli->error, or problem
 *               description).  A monitoring system need only check for:
 *              /^BAD/ (alert) or /^OK/ (everybody happy)
 */
 
/* **************************
 * Change related value below
 * **************************
 */
    $host = "";
    $user = "";
    $pass = "";
    $mailto = ""; // Your email address
    $mailsubject = "";
    $mailfrom = "";
 
/* ******************************************
 * No need to change anything below this line
 * ******************************************
 */ $mailmessage = "BAD: ".$err_msg."\n";
    $mailheaders = "From:" . $mailfrom;
    error_reporting(E_ALL);
    header("Content-Type: text/plain"); # Not HTML
    $sql = "SHOW SLAVE STATUS";
    $skip_file = 'skip_alerts';
    $link = mysql_connect($host, $user, $pass, null);
 
    if($link)
        $result = mysql_query($sql, $link);
    else {
        printf("BAD: Connection Failed %s", mysql_error());
        mysql_close($link);
        return;
    }
 
    if($result)
        $status = mysql_fetch_assoc($result);
    else {
        printf("BAD: Query failed - %s\n", mysql_error($link));
        mysql_close($link);
        return;
    }
 
    mysql_close($link);
 
    $slave_lag_threshold = 120;
 
    $tests = array(
        'test_slave_io_thread' => array('Slave_IO_Running', "\$var === 'Yes'",
                                        'Slave IO Thread is not running'),
        'test_slave_sql_thread' => array('Slave_SQL_Running', "\$var === 'Yes'",
                                        'Slave SQL Thread is not running'),
        'test_last_err' => array('Last_Errno', "\$var == 0",
                                 "Error encountered during replication - "
                                 .$status['Last_Error']),
        'test_master_status' => array('Seconds_Behind_Master', "isset(\$var)",
                                        'Unknown master status (Seconds_Behind_Master IS NULL)'),
        'test_slave_lag' => array('Seconds_Behind_Master',
                                  "\$var < \$slave_lag_threshold",
                                  "Slave is ${status['Seconds_Behind_Master']}s behind master (threshold=$slave_lag_threshold)")
    );
 
    $epic_fail = false;
    if(is_file($skip_file))
        $epic_fail = false;
    else
    {
        foreach($tests as $test_name => $data) {
            list($field, $expr, $err_msg) = $data;
            $var = $status[$field];
            $val = eval("return $expr;");
            if(!$val) {
                mail($mailto,$mailsubject,$mailmessage,$mailheaders);
                $epic_fail = true;
            }
        }
    }
 
    if(!$epic_fail) {
        print "OK: Checks all completed successfully\n";
    }
?>

3. Change related value on following line:

/* **************************
 * Change related value below
 * **************************
 */
    $host = "localhost";
    $user = "root";
    $pass = "[email protected]";
    $mailto = "[email protected]"; // Your email address
    $mailsubject = "mysql.mydomain.org Replication Alert";
    $mailfrom = "[email protected]";

4. Setup cron to execute the script every 15 minutes:

*/15 * * * * /usr/bin/php -q /root/scripts/check_slave.php

5. Restart cron service:

$ service crond restart

Done. You should receive notification email if the master/slave replication failed!

 

Update

Following script is being modified by Joel Brock as stated in comment section below:

<!--?php 
/**
* Description: This script checks the slave status on the configured host
* printing "BAD ” or “OK ” for failure
* or success respectively.
* The possible failures could include:
* 1) connection failure
* 2) query failure (permissions, network, etc.)
* 3) fetch failure (???)
* 4) slave or io thread is not running
* 5) Unknown master state (seconds_behind_master is null)
* 6) seconds_behind_master has exceeded the configured threshold
*
* If none of these condition occur, we asssume success and return
* an “OK” response, otherwise we include the error we can find
* (mysqli_connect_error() or $mysqli--->error, or problem
* description). A monitoring system need only check for:
* /^BAD/ (alert) or /^OK/ (everybody happy)
*/
 
/* **************************
* Change related value below
* **************************
*/
$host = array(
// “cr-1″ => “192.168.0.81″,
// “cr-2″ => “192.168.0.82″,
// “cr-3″ => “192.168.0.83″,
// “cr-4″ => “192.168.0.84″,
// “cr-5″ => “192.168.0.85″,
// “jp-1″ => “192.168.100.81″,
// “jp-2″ => “192.168.100.82″,
// “jp-3″ => “192.168.100.83″,
// “jp-4″ => “192.168.100.84″,
// “jp-5″ => “192.168.100.85″,
“cr-test” => “192.168.0.87″
);
$user = “”;
$pass = “”;
$mailto = “”;
$mailfrom = “”;
 
/* ******************************************
* No need to change anything below this line
* ******************************************
*/
 
error_reporting(E_ALL);
header(“Content-Type: text/plain”); # Not HTML
foreach ($host as $key => $value) {
 
$mailsubject =[".$key."] SLAVE REPLICATION ALERT”;
$mailheaders = “From:. $mailfrom;
$sql = “SHOW SLAVE STATUS”;
$skip_file = ‘skip_alerts’;
$link = mysql_connect($value, $user, $pass, null);
 
if($link)
    $result = mysql_query($sql, $link);
else {
    printf(“BAD: Connection Failed %s”, mysql_error());
    mysql_close($link);
    return;
}
 
if($result)
    $status = mysql_fetch_assoc($result);
else {
    printf(“BAD: Query failed – %s\n”, mysql_error($link));
    mysql_close($link);
    return;
}
 
mysql_close($link);
 
$slave_lag_threshold = 120;
 
$tests = array(
    ‘test_slave_io_thread’ => array(‘Slave_IO_Running’, “\$var === ‘Yes’”,
    ‘Slave IO Thread is not running’),
    ‘test_slave_sql_thread’ => array(‘Slave_SQL_Running’, “\$var === ‘Yes’”,
    ‘Slave SQL Thread is not running’),
    ‘test_last_err’ => array(‘Last_Errno’, “\$var == 0,
    “Error encountered during replication – ”
    .$status['Last_Error']),
    ‘test_master_status’ => array(‘Seconds_Behind_Master’,isset(\$var),
    ‘Unknown master status (Seconds_Behind_Master IS NULL)),
    ‘test_slave_lag’ => array(‘Seconds_Behind_Master’,
    “\$var $data) {
        list($field, $expr, $err_msg) = $data;
        $var = $status[$field];
        $val = eval(return $expr;);
        $val1 = (!$val) ? $val1 + 1 : $val11;
        $mailmessage .= “BAD:. $key . ” replication failed. Reason:. $err_msg . “\n”;
        }
    if ($val1 > 0) {
        mail($mailto,$mailsubject,$mailmessage,$mailheaders);
        print $mailmessage . “\n”;
        $epic_fail = true;
        }
    }
 
    if(!$epic_fail) {
        print “OK: Checks all completed successfully on [".$key."]\n”;
    }
}
?>

System Administration: Managing Remote Location

As a system administrator which administrating many branches, I need to support the end-user environment as well. Doing this from single location is quite hard and I need to create the best environment to manage all of these stuffs efficiently.

I am listing out some tips or what we can do to improve communication and collaboration between branches:

VPN between branches

  • VPN has ability to bring all of the computers in different branches connected to each other via a secured network. This will make sure that data communication between colleagues is protected and user can feel like they are in one single place.
  • The recommended way to do this is to setup a VPN (PPTP) server at one location (lets say head quarter). Create VPN account and assign to everyone in the company with dedicated internal IP (for better tracking).
  • All sensitive information should be located in one place and can only been accessed via VPN connectivity. This will prevenet data leakage and you have logs to every access to the internal system via VPN server.

Internal instant messaging system (chat)

  • Instant messaging is important to improve communication and collaboration. You can use any messaging service available online like GTalk, MSN Messenger, Yahoo Messenger and Skype. It depends on you, but it is highly recommended to use internal instant messenger system like Microsoft Lync 2010, BigAnt Office Messenger, Outlook Messenger and many more.
  • Using internal messenger will give some advantages like:
    • Prevent employee to chat with gossip friends (if you are using public messenger like GTalk and MSN)
    • Simple file transfer and sharing
    • Can back trace the chat history, if the boss suspected something is not right with employee (Good for the boss!)
    • Prevent outsider from sniffing your conversation

DMZ zone

  • Depending on how your network infrastructure being setup, you may need to have DMZ zone available to secure the internal network. DMZ zone is something that we called ‘another network zone that exposed to the public network’. Basically, it help you to isolate your internal network and in the same time able can connect to the web server that exposed to the public network.
  • Example of simple DMZ setup:
  • This is example if you not setup a DMZ with same peripherals as above:
     You can notice how unsecured it can be if you include the servers in one internal network.
  • To setup DMZ, what you need to do is just:
    • Create another network in your router with another subnet and IP range
    • Make sure the incoming connection from public network to web and email service to DMZ only via router
    • Make sure your internal LAN can be connected to DMZ via router
    • Make sure your external firewall blocks all incoming connection unless for web, email and NAT

Network drive and file sharing

  • File sharing and network drive is needed whenever users need to send big files, usually more than 10MB which usually not recommended to be sent via instant messenger or email.
  • The most popular and easy to setup file sharing is SAMBA, where you can map directly in each PC to the public sharing directory. SAMBA server/client comes by default for Windows, Mac and Linux.
  • Using VPN which connect all employees in one secure network will make SAMBA easier to setup and implement.
  • Other file sharing protocol like FTP might be time consuming to setup and slow due to binary data transfer. NFS in other hand is not come by default in Windows and you need to install the client to connect.

Collaboration portal

  • This is really important if you rely on the teamwork. Collaboration portal is something that most of companies think that is is a waste and should not be implement. This is wrong. I am suggesting you to try any collaboration portal and install it in your private server. Play around with that and you will see the importance of it.
  • Collaboration portal can help you to achieve:
    • Create, manage and monitor project, task and  report and assign it to users
    • One central point to store and share confidential document
    • Applying leave and check leave balance
    • Synchronize and connect the account to mail, calendar, instant messaging, active directory, CRM and other services
    • Edit the any document online, without need to download and resend it back
  • There are a lot of collaborative software available in the market and some of it is open-source. You can browse the list at http://en.wikipedia.org/wiki/List_of_collaborative_software

This is what I manage to setup the network office on company that I am working for. Do share with us if you have more point to highlight!

Setup Mail Gateway/Forwarding using Postfix

I will show you on how to setup a mail forwarding run in Postfix, which is my MX record will be the email gateway and this server will forward all emails to my mail server which run under cPanel.

What we really need is an MTA (mail transfer agent), application which route your email here and there until all the transaction complete and the email reach the destination. Variables as below:

OS: CentOS 5.5 64bit
MTA version: Postfix 2.3.3
Mail gateway IP: 28.90.150.2
Mail gateway IP: forwarder.getmail.com
Destination server (cPanel): 28.90.166.73
Domain: getmail.com, yoursetup.net and mymouse.biz

1. In this case, I already have MX record which pointing to my cPanel server for 3 domains as below:

getmail.com.    MX    10    mail.getmail.com.
mail             A          28.90.166.173
 
yoursetup.net.    MX    10    mail.yoursetup.net.
mail               A          28.90.166.173
 
mymouse.biz.    MX    10    mail.mymouse.biz.
mail             A          28.90.166.173

2. Lets setup and configure MTA and all required applications. We also need to stop sendmail (by default has been enabled by system), remove sendmail from start-up service, disable SElinux and install Postfix using yum:

service sendmail stop
chkconfig sendmail off
setenforce 0
yum install postfix -y

3. We need to do some configuration to tell Postfix what type of MTA it should be. Edit /etc/postfix/main.cf with text editor and change or uncomment following value: Continue reading “Setup Mail Gateway/Forwarding using Postfix” »

Spam, Spammer, Spambots = Money

Spams, spammers and spambots are exist for only one purpose, money.

1. Spam is email that is sent to other people without being requested. Why they want to disturb our life? Because this is one way of advertising.

2. Spammer will mostly send you something that you don’t know and don’t want to know, and turn to make you know, which equal to advertising. Advertising agency usually get paid for publishing advertisement, newsletter, social announcement and many more, so do spammer. Spammer get money for doing advertising on bad things like replica stuffs, pills, porn, multi-level marketing etc. In short word, spammer is ‘bad advertising agency’.

3. Spammers are not stupid. They have capabilities to be hackers, software developers, system engineers, researchers who tend to get more money which will bring themselves happiness, with less effort. They know how to do things right, do automation for their spamming task, bypass all security features and build many ‘add-on features’ in order to bring the ‘advertisement’ directly to you.

Continue reading “Spam, Spammer, Spambots = Money” »

How to Increase Email Reliability

Have you encountered valid email being delivered to your Junk/Bulk/Spam? Why is this happened since you are not a spammer? How to make sure my email going through to the Inbox?

We called this as false-positive. False positives are innocent emails that get mistakenly identified as spams. Recent mail system security has incredibly tighten due to number of spam pattern increase. Before your email being delivered to respective mailbox, the email being filtered based on recipient mail server rules.

Example of filtering that can happened in recipient mail server:

1. PTR checking (pointer record or Reverse DNS)
2. SPF checking (Sender Policy Framework)
3. Bayesian Filtering
4. SpamAssassin Server Scoring and Filtering
5. RBL (Real-time Blackhole List)

What we can do?

Depending on how tight is the filtering level, false-positive can happen in any mail server. This is quite annoying since you cannot do anything from your side to fix this. What you can do from your side then? You can use following tips to bring up email’s reliability:

  • PTR – You must use a SMTP server to relay your email to the recipient. That server must have a public IP which recipient can see. That public IP must have a reverse lookup value. Example:
    • Public IP: 154.80.143.22. Hostname: mail.myserver.net
    • When you lookup mail.myserver.net, you surely can get 154.80.143.22, but when you reverse lookup 154.80.143.22, do you get the same result (mail.myserver.net)?
    • How to create PTR records? You MUST contact the IP owner, which can be found from whois page.
  • SPF – This one is useful to tell the world that your domain’s email address should come from certain IP address. Every spammer can use your domain as “FROM:” field, SPF checking will make sure the domain send from, match the sender IP specified in SPF record. You can generate the SPF records from OpenSPF and apply into DNS records of your domain (TXT records).
  • DKIM – This is quite new technology, where sender prove the email comes from them by signing the email with digital signature. You can browse around to see how to enable DKIM for your domain/server.
  • dnsbl.info – Make sure your SMTP IP address is not listed in this website, http://www.dnsbl.info/ . This website can tell whether your IP is in any RBL list or not. If listed, contact the anti-spam organization that list your IP and request for removal. You might need to follow their requirement for that.
  • Click “Not Spam” – Usually, if the sender is using a new domain and do not have any transaction with that particular mail server previously, it will mark your email as spam especially for email service provider like Yahoo, Gmail and Hotmail. If it happens, make sure you click “Not Spam” or “Not Junk” to let the the mail server know that this is a valid email and should be sent to inbox.

If you have done everything as list above but still cannot pass through the inbox, something is not right on the recipient side. Contact their system administrator and let them know about this so they can whitelist you inside their server.

Leave a comment if you have more point to share. Cheers!