Linux: 2 Way File Synchronization and Replication using Unison + FTP

Usually, when we want to replicate or synchronize files between network, we will use rsync, scp or sftp. This kind of replication is one-way replication method, it will sync from master (source) to slave (destination) only. What about if we want to have 2 folders which both are masters? So we need a two-way replication method applied.

Why do I need to have 2 folders which sync each other? Because I already have a load balancer run on top of my web server. In this case, I need to do 2 way replication so the web contents will always be the same to any users who access the web site. The load balancer is run on Pound with normal round-robin algorithm. Following diagram might give us some better understanding:

I will use the HTTP load balancer server as the middle man to execute synchronization via FTP. I need to create 2 FTP account (each in Web#1 and Web#2) and using CurlFTPFS, I will mount both FTP account inside the HTTP load balancer server. Then Unison will do the 2 way replications.

Before start, make sure HTTP load balancer server has load balancer running which you can refer to this post and CurlFTPFS is running which you can refer to this post. Variables as below:

OS: CentOS 6.2 64bit
HTTP Load Balancer IP: 192.168.20.20
Web Server #1: 192.168.20.21
Web Server #2: 192.168.20.22
Directory to be sync: /home/mywebfile/

1. We will install using the simplest method which is yum. Make sure RPMforge is installed in your system. Follow this step if you have no idea on how to enable RPMforge repository:

$ yum install -y unison

2. I am assuming that you have install and configure CurlFTPFS. Mount both FTP accounts:

$ curlftpfs 192.168.20.21 /mnt/ftp/ftpuser1 -o allow_other
$ curlftpfs 192.168.0.22 /mnt/ftp/ftpuser2 -o allow_other

3. Configure Unison. Since we want to synchronize both folders /mnt/ftp/ftpuser1 and /mnt/ftp/ftpuser2 using root user, we need to create a default profile so Unison knows what to sync, where to sync and how to sync. Using text editor, open following files:

$ vim /root/.unison/default.prf

And add following line:

root=/mnt/ftp/ftpuser1
root=/mnt/ftp/ftpuser2
batch=true

4. Start and run Unison for first synchronization:

$ unison default

5. You will notice that both directories will now sync. But this one need to be done manually. To automate this, we can use cron job and setup to run unison every minutes:

$ crontab -e

And add following line:

* * * * * /usr/bin/unison default

Save the file and restart crond:

$ service crond restart

Or, we can use Fsniper to trigger “unison default” command. You may see this post on how to install and configure Fsniper. For more information on Unison, you can refer to the manual page at here.

Warning: CurlFTPFS is not really good in handling remote files synchronization if the connection between those FTP servers are slow. You might need to consider using other network file system like NFS and Samba to make sure the synchronization works smoothly.

2 thoughts on “Linux: 2 Way File Synchronization and Replication using Unison + FTP

  1. hello, how to sync using ftp and unison automatic eg : every 24 hours
    thanks

    Reply

    1. Hi,

      This tutorial shows on how to trigger unison syncing every minutes through cron job. If you want it to automatically sync every 24 hours, you need to change the cron job scheduling similar to below:
      0 0 * * * /usr/bin/unison default

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *