Linux: Log Rotation Customization

Problem Description

Our server has facing problem due to heavy development on web applications project. As the result, it cause our server hard disk turn full within a week. It all cause by the error log, which actually being rotated on weekly basis and has grow to 103GB within a week after rotation!

In the same time, it will consume CPU processing to the peak because the logrotate service will automatically compress the huge log file as what has been setup in /etc/logrotate.conf by default.

Symptom

Following process has been captured when I encounter this problem:

root     13324 38.2  0.0   4028   620 ?        RN   18:59  11:10      \_ gzip -f -

By checking the PID environment, we can see that this process is creating .gz (Gunzip) file for error log in Apache archive directory:

root@server [~] ll /proc/13324/fd
total 0
dr-x------ 2 root root  0 Jan 11 19:00 ./
dr-xr-xr-x 5 root root  0 Jan 11 19:00 ../
lr-x------ 1 root root 64 Jan 11 19:29 0 -> pipe:[1815813470]
l-wx------ 1 root root 64 Jan 11 19:29 1 -> /usr/local/apache/logs/archive/error_log-01-2012.gz
l-wx------ 1 root root 64 Jan 11 19:29 2 -> /usr/local/cpanel/logs/stats_log

Analysis

Following is what has been setup by default on my /etc/logrotate.d/httpd :

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

The log rotation will find any files which ended with ‘log’ (eg: access.log or error.log) under directory /var/log/httpd:

  • missingok – if the log file is missing, go on to the next log file without issuing error
  • notifempty – disable log rotation for empty file
  • sharedscripts – once all log files has been rotated, it will execute postrotate script once. If you are not specifying this option, logrotate will reload httpd for every log file which has been rotated
  • postrotate – bash script to what logrotate will do once the log file rotated. Apache need to be reloaded in order to write to the new files after log rotation complete
  • endscript – means the end line of the postrotate script

 

Solution

So I need to change my log rotation configuration for httpd by adding following line into /etc/logrotate.d/httpd:

/etc/httpd/logs/error_log {
    rotate 5
    nocompress
    notifempty
    size 1024M
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Since I am using cPanel, the Apache log is located under /etc/httpd/logs/error_log:

  • rotate 5 – log rotation will kept the last 5 log files. The rest will be deleted -> For backup purpose
  • nocompress – the last log files will not be compress -> Save CPU consumption
  • notifempty – disable log rotation for empty file -> Do not waste time to process empty file
  • size 1024M – rotate when the file size reach 1GB -> Do not waste disk space
  • postrotate – bash script to what logrotate will do once the log file rotated. Httpd need to be reloaded in order to write to the new files after rotated -> Restart httpd so the new log will take place in new file
  • endscript – means the end line of the postrotate script

Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *