Logstash logrotate

Elastic Stack Logstash
1

Hello all,

I have been googling topic about logrotate on logstash file, such as logstash.log and logstash.stdout, for some reason I couldn't make logrotate working.

My logstash.log and logstash.stdout are pretty intensive, the size that is created in minute are quite big. For logstash.log, every 2 minutes it create approximately 40M, while for logstash.stdout is 90M.

I have logstash configuration for logrotate as follows:

/var/log/logstash/*.log {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
            /sbin/service logstash restart > /dev/null
       endscript
}

When I ran it manually from command line as the following:

logrotate -vdf /etc/logrotate.conf

I got the following error:

...
rotating pattern: /var/log/logstash/*.log  forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/logstash/logstash.log
  log needs rotating
rotating log /var/log/logstash/logstash.log, log->rotateCount is 7
dateext suffix '-20150611'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding logs to compress failed
glob finding old rotated logs failed
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/logstash/logstash.log to /var/log/logstash/logstash.log-20150611
creating new /var/log/logstash/logstash.log mode = 0644 uid = 996 gid = 995
running postrotate script
running script with arg /var/log/logstash/*.log : "
            /sbin/service logstash restart > /dev/null
"
...

If I removed postrotate ... endscript, I got the following message:

rotating pattern: /var/log/logstash/*.log  forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/logstash/logstash.log
  log needs rotating
rotating log /var/log/logstash/logstash.log, log->rotateCount is 7
dateext suffix '-20150611'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding logs to compress failed
glob finding old rotated logs failed
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/logstash/logstash.log to /var/log/logstash/logstash.log-20150611
creating new /var/log/logstash/logstash.log mode = 0644 uid = 996 gid = 995

But the logstash.log file was not reset to zero size file, and no new file such as /var/log/logstash/logstash.log-20150611 was created.

Please share your knowledge and experience if you encountered this issue before.

Thanks!
-Laurentius

2

You might want to check how you are starting Logstash and ensure you aren't passing it any parameters that cause it to do extra logging as that amount of output sounds excessive and is definitely not the average. Check you aren't passing --verbose, --debug or any number of -v parameters to Logstash. I'd also check your Logstash configuration to ensure you aren't using the stdout plugin in your output section.

When you use the -d option with logrotate, it won't do any rotation even though it says it does. Are you always specifying this parameter when running logrotate in your tests?

3

Joshua,

Thanks for sharing your knowledge. I will look into these settings.

4

Hi,

You should try using the copytruncate directive.
See below man page for that:

          Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one.  It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever.  Note that there is a very small time slice between copying the file and  truncating  it, so some logging data might be lost.  When this option is used, the create option will have no effect, as the old log file stays in place.

You can try using something like this (of course, adapt it to your needs):

    size 100M
    rotate 7
    copytruncate
    compress
    delaycompress
    missingok
    notifempty
    nodateext

Also, be careful that it can lead to some logs being lost, from the moment the file get's copied to the one that is truncated (very small amount of time, you should not really be impacted).

Let us know if it helps.