Processing log files while using logrotate

Elastic Stack Logstash
1

I was wondering what everybody's opinion is on processing log files read by logstash but need to be rotated using logrotate. It appears that the two don't work very well together. I'm trying to make sure we don't loose any log data during the log rotation but it appears that if we stop ELK we can't make sure it reads to the end of the file. I did see in the documentation where it says to add multiple log files in the Input - File section, but it seems like there still might be a problem. Thanks

2

Logstash's behavior when logs are rotated is very much underdocumented. There are probably cases where rotations aren't dealt with gracefully.

It appears that the two don't work very well together.

Could you be more specific?

3

If I understand logrotate correctly as it relates to logstash specifically, here is my example: if you are rotating a file called snare.log, logrotate will either make a copy of the file and then truncate the original file or just make a copy of the file and leave the original alone. So, after the first iteration of logrotate, you would have snare.log and snare.log.1 and snare.log may be empty or not depending on what parameter you used. If you stopped ELK before logrotate started and then restarted it after logrotate finished, you need to finish processing anything in the original snare.log BEFORE you start processing what is in snare.log.1. If I'm reading the documentation of Logstash -> Input -> File correctly, there is a chance that you might miss some log entries. I'm including the link here:

https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html#start_position

Thanks

4

If I understand logrotate correctly as it relates to logstash specifically, here is my example: if you are rotating a file called snare.log, logrotate will either make a copy of the file and then truncate the original file or just make a copy of the file and leave the original alone.

Copy & truncate is indeed one way of rotating, but I've never heard of rotation by merely copying. What would be the point of that? The other dominant way of rotating is to rename the log file and either let the program discover the rotation by itself so that it can reopen the file, or HUPing or otherwise restarting or reloading the program.

If you stopped ELK before logrotate started and then restarted it after logrotate finished, you need to finish processing anything in the original snare.log BEFORE you start processing what is in snare.log.1.

You mean that Logstash should read the remainder of (what's now) snare.log.1 before starting with the (more or less empty) snare.log? Yes, if you have order-dependent filtering of events.

5

correct, can you explain what you mean by order-dependent filtering of events?

6

Sorry. I meant filters like multiline, aggregate, or similar that care about the processing order. For independent events it doesn't matter in which order events are processed.

7

I think part of the problem is the since_db file. Using my earlier example, if I have snare.log:

  1. rsyslog is writing to snare.log
  2. At the same time logstash is reading thru it and updating the since_db file for snare.log
  3. Stop ELK and then run logrotate (which forces rsyslog to stop momentarily)
  4. logrotate copies what's in snare.log to snare.log.1 and truncates snare.log and then restarts rsyslog
  5. When ELK restarts, does it see snare.log.1 as the file it was reading previously as snare.log and does it use the same since_db?
    I know the since_db uses the inode and byte offset to know where it is in the file (assuming the inode number follows the new file name but I think rsyslog uses the file handle to keep track of files.

Thanks for the dialog on this!!!

8

Hi

I am really confused. I have read probably every post about this but came out with different conclusion every time !

consider (path => "mylog.log*") && start_position => "end". Now if I have a log file called "mylog.log" with size Y
at some point in time logstash is reading at position X of "mylog.log" and logrotate executed the following (1.) cp mylog.log mylog.log1 (2.) echo "" > mylog.log
question is: what will happen to bytes X-->Y will logstash read them or not ??

9

I too would like to understand how not to lose data in the face logrotation.