Logstash input file plugin reprocesses file from beginning whenever new line gets appended to log

Elastic Stack Logstash
1

Hi,
I tried with both start_position as beginning and end. Both behave in the same way. Only difference is, if there are existing logs and start_position is set as end - logstash won't raise event until a new line gets appended to log. But once that happens, log file is read from beginning. Is there any support in file plugin so that only new lines could be parsed?

Actually, I'm generating some accumulators every 15 mins using aggregate plugin. Accumulators are count of occurrences of specific patterns in logs. My requirement is, accumulator should give counts based on only those events which occurred in last 15 mins.
e.g. let's assume, acum1 is counting "X" event.

  1. Between 11:00 - 11:15, 10 X events are generated. Between 11:00 - 11:15, acum1 value should be 10
  2. Between 11:15 - 11:30, 5 X events are generated. Between 11:15 - 11:30, acum1 value should be 5, NOT 15 (10 + 5).
    But as per input file plugin behavior, it'e being computed as 15 not 5. Is it some how achievable?

I believe, tail.rb (https://github.com/jordansissel/ruby-filewatch/blob/v0.6.5/lib/filewatch/tail.rb) is doing the job, could I change it somehow to achieve my required functionality? Any help would be highly appreciated as I 'm stuck in to it.
I'm using 6.2.4 version of Logstash.

Thanks,
Suresh

2

What platform is this on?

3

It's Linux (2.6.32-696.18.7.el6.x86_64).

4

If it is re-reading the file that suggests that when it gets "appended" to the inode is changing. Can you check that using "ls -li" before and after?

5

Thanks a lot Badger!
This was the issue. If I manually add a line at the end of log file using vi editor, every time file's inode get changed and file is re-read from beginning. If I append to file using >>, inode doesn't get changed and only new records get processed by Logstash. I guess, log4j2 is also not appending at end of file and inode is getting changed. I'll look for such configuration in log4j2. Do you have any idea of such configuration?

Thanks again!

6

What appender are you using?

7

RollingFileAppender. append is not set but by default it should be true.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.