I tried with both start_position as beginning and end. Both behave in the same way. Only difference is, if there are existing logs and start_position is set as end - logstash won't raise event until a new line gets appended to log. But once that happens, log file is read from beginning. Is there any support in file plugin so that only new lines could be parsed?
Actually, I'm generating some accumulators every 15 mins using aggregate plugin. Accumulators are count of occurrences of specific patterns in logs. My requirement is, accumulator should give counts based on only those events which occurred in last 15 mins.
e.g. let's assume, acum1 is counting "X" event.
- Between 11:00 - 11:15, 10 X events are generated. Between 11:00 - 11:15, acum1 value should be 10
- Between 11:15 - 11:30, 5 X events are generated. Between 11:15 - 11:30, acum1 value should be 5, NOT 15 (10 + 5).
But as per input file plugin behavior, it'e being computed as 15 not 5. Is it some how achievable?
I believe, tail.rb (https://github.com/jordansissel/ruby-filewatch/blob/v0.6.5/lib/filewatch/tail.rb) is doing the job, could I change it somehow to achieve my required functionality? Any help would be highly appreciated as I 'm stuck in to it.
I'm using 6.2.4 version of Logstash.