Aggregate filter doesn't find last line of file
I am trying to identify the last line of my log files using aggregate timeout. Then I am setting set it a "_aggregatetimeout" tag. (I want to later use it in a 'elapsed' plugin to calculate the total time a user spends on his session)
I have timeout_task_id_field => "path" so that when new entries stop coming with the same filename, it would assume a timeout. I'm facing below issues:
- It almost never hits the last event from the file, but assumes timeout on an event a few lines before the last.
- If I set a very small value for timeout, say inactivity_timeout => 60, it detects the final event almost the last one, but it also hits those other lines the user might have taken a 1+ minutes break. Thus I must increase the timeout number
- If i increase the timeout to a bit bigger number (inactivity_timeout => 7200) it doesnt find any hits on more than half of the files.
aggregate {
periodic_flush => true
task_id => "%{path}"
code => "map['lasttimestamp'] = event.get('time_stamp')"
push_map_as_event_on_timeout => true
timeout_task_id_field => "path"
timeout_timestamp_field => "@timestamp"
inactivity_timeout => 7200 # x sec timeout
##timeout => 1000 # x sec timeout from first entry
timeout_tags => ['_aggregatetimeout'] #when timeout happens this will be added to tags of the new event
timeout_code => "event.set('time_stamp', event.get('lasttimestamp'))"
}
date{
match => ["time_stamp", "yyyy/MM/dd-HH:mm:ss.SSS"]
}