Aggregate filter doesn't find last line of file

Aggregate filter doesn't find last line of file

I am trying to identify the last line of my log files using aggregate timeout. Then I am setting set it a "_aggregatetimeout" tag. (I want to later use it in a 'elapsed' plugin to calculate the total time a user spends on his session)
I have timeout_task_id_field => "path" so that when new entries stop coming with the same filename, it would assume a timeout. I'm facing below issues:

  1. It almost never hits the last event from the file, but assumes timeout on an event a few lines before the last.
  2. If I set a very small value for timeout, say inactivity_timeout => 60, it detects the final event almost the last one, but it also hits those other lines the user might have taken a 1+ minutes break. Thus I must increase the timeout number
  3. If i increase the timeout to a bit bigger number (inactivity_timeout => 7200) it doesnt find any hits on more than half of the files.
aggregate {
					periodic_flush => true
			         task_id => "%{path}"
					 code => "map['lasttimestamp'] = event.get('time_stamp')"
			         push_map_as_event_on_timeout => true 					 
					 timeout_task_id_field => "path"
					 timeout_timestamp_field => "@timestamp"
					 inactivity_timeout => 7200 # x sec timeout
					 ##timeout => 1000 # x sec timeout from first entry
					 timeout_tags => ['_aggregatetimeout']		#when timeout happens this will be added to tags of the new event			 
					 timeout_code => "event.set('time_stamp', event.get('lasttimestamp'))"
		   match => ["time_stamp", "yyyy/MM/dd-HH:mm:ss.SSS"]   

For it to find the last line the order of lines has to be preserved. Have you set pipeline.workers 1 and disabled java_execution?

If you set inactivity_timeout to 7200 I would expect you to get an error since that would then be greater than the default value of timeout.

I did both ( pipeline.workers: 1 pipeline.java_execution: false on my logstash.yml , but still the same results. Is there any better ways to directly read the 'last line' from a log file?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.