# Aggregate filter doesn't find last line of file

**URL:** <https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962>\
**Category:** Logstash\
**Created:** [June 5, 2020, 1:56pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962 "2020-06-05T13:56:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![anoop](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@anoop](https://discuss.elastic.co/u/anoop)\
**Post date:** [June 5, 2020, 1:56pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962/1 "2020-06-05T13:56:57Z")

</div>

Aggregate filter doesn't find last line of file

I am trying to identify the last line of my log files using aggregate timeout. Then I am setting set it a "\_aggregatetimeout" tag. (I want to later use it in a 'elapsed' plugin to calculate the total time a user spends on his session)  
I have timeout\_task\_id\_field =\> "path" so that when new entries stop coming with the same filename, it would assume a timeout. I'm facing below issues:

1. It almost never hits the last event from the file, but assumes timeout on an event a few lines before the last.
2. If I set a very small value for timeout, say inactivity\_timeout =\> 60, it detects the final event almost the last one, but it also hits those other lines the user might have taken a 1+ minutes break. Thus I must increase the timeout number
3. If i increase the timeout to a bit bigger number (inactivity\_timeout =\> 7200) it doesnt find any hits on more than half of the files.

```auto
aggregate {
					periodic_flush => true
			         task_id => "%{path}"
					 code => "map['lasttimestamp'] = event.get('time_stamp')"
			         push_map_as_event_on_timeout => true 					 
					 timeout_task_id_field => "path"
					 timeout_timestamp_field => "@timestamp"
					 inactivity_timeout => 7200 # x sec timeout
					 ##timeout => 1000 # x sec timeout from first entry
					 timeout_tags => ['_aggregatetimeout'] #when timeout happens this will be added to tags of the new event			 
					 timeout_code => "event.set('time_stamp', event.get('lasttimestamp'))"
		}
			date{
		   match => ["time_stamp", "yyyy/MM/dd-HH:mm:ss.SSS"]   
		} 

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 5, 2020, 2:06pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962/2 "2020-06-05T14:06:23Z")

</div>

For it to find the last line the order of lines has to be preserved. Have you set pipeline.workers 1 and disabled java\_execution?

If you set inactivity\_timeout to 7200 I would expect you to get an error since that would then be greater than the default value of timeout.

---

<div class="post-metadata">

**Author:** ![anoop](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@anoop](https://discuss.elastic.co/u/anoop)\
**Post date:** [June 9, 2020, 12:00pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962/3 "2020-06-09T12:00:22Z")

</div>

I did both ( pipeline.workers: 1 pipeline.java\_execution: false on my logstash.yml , but still the same results. Is there any better ways to directly read the 'last line' from a log file?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2020, 12:00pm UTC](https://discuss.elastic.co/t/aggregate-filter-doesnt-find-last-line-of-file/235962/4 "2020-07-07T12:00:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.