# Aggregate Filter Timeout issue

**URL:** <https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847>\
**Category:** Logstash\
**Created:** [October 17, 2018, 1:22pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847 "2018-10-17T13:22:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dan-at-mimas](https://avatars.discourse-cdn.com/v4/letter/d/94ad74/32.png) [@dan-at-mimas](https://discuss.elastic.co/u/dan-at-mimas)\
**Post date:** [October 17, 2018, 1:22pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847/1 "2018-10-17T13:22:51Z")

</div>

Hello,

I'm processing some historical apache logs using logstash and attempting to use the aggregate filter to identify double-clicks. In this case, a double-click is identified as the same request, from the same client IP and user-agent, occurring within 30 seconds of each other.

Here is example apache log entries that I am processing, with IP and request details anonymised:

[Test logs](https://gist.github.com/dan-at-mimas/b2e2a44259a0482450ca3d229b5929f4)

Here is a reduced version of my logstash config:

[logstash config](https://gist.github.com/dan-at-mimas/c7a0c65c5f841c0910610ae9de41314e)

And here is the output I am getting:

[aggregation output](https://gist.github.com/dan-at-mimas/0e0496d2414bfdc29232bdc975948ddc)

Looking at the original Apache log data, none of the events seem to occur within 30 seconds of each other. In the logstash config I am setting the aggregate filter to a 30-second timeout (this is based on the 'iso\_timestamp' field, which is an ISO date created from the timestamp in the actual log entry).

Given the above information I would have expected there to be an output event for each of the log entries, and no double-clicks - however, this appears in the output:

```
{
	"_id" : ObjectId("5bc72b21962d740353000006"),
	"@timestamp" : ISODate("2018-10-17T12:29:21.251Z"),
	"task_id" : "0.0.0.0Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko/results?terms=test",
	"@version" : "1",
	"double_click" : true,
	"clicks" : 2,
	"iso_timestamps" : [
		{
			"iso_timestamp" : ISODate("2018-10-08T14:18:57Z")
		},
		{
			"iso_timestamp" : ISODate("2018-10-08T14:14:08Z")
		}
	]
}

```

Here these events seem to have been aggregated despite the timestamps being over 4 minutes apart.

I should add that i've also tried using an inactivity\_timeout, which yielded similar results.

Yet for some of my other log processing, it seems to correctly identify double-clicks as defined above.

I am most likely doing something very wrong, but I can't see what it is!

Any help would be gratefully received.

Thanks

---

<div class="post-metadata">

**Author:** ![dan-at-mimas](https://avatars.discourse-cdn.com/v4/letter/d/94ad74/32.png) [@dan-at-mimas](https://discuss.elastic.co/u/dan-at-mimas)\
**Post date:** [October 17, 2018, 3:06pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847/2 "2018-10-17T15:06:52Z")

</div>

I have possibly made some progressed, by not setting iso\_timestamps in the timeout\_code. Not sure why I was doing this in the first place. This seems to now yield the expected results. I just need to confirm that genuine double-clicks are still picked up.

I have updated the gists accordingly

---

<div class="post-metadata">

**Author:** ![dan-at-mimas](https://avatars.discourse-cdn.com/v4/letter/d/94ad74/32.png) [@dan-at-mimas](https://discuss.elastic.co/u/dan-at-mimas)\
**Post date:** [October 17, 2018, 3:20pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847/3 "2018-10-17T15:20:27Z")

</div>

Ok, so weirdly the aggregation doesn't seem to consistently produce the same results.

I've run a couple of times with the same data set and configuration and received different results each time, sometimes

My only guess at this point is something to do with the number of filter workers I am using (i.e. \> 1), however I tried setting the number of workers to 1 in the logstash config, and everything just seems to grind to a halt. And even if this is the case, why is the timeout value not being respected?

Help!

---

<div class="post-metadata">

**Author:** ![dan-at-mimas](https://avatars.discourse-cdn.com/v4/letter/d/94ad74/32.png) [@dan-at-mimas](https://discuss.elastic.co/u/dan-at-mimas)\
**Post date:** [October 18, 2018, 3:18pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847/4 "2018-10-18T15:18:33Z")

</div>

It looks like it was an issue with using multiple workers.

The problem I was having was that when I reduced workers to 1 logstash seemed to grind to a halt, and wouldn't come back up.

I am currently running logstash inside a Vagrant VM, and after investigating further I discovered that while I had the vm configured for multiple cores, I needed this configuration to get it to work effectively:

vb.customize ["modifyvm", :id, "--ioapic", "on"]

After setting that, and lowering the logstash pipeline workers to 1, everything seems to now work as expected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2018, 3:22pm UTC](https://discuss.elastic.co/t/aggregate-filter-timeout-issue/152847/5 "2018-11-15T15:22:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.