# Aggregate filter gets different output even with same input

**URL:** <https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394>\
**Category:** Logstash\
**Created:** [July 16, 2020, 6:42am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394 "2020-07-16T06:42:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 16, 2020, 6:42am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/1 "2020-07-16T06:42:24Z")

</div>

I encountered similar problem again while using a mass of test data. This time to set pipeline.workers=1 and pipeline.java\_execution = false in logstash.yml cannot help anymore.

I have 2 input files like below -  
inbound.txt:  
17:03:56.662 53897 55931  
17:03:56.737 53898 55932  
17:03:56.814 53899 55933  
17:03:56.889 53900 55934  
outbound.txt:  
17:03:56.887 77307 55931  
17:03:56.953 77308 55932  
17:03:57.028 77309 55933  
17:03:57.105 77310 55934

So my current status is to test with 10/20/30 lines data, it works perfect. However, to use 100+ lines test data, the output results become random, for example, both 100 lines test data in 2 files, run the logstash config twice, the first time 87 lines can matched successfully and left 13 records not matched. The the 2nd time, 58 records can matched but other 42 records not matched.

What's going wrong in my config or is it a bug when logstash aggregate filter handle mass of data? Please advise and help!

、、、  
filter {  
if [type] == "inbound" {  
grok {match =\> {"message" =\>"%{TIME: TIME1}\s+.\*\s+%{NOTSPACE: FIXinID}"}}  
}

if [type] == "outbound" {  
grok {match =\> {"message" =\>" %{TIME: TIME6} {%NOTSPACE: OutID} %{NOTSPACE: FIXinID} " }}  
}

mutate { remove\_field =\> ["@version","host","message","path"] }

if [type] == "inbound" {  
aggregate {  
task\_id =\> "%{FIXinID}"  
code =\> "map['time1\_a']=event.get('TIME1');"  
map\_action =\> "create"  
}  
}

```
if [type] == "outbound" {
       aggregate { 
                  task_id => "%{FIXinID}"
                  code => "event.set( 'time6_a', event.get('TIME6') ); event.set( 'time1_a', map['time1_a'] );"
                  map_action => "update"
                  end_of_task =>true
                  timeout => 120
                      }
}

```

}  
、、、

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 16, 2020, 6:07pm UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/2 "2020-07-16T18:07:30Z")

</div>

Are you relying on the relative order of events from two different inputs? I do not believe that is supported.

---

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 16, 2020, 11:40pm UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/3 "2020-07-16T23:40:24Z")

</div>

> [@Badger](#):
>
> Are you relying on the relative order of events from two different inputs? I do not believe that is supported

I imported data from 2 different files in one input, just like below. Not supported, either?

···  
input {  
file{  
path =\> ["/vars/logstash/rowdata/inbound.txt"]  
sincedb\_path =\> ["/vars/logstash/test.sincedb"]  
start\_position =\> "beginning"  
mode =\> "read"  
type =\> "inbound"  
}  
file{  
path =\> ["/vars/logstash/rowdata/outbound.txt"]  
sincedb\_path =\> ["/vars/logstash/test.sincedb"]  
start\_position =\> "beginning"  
mode =\> "read"  
type =\> "outbound"  
}  
}  
···

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 17, 2020, 1:56am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/4 "2020-07-17T01:56:36Z")

</div>

That is two different input plugins. You cannot control the ordering of events.

---

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 17, 2020, 2:33am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/5 "2020-07-17T02:33:42Z")

</div>

Got it. Thanks Badger for the info! It”s very useful for me. Let me try combine those file into one input first and then run the Logstash config to aggregate all events.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2020, 2:33am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394/6 "2020-08-14T02:33:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.