# Logstash Aggregate Filter output different result each time

**URL:** <https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303>\
**Category:** Logstash\
**Created:** [June 30, 2020, 12:52pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303 "2020-06-30T12:52:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [June 30, 2020, 12:52pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/1 "2020-06-30T12:52:58Z")

</div>

I have 2 input files like below -  
inbound.txt:  
17:03:56.662 53897 55931  
17:03:56.737 53898 55932  
17:03:56.814 53899 55933  
17:03:56.889 53900 55934  
outbound.txt:  
17:03:56.887 77307 55931  
17:03:56.953 77308 55932  
17:03:57.028 77309 55933  
17:03:57.105 77310 55934  
17:03:57.180 77311 55935

My aggregate filter part in Logstash config is like below. Purpose is to merge the lines with same 3rd ID ( I named it FIXinID ) into one events. After that I can calculate the time gab for each InID. My current trouble is, the aggregate filter sometimes worked perfect, sometimes not. I don't know how to finger it out. Can anyone help advise please?

```auto
filter {
    if [type] == "inbound" {
          grok {match => {"message" =>"%{TIME: TIME1}\s+.*\s+%{NOTSPACE: FIXinID}"}}
     }

   if [type] == "outbound" {
          grok {match => {"message" =>" %{TIME: TIME6} {%NOTSPACE: OutID} %{NOTSPACE: FIXinID} " }}
     }

   mutate { remove_field => ["@version","host","message","path"] }

   if [type] == "inbound" {
       aggregate { 
                      task_id => "%{FIXinID}"
                      code => "map['time1_a']=event.get('TIME1');"
                      map_action => "create"
                          }
    }
  
    if [type] == "outbound" {
           aggregate { 
                      task_id => "%{FIXinID}"
                      code => "event.set( 'time6_a', event.get('TIME6') ); event.set( 'time1_a', map['time1_a'] );"
                      map_action => "update"
                      end_of_task =>true
                      timeout => 120
                          }
    }
}

```

The combination result sometimes is perfect that all events merge successfully except the alone one FIXinID=55935. However, sometimes only 3 or 2 inbound/outbound events can combine together.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 30, 2020, 4:10pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/2 "2020-06-30T16:10:01Z")

</div>

Have you set pipeline.workers to 1? Are you running a version where you need to disable [java\_execution](https://github.com/elastic/logstash/issues/10938) (anything from 7.0 to 7.6).

---

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 1, 2020, 10:42am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/3 "2020-07-01T10:42:47Z")

</div>

I tried "-w 1" in my execution cmd, it worked! Thanks Badger so much!

---

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 15, 2020, 3:36pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/4 "2020-07-15T15:36:19Z")

</div>

I encountered similar problem again while using a mass of test data. This time to set pipeline.workers=1 and pipeline.java\_execution = false in logstash.yml cannot help anymore.

So my current status is - to test with 10/20/30 lines data, it works perfect. However, to use 100+ lines test data, the output results become random, for example, both 100 lines test data in 2 files, run the logstash config twice, the first time 87 lines can matched successfully and left 13 records not matched. The the 2nd time, 58 records can matched but other 42 records not matched.

What's going wrong in my config or is it a bug when logstash aggregate filter handle mass of data? Please advise and help!

---

<div class="post-metadata">

**Author:** ![Aberry](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@Aberry](https://discuss.elastic.co/u/Aberry)\
**Post date:** [July 15, 2020, 3:37pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/5 "2020-07-15T15:37:11Z")

</div>

Hello Badger, I encountered similar problem with above config. could you please advise? thanks in advanced!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 15, 2020, 5:19pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/6 "2020-07-15T17:19:39Z")

</div>

If you have set pipeline.workers=1 and pipeline.java\_execution = false I am not aware of any other reason why data might not be ordered.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2020, 5:19pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303/7 "2020-08-12T17:19:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.