# Logstash aggregate filter is inconsistent

**URL:** <https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925>\
**Category:** Logstash\
**Created:** [December 24, 2019, 7:08am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925 "2019-12-24T07:08:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![karfei00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karfei00/32/59910_2.png) [@karfei00](https://discuss.elastic.co/u/karfei00)\
**Post date:** [December 24, 2019, 7:08am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925/1 "2019-12-24T07:08:27Z")

</div>

I'm trying to group multiple cities into the same country using data from jdbc. I'm making use of the config push\_previous\_map\_as\_event, but result is inconsistent even though SQL is ordered correctly by country code.

This is my logstash conf file

> input {  
> jdbc {  
> jdbc\_driver\_library =\> "C:\logstash\ifxjdbc.jar"  
> jdbc\_driver\_class =\> "com.informix.jdbc.IfxDriver"  
> jdbc\_connection\_string =\> "jdbc:informix-sqli://HOST:PORT/:INFORMIXSERVER=\_soc;DBDATE=MDY4/;DB\_LOCALE=en\_US.57372"  
> jdbc\_user =\> "user"  
> jdbc\_password =\> "password"  
> schedule =\> "\*/30 \* \* \* \* \*"  
> statement =\> "select distinct co.co\_ctry\_cd, ct\_city\_nm from city ct  
> inner join country co on co.co\_ctry\_lk = ct.co\_ctry\_lk AND (co.CO\_REC\_EFFECT\_DT \<= today AND(co.CO\_REC\_EXPIRY\_DT \>= today OR co.CO\_REC\_EXPIRY\_DT IS NULL))  
> where (ct.CT\_REC\_EFFECT\_DT \<= today AND(ct.CT\_REC\_EXPIRY\_DT \>= today OR ct.CT\_REC\_EXPIRY\_DT IS NULL))   
> and ct.co\_ctry\_lk in(111,113) order by co.co\_ctry\_cd"  
> }  
> }  
> filter {  
> aggregate {  
> task\_id =\> "%{co\_ctry\_cd}"  
> code =\> "  
> map['co\_ctry\_cd'] ||= event.get('co\_ctry\_cd')  
> map['cities'] ||= `[]`  
> map['cities'] \<\< {  
> 'ct\_city\_nm' =\> event.get('ct\_city\_nm'),  
> }  
> event.cancel()  
> "  
> push\_previous\_map\_as\_event =\> true  
> timeout =\> 10  
> }  
> }  
> output {   
> stdout {  
> }  
> }

The aggregated result is inconsistent, first 3 iterations are good, starting from 4th multiple maps of the same country are pushed as different events instead of one. Please scroll to the bottom and refer to the **bold** section.

Below is the logstash output, anyone know what is the problem ?

> C:\Development\logstash-7.5.1\bin\>logstash -f C:\logstash\informix\_2.conf  
> Thread.exclusive is deprecated, use Thread::Mutex  
> Sending Logstash logs to C:/Development/logstash-7.5.1/logs which is now configured via log4j2.properties  
> [2019-12-24T14:40:34,189][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
> [2019-12-24T14:40:34,450][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.5.1"}  
> [2019-12-24T14:40:37,855][INFO][org.reflections.Reflections] Reflections took 84 ms to scan 1 urls, producing 20 keys and 40 values  
> [2019-12-24T14:40:39,427][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster\_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.  
> [2019-12-24T14:40:39,453][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, "pipeline.sources"=\>["C:/logstash/informix\_2.conf"], :thread=\>"#\<Thread:0x7220fc run\>"}  
> [2019-12-24T14:40:39,882][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"}  
> [2019-12-24T14:40:40,019][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
> [2019-12-24T14:40:40,711][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
> C:/Development/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated  
> [2019-12-24T14:41:02,670][INFO][logstash.inputs.jdbc][main] (0.183461s) select distinct co.co\_ctry\_cd, ct\_city\_nm from city ct  
> inner join country co on co.co\_ctry\_lk = ct.co\_ctry\_lk AND (co.CO\_REC\_EFFECT\_DT \<= today AND(co.CO\_REC\_EXPIRY\_DT \>= today OR co.CO\_REC\_EXPIRY\_DT IS NULL))  
> where (ct.CT\_REC\_EFFECT\_DT \<= today AND(ct.CT\_REC\_EXPIRY\_DT \>= today OR ct.CT\_REC\_EXPIRY\_DT IS NULL))  
> and ct.co\_ctry\_lk in(111,113) order by co.co\_ctry\_cd  
> C:/Development/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/awesome\_print-1.7.0/lib/awesome\_print/formatters/base\_formatter.rb:31: warning: constant ::Fixnum is deprecated  
> **`INTERATION #1 - AW has 8 cities, BB has 12, which is the expected result`**  
> {  
> "co\_ctry\_cd" =\> "AW",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "ANGOCHI"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ORANJESTAD"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "NOORD"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "ARUBA"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "PARADERA"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "SAVANETA"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "SANTA CRUZ"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "SAN NICOLAS"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:41:02.959Z  
> }  
> {  
> "co\_ctry\_cd" =\> "BB",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "BRIDGETOWN"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ST GEORGE"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ST ANDREW"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "CHRIST CHURCH"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "ST LUCY"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "ST JAMES"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "ST JOSEPH"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "ST JOHN"  
> },  
> [8] {  
> "ct\_city\_nm" =\> "ST THOMAS"  
> },  
> [9] {  
> "ct\_city\_nm" =\> "ST MICHAEL"  
> },  
> [10] {  
> "ct\_city\_nm" =\> "ST PHILIP"  
> },  
> [11] {  
> "ct\_city\_nm" =\> "ST PETER"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:41:14.918Z  
> }  
> [2019-12-24T14:41:30,856][INFO][logstash.inputs.jdbc][main] (0.221281s) select distinct co.co\_ctry\_cd, ct\_city\_nm from city ct  
> inner join country co on co.co\_ctry\_lk = ct.co\_ctry\_lk AND (co.CO\_REC\_EFFECT\_DT \<= today AND(co.CO\_REC\_EXPIRY\_DT \>= today OR co.CO\_REC\_EXPIRY\_DT IS NULL))  
> where (ct.CT\_REC\_EFFECT\_DT \<= today AND(ct.CT\_REC\_EXPIRY\_DT \>= today OR ct.CT\_REC\_EXPIRY\_DT IS NULL))  
> and ct.co\_ctry\_lk in(111,113) order by co.co\_ctry\_cd  
> **`INTERATION #2 - AW has 8 cities, BB has 12, which is the expected result`**  
> {  
> "co\_ctry\_cd" =\> "AW",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "ANGOCHI"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ARUBA"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ORANJESTAD"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "NOORD"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "PARADERA"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "SAVANETA"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "SANTA CRUZ"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "SAN NICOLAS"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:41:30.983Z  
> }  
> {  
> "co\_ctry\_cd" =\> "BB",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "BRIDGETOWN"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ST GEORGE"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ST ANDREW"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "CHRIST CHURCH"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "ST JAMES"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "ST LUCY"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "ST JOSEPH"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "ST JOHN"  
> },  
> [8] {  
> "ct\_city\_nm" =\> "ST MICHAEL"  
> },  
> [9] {  
> "ct\_city\_nm" =\> "ST THOMAS"  
> },  
> [10] {  
> "ct\_city\_nm" =\> "ST PHILIP"  
> },  
> [11] {  
> "ct\_city\_nm" =\> "ST PETER"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:41:49.918Z  
> }  
> [2019-12-24T14:42:00,823][INFO][logstash.inputs.jdbc][main] (0.137559s) select distinct co.co\_ctry\_cd, ct\_city\_nm from city ct  
> inner join country co on co.co\_ctry\_lk = ct.co\_ctry\_lk AND (co.CO\_REC\_EFFECT\_DT \<= today AND(co.CO\_REC\_EXPIRY\_DT \>= today OR co.CO\_REC\_EXPIRY\_DT IS NULL))  
> where (ct.CT\_REC\_EFFECT\_DT \<= today AND(ct.CT\_REC\_EXPIRY\_DT \>= today OR ct.CT\_REC\_EXPIRY\_DT IS NULL))  
> and ct.co\_ctry\_lk in(111,113) order by co.co\_ctry\_cd  
> **`INTERATION #3 - AW has 8 cities, BB has 12, which is the expected result`**  
> {  
> "co\_ctry\_cd" =\> "AW",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "ANGOCHI"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "NOORD"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ORANJESTAD"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "ARUBA"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "PARADERA"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "SANTA CRUZ"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "SAVANETA"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "SAN NICOLAS"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:00.937Z  
> }  
> {  
> "co\_ctry\_cd" =\> "BB",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "BRIDGETOWN"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ST ANDREW"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ST GEORGE"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "CHRIST CHURCH"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "ST JAMES"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "ST JOSEPH"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "ST LUCY"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "ST JOHN"  
> },  
> [8] {  
> "ct\_city\_nm" =\> "ST MICHAEL"  
> },  
> [9] {  
> "ct\_city\_nm" =\> "ST PHILIP"  
> },  
> [10] {  
> "ct\_city\_nm" =\> "ST THOMAS"  
> },  
> [11] {  
> "ct\_city\_nm" =\> "ST PETER"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:19.933Z  
> }  
> [2019-12-24T14:42:31,068][INFO][logstash.inputs.jdbc][main] (0.359830s) select distinct co.co\_ctry\_cd, ct\_city\_nm from city ct  
> inner join country co on co.co\_ctry\_lk = ct.co\_ctry\_lk AND (co.CO\_REC\_EFFECT\_DT \<= today AND(co.CO\_REC\_EXPIRY\_DT \>= today OR co.CO\_REC\_EXPIRY\_DT IS NULL))  
> where (ct.CT\_REC\_EFFECT\_DT \<= today AND(ct.CT\_REC\_EXPIRY\_DT \>= today OR ct.CT\_REC\_EXPIRY\_DT IS NULL))  
> and ct.co\_ctry\_lk in(111,113) order by co.co\_ctry\_cd  
> **`INTERATION #4 - Multiple country AW/BB events, results not aggregated correctly**  
> {  
> "co\_ctry\_cd" =\> "AW",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "ARUBA"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ANGOCHI"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ORANJESTAD"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "SAN NICOLAS"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "NOORD"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "PARADERA"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "SAVANETA"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:31.183Z  
> }  
> {  
> **"co\_ctry\_cd" =\> "BB",**  
> **"cities" =\> [**  
> **[0] {**  
> **"ct\_city\_nm" =\> "CHRIST CHURCH"**  
> **}**  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:31.184Z  
> }  
> {  
> **"co\_ctry\_cd" =\> "AW",**  
> **"cities" =\> [**  
> **[0] {**  
> **"ct\_city\_nm" =\> "SANTA CRUZ"**  
> **}**  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:31.184Z  
> }  
> {  
> "co\_ctry\_cd" =\> "BB",  
> "cities" =\> [  
> [0] {  
> "ct\_city\_nm" =\> "BRIDGETOWN"  
> },  
> [1] {  
> "ct\_city\_nm" =\> "ST GEORGE"  
> },  
> [2] {  
> "ct\_city\_nm" =\> "ST JOHN"  
> },  
> [3] {  
> "ct\_city\_nm" =\> "ST ANDREW"  
> },  
> [4] {  
> "ct\_city\_nm" =\> "ST JAMES"  
> },  
> [5] {  
> "ct\_city\_nm" =\> "ST LUCY"  
> },  
> [6] {  
> "ct\_city\_nm" =\> "ST PETER"  
> },  
> [7] {  
> "ct\_city\_nm" =\> "ST JOSEPH"  
> },  
> [8] {  
> "ct\_city\_nm" =\> "ST MICHAEL"  
> },  
> [9] {  
> "ct\_city\_nm" =\> "ST THOMAS"  
> },  
> [10] {  
> "ct\_city\_nm" =\> "ST PHILIP"  
> }  
> ],  
> "@version" =\> "1",  
> "@timestamp" =\> 2019-12-24T06:42:44.955Z  
> }......

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 24, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925/2 "2019-12-24T13:14:48Z")

</div>

> [@karfei00](#):
>
> [INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4

If you have pipeline.workers set to 4, then you have 4 separate aggregates, each potentially aggregating a subset of the events. You have to set pipeline.workers to 1, and if your aggregation depends on event ordering (which I don't think yours does) then you [must](https://github.com/elastic/logstash/issues/10938) also disable the new java execution engine.

---

<div class="post-metadata">

**Author:** ![karfei00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karfei00/32/59910_2.png) [@karfei00](https://discuss.elastic.co/u/karfei00)\
**Post date:** [December 27, 2019, 6:02am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925/3 "2019-12-27T06:02:10Z")

</div>

thanks, after changing pipeline.workers: 1, now it is aggregating correctly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 24, 2020, 6:02am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-is-inconsistent/212925/4 "2020-01-24T06:02:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.