How to aggregate many events?

Elastic Stack Logstash
1

hi,
i'm searching how to aggregate 2 or more events issued from a log file. The events have an common id. This is the criteria to aggregate events.

I has tried the example of logstash 6.8 documentation, but it doesn't work ! :frowning:

image

And I would like these 2 result events to push them into elasticsearch :

{ "country_name": "France", "towns": [ {"town_name": "Paris"}, {"town_name": "Marseille"} ] }
{ "country_name": "USA", "towns": [ {"town_name": "New-York"} ] }
But in fact, the result obtained is 3 lines in Elastic index ! Why ?

i'm using this filter :
image
info : logstash 6.8.3, Elastic 6.8

is there someone to help me please ?
Thanks

2

Have you disabled pipeline.java_execution?

3

Hi Badger,

No pipeline.java, but logstash.pipeline declared :

[2019-10-16T15:59:01,392][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-10-16T15:59:01,410][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.3"}
[2019-10-16T15:59:12,385][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

is there a relation with the malfunction ?

4

Add

pipeline.java_execution: false

in logstash.yml or add '--java-execution false' to the command line to work around this issue. Also you must set pipeline.workers to 1.

5

i tried :

logstash -f D:\LogstashConf\fic_info2.conf --java-execution false --pipeline.workers 1

[2019-10-16T17:16:28,385][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.3"}
[2019-10-16T17:16:39,973][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

But the result is always 3 events in Elastic. Too bad ! :slightly_frowning_face:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.