Logstash aggregate without unique id

Hello, supposed il have current log

 INFO - 12345 - TASK_START - start
 INFO - 12345 - SQL - sqlQuery1 - 12
 INFO - 12345 - SQL - sqlQuery2 - 34
 INFO - 12345 - TASK_END - end
 INFO - 12345 - TASK_START - start
 INFO - 12345 - SQL - sqlQuery1 - 24
 INFO - 12345 - SQL - sqlQuery2 - 48
 INFO - 12345 - TASK_END - end

What Il would do is to use logstash aggregate plugin for this where Il don't have unique id.
For this lines of log, my desired output is two documents :slight_smile:

  INFO - 12345 - 46 (Result of 12+ 34) 
  INFO - 12345 - 72 ( Result offre 24+48) 

Thanks in advance.
Regards

That exactly matches the use case in example 1 of the documentation.

When I use example 1 configuration, This give me one document

INFO - 12345 - 118 ( result of 12+34+24+48)

That surprises me, since the code deletes the map entry if end_of_task is true.

You can get it to work using --pipeline.batch.size 1.

By default, logstash works in batches of 125 events, so 125 events are parsed using grok, then 125 events go through the aggregate that creates the map if it does not exist, then 125 events update the map, then 125 events go through the aggregate that ends the aggregation if it is a TASK_END. So the sql_duration with the value of 118 is added to the first TASK_END, not the second.

Great, it works Fine.
Thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.