Logstash Combine two fields of different documents based on another field (without an ID)

I am using the latest version of logstash(7.6.2). I am trying to add two fields based on values in an another field. Find below my sample data with fields TimeStamp, Value and Event. Also, find below my desired output.

Sample Data

18/May/2015:02:05:10 +0000 10 a
18/May/2015:02:05:15 +0000 20 b
18/May/2015:02:05:26 +0000 30 a
18/May/2015:02:05:48 +0000 40 b
18/May/2015:15:05:08 +0000 50 a
18/May/2015:16:05:03 +0000 60 b

Desired Output with extra field

18/May/2015:02:05:10 +0000 10 a
18/May/2015:02:05:15 +0000 20 b 30
18/May/2015:02:05:26 +0000 30 a
18/May/2015:02:05:48 +0000 40 b 70
18/May/2015:15:05:08 +0000 50 a
18/May/2015:16:05:03 +0000 60 b 110

What I was trying to do is to add the field Value based on Event and create a new field for Event=b.

Elapsed filter does something like this but it finds the difference of timestamps. But, what i want is to add Value fields based on Event tags. I tried using aggregate but it requires a unique ID. Is there any way to do this without using ID or creating dynamic IDS?

Any help here is appreciated. Thanks in advance

You can do that with an aggregate filter. Use mutate+add_field to add a task_id (which can be a constant) then if the last field is a then set map_action => "create" and if the last field is b then set map_action => "update" and end_of_task => true

Make sure you disable java_execution in addition to setting pipeline.workers 1 so that the order of events is preserved.

Thanks for your reply @Badger. I have used a Ruby class variable to capture the value between the events. It worked wonderfully. I am sure the aggregate filter you suggested will also do the task. But, I want to understand which one would be efficient in terms of performance? Thanks in advance!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.