Compare current event against previously one

Hello Guys!

We are processing some MSSQL tables via JDBC input plugin. Some times, 3 rolls will be almost equal. Our goal is merge these lines together and transform the values that are different in an array. So, given this 3 events

event1 = { "id" => 1 , "name" => "John", "country" => "Texas" }

event2 = {
"id" => 1 ,
"name" => "John",
"state" => "Alaska"
}

event3 = {
"id" => 1 ,
"name" => "John",
"state" => "Oregon"
}

It need to be merged in something such as:

finalEvent = { "id" => 1 , "name" => "John", "state" => ["Texas", "Alaske", "Oregon"] }

And finally saved into ES.

It's possible? Another question, how to achieve two stages processing into LogStash, something as:

Event -> UnWindKeys () -> MergeObjectsWithSameId () -> Save to ES?

Thanks! :slightly_smiling:

Have you seen https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html?

1 Like

@warkolm Wow! It's awesome! Can you help-me with some piece of code, to achieve the finalEvent? I don't know Ruby, also I'm new to LogStash. Thanks!

Unfortunaly, aggregate plugin can't help you as it works today.
It needs a clearly identified "task end event" to push aggregate map into end task event.

In that case, we can clearly detect "task start event" : it is the first one with id=1.

To fill the need, I plan to add a new option in aggregate plugin :
push_previous_map_as_event => true

When this option is activated, each time a new task is detected (in your case, each time a new id is detected), it automatically push previous aggregate map as a new event in logstash.

Using that feature in your case, you can enrich aggregate map with each db line result, and when aggregate detects a new id, it push map with all aggregated data info as new event.

1 Like

Thanks for you answer, I also answered you via email :wink: Cheers!

This option is now released in logstash-filter-aggregate version 2.2.0.

You can find an example here :

1 Like