How to aggregate multiple events into single output

Hi All -

I am trying to create a logstash pipeline, that reads all the events in that pipeline, aggregates and produces one single output to elastic search.
i/p looks like this -
'''
{
'entity': '1212',
...
},
{
'entity': '1313',
...
},
{
'entity': '1414',
...
}
'''
My output should look something like this -
'''
{
'entries': {
'entity': [1212, 1313, 1414]
},
'count' : 3
}
'''
How can i achieve this? Any help is appreciated. Thanks.

Use an aggregate filter, something like example three in the documentation.

In the code option do something like

map["entity"] ||= []
map["entity"] << event.get("entity")

and in the timeout_code option do an event.set of count based on the array length.

Thank you. But is there an option like end of event instead of giving a timeout option? In my case, i gave a inactive time of 3 secs and then push the event. But ideally, it should be end of that specific event.

If you can detect the end of an event you might be able to do something like example 4, or even example 1 or 2.

Thank you. Will check that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.