How to aggregate multiple events into single output

[pvaithee07 (vaithee)]

Hi All -

I am trying to create a logstash pipeline, that reads all the events in that pipeline, aggregates and produces one single output to elastic search.
i/p looks like this -
'''
{
'entity': '1212',
...
},
{
'entity': '1313',
...
},
{
'entity': '1414',
...
}
'''
My output should look something like this -
'''
{
'entries': {
'entity': [1212, 1313, 1414]
},
'count' : 3
}
'''
How can i achieve this? Any help is appreciated. Thanks.

[Badger]

Use an aggregate filter, something like example three in the documentation.

In the code option do something like

map["entity"] ||= []
map["entity"] << event.get("entity")

and in the timeout_code option do an event.set of count based on the array length.

[pvaithee07 (vaithee)]

Thank you. But is there an option like end of event instead of giving a timeout option? In my case, i gave a inactive time of 3 secs and then push the event. But ideally, it should be end of that specific event.

[Badger]

If you can detect the end of an event you might be able to do something like example 4, or even example 1 or 2.

[pvaithee07 (vaithee)]

Thank you. Will check that.