Collect multiple events for processing in the filter and send to output (bulk)

Hi folks,

I have following use case:

I receive events from beat and would like to enrich them with additional information using a filter plugin in logstash. The filter plugin must send a request to an endpoint and retrieve information based on the event information I have received in the input. Due to massive amount if events it is not possible to send an API call for each and every one by one (otherwise I would just use http filter plugin) Usually I would collect a given amount of events (lets say 200) and send them as a bulk.

Is there such a mechanism in logstash which will allow me to collect incoming events, process them all at once in a logstash filter and proceed with output?

Thanks for any suggestions

You may be able to do it using an aggregate filter. This thread and the ones it links to may help you with some ideas.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.