Collect multiple events for processing in the filter and send to output (bulk)

Elastic Stack Logstash
1

Hi folks,

I have following use case:

I receive events from beat and would like to enrich them with additional information using a filter plugin in logstash. The filter plugin must send a request to an endpoint and retrieve information based on the event information I have received in the input. Due to massive amount if events it is not possible to send an API call for each and every one by one (otherwise I would just use http filter plugin) Usually I would collect a given amount of events (lets say 200) and send them as a bulk.

Is there such a mechanism in logstash which will allow me to collect incoming events, process them all at once in a logstash filter and proceed with output?

Thanks for any suggestions

2

You may be able to do it using an aggregate filter. This thread and the ones it links to may help you with some ideas.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.