I have configured logstash on multiple AWS instances. These logstash instances receive thousand of logs and I want a way to wait for the entire trace of logs in logstash and take action on the batch instead of a single log event. I thought of using persistent queues to buffer these logs in logstash but since I am having multiple instances the queue will not be a good option.
Is there some way to buffer logs for some time in logstash and after that send the batches of logs that I want to send to Elastic search?
You mean like a stack trace?
What is sending the logs to Logstash?
Logstash forwarder is sending the logs to logstash. I want a way to buffer logs in logstash for some time and do batch checking on them. If entire batch satisfies my criterion then i need to pass entire batch of logs to elastic search otherwise drop the entire batch.
This is no longer supported, you should upgrade to Beats urgently.
Logstash is not really made for this, I am not sure what you can do.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.