Filebeat Aggregate & LoadBalance

Elastic Stack Beats
1

Hello,

we try to do some simple kind of fail over for Logstash so we setup a second instance and now we activated the loadbalance on filebeat. We also have some pipelines where we´re using aggregate and relying on previous events.

Is loadbalance even possible with this circumstances or should we link those pipelines with aggregate only to a single logstash instance?

2

The aggregate filter requires all related events to pass through a single thread, which limits performance, scalability and the ability to build a highly available pipeline. If you can describe your data and what you are looking to use the aggregate filter for, maybe someone can come up with alternate approaches that does not suffer from these quite severe restrictions.

3

Hey,
thanks for your reply!

We´ll consider this for future pipelines. Currently it´s just affecting 1 (almost outdated) pipeline so we´ll take the risk of a downtime :slight_smile:

regards

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.