Running multiple filebeat instances to handle netflow load

[hjazz6]

Hi,

I'm currently running Filebeat v7.14 with the Netflow module to send Netflow traffic directly into Elasticsearch. However, when I look at the Filebeat monitoring stats, it appears that I'm dropping packets. I'm thinking of running multiple (maybe 3 or 4 more) instances of filebeat to handle the load.

I've seen the recommendation of using systemd to start multiple filebeat services. Is this a feasible way to load-balance the netflow traffic across multiple filebeat instances?

It seems that I have to set different path.data for each instance? Is that the only thing I have to change, or can I use the same configuration for all the instances?

[tomx1]

Keep in mind that more than one instance will not help you if they should use the same input. Except you are able to send your netflow events to different ports per instance. I have also a performanceissue with filebeat and netflow, see here: Performanceissue with Filebeat and Netflow Input

May I ask you how much Events are you able to send through filebeat? In my case it's ~20.000 flows per second, I have to add an additional VM to get 20.000 more which is not a solution. Somehow it must be possible to get better performance with a single filebeat instance.

[techie.antonio]

We use the new ElastiFlow collector because it is provides much better throughput than Logstash (x16) or Filebeat (x4) on the same hardware and also has a lot more features.