2 instances of Filebeat on same Linux server output to same ES

Elastic Stack Beats
1

Hi,

I have a Linux server running Filebeat 8.3.3 taking Netflow as input and writing it out to ES on another server. As the Netflow load is much more than what Filebeat can handle, I'm thinking of splitting the Netflow input into 2, i.e. input via 2 physical ports on my Netflow server, then run 2 instances of Filebeat, one for each port, and having both write to the same ES.

Is this feasible? If so, how should I go about configuring it? If not, is there a better way?

Thank you.

2

I think you can configure multiple instances of the filebeat running on different ports and sending data to the same.As for the netflow you can use a load balancer which will distribute the data across these filebeat.
You can also try optimizing the filebeat

3

When you say "different ports", do you mean different UDP ports in the netflow packets? The packets are all sending to the same UDP port as the port that filebeat is listening on. Is there a way to "tie" different filebeat instances to different physical ports?

Yes, I've also tried to optimize filebeat, and the best I can do thus far is 13K records indexed on ES per second, but that's still not enough to handle the load.

4

Can you try with logstash

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.