Filebeat on central Syslog Server for loadbalancing

Elastic Stack Beats
1

I'm planning to get rid of our kafka messaging queue since we just set it up it because it was part of a hdfs, wich will be removed. I now want to scale up from one to two logstash with persisted queue and setup load balancing to be safe. Would it be a good Idea to install filebeat on the central syslog servers, so I can utilize the loadbalancing of filebeat ? The Setup would be as folows :

[Clients] ----syslog---> [Syslog Servers with filebeat installed x2 ]--------beats-------> [Logstash Servers x2]--------> [Elastic Servers x3]

2

I guess this makes sense, also filebeat enables to be configured in a way to support load balance outputs: https://www.elastic.co/guide/en/beats/filebeat/current/load-balancing.html

3

Do you think Filebeat could handle the load ?
We're talking about roughly 1100 events per second, but since filebeat would not do a lot of processing the load should not be that high.

4

you can also configure filebeat to use spooling to disk to deal with traffic spikes.
filebeat should be able to handle this load pretty well. but the only way to be sure is to make a small test with your load.
you might as well be getting some backpressure from upstream (but this you can load balance and scale)

1 Like
5

Thanks a lot for your quick answer.
We plan to test Filebeat with the following scenario:

  • We will deploy a second logstash server
  • We will configure filebeats on both syslog servers to send everything to the new logstash
  • We will keep rsyslog sending messages to kafka
  • The new Logstash will be configured to write messages in a file or on stdout (so we dont save duplicate data in our elastic cluster and dont put double the load on elasticsearch)
  • If filebeat and the new logstash handle the load fine, we will try to switch to just using filebeat and logstash without kafka.
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.