How to configure filebeat for logstash cluster environment?

Elastic Stack Logstash
1

I am missing something very basic when I think of how Filebeat will be configured in a clustered logstash setup.

As per the article https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html and this architecture diagram in the article, I think that there is some kind of load balancer in front of the logstash cluster. However, the Filebeat output documentation suggests that there must be an array of all the Logstatsh nodes specified. Using this list of nodes, Filebeat will do the load balancing from the client-side.

Also as per https://github.com/elastic/logstash/issues/2632 GitHub issue, there is no native logstash clustering available yet.

So, my question is, what kind of setup do I need to be able to point my multiple Filebeat to one logstash service endpoint without specifying the logstash nodes in the cluster?

  • Is it possible?
  • Would having load balancer in front of Logstash cluster be of any help?

Thanks,
Manish

2

Hello @manish.sapariya

Filebeat is able to load balance across multiple Logstash instances using the loadbalance option (see documentation).

It is possible to use a Load balancer, but it requires sticky TCP sessions and enabling ttl to ensure the requests are well balanced across all hosts (see documentation).

3

Hi @Luca_Belluccini,
Thank you for your response. I am aware of the loadbalance option. Here are some specifics which confuses me.

We plan to run the logstash in the AWS autoscale service. As the instances go down and come up dynamically I am not sure how do we keep the filebeat aware of the changes that have taken place in the logstash instances.

Hope this helps understand my question.
-Manish

4

No, the expectation is that filebeat will load balance across an array of logstash endpoints listed in its configuration.