Multiple Logstash Nodes

Elastic Stack Logstash
1

Is is possible to create multiple logstash nodes running the same pipeline to provide high availability? Like if one of the logstash node is down, the beat will push data into another node to make it work.

I look at this page Deploying and Scaling Logstash | Logstash Reference [8.11] | Elastic, but does not understand the "scalability" section very clearly. It says:

It’s common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources.

So it sounds like each Beats will write to one specific Logstash node. When this node is down, the data from this Beat will be blocked.

Any help or thoughts will be appreciated. Thanks!

2

A Beat's Logstash output can be configured with multiple Logstash hosts. If one of them is down it'll try with one of the other ones. See also the loadbalance option.

3

Thanks! Another question: Right now we run Filebeat on about 30 hosts and push to 3 Logstash nodes. Mostly I use the default settings for those configs and it works fine. We are planning to extend the Filebeat to about 400 hosts. Is there anything we probably need to configure when the input from Filebeat is that large?

4

The number of Filebeat clients doesn't really matter, but if their combined event rate is too high some tuning might be in order.

5

Agree. Which section of Logstash Doc should I look into for tuning issues?

6

Start with Tuning and Profiling Logstash Performance?

7

Get it. Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.