Multiple Logstash Nodes

(Reggie Yang) #1

Is is possible to create multiple logstash nodes running the same pipeline to provide high availability? Like if one of the logstash node is down, the beat will push data into another node to make it work.

I look at this page, but does not understand the "scalability" section very clearly. It says:

It’s common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources.

So it sounds like each Beats will write to one specific Logstash node. When this node is down, the data from this Beat will be blocked.

Any help or thoughts will be appreciated. Thanks!

(Magnus Bäck) #2

A Beat's Logstash output can be configured with multiple Logstash hosts. If one of them is down it'll try with one of the other ones. See also the loadbalance option.

(Reggie Yang) #3

Thanks! Another question: Right now we run Filebeat on about 30 hosts and push to 3 Logstash nodes. Mostly I use the default settings for those configs and it works fine. We are planning to extend the Filebeat to about 400 hosts. Is there anything we probably need to configure when the input from Filebeat is that large?

(Magnus Bäck) #4

The number of Filebeat clients doesn't really matter, but if their combined event rate is too high some tuning might be in order.

(Reggie Yang) #5

Agree. Which section of Logstash Doc should I look into for tuning issues?

(Magnus Bäck) #6

Start with Tuning and Profiling Logstash Performance?

(Reggie Yang) #7

Get it. Thanks!

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.