Logstash HA

[ksaha (Koushik Saha)]

Hi,

Can anyone please provide what are the various options of implementing Logstash High Availability?
I am aware of using at least two Logstash instance for Logstash HA. Is there any other way by which I can achieve Logstash HA?

Thanks

[warkolm (Mark Walkom)]

That will depend on what you are doing with your Logstash config.

eg can you use a load balancer in front of them both, if you are using a TCP input.

[djb000m]

Hi, sorry to jump on someone else's thread but I'm also looking to horizontally scale Logstash.

@warkolm

That will depend on what you are doing with your Logstash config.

eg can you use a load balancer in front of them both, if you are using a TCP input.

Is using HAProxy in front of the Logstash nodes the preferred method for HA? Do the Logstash nodes require any configuration so they are aware of the others and to share pipelines?

I've checked the documentation but all I can find is a description of what I want, but not the details on how to achieve it.

Thanks

[warkolm (Mark Walkom)]

We don't provide pointed advice there given the number of options. If HAProxy is what you know, then use it

[djb000m]

Wow, you're quick Mark, thanks!

I was curious about this line (in the documentation),

Logstash is horizontally scalable and can form groups of nodes running the same pipeline.

Is there some configuration within Logstash for awareness of other nodes and creating shared pipelines, or is it just a case of manually making sure the configurations match across Logstash nodes?

thanks

[leandrojmp (Leandro Pereira)]

@djb000m

Is there some configuration within Logstash for awareness of other nodes and creating shared pipelines, or is it just a case of manually making sure the configurations match across Logstash nodes?

No, there is not, you need to make sure that your nodes are using the same configuration, this line in documentation basically only says that you can have multiple nodes running the same configuration, but they do not communicate between them and are not aware of the existence of other nodes.

If you pay for the Gold License and beyond, you can use the Centralized Pipeline Management to update your pipelines directly from Kibana, but sill, each node needs to be configured to use a pipeline or note. This feature is not available on the Basic License.