How to make Logstash highly available

Hi Experts,

My plan is to make LS highly available as I did it for ES and Kibana.
So I have load balancer for ES and for Kibana. Not sure how I achieve this for LS. Please suggest.

Thanks
VG

Since Logstash does not serve user requests in the same way as ES and Kibana it's not clear what "high availability" means.

What kind of inputs and filters do you have? If you have natural buffering of incoming messages because they come from files on a file system and you ship messages via TCP it's typically fine if Logstash is unavailable for short periods of time. The log shipper will pause until Logstash is available again and resend what has been queued up. This strategy obviously won't work for UDP datagrams sent directly from log clients.

With TCP connections you can of course put Logstash behind a load balancer, much like you've apparently done with ES and Kibana.

Thanks for the response Magnus,

Actually I have UDP port in picture , so any solution in UDP case ?

@vikas_gopal @magnusbaeck

Check out this discussion as well:

Actually I have UDP port in picture , so any solution in UDP case ?

If you have a load balancer that supports UDP proxying then you should be good. Otherwise you can use DNS; a service would have to probe each Logstash instance and take it out of the rotation if it goes down. Perhaps Consul or etcd would be useful here. Otherwise there are certainly commercial options like F5 GTM.

You should also have a read through the Deploying and Scaling Logstash chapter of the documentation which explains some possible scenarios for scaling out for high availability and load balancing.

1 Like