How to make Logstash highly available

[vikas_gopal]

Hi Experts,

My plan is to make LS highly available as I did it for ES and Kibana.
So I have load balancer for ES and for Kibana. Not sure how I achieve this for LS. Please suggest.

Thanks
VG

[magnusbaeck (Magnus Bäck)]

Since Logstash does not serve user requests in the same way as ES and Kibana it's not clear what "high availability" means.

What kind of inputs and filters do you have? If you have natural buffering of incoming messages because they come from files on a file system and you ship messages via TCP it's typically fine if Logstash is unavailable for short periods of time. The log shipper will pause until Logstash is available again and resend what has been queued up. This strategy obviously won't work for UDP datagrams sent directly from log clients.

With TCP connections you can of course put Logstash behind a load balancer, much like you've apparently done with ES and Kibana.

[vikas_gopal]

Thanks for the response Magnus,

Actually I have UDP port in picture , so any solution in UDP case ?

[PhaedrusTheGreek (Jay Greenberg)]

@vikas_gopal @magnusbaeck

Check out this discussion as well:

Question on "Multiple Connections for Logstash High Availability" diagram published in logstash documentation

@RajkumarV There are a few ways that you can design this, but generally you want to have an intermediary device such as a load balancer to distribute traffic across your logstash instances. You could use any commercial hardware load balancer, or something like HAProxy. Send your UDP traffic to the Virtual IP on the Load balancer, which then distributes traffic evenly between logstash instances. Logstash does not have any awareness of the load balancing, it just simply listens for UDP messag…

[magnusbaeck (Magnus Bäck)]

Actually I have UDP port in picture , so any solution in UDP case ?

If you have a load balancer that supports UDP proxying then you should be good. Otherwise you can use DNS; a service would have to probe each Logstash instance and take it out of the rotation if it goes down. Perhaps Consul or etcd would be useful here. Otherwise there are certainly commercial options like F5 GTM.

[Joshua_Rich]

You should also have a read through the Deploying and Scaling Logstash chapter of the documentation which explains some possible scenarios for scaling out for high availability and load balancing.