How can I configure logstash in this way?

Elastic Stack Logstash
1

I need a way to configure a logstash instance to work as load balancer for other 5 logstash instances.

How can I do that?
and what are the advantages and the disavantages to use multiple instances instead of only one?

ls
ls600×600 9.41 KB

Thanks

2

Logstash is not a load balancer, some outputs have load balancing capabilities, but not all.

The logstash-to-logstash communication can be done using the lumberjack output and input, it seems that you can provide a list of hosts to the lumberjack output.

What is your use case and why you want to use logstash to do Load Balancing instead of a dedicated tool like NGINX or HAProxy?

The advantages or disadvantages of multiple instances depends entirely on your use case and your infrastructure.

You need to share more information about what you are trying to do.

3

Thanks for your reply :slight_smile:
I just need a way to increase scalability and resilience. So I would add more logstash instances. But I want to know what are the disadvanteges too.

I just noticed that I can configure the beats to do the load balancing with multiple instances of logstash: How to make Filebeat send events to multiple logstash in roundrobin

So at this point what is the best option to get more scalability and resilience?

  • Add multiple host and loadbalance=true in the beats configuration
  • Use logstash as loadbalancer with lumberjack plugin
  • Use a dedicated tool like NGINX or HAProxy
  • or other
4

If your input is one of the beats, then use the load balancing directly in beats configuration.

If you have other inputs, like TCP, then I would recommend using NGINX or HAProxy as a load balancer.

The advantages of running multiple instances in different servers is that if one of the instances goes down, the other will continue to receive and process the data, the disadvantage is that you need to make sure that all the configuration are the same, and you also have more servers to take care.

It really depends on your infrastructure as you can automate almost everything.

6

Ok thanks.
So if a logstash instance go down, then the other logstash instances will manage the load? In round robin or other order?

7

If you are using filebeat to ship the logs to logstash and configured your hosts, it will load balance the events using round robin, if one instance goes down, the other will share the events.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.