Logstash-to-logstash communication in 2020 - with load balancing

[nico127]

Hello,

My problem is pretty similar to this one: Logstash to logstash cluster with or without load balancer

I contacted the author, who told me he hadn't found a good solution yet. I'll give a short summary of my problem:

• There are "forwarder" machines in several networks. Many sources from inside their respective networks send them data.
• Those machines then forward the data to either an ES cluster in a completely different network, or to one inside their network. Both cases coexist.
• Depending on the situation, more than one logstash instance might be required in front of those ES clusters - either to handle the load, or for availability purposes.

Most of this is already in place, without logstash.

So my design idea was to have several instances of logstash:

• one on the "forwarder", to have different inputs for different kinds of data, which would then forward to
• one or more for each ES cluster, doing the data operations (extraction, normalisation, etc)

I can't really have all the data operations on the forwarders, as they aren't powerful enough.

So my first question is how to do proper logstash-to-logstash communication? I know the typical answer is the lumberjack input/output plugin, but they haven't been updated in several years, and many problems are reported. I'd rather not move to a deprecated, or unsupported, technology. What's more, the lumberjack output plugin does not support load balancing. An issue is opened since 2016.

Which brings the second question, of course: how to load-balance logstash-to-logstash communication? Is it even possible without external aid?

Thanks in advance for any pointers!
Nico

[PraveenKT (Praveen Kumar Teegala)]

create 2nd config file in forwarder logstash. in beats point output same logstash forwarder with diffeent ports (becase 2 configs are there). can you please explain in more simple way?

Server logstash and CLient logstash. Do you need load balance at client/logstash ?

[nico127]

Hello, thank you for answering.

I don't have beats: I have many different data sources sending to one logstash client that then sends to one or more logstash servers. The logstash client should be able to load balance between the logstash servers.

[PraveenKT (Praveen Kumar Teegala)]

like sending to multiple elasticsearch nodes, send to multiple logstash servers. Is it not working?

another option
From client logstash to send a loadbalancer, from loadbalancer to logstash servers.

another option is like loadbalance but not.
in client logstash create 2 config files, one config file output to one logstash server another one to another logstash server.

[nico127]

No, it is not. Having an external load balancer might work, although I am not sure. Your other option would work for load balancing, but not for failover: if one of the receiving logstash server dies, some data will be lost, which isn't the goal.