Logstash to logstash cluster with or without load balancer

Elastic Stack Logstash
1

My current setup is:

Local Filebeats Logstash Output => 
Local Logstash TCP Input => 
Local Logstash TCP Output => 
Cloud Load Balancer =>
Cloud Cluster (Logstash1 TCP Input, Logstash2 TCP Input) =>
Cloud Elastic Search

The problem is that the cloud load balancer will not really load balance the traffic as Logstash uses persistent TCP connections. So once a Local Logstash server connects to a Cloud Logstash server, it will not renegotiate a connection until one of them goes down.
So in my example above, if I reboot Cloud Logstash1, then all of my traffic will switch over to Cloud Logstash2. Yay!
But when Cloud Logstash1 comes back online, nothing will switch over to Cloud Logstash1 until Cloud Logstash2 goes offline. Boo!

TCP output only has an option for 1 host, and no load balancing.

So my question is, how do I and or you, handle load balancing in this situation? And for the love of everything Holy, don't recommend Lumberjack, it doesn't work.

2

It looks like I either need to build my own cluster solution, like Kafka, or use Lumberjack.

I have e-mailed Jordan, I think, about trying to resurrect Lumberjack from the dead. I finally found where it says that Lumberjack has been deprecated.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.