Hi,
I'm seeing lots of 429 errors in our Elk stack version 2.2. We have 5 clusters set up each with 24 data nodes. So the current data pipeline is: log_shipper -> kafka <- logstash (as kafka consumer) -> elasticsearch data nodes. So based on this pipeline I have each Logstash consumer's output plugin sending to an array of 24 elasticsearch data nodes. I believe logstash load balances in a round-robin fashion but will exhaust a data node until the data node returns several 429's and then moves onto another data node. Meaning the data flow will not be evenly distributed between all nodes in the array. Is this accurate?
Assuming this is accurate, I'm looking at adding HA Proxy where the Logstash consumers will send to the 1 or 2 haproxies and allowing HA Proxy to do the load balancing.
Has anyone tried this?