I have a 3 node Elasticsearch cluster, 2 Logstash nodes and about 100 filebeats sending data to Logstash. Every piece is 7.17
Both Logstash nodes have the exact same configuration. There is a 16 GB persistent queue on each node.
All Filebeats send data to both Logstash nodes using their internal Load Balancer (I mean all the filebeat.yml config files have this: loadbalance: true)
Both Logstash nodes send data to all three Elasticsearch nodes, I mean everything is balanced
When too many events arrive to Elasticsearch from Logstash (more than around 10 K events / sec), persistent queues are there to fill,
however only the node 1 persistent queue fills
Is this the expected behaviour? We upgraded from ELK 7.5 to 7.17, when we had 7.5 both queues filled.
Both Logstash send events to Elasticsearch, and when events arrive slower, persistent queue drains
What could be wrong in my ELK Stack?