ES cluster stops working when enabling too many Logstash workers to send bulk requests to it

I've noticed that when I scale-out the number of Logstash apps I have in containers writing to same ES cluster it stops all activity (index & search)

But as soon as I go back to the reduced number of workers (from 20 to 8) the cluster works back again. The problem is that # of apps is not enough to cope with the load and I want to send more data to ES.

why is this happening ? I though it was because I didn't have enough ingest nodes (my Logstash apps are writing to 2 of them) but then I no longer think that because it is "bulk" thread pools in many data nodes (not ingest) who are completely loaded and rejecting requests.

appreciate any inputs,


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.