How do I increase bulk index rate/throughput to my cluster? Elasticsearch keeps throttling back sending 429 errors seen from Logstash debug logs. What settings do I need to tweak in either Elasticsearch or logstash to get better throughput performance. I am using 2 beefy Logstash nodes reading from file archived EVTX Windows Event Logs in JSON files, and archived BRO Logs (text). I also have filebeats sending bro logs from 2 live sensors feeding the cluster. The best I have been able to get is 23,000 Primary shards index per second. I currently have 5 shards with 1 replica per index and i am indexing by day based off event time (logs).
I have have tried a lot of things. I tweaked logstash configs and elasticsearch settings in the .yml and no change. the only positive change i got was when i added the following under the output elasticsearch module:
flush_size => 10000
pool_max => 5000
pool_max_per_route => 2500
timeout => 120
Here are some details about my cluster:
Elastic Stack 5.0; 15 Nodes; 12 DN; 2 Coordinate Nodes; 1 master (3 Eligible Masters)
Hardware: on average each data node has 50 GB RAM with 3-8 TB of SSD and 11-16 CPU cores each. I have 2 logstash Nodes that have 16 CPU Cores, 64 GB of RAM and 800 TB SSD.