Hi,
I have seen several other threads talking about the 429 errors from the elasticsearch output, but I'm not finding any kind of solution. So I'm starting a new thread.
My Environment:
2 Kibana Servers
4 Dedicated masters
6 Ingest/Logstash nodes, logstash 2GB, Ingest ES 8GB heap, 16 GB RAM
9 Data nodes 16gb heap 32GB RAM
I'm indexing about 20-25K/s
The logstash servers send events to their local elasticsearch instance running in ingest only mode.
Each logstash server is reporting these all of the time,
[2017-01-30T20:03:22,323][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$6@653b75 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2ee65694[Running, pool size = 4, active threads = 4, queued tasks = 50, completed tasks = 8057722]]"})
Looking at the metrics on the Elasticsearch instance on the Logstash server the JVM isn't busy and the load on the server is low.
Can this queue be increased from 50? I have seen in other threads that it is not recommended.
Messages are getting through, but I don't like to see these errors and I think something needs to be corrected.
Any help would be appreciated.
Thanks,
Tim