Hi everyone,
I have a cluster with 3 dedicated masters and 17 data nodes. Several Logstash servers are sending events to the Elastic cluster using the load balancing provided by Logstash.
I see this error in some elastic nodes:
[2017-11-07T15:01:41,740][ERROR][o.e.a.b.TransportBulkAction] [node08] failed to execute pipeline for a bulk request
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.ingest.PipelineExecutionService$2@55d30f9a on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2e2300f0[Running, pool size = 8, active threads = 8, queued tasks = 231, completed tasks = 17927703]]
It looks like that elastic node is over its capacity, but the CPU, RAM and disk usage are not at 100%. Also, almost all nodes are under the 10% of CPU. Why I have queued tasks if there are available resources?.
What is happening?.
Thanks.