Elastic search Q'ing issue

Elastic Stack Elasticsearch
1

I have a kafka consumer that reads from broker and send it to ElasticSearch.The QPS is pretty high.
My consumer is in Node.js and it asynchronously post to Elasticsearch for all events incoming.

Getting this error while trying to post to ES.

{"error":{"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@7a32e3a3 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@232df9ef[Running, pool size = 4, active threads = 4, queued tasks = 258, completed tasks = 12838]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@7a32e3a3 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@232df9ef[Running, pool size = 4, active threads = 4, queued tasks = 258, completed tasks = 12838]]"},"status":429} null 10.117.24.229

Googled the above and learn about Elasticsearch thread pool and Q size and tried changing configs.
Could you guide on what to do next ?

2

The best bet is to make sure you are using _bulk and if you are, add more nodes to your cluster, as it over overloaded.

3

Using bulk requests as mark suggests is a great idea to improve performance and get better throughput. Also read this blog post on the topic.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.