[27]: index [.marvel-es-1-2017.01.05], type [node], id [AVlvCMR8jzJUIDxT5zD3], message [RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@1d2e5bb4 on EsThreadPoolExecutor[bulk, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6eff5538[Running, pool size = 8, active threads = 8, queued tasks = 501, completed tasks = 7115945]]];]
[29]: index [.marvel-es-1-2017.01.05], type [node], id [AVlvCMR8jzJUIDxT5zD4], message [RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@1d2e5bb4 on EsThreadPoolExecutor[bulk, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6eff5538[Running, pool size = 8, active threads = 8, queued tasks = 501, completed tasks = 7115945]]];]
[31]: index [.marvel-es-1-2017.01.05], type [node], id [AVlvCMR8jzJUIDxT5zD5], message [RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[ElasticSearch-28][10.10.0.133:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@1d2e5bb4 on EsThreadPoolExecutor[bulk, queue capacity = 500, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6eff5538[Running, pool size = 8, active threads = 8, queued tasks = 501, completed tasks = 7115945]]];]
I'm not sure, maybe it indicates, that you overflowed writing queue while mass inserting?
EsThreadPoolExecutor[bulk, queue capacity = 500,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6eff5538[Running,
pool size = 8, active threads = 8, queued tasks = 501, completed tasks =
7115945]]];]
Thanks for the quick response. Is it possible to increase the queue size?
We have a 30 nodes cluster.
Thanks
Try these settings.
I suggest, you need the bulk one
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-threadpool.html
It's possible to increase the queue size but let's talk about something here.
Why is your queue filling up? Because your producers are adding work items to the queue faster than the consumers can take them off the queue.
What would happen if you increase the queue size? The producers will still be producing at the rate they are currently producing, and the consumers will still be consuming at the rate they are currently consuming. This means that the queue will still fill up. Except now, the queue is bigger which will put more memory pressure on the system. In fact, this means the consumers might slow down, which might exacerbate the production rate greater than consumption rate problem. From there, a vicious feedback loop will form.
Queues are helpful for handling variable load. They are not helpful for handing scenarios where the production rate exceeds the consumption rate. In this scenario, a queue filling up is a form of back pressure. It is telling you that you need to slow down the rate of production (or speed up the rate of consumption).
Only increase the queue size if you are in a situation where you have variable load that sometimes exceeds capacity. Do not increase the queue size if you are in a situation where production rate exceeds consumption rate.
I am little confused with this definition:
bulk
For bulk operations. Thread pool type is fixed with a size of # of available processors, queue_size of 50.
I have 26 data nodes and 3 masters instance type: r3.2xlarge with 8 cores / each
What do they mean by " Thread pool type is fixed with a size of # of available processors"?
A fixed thread pool means it always has N (= number of CPU cores here) threads to execute tasks it pulls from the queue. Since each of your nodes has 8 cores, this means 8 threads per node for indexing ops.
The queue handles the back log, holding onto new indexing requests until an indexing thread takes it.
I agree with @jasontedor: you should not just blindly increase the queue size. You should understand which of the two possible root causes explain your rejected execution exceptions.
Mike McCandless
Thank you all
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.