# EsThreadPoolExecutor err

**URL:** <https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464>\
**Category:** Elasticsearch\
**Created:** [August 29, 2018, 4:50am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464 "2018-08-29T04:50:32Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [August 29, 2018, 4:50am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/1 "2018-08-29T04:50:32Z")

</div>

hi there!

this error i can't solve while using kibana

> rejected execution of org.elasticsearch.action.search.FetchSearchPhase$1@3f6d8e9e on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@71704db3[Running, pool size = 2, active threads = 2, queued tasks = 1000, completed tasks = 259214]]

this is `GET /_cat/thread_pool?v` result

> node\_name name active queue rejected  
> xVfqIa9 bulk 0 0 0  
> xVfqIa9 fetch\_shard\_started 0 0 0  
> xVfqIa9 fetch\_shard\_store 0 0 0  
> xVfqIa9 flush 0 0 0  
> xVfqIa9 force\_merge 0 0 0  
> xVfqIa9 generic 0 0 0  
> xVfqIa9 get 0 0 0  
> xVfqIa9 index 0 0 0  
> xVfqIa9 listener 0 0 0  
> xVfqIa9 management 1 0 0  
> xVfqIa9 refresh 0 0 0  
> xVfqIa9 search 0 0 106502  
> xVfqIa9 snapshot 0 0 0  
> xVfqIa9 warmer 0 0 0  
> jZURHTt bulk 0 0 0  
> jZURHTt fetch\_shard\_started 0 0 0  
> jZURHTt fetch\_shard\_store 0 0 0  
> jZURHTt flush 0 0 0  
> jZURHTt force\_merge 0 0 0  
> jZURHTt generic 0 0 0  
> jZURHTt get 0 0 0  
> jZURHTt index 0 0 0  
> jZURHTt listener 0 0 0  
> jZURHTt management 1 0 0  
> jZURHTt refresh 0 0 0  
> jZURHTt search 0 0 40538  
> jZURHTt snapshot 0 0 0  
> jZURHTt warmer 0 0 0  
> hmMrMMb bulk 0 0 0  
> hmMrMMb fetch\_shard\_started 0 0 0  
> hmMrMMb fetch\_shard\_store 0 0 0  
> hmMrMMb flush 0 0 0  
> hmMrMMb force\_merge 0 0 0  
> hmMrMMb generic 0 0 0  
> hmMrMMb get 0 0 0  
> hmMrMMb index 0 0 0  
> hmMrMMb listener 0 0 0  
> hmMrMMb management 1 0 0  
> hmMrMMb refresh 0 0 0  
> hmMrMMb search 0 0 213796  
> hmMrMMb snapshot 0 0 0  
> hmMrMMb warmer 0 0 0  
> xaGXJwM bulk 0 0 0  
> xaGXJwM fetch\_shard\_started 0 0 0  
> xaGXJwM fetch\_shard\_store 0 0 0  
> xaGXJwM flush 0 0 0  
> xaGXJwM force\_merge 0 0 0  
> xaGXJwM generic 0 0 0  
> xaGXJwM get 0 0 0  
> xaGXJwM index 0 0 0  
> xaGXJwM listener 0 0 0  
> xaGXJwM management 1 0 0  
> xaGXJwM refresh 0 0 0  
> xaGXJwM search 0 0 41379  
> xaGXJwM snapshot 0 0 0  
> xaGXJwM warmer 0 0 0

plz point out to me how to fix EsThreadPoolExecutor err.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2018, 5:10am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/2 "2018-08-29T05:10:30Z")

</div>

It means your cluster is overloaded and cannot process the requests that Kibana is sending.

You'd need to describe more about your cluster and the use case for us to provide further assistance.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 29, 2018, 5:40am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/3 "2018-08-29T05:40:24Z")

</div>

Can you please provide the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [August 29, 2018, 6:52am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/4 "2018-08-29T06:52:53Z")

</div>

`GET _cluster/stats?pretty`

> {  
> "\_nodes": {  
> "total": 9,  
> "successful": 9,  
> "failed": 0  
> },  
> "cluster\_name": "066789247273:cluster-name",  
> "timestamp": 1535525504791,  
> "status": "green",  
> "indices": {  
> "count": 194,  
> "shards": {  
> "total": 1932,  
> "primaries": 966,  
> "replication": 1,  
> "index": {  
> "shards": {  
> "min": 2,  
> "max": 10,  
> "avg": 9.958762886597938  
> },  
> "primaries": {  
> "min": 1,  
> "max": 5,  
> "avg": 4.979381443298969  
> },  
> "replication": {  
> "min": 1,  
> "max": 1,  
> "avg": 1  
> }  
> }  
> },  
> "docs": {  
> "count": 121074770,  
> "deleted": 4  
> },  
> "store": {  
> "size\_in\_bytes": 269101490152,  
> "throttle\_time\_in\_millis": 0  
> },  
> "fielddata": {  
> "memory\_size\_in\_bytes": 14201952,  
> "evictions": 0  
> },  
> "query\_cache": {  
> "memory\_size\_in\_bytes": 367089553,  
> "total\_count": 8813864,  
> "hit\_count": 4746820,  
> "miss\_count": 4067044,  
> "cache\_size": 205870,  
> "cache\_count": 205870,  
> "evictions": 0  
> },  
> "completion": {  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 21829,  
> "memory\_in\_bytes": 1265913086,  
> "terms\_memory\_in\_bytes": 1157292887,  
> "stored\_fields\_memory\_in\_bytes": 63663248,  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory\_in\_bytes": 3968,  
> "points\_memory\_in\_bytes": 2535659,  
> "doc\_values\_memory\_in\_bytes": 42417324,  
> "index\_writer\_memory\_in\_bytes": 0,  
> "version\_map\_memory\_in\_bytes": 0,  
> "fixed\_bit\_set\_memory\_in\_bytes": 0,  
> "max\_unsafe\_auto\_id\_timestamp": 1535445184828,  
> "file\_sizes": {}  
> }  
> },  
> "nodes": {  
> "count": {  
> "total": 9,  
> "data": 9,  
> "coordinating\_only": 0,  
> "master": 9,  
> "ingest": 9  
> },  
> "versions": [  
> "5.5.2"  
> ],  
> "os": {  
> "available\_processors": 9,  
> "allocated\_processors": 9,  
> "names": [  
> {  
> "count": 9  
> }  
> ],  
> "mem": {  
> "total\_in\_bytes": 35488604160,  
> "free\_in\_bytes": 1055694848,  
> "used\_in\_bytes": 34432909312,  
> "free\_percent": 3,  
> "used\_percent": 97  
> }  
> },  
> "process": {  
> "cpu": {  
> "percent": 484  
> },  
> "open\_file\_descriptors": {  
> "min": 829,  
> "max": 1812,  
> "avg": 1263  
> }  
> },  
> "jvm": {  
> "max\_uptime\_in\_millis": 86833492,  
> "mem": {  
> "heap\_used\_in\_bytes": 5919426048,  
> "heap\_max\_in\_bytes": 14417068032  
> },  
> "threads": 707  
> },  
> "fs": {  
> "total\_in\_bytes": 949996781568,  
> "free\_in\_bytes": 677929021440,  
> "available\_in\_bytes": 629459644416  
> },  
> "network\_types": {  
> "transport\_types": {  
> "netty4": 9  
> },  
> "http\_types": {  
> "filter-jetty": 9  
> }  
> }  
> }  
> }

`GET _cluster/settings?pretty`

> {  
> "persistent": {  
> "cluster": {  
> "routing": {  
> "allocation": {  
> "cluster\_concurrent\_rebalance": "2",  
> "node\_concurrent\_recoveries": "2",  
> "disk": {  
> "watermark": {  
> "low": "14.7gb",  
> "high": "9.8gb"  
> }  
> },  
> "node\_initial\_primaries\_recoveries": "4"  
> }  
> },  
> "blocks": {  
> "create\_index": "false"  
> }  
> },  
> "indices": {  
> "recovery": {  
> "max\_bytes\_per\_sec": "20mb"  
> }  
> }  
> },  
> "transient": {  
> "cluster": {  
> "routing": {  
> "allocation": {  
> "cluster\_concurrent\_rebalance": "2",  
> "node\_concurrent\_recoveries": "2",  
> "disk": {  
> "watermark": {  
> "low": "14.7gb",  
> "high": "9.8gb"  
> }  
> },  
> "exclude": {},  
> "node\_initial\_primaries\_recoveries": "4"  
> }  
> }  
> },  
> "indices": {  
> "recovery": {  
> "max\_bytes\_per\_sec": "20mb"  
> }  
> }  
> }  
> }

`GET _cluster/health`

> {  
> "cluster\_name": "066789247273:cluster-name",  
> "status": "green",  
> "timed\_out": false,  
> "number\_of\_nodes": 9,  
> "number\_of\_data\_nodes": 9,  
> "active\_primary\_shards": 966,  
> "active\_shards": 1932,  
> "relocating\_shards": 2,  
> "initializing\_shards": 0,  
> "unassigned\_shards": 0,  
> "delayed\_unassigned\_shards": 0,  
> "number\_of\_pending\_tasks": 1,  
> "number\_of\_in\_flight\_fetch": 0,  
> "task\_max\_waiting\_in\_queue\_millis": 0,  
> "active\_shards\_percent\_as\_number": 100  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 29, 2018, 7:53am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/5 "2018-08-29T07:53:29Z")

</div>

> [@shwesinhan](#):
>
> "store": {  
> "size\_in\_bytes": 269101490152,  
> "throttle\_time\_in\_millis": 0  
> },

> [@shwesinhan](#):
>
> "indices": {  
> "count": 194,  
> "shards": {  
> "total": 1932,  
> "primaries": 966,  
> "replication": 1,  
> "index": {  
> "shards": {  
> "min": 2,  
> "max": 10,  
> "avg": 9.958762886597938  
> },

Given the amount of data in the cluster, you seem to have a very large number of shards. Each visualisation will generate an aggregation and each shard this addresses will basically take up one search queue slot. You should therefore be able to get rid of this error by significantly reducing the number of shards in your cluster. Read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for guidance on shards and sharding.

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [September 5, 2018, 7:56am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/6 "2018-09-05T07:56:16Z")

</div>

To reduce shards, i've changed from daily-indices-format to monthly-indices-format.  
Then 1412 shards active after deleting two months indices doc.  
Others are I can't delete.  
So, now i have daily and month indices but i still have `EsThreadPolExecutor` err  
Could you advise me what kind of api should i use to reduce shards??  
And may I know which indices format is suitable for 3GB daily usage.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 5, 2018, 8:23am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/7 "2018-09-05T08:23:30Z")

</div>

You might be able to use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/indices-shrink-index.html) to reduce the number of primary shards per index. Another option is to [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html) the data you have in daily indices into monthly ones instead.

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [September 6, 2018, 4:00am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/8 "2018-09-06T04:00:54Z")

</div>

thank you @Christian_Dahlqvist

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [September 10, 2018, 4:19am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/9 "2018-09-10T04:19:24Z")

</div>

i tried re-index but i got `Request failed to get to the server (status code: 504)` msg  
how to solve this problem?  
my daily indices size are generally 2Gi  
i use aws es

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 10, 2018, 5:20am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/10 "2018-09-10T05:20:00Z")

</div>

I do not know why the reindex request failed. I believe AWS ES has some APIs disabled, so you may want to bring this issue up with them. Each index by default has 5 primary shard and also 5 replica shards, so it is looks like the average shards size in your cluster (based on stats provided before your delete) is around 130 MB, which is very small.

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [September 10, 2018, 8:09am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/11 "2018-09-10T08:09:48Z")

</div>

thank you for help @Christian_Dahlqvist  
even i got `Request failed to get to the server (status code: 504)` msg, aws es does background process. so i can re-index properly after i wait a few moment by monitoring task api

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 8, 2018, 8:09am UTC](https://discuss.elastic.co/t/esthreadpoolexecutor-err/146464/12 "2018-10-08T08:09:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.