# X of x shards failed **URL:** \ **Category:** Elasticsearch\ **Created:** [June 14, 2019, 1:41pm UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863 "2019-06-14T13:41:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Post date:** [June 14, 2019, 1:41pm UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/1 "2019-06-14T13:41:29Z")
Hi I have a single node setup. It looks like each index has 5 primary shards and when loading data on the kibana dashboard, we keep getting x of x shards failed error messages. Can anyone advise how to solve this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 14, 2019, 10:22pm UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/2 "2019-06-14T22:22:57Z")
What do your Elasticsearch logs show? ---
**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Post date:** [June 19, 2019, 8:20am UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/3 "2019-06-19T08:20:15Z")
I read somewhere I probably shouldn't have 5 shards on a single node deployment. would adding more ES nodes solve the issue? The whole log message is too long to post here but here is part of it - ```nohighlight [2019-06-19T08:18:08,250][DEBUG][o.e.a.s.TransportSearchAction] [nre3ddu] [index-2019.06.18][3], node[nre3dduXSIeBsIU67w9kQQ], [P], s[STARTED], a[id=GbCtqh0MRfG1r2gzwWhzkA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[index-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='1560932209094', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=9, allowPartialSearchResults=true, source={"size":0,"query":{"bool":{"must":[{"match_all":{"boost":1.0}},{"match_all":{"boost":1.0}},{"range":{"@timestamp":{"from":1560845810780,"to":1560932210780,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"uri-path":{"query":"releases","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"bool":{"should":[{"match_phrase":{"uri-type":{"query":"sdk","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"uri-type":{"query":"client","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"uri-type":{"query":"server","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}},{"bool":{"should":[{"match_phrase":{"cs-uri-stem":{"query":"zip","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"cs-uri-stem":{"query":"tar.bz2","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"cs-uri-stem":{"query":"run","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"cs-uri-stem":{"query":"dmg","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"cs-uri-stem":{"query":"unitypackage","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"cs-uri-stem":{"query":"exe","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"must_not":[{"match_phrase":{"sc-status":{"query":403,"slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":[{"field":"@timestamp","format":"date_time"},{"field":"date","format":"date_time"}],"script_fields":{},"aggregations":{"2":{"terms":{"field":"geoip.country_name.keyword","size":20,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}] lastShard [true] org.elasticsearch.transport.RemoteTransportException: [nre3ddu][127.0.0.1:9300][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@55384cfb on QueueResizingEsThreadPoolExecutor[name = nre3ddu/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 405nanos, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@a8117c8[Running, pool size = 13, active threads = 13, queued tasks = 1029, completed tasks = 24350639]] at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.5.1.jar:6.5.1] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_191] at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:98) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:86) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:93) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$4(SearchService.java:1074) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) ~[elasticsearch-6.5.1.jar:6.5.1] ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2019, 9:28am UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/4 "2019-06-19T09:28:59Z")
I reformatted your message using the `` button because it was unreadable as posted. It's a good idea to format messages well, as they're much more likely to get a useful response that way. I see this: > [@jamesp220291](#): > > EsRejectedExecutionException ... queue capacity = 1000 ... queued tasks = 1029 This means that the node is overloaded, having over 1000 tasks in its search queue, and so it is rejecting further searches. ---
**Author:** ![jamesp220291](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jamesp220291](https://discuss.elastic.co/u/jamesp220291)\ **Post date:** [June 19, 2019, 9:48am UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/5 "2019-06-19T09:48:10Z")
Thanks. Is it possible to increase this? I have read this and it isn't clear - [How i can increase thread pool queue size in Elasticsearch 5.6](https://discuss.elastic.co/t/how-i-can-increase-thread-pool-queue-size-in-elasticsearch-5-6/145661) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 19, 2019, 10:22am UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/6 "2019-06-19T10:22:38Z")
A longer queue is normally not the solution. If the node can't keep up with the workload then it'll overflow the queue no matter how long you make it. I'd suggest either reducing the workload or else to scale out to multiple nodes. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2019, 10:22am UTC](https://discuss.elastic.co/t/x-of-x-shards-failed/185863/7 "2019-07-17T10:22:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.