# All shards failed for phase: \[query\] on Elasticsearch 7.2.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [August 1, 2019, 7:46am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270 "2019-08-01T07:46:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Extinguo](https://avatars.discourse-cdn.com/v4/letter/e/5e9695/32.png) [@Extinguo](https://discuss.elastic.co/u/Extinguo)\ **Post date:** [August 1, 2019, 7:46am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/1 "2019-08-01T07:46:07Z")
Hey guys, I'm running Elastic 7.2.0 on a single machine in docker containers. It works fine except for some problems with the shards. I let Elastic run for a few days and looked at the logs today. The look fine, except for one thing: Elasticsearch threw a lot of "All shards failed"-Exceptions at some points. Please find the stacktrace below: ``` {"type": "server", "timestamp": "2019-07-30T15:03:57,209+0200", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "A-Elastic-Stack", "node.name": "es01", "cluster.uuid": "qxxxxxxxxxxxxxx", "node.id": "0xxxxxxxxxxx", "message": "All shards failed for phase: [query]" } {"type": "server", "timestamp": "2019-07-30T15:03:57,213+0200", "level": "WARN", "component": "r.suppressed", "cluster.name": "A-Elastic-Stack", "node.name": "es01", "cluster.uuid": "qxxxxxxxxxxxx", "node.id": "0xxxxxxxxxxxxx", "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:835) [?:?]"] } ``` The results of `GET /_cluster/health/?level=shards` look fine to me. Please find a snippet of the result below: ``` "cluster_name" : "A-Elastic-Stack", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 37, "active_shards" : 37, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0, "indices" : { ".monitoring-logstash-7-2019.07.28" : { "status" : "green", "number_of_shards" : 1, "number_of_replicas" : 0, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "shards" : { "0" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } } }, ... ".kibana_task_manager" : { "status" : "green", "number_of_shards" : 1, "number_of_replicas" : 0, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "shards" : { "0" : { "status" : "green", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0 } } }, ``` All of the shards are green. I also noticed that Elasticsearch throws similar errors during start up occasionally. However, during the startup it differs between "are shards failed" ".security-7 failed" and no exceptions whatsoever. Can somebody give me a hint what to look at? Could it be relationed to the setting auto\_expand\_replicas, which is set to "0-1"? Is ES trying to create a replica but failing to do so since there are no other instances of ES up and running? Thanks in advance! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 1, 2019, 8:08am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/2 "2019-08-01T08:08:45Z")
Were there any other log messages around the same times? I might expect to see this while a node is starting up, or maybe shutting down, but not otherwise in a one-node cluster. > [@Extinguo](#): > > Is ES trying to create a replica but failing to do so since there are no other instances of ES up and running? No, if Elasticsearch has only ever seen a single node then it won't have been trying to create a replica. ---
**Author:** ![Extinguo](https://avatars.discourse-cdn.com/v4/letter/e/5e9695/32.png) [@Extinguo](https://discuss.elastic.co/u/Extinguo)\ **Post date:** [August 1, 2019, 9:12am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/3 "2019-08-01T09:12:00Z")
Hi David, thanks for your answer! Turns out I misread the datestamp. You are correct, the cluster was rebootet at the time. Do you know why the shards fail while starting up? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 1, 2019, 10:53am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/4 "2019-08-01T10:53:40Z")
The log message is indicating that a search has failed to search any of the shards it tried. The shards themselves are still starting up at this point, which can take some time. Arguably the logs are being overly dramatic here, there's no need for such a noisy warning in this case. I opened [an issue to discuss this further](https://github.com/elastic/elasticsearch/issues/45082). ---
**Author:** ![Extinguo](https://avatars.discourse-cdn.com/v4/letter/e/5e9695/32.png) [@Extinguo](https://discuss.elastic.co/u/Extinguo)\ **Post date:** [August 1, 2019, 11:23am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/5 "2019-08-01T11:23:47Z")
Ok great! Thank you for the clarification. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 29, 2019, 11:23am UTC](https://discuss.elastic.co/t/all-shards-failed-for-phase-query-on-elasticsearch-7-2-0/193270/6 "2019-08-29T11:23:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.