# ALL SHARD FAILED - SEARCH PHASE EXECUTION EXCEPTION **URL:** \ **Category:** Kibana\ **Created:** [April 2, 2019, 7:55am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933 "2019-04-02T07:55:29Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![elborni96](https://avatars.discourse-cdn.com/v4/letter/e/b19c9b/32.png) [@elborni96](https://discuss.elastic.co/u/elborni96)\ **Post date:** [April 2, 2019, 7:55am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/1 "2019-04-02T07:55:29Z")
Hi, when I try to connect to Kibana interface the following message is displayed: **{"message":"all shards failed: [search\_phase\_execution\_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}** How can I resolve this? Thanks, Mirko Bornivelli ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 8:14am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/2 "2019-04-02T08:14:28Z")
Is there anything in the Elasticsearch logs? ---
**Author:** ![elborni96](https://avatars.discourse-cdn.com/v4/letter/e/b19c9b/32.png) [@elborni96](https://discuss.elastic.co/u/elborni96)\ **Post date:** [April 2, 2019, 8:28am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/3 "2019-04-02T08:28:44Z")
These are latest ,message from **/var/log/elasticsearch/elasticsearch.log** [2019-04-02T10:21:14,933][DEBUG][o.e.a.s.TransportSearchAction] [a1VVZj2] All shards failed for phase: [query] [2019-04-02T10:21:14,933][WARN][r.suppressed] [a1VVZj2] path: /.kibana/\_search, params: {rest\_total\_hits\_as\_int=true, size=20, index=.kibana, from=0} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.1.jar:6.6.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0\_201] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0\_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_201] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 8:30am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/4 "2019-04-02T08:30:02Z")
What is the output of the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)? ---
**Author:** ![elborni96](https://avatars.discourse-cdn.com/v4/letter/e/b19c9b/32.png) [@elborni96](https://discuss.elastic.co/u/elborni96)\ **Post date:** [April 2, 2019, 8:35am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/5 "2019-04-02T08:35:11Z")
This is the output : { "cluster\_name" : "elasticsearch", "status" : "red", "timed\_out" : true, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 0, "active\_shards" : 0, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 100.0 } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 8:53am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/6 "2019-04-02T08:53:15Z")
What happened to get your Elasticsearch cluster into this state? Where is your Elasticsearch node deployed? ---
**Author:** ![elborni96](https://avatars.discourse-cdn.com/v4/letter/e/b19c9b/32.png) [@elborni96](https://discuss.elastic.co/u/elborni96)\ **Post date:** [April 2, 2019, 9:48am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/7 "2019-04-02T09:48:14Z")
I have a **test** server that contain the following instances: - Elastic Search - Kibana - Logstash - FIlebeat Using Filebeat I go to monitor specific file; Logstash parse the events and send the output to ElasticSearch. When I tried to replicate the ELK stack configuration on the **production** server the following error appear and I can't execute search on Kibana: **{"message":"all shards failed: [search\_phase\_execution\_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}** I'm a new user of ELK Stack and i don't have understand what " **shards**" are. How can I solve this problem? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 2, 2019, 10:06am UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/8 "2019-04-02T10:06:46Z")
How did you install the components? What kind of hardware are the components running on? How did you replicate the configuration to production? ---
**Author:** ![elborni96](https://avatars.discourse-cdn.com/v4/letter/e/b19c9b/32.png) [@elborni96](https://discuss.elastic.co/u/elborni96)\ **Post date:** [April 2, 2019, 12:22pm UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/9 "2019-04-02T12:22:51Z")
I have tested my configurations on test environment (CentOS virtual machine), then i have replicate the configurations on the production server ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 30, 2019, 12:28pm UTC](https://discuss.elastic.co/t/all-shard-failed-search-phase-execution-exception/174933/10 "2019-04-30T12:28:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.