ALL SHARD FAILED - SEARCH PHASE EXECUTION EXCEPTION

Elastic Stack Kibana
1

Hi,

when I try to connect to Kibana interface the following message is displayed:

{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}

How can I resolve this?

Thanks,
Mirko Bornivelli

2

Is there anything in the Elasticsearch logs?

3

These are latest ,message from /var/log/elasticsearch/elasticsearch.log

[2019-04-02T10:21:14,933][DEBUG][o.e.a.s.TransportSearchAction] [a1VVZj2] All shards failed for phase: [query]
[2019-04-02T10:21:14,933][WARN ][r.suppressed ] [a1VVZj2] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=20, index=.kibana, from=0}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:221) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:175) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:221) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:462) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:368) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:362) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

4

What is the output of the cluster health API?

5

This is the output :

{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : true,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

6

What happened to get your Elasticsearch cluster into this state? Where is your Elasticsearch node deployed?

7

I have a test server that contain the following instances:

  • Elastic Search
  • Kibana
  • Logstash
  • FIlebeat

Using Filebeat I go to monitor specific file; Logstash parse the events and send the output to ElasticSearch.

When I tried to replicate the ELK stack configuration on the production
server the following error appear and I can't execute search on Kibana:

{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}

I'm a new user of ELK Stack and i don't have understand what "shards" are.

How can I solve this problem?

8

How did you install the components? What kind of hardware are the components running on? How did you replicate the configuration to production?

9

I have tested my configurations on test environment (CentOS virtual machine), then i have replicate the configurations on the production server

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.