# Elasticsearch node fails after kibana startup .action.search.SearchPhaseExecutionException: all shards failed

**URL:** <https://discuss.elastic.co/t/elasticsearch-node-fails-after-kibana-startup-action-search-searchphaseexecutionexception-all-shards-failed/199658>\
**Category:** Elasticsearch\
**Created:** [September 16, 2019, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-node-fails-after-kibana-startup-action-search-searchphaseexecutionexception-all-shards-failed/199658 "2019-09-16T12:26:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![O\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/o_k/32/52424_2.png) [@O\_K](https://discuss.elastic.co/u/O_K)\
**Post date:** [September 16, 2019, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-node-fails-after-kibana-startup-action-search-searchphaseexecutionexception-all-shards-failed/199658/1 "2019-09-16T12:26:33Z")

</div>

I have 2 nodes

```auto
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 3,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

```

Once i start Kibana, I receive 2 different message from master nodes.

Just want to understand if this is as per design or something is wrong

```auto
{"type": "server", "timestamp": "2019-09-16T12:13:23,505+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "58CjS3W3R1m4FYwnPfPfOw", "message": "Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...])." }
^R

```

```auto
{"type": "server", "timestamp": "2019-09-16T12:09:57,555+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "g3MOxW7_TnenDyigGrWEyw", "message": "All shards failed for phase: [query]" }
{"type": "server", "timestamp": "2019-09-16T12:09:57,556+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "wbPF17P8SVKoOxbu9ESiWA", "node.id": "g3MOxW7_TnenDyigGrWEyw", "message": "path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager}" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",

```

After that I see elasticsearch node went down

```auto
k exec -ti elasticsearch-master-1 bash -n logging
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d2b1bf3399cb5c10a41215bf82c304d660e69ee6bef8811df4131ef0a614d8b8": OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

```

but pod looks like still running, very strange

```auto
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 45m
elasticsearch-master-1 1/1 Running 0 45m
kibana-788597cf8-ctqbb 1/1 Running 0 21m

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 14, 2019, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-node-fails-after-kibana-startup-action-search-searchphaseexecutionexception-all-shards-failed/199658/2 "2019-10-14T12:26:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.