# Kibana Pod going into CrashLoopBackOff

**URL:** <https://discuss.elastic.co/t/kibana-pod-going-into-crashloopbackoff/228214>\
**Category:** Kibana\
**Created:** [April 15, 2020, 10:03pm UTC](https://discuss.elastic.co/t/kibana-pod-going-into-crashloopbackoff/228214 "2020-04-15T22:03:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ntsh999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ntsh999/32/51172_2.png) [@ntsh999](https://discuss.elastic.co/u/ntsh999)\
**Post date:** [April 15, 2020, 10:03pm UTC](https://discuss.elastic.co/t/kibana-pod-going-into-crashloopbackoff/228214/1 "2020-04-15T22:03:08Z")

</div>

I am running Elastic stack v7.2 on Kubernetes, my ES cluster is in RED state. As a reason my Kibana application never comes up as the Kibana pod is down with the error CrashLoopBackOff. Kibana pod shows the following error log :

```auto
{"type":"log","@timestamp":"2020-04-15T21:50:28Z","tags":["fatal","root"],"pid":1,"message":"{ Error: [search_phase_execution_exception] all shards failed\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:274:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 503,\n displayName: 'ServiceUnavailable',\n message: '[search_phase_execution_exception] all shards failed',\n path: '/.kibana/_count',\n query: {},\n body:\n { error:\n { root_cause: [],\n type: 'search_phase_execution_exception',\n reason: 'all shards failed',\n phase: 'query',\n grouped: true,\n failed_shards: [] },\n status: 503 },\n statusCode: 503,\n response:\n '{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}',\n toString: [Function],\n toJSON: [Function] }"}

 FATAL [search_phase_execution_exception] all shards failed :: {"path":"/.kibana/_count","query":{},"body":"{\"query\":{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"exists\":{\"field\":\"index-pattern\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.index-pattern\":\"6.5.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"visualization\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.visualization\":\"7.2.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"dashboard\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.dashboard\":\"7.0.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"search\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.search\":\"7.0.0\"}}}}]}}]}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}"}

```

As I am running this on Kubernetes which adds another layer of abstraction how am I supposed to troubleshoot this without kibana?

Below are the last log from the data node-

```auto
{"type": "server", "timestamp": "2020-04-15T21:50:28,854+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "es-eic-logs", "node.name": "elasticsearch-data-0", "cluster.uuid": "uAR2d-NXS6i95ZCxxA6m1A", "node.id": "v38AMmu3TSaZhm3JbzE-Jg", "message": "All shards failed for phase: [query]" }
{"type": "server", "timestamp": "2020-04-15T21:50:28,854+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "es-eic-logs", "node.name": "elasticsearch-data-0", "cluster.uuid": "uAR2d-NXS6i95ZCxxA6m1A", "node.id": "v38AMmu3TSaZhm3JbzE-Jg", "message": "path: /.kibana/_count, params: {index=.kibana}" ,
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }

```

Right now I am not indexing any new data into the ES Cluster, I just want it to be in GREEN so that I can start indexing again. Any pointers on how troubleshoot and fix this issue?

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [April 16, 2020, 12:50pm UTC](https://discuss.elastic.co/t/kibana-pod-going-into-crashloopbackoff/228214/2 "2020-04-16T12:50:03Z")

</div>

Kibana won't work if your ES cluster is in RED state. If you solve the problems with your ES cluster, Kibana should just work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2020, 12:50pm UTC](https://discuss.elastic.co/t/kibana-pod-going-into-crashloopbackoff/228214/3 "2020-05-14T12:50:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.