Kibana Pod going into CrashLoopBackOff

I am running Elastic stack v7.2 on Kubernetes, my ES cluster is in RED state. As a reason my Kibana application never comes up as the Kibana pod is down with the error CrashLoopBackOff. Kibana pod shows the following error log :

{"type":"log","@timestamp":"2020-04-15T21:50:28Z","tags":["fatal","root"],"pid":1,"message":"{ Error: [search_phase_execution_exception] all shards failed\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:274:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 503,\n  displayName: 'ServiceUnavailable',\n  message: '[search_phase_execution_exception] all shards failed',\n  path: '/.kibana/_count',\n  query: {},\n  body:\n   { error:\n      { root_cause: [],\n        type: 'search_phase_execution_exception',\n        reason: 'all shards failed',\n        phase: 'query',\n        grouped: true,\n        failed_shards: [] },\n     status: 503 },\n  statusCode: 503,\n  response:\n   '{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}',\n  toString: [Function],\n  toJSON: [Function] }"}

 FATAL  [search_phase_execution_exception] all shards failed :: {"path":"/.kibana/_count","query":{},"body":"{\"query\":{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"exists\":{\"field\":\"index-pattern\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.index-pattern\":\"6.5.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"visualization\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.visualization\":\"7.2.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"dashboard\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.dashboard\":\"7.0.0\"}}}}]}},{\"bool\":{\"must\":[{\"exists\":{\"field\":\"search\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.search\":\"7.0.0\"}}}}]}}]}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}"}

As I am running this on Kubernetes which adds another layer of abstraction how am I supposed to troubleshoot this without kibana?

Below are the last log from the data node-

{"type": "server", "timestamp": "2020-04-15T21:50:28,854+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "es-eic-logs", "node.name": "elasticsearch-data-0", "cluster.uuid": "uAR2d-NXS6i95ZCxxA6m1A", "node.id": "v38AMmu3TSaZhm3JbzE-Jg",  "message": "All shards failed for phase: [query]"  }
{"type": "server", "timestamp": "2020-04-15T21:50:28,854+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "es-eic-logs", "node.name": "elasticsearch-data-0", "cluster.uuid": "uAR2d-NXS6i95ZCxxA6m1A", "node.id": "v38AMmu3TSaZhm3JbzE-Jg",  "message": "path: /.kibana/_count, params: {index=.kibana}" ,
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }

Right now I am not indexing any new data into the ES Cluster, I just want it to be in GREEN so that I can start indexing again. Any pointers on how troubleshoot and fix this issue?

Kibana won't work if your ES cluster is in RED state. If you solve the problems with your ES cluster, Kibana should just work.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.