# Kibana is unable to start with all shards failed message

**URL:** <https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046>\
**Category:** Elasticsearch\
**Created:** [April 8, 2020, 3:51am UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046 "2020-04-08T03:51:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![brb44](https://avatars.discourse-cdn.com/v4/letter/b/c6cbf5/32.png) [@brb44](https://discuss.elastic.co/u/brb44)\
**Post date:** [April 8, 2020, 3:51am UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/1 "2020-04-08T03:51:31Z")

</div>

After encrypting elasticsearch and changing the ALB from HTTP to https for ES and Kibana. I'm getting below error. Also, metricbeat is also in a failed state.

`kibana: {"type":"error","@timestamp":"2020-04-08T03:33:47Z","tags":["fatal","root"],"pid":22235,"level":"fatal","error":{"message":"all shards failed: [search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/doc/_count\",\"query\":{},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"index-pattern\\\"}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"term\\\":{\\\"migrationVersion.index-pattern\\\":\\\"6.5.0\\\"}}}}]}}]}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickCallback (internal/process/next_tick.js:180:9)"},"message":"all shards failed: [search_phase_execution_exception] all shards failed"}`

Can someone please help me in getting this resolved.

---

<div class="post-metadata">

**Author:** ![joelgriffith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joelgriffith/32/53738_2.png) [@joelgriffith](https://discuss.elastic.co/u/joelgriffith)\
**Post date:** [April 8, 2020, 9:32pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/2 "2020-04-08T21:32:19Z")

</div>

@brb44 this appears to be more of an issue in your ElasticSearch cluster/nodes. Can you tell us more about your ElasticSearch setup?

---

<div class="post-metadata">

**Author:** ![brb44](https://avatars.discourse-cdn.com/v4/letter/b/c6cbf5/32.png) [@brb44](https://discuss.elastic.co/u/brb44)\
**Post date:** [April 8, 2020, 10:16pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/3 "2020-04-08T22:16:59Z")

</div>

@joelgriffith Thanks for the prompt response.  
Sure, Mine its a 3 node cluster and recently enabled xpack tls on all the nodes. version I'm using is 6.5.1 and license is basic. Below is the elasticsearch.yml.

cluster.name: elasticsearch-dev  
node.name: es-masterdata-dev  
path.data: "/data/elasticsearch"  
path.logs: "/var/log/elasticsearch"  
node.master: true  
node.data: true  
network.host: _ec2:privateIpv4_  
discovery.ec2.groups: sg-XXXXXXX  
discovery.ec2.availability\_zones: us-west-2a,us-west-2b  
discovery.ec2.protocol: http  
discovery.ec2.endpoint: [ec2.us-west-2.amazonaws.com](http://ec2.us-west-2.amazonaws.com)  
discovery.ec2.host\_type: private\_ip  
discovery.zen.hosts\_provider: ec2  
xpack.security.enabled: false  
xpack.license.self\_generated.type: basic  
xpack.security.transport.ssl.enabled: true  
xpack.security.transport.ssl.verification\_mode: full  
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elastic.key  
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elastic.crt  
xpack.security.transport.ssl.certificate\_authorities: /etc/elasticsearch/certs/elasticca.crt

---

<div class="post-metadata">

**Author:** ![brb44](https://avatars.discourse-cdn.com/v4/letter/b/c6cbf5/32.png) [@brb44](https://discuss.elastic.co/u/brb44)\
**Post date:** [April 9, 2020, 6:29pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/4 "2020-04-09T18:29:09Z")

</div>

Adding to the above message  
I'm getting this error message in elasticsearch logs  
`/.kibana/doc/config%3A6.5.1, params: {index=.kibana, id=config:6.5.1, type=doc} org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.5.1]: routing [null]] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:224) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:203) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:97) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:61) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:395) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:487) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.rest.action.document.RestGetAction.lambda$prepareRequest$0(RestGetAction.java:81) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:239) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:335) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:545) [transport-netty4-client-6.5.1.jar:6.5.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-client-6.5.1.jar:6.5.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.5.1.jar:6.5.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]`

---

<div class="post-metadata">

**Author:** ![joelgriffith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joelgriffith/32/53738_2.png) [@joelgriffith](https://discuss.elastic.co/u/joelgriffith)\
**Post date:** [April 9, 2020, 7:03pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/5 "2020-04-09T19:03:53Z")

</div>

It's possible that Kibana shards are not allocated, and to quickly check you can do:

`curl -XGET <host>:<port>/_cat/shards?v`

Once done, you can use something like the explain API to help detail what's going one [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 9, 2020, 7:15pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/6 "2020-04-09T19:15:45Z")

</div>

In order to use security with a Basic license you need to be on Elasticsearch 6.8 or higher. On 6.5.1 security still requires a commercial license. I would recommend you upgrade to the latest 6.8 version.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2020, 7:15pm UTC](https://discuss.elastic.co/t/kibana-is-unable-to-start-with-all-shards-failed-message/227046/7 "2020-05-07T19:15:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.