# After upgrading from Elasticsearch 6.5 to 6.6 - All shards failed

**URL:** <https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366>\
**Category:** Elasticsearch\
**Created:** [February 6, 2019, 10:56pm UTC](https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366 "2019-02-06T22:56:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rafael.gumiero](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@rafael.gumiero](https://discuss.elastic.co/u/rafael.gumiero)\
**Post date:** [February 6, 2019, 10:56pm UTC](https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366/1 "2019-02-06T22:56:18Z")

</div>

Hi Team,

I have performed update via repository and the following error is occurring in Elasticsearch:

Environment:

# uname -r

3.10.0-957.5.1.el7.x86\_64

# cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.6 (Maipo)

# rpm -qa |egrep 'elastic|beat|kibana'

auditbeat-6.6.0-1.x86\_64  
packetbeat-6.6.0-1.x86\_64  
filebeat-6.6.0-1.x86\_64  
metricbeat-6.6.0-1.x86\_64  
journalbeat-6.6.0-1.x86\_64  
kibana-6.6.0-1.x86\_64  
elasticsearch-6.6.0-1.noarch  
heartbeat-elastic-6.6.0-1.x86\_64

[2019-02-06T22:32:30,099][INFO][o.e.n.Node] [RV4BgCv] initialized  
[2019-02-06T22:32:30,099][INFO][o.e.n.Node] [RV4BgCv] starting ...  
[2019-02-06T22:32:30,255][INFO][o.e.t.TransportService] [RV4BgCv] publish\_address {172.31.32.144:9300}, bound\_addresses {172.31.32.144:9300}  
[2019-02-06T22:32:30,359][INFO][o.e.b.BootstrapChecks] [RV4BgCv] bound or publishing to a non-loopback address, enforcing bootstrap checks  
[2019-02-06T22:32:33,427][INFO][o.e.c.s.MasterService] [RV4BgCv] zen-disco-elected-as-master ([0] nodes joined), reason: new\_master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine\_memory=3971969024, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}  
[2019-02-06T22:32:33,431][INFO][o.e.c.s.ClusterApplierService] [RV4BgCv] new\_master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine\_memory=3971969024, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine\_memory=3971969024, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])  
[2019-02-06T22:32:33,558][INFO][o.e.h.n.Netty4HttpServerTransport] [RV4BgCv] publish\_address {172.31.32.144:9200}, bound\_addresses {172.31.32.144:9200}  
[2019-02-06T22:32:33,558][INFO][o.e.n.Node] [RV4BgCv] started  
[2019-02-06T22:32:35,894][WARN][o.e.x.s.a.s.m.NativeRoleMappingStore] [RV4BgCv] Failed to clear cache for realms []  
[2019-02-06T22:32:35,929][INFO][o.e.l.LicenseService] [RV4BgCv] license [133eabd5-a347-4bc4-8f84-dade53a9cf99] mode [basic] - valid  
[2019-02-06T22:32:35,937][INFO][o.e.g.GatewayService] [RV4BgCv] recovered [30] indices into cluster\_state  
[2019-02-06T22:32:36,305][WARN][r.suppressed] [RV4BgCv] path: /.kibana/doc/kql-telemetry%3Akql-telemetry, params: {index=.kibana, id=kql-telemetry:kql-telemetry, type=doc}  
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:230) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:217) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1200(TransportSingleShardAction.java:143) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:273) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.onFailure(TransportSingleShardAction.java:110) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:744) [elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.6.0.jar:6.6.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]  
at java.lang.Thread.run(Thread.java:834) [?:?]  
Caused by: org.elasticsearch.transport.RemoteTransportException: [RV4BgCv][172.31.32.144:9300][indices:data/read/get[s]]  
Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST\_RECOVERY, STARTED]  
at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1550) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:911) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:169) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:90) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:89) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:115) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.0.jar:6.6.0]  
... 3 more

---

<div class="post-metadata">

**Author:** ![Fr3dY](https://avatars.discourse-cdn.com/v4/letter/f/cdc98d/32.png) [@Fr3dY](https://discuss.elastic.co/u/Fr3dY)\
**Post date:** [February 12, 2019, 9:36am UTC](https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366/2 "2019-02-12T09:36:30Z")

</div>

Same issue here after upgrading from 6.5 to 6.6, using CentOS Linux release 7.6.1810 (Core) - 4.4.174-1.el7.elrepo.x86\_64.

Regards,

---

<div class="post-metadata">

**Author:** ![Fr3dY](https://avatars.discourse-cdn.com/v4/letter/f/cdc98d/32.png) [@Fr3dY](https://discuss.elastic.co/u/Fr3dY)\
**Post date:** [February 25, 2019, 8:27am UTC](https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366/3 "2019-02-25T08:27:59Z")

</div>

Update: Upgraded to 6.6.1 but problem persists ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2019, 8:27am UTC](https://discuss.elastic.co/t/after-upgrading-from-elasticsearch-6-5-to-6-6-all-shards-failed/167366/4 "2019-03-25T08:27:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
