After upgrading from Elasticsearch 6.5 to 6.6 - All shards failed

Hi Team,

I have performed update via repository and the following error is occurring in Elasticsearch:

Environment:

uname -r

3.10.0-957.5.1.el7.x86_64

cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.6 (Maipo)

rpm -qa |egrep 'elastic|beat|kibana'

auditbeat-6.6.0-1.x86_64
packetbeat-6.6.0-1.x86_64
filebeat-6.6.0-1.x86_64
metricbeat-6.6.0-1.x86_64
journalbeat-6.6.0-1.x86_64
kibana-6.6.0-1.x86_64
elasticsearch-6.6.0-1.noarch
heartbeat-elastic-6.6.0-1.x86_64

[2019-02-06T22:32:30,099][INFO ][o.e.n.Node ] [RV4BgCv] initialized
[2019-02-06T22:32:30,099][INFO ][o.e.n.Node ] [RV4BgCv] starting ...
[2019-02-06T22:32:30,255][INFO ][o.e.t.TransportService ] [RV4BgCv] publish_address {172.31.32.144:9300}, bound_addresses {172.31.32.144:9300}
[2019-02-06T22:32:30,359][INFO ][o.e.b.BootstrapChecks ] [RV4BgCv] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-06T22:32:33,427][INFO ][o.e.c.s.MasterService ] [RV4BgCv] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine_memory=3971969024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-02-06T22:32:33,431][INFO ][o.e.c.s.ClusterApplierService] [RV4BgCv] new_master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine_memory=3971969024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {RV4BgCv}{RV4BgCvYTuy4OaQRv4ThFQ}{y51aDoCDQWujSxJPrPme4g}{172.31.32.144}{172.31.32.144:9300}{ml.machine_memory=3971969024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-02-06T22:32:33,558][INFO ][o.e.h.n.Netty4HttpServerTransport] [RV4BgCv] publish_address {172.31.32.144:9200}, bound_addresses {172.31.32.144:9200}
[2019-02-06T22:32:33,558][INFO ][o.e.n.Node ] [RV4BgCv] started
[2019-02-06T22:32:35,894][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [RV4BgCv] Failed to clear cache for realms []
[2019-02-06T22:32:35,929][INFO ][o.e.l.LicenseService ] [RV4BgCv] license [133eabd5-a347-4bc4-8f84-dade53a9cf99] mode [basic] - valid
[2019-02-06T22:32:35,937][INFO ][o.e.g.GatewayService ] [RV4BgCv] recovered [30] indices into cluster_state
[2019-02-06T22:32:36,305][WARN ][r.suppressed ] [RV4BgCv] path: /.kibana/doc/kql-telemetry%3Akql-telemetry, params: {index=.kibana, id=kql-telemetry:kql-telemetry, type=doc}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:230) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:217) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1200(TransportSingleShardAction.java:143) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:273) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1103) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1215) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1189) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.onFailure(TransportSingleShardAction.java:110) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:744) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.6.0.jar:6.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [RV4BgCv][172.31.32.144:9300][indices:data/read/get[s]]
Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]
at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1550) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:911) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:169) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:90) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:82) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:89) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:115) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.0.jar:6.6.0]
... 3 more

Same issue here after upgrading from 6.5 to 6.6, using CentOS Linux release 7.6.1810 (Core) - 4.4.174-1.el7.elrepo.x86_64.

Regards,

Update: Upgraded to 6.6.1 but problem persists :frowning:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.