Elasticsearch 5.6.5 upgrade to 6.4.0 Node will not start “failed to read local state”

Elastic Stack Elasticsearch
1

Hi,

I tried upgrading my Elasticsearch 5.6.5 to 6.4.0 on Centos 7 using the instructions describe here but everytime I start the service it fails with this exception:

...
[2018-08-24T10:16:42,096][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic-node1] parsed [0] roles from file [/opt/elasticsearch-6.4.0/config/roles.yml]
[2018-08-24T10:16:42,937][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/31701] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-08-24T10:16:43,503][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-08-24T10:16:44,307][ERROR][o.e.g.GatewayMetaState   ] [elastic-node1] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:145, file:/var/elastic/data/nodes/0/_state/global-145.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:56) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.node.Node.<init>(Node.java:447) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.node.Node.<init>(Node.java:256) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.0.jar:6.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.io.IOException: failed to read [id:145, file:/var/elastic/data/nodes/0/_state/global-145.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
        ... 15 more
Caused by: java.lang.IllegalArgumentException: Index patterns must not be null or empty; got null
        at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:98) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:385) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:520) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1215) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1262) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1253) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.4.0.jar:6.4.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.4.0.jar:6.4.0]
        ... 15 more
  ...

I've seen this same problem in another topic (Upgrade from 5.6 to 6.2.1: Failed to read local state), but it's unsolved.

Any idea how to solve it?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.