We had 3 node ES 6.4.0 cluster with Xpack security enabled. The license had just expired and we had temporarily shutdown the cluster for a planned host migration (hosts were physically moved to another location, no change in IP or FS). Now that the migration is complete and we want to bring up the cluster to upgrade the license, the ES process does not start. I see the below exception on all the 3 hosts. I tried checking permissions and file existence, both seem alright. Also, tried disabling XPack security and starting the process - that didn't work too.
Please let me know what might I be missing here.
[2019-01-31T19:03:00,449][ERROR][o.e.g.GatewayMetaState ] [lcb11.com] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:8, file:/home/appuser/elastic/es_on_9201/data/nodes/0/indices/L2uMUBH1SlKJAbM98RY8fg/_state/state-8.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:199) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:89) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.IndexFolderUpgrader.upgradeIndicesIfNeeded(IndexFolderUpgrader.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:87) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:447) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.node.Node.<init>(Node.java:256) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.io.IOException: failed to read [id:8, file:/home/appuser/elastic/es_on_9201/data/nodes/0/indices/L2uMUBH1SlKJAbM98RY8fg/_state/state-8.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:1521] [rollover_info] failed to parse field [met_conditions]
at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:316) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
...
... 15 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:1521] [met_conditions] failed to parse field [max_age]
at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$3(ObjectParser.java:244) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
..
... 15 more
Caused by: org.elasticsearch.common.xcontent.NamedObjectNotFoundException: named objects are not supported for this parser
at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:126) ~[elasticsearch-x-content-6.4.0.jar:6.4.0]
..
... 15 more