I performed a rolling upgrade in a cluster of 3 nodes, moving from version 5.6.5 to 6.7.0 and ended up seeing the following exception in log file,
[2019-04-03T12:04:06,926][ERROR][o.e.b.Bootstrap ] [node-1] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.0.jar:6.7.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.7.0.jar:6.7.0]
... 15 more
Caused by: java.lang.IllegalStateException: only value lists are allowed in serialized settings
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:692) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:650) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:640) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:443) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1253) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.7.0.jar:6.7.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.7.0.jar:6.7.0]
... 15 more
Here's the output of ls -l /var/lib/elasticsearch/nodes/0/_state/
,
-rw-r--r--. 1 elasticsearch elasticsearch 67089 Apr 3 12:05 global-1.st
-rw-r--r--. 1 elasticsearch elasticsearch 71 Apr 3 12:04 node-0.st
Here's what I have in elasticsearch.yml
,
cluster.name: search
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host:
- 172.16.90.94
- _local_
discovery.zen.ping.unicast.hosts:
- 172.16.90.95
I wasn't using any subscription earlier, no ml jobs, etc.
If I clear out the /var/lib/elasticsearch/nodes/
path then node comes back without complaining,
sudo rm -rfv /var/lib/elasticsearch/nodes/*
Any help will be appreciated