"Failed to read or upgrade local state, exiting..." when starting ES instance

Hello,

I'm developing a kibana plugin, without my ES instance I'm completely dead. Yesterday ES instance would start normally, but today it doesn't work for some reason.
Error logs are:

> [2018-12-28T10:38:41,715][ERROR][o.e.g.GatewayMetaState   ] [LFR016701] failed to read or upgrade local state, exiting...
> java.io.IOException: failed to find global metadata [generation: 68]
>         at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:158) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:110) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:472) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
> [2018-12-28T10:38:41,731][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [LFR016701] uncaught exception in thread [main]
> org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find global metadata [generation: 68]];
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
> Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
>         at org.elasticsearch.node.Node.<init>(Node.java:579) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         ... 6 more
> Caused by: java.io.IOException: failed to find global metadata [generation: 68]
>         at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:110) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:472) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         ... 6 more

What should I do about this?

(when I google "failed to find global metadata", nothing is found.)

Did you pull/build a new version of elasticsearch recently? The data-level compatibility between snapshot version is not guaranteed. So, it's possible that some modification broke backward-compatibility with the previous revision. The simplest workaround here is to delete data directory in elasticsearch installation, but unfortunately, it will also wipe out all your data indexed in elasticsearch.

Thanks for responding.
In fact when it was working yesterday I was already on 7.0 0-snapshot.
I'll try wour workaround soon enough and tell you if that fixes (I can do the rolling upgrade once again to get data back, I have a save.).

Using a new data directory did work. I still don't understand why it broke though, since when I copied old data directory's indices in the new one, it still worked. Worth keeping an eye on it.

Is it possible that you downgraded elasticsearch somehow?

Not really downgraded.
But what I think happened is that the other elasticsearch installed on machine (es 6.5.3, that I thought was uninstalled, I just noticed it right now) started on its own on windows startup (as a service). And it might have broken data folder somehow.
Is there a way I can uninstall this service? I'm not used to windows.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.