"Failed to read or upgrade local state, exiting..." when starting ES instance

Elastic Stack Elasticsearch
1

Hello,

I'm developing a kibana plugin, without my ES instance I'm completely dead. Yesterday ES instance would start normally, but today it doesn't work for some reason.
Error logs are:

> [2018-12-28T10:38:41,715][ERROR][o.e.g.GatewayMetaState   ] [LFR016701] failed to read or upgrade local state, exiting...
> java.io.IOException: failed to find global metadata [generation: 68]
>         at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:158) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:110) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:472) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
> [2018-12-28T10:38:41,731][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [LFR016701] uncaught exception in thread [main]
> org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find global metadata [generation: 68]];
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
> Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
>         at org.elasticsearch.node.Node.<init>(Node.java:579) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         ... 6 more
> Caused by: java.io.IOException: failed to find global metadata [generation: 68]
>         at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:158) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:110) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:472) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
>         ... 6 more

What should I do about this?

(when I google "failed to find global metadata", nothing is found.)

2

Did you pull/build a new version of elasticsearch recently? The data-level compatibility between snapshot version is not guaranteed. So, it's possible that some modification broke backward-compatibility with the previous revision. The simplest workaround here is to delete data directory in elasticsearch installation, but unfortunately, it will also wipe out all your data indexed in elasticsearch.

3

Thanks for responding.
In fact when it was working yesterday I was already on 7.0 0-snapshot.
I'll try wour workaround soon enough and tell you if that fixes (I can do the rolling upgrade once again to get data back, I have a save.).

4

Using a new data directory did work. I still don't understand why it broke though, since when I copied old data directory's indices in the new one, it still worked. Worth keeping an eye on it.

5

Is it possible that you downgraded elasticsearch somehow?

6

Not really downgraded.
But what I think happened is that the other elasticsearch installed on machine (es 6.5.3, that I thought was uninstalled, I just noticed it right now) started on its own on windows startup (as a service). And it might have broken data folder somehow.
Is there a way I can uninstall this service? I'm not used to windows.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.