# Failed to find metadata for existing index after node restart

**URL:** https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507
**Category:** Elasticsearch
**Created:** [July 21, 2019, 8:41am UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507 "2019-07-21T08:41:28Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Vertigo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vertigo/32/53875_2.png) [@Vertigo](https://discuss.elastic.co/u/Vertigo)
#### Post date: [July 21, 2019, 8:41am UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/1 "2019-07-21T08:41:28Z")

</div>

Recently we upgraded from 6.7.1 to 7.2.0. After the upgrade, if a node leaves the cluster it sometimes runs into the below error. The only way we have found to fix this issue is delete the data directly and lose any data on that node.

```
[2019-07-21T08:07:13,264][ERROR][o.e.g.GatewayMetaState] [dcmipvmnsm003] failed to read or upgrade local state, exiting...
java.io.IOException: failed to find metadata for existing index alert-2019.04.04 [location: glGVcnWpRg2PhzxB--5WAw, generation: 12]
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:473) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
[2019-07-21T08:07:13,270][ERROR][o.e.b.Bootstrap] [dcmipvmnsm003] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
	at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
Caused by: java.io.IOException: failed to find metadata for existing index alert-2019.04.04 [location: glGVcnWpRg2PhzxB--5WAw, generation: 12]
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.2.0.jar:7.2.0]
	... 11 more
[2019-07-21T08:07:13,273][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [dcmipvmnsm003] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find metadata for existing index alert-2019.04.04 [location: glGVcnWpRg2PhzxB--5WAw, generation: 12]];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
	at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
	... 6 more
Caused by: java.io.IOException: failed to find metadata for existing index alert-2019.04.04 [location: glGVcnWpRg2PhzxB--5WAw, generation: 12]
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~
```

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 21, 2019, 9:15am UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/2 "2019-07-21T09:15:28Z")

</div>

Hi @Vertigo and welcome!

This error indicates that some of the data in the data path is unexpectedly going missing during the restart. Elasticsearch 7 has stricter consistency checks on the contents of the data path to detect this kind of loss compared to earlier versions. You need to make sure that everything in the data path remains intact across a restart.

---

<div class="post-metadata">

### Author: ![Vertigo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vertigo/32/53875_2.png) [@Vertigo](https://discuss.elastic.co/u/Vertigo)
#### Post date: [July 21, 2019, 1:57pm UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/3 "2019-07-21T13:57:04Z")

</div>

Ok, is there a way to ignore the inconsistency or move past this error, or Just delete that shard that is having issues?

Also, how do we ensure that the data path remains intact? The only service that has access to that directory is elasticsearch.

Thanks!

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 21, 2019, 5:55pm UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/4 "2019-07-21T17:55:22Z")

</div>

No, ignoring this problem is a very bad idea. If this data is unexpectedly going missing then who knows what else is going wrong?

To be clear, Elasticsearch does not get into this broken state on its own. There's something else involved too. Can you share the full content of your `elasticsearch.yml`? Avoid redacting as much as you can, and identify any redactions as clearly as possible.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 22, 2019, 2:55pm UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/6 "2019-07-22T14:55:42Z")

</div>

Thanks. I don't see anything in your config that'd make this error more likely. It really does look like some files are going missing from `/data/nsm/elasticsearch` outside of Elasticsearch's control.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 19, 2019, 2:55pm UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/7 "2019-08-19T14:55:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

### Author: ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)
#### Post date: [September 30, 2019, 12:25pm UTC](https://discuss.elastic.co/t/failed-to-find-metadata-for-existing-index-after-node-restart/191507/8 "2019-09-30T12:25:08Z")

</div>

This turned out to be a bug, see [https://github.com/elastic/elasticsearch/issues/47276](https://github.com/elastic/elasticsearch/issues/47276) for more info and a workaround.