Failed to find metadata for existing index

Elastic Stack Elasticsearch
1

I'm having problem on starting elasticsearch
I removed a folder by mistake in elasticsearch data folder(/usr/share/elasticsearch/data/nodes/0) and unfortunately it does not start again!!
it throws exception. How can I solve this?

elasticsearch error log is as below:

"stacktrace": ["java.io.IOException: failed to find metadata for existing index microservice-2019.04.30 [location: 7bm7JXHSRlaSWDsS1Afm-g, generation: 62]",
"at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:473) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.0.jar:7.0.0]"] }
{"type": "server", "timestamp": "2019-05-01T10:10:02,976+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "a3c13b205bd0",  "message": "uncaught exception in thread [main]" , 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find metadata for existing index microservice-2019.04.30 [location: 7bm7JXHSRlaSWDsS1Afm-g, generation: 62]];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0.jar:7.0.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0.jar:7.0.0]",
"... 6 more",
"Caused by: java.io.IOException: failed to find metadata for existing index microservice-2019.04.30 [location: 7bm7JXHSRlaSWDsS1Afm-g, generation: 62]",
"at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0.jar:7.0.0]",
"... 6 more"] }```
2

I can't think of an easy way to safely recover any of the data on this node. If you start a new node with an empty data path then Elasticsearch will rebuild any missing shard copies from the other nodes in the cluster.

1 Like
3

Thanks for your reply David,
I can't add new node, as my elasticsearch is failed on boot.
Is there any way I could remove microservice-2019.04.30 index from files?

4

Sorry, not really, there will be references to this index in other metadata files so if you remove the index then those cross-references will be broken and the node will still fail to start. Modifying the contents of the data directory by hand isn't really supported.

If you change the path.data directory to another, empty, directory then Elasticsearch will make a brand-new node and will then recover any missing shards from the redundant copies held on other nodes in the cluster.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.