Upgrade 6.8.4 -> 7.4.2: forgot to reindex

Elastic Stack Elasticsearch
1

Hi,

i just did a upgrade of my elasticsearch 6.8.4 installation (only one node!). I cant start elasticsearch now because i have old indices which i did not reindex before:

    The index [[index_xyz]] was created with version [5.0.1] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.4.2.

If i try to downgrade to 6.8.4 which was installed before i cannot start either:

[2019-11-28T14:54:15,909][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:51, file:/var/lib/elasticsearch/elasticsearch/nodes/0/_state/node-51.st]]; nested: IOException[failed to read [id:51, file:/var/lib/elasticsearch/elasticsearch/nodes/0/_state/node-51.st]]; nested: XContentParseException[[-1:36] [node_meta_data] unknown field [node_version], parser not found];
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:51, file:/var/lib/elasticsearch/elasticsearch/nodes/0/_state/node-51.st]
    at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
    ... 6 more
Caused by: java.io.IOException: failed to read [id:51, file:/var/lib/elasticsearch/elasticsearch/nodes/0/_state/node-51.st]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
    ... 6 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not found
    at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-6.8.4.jar:6.8.4]
    at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.8.4.jar:6.8.4]
    at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:197) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]

How can i get my elasticsearch running again that i can reindex the indices?

Thanks,
Greets
Kilian

2

You can't downgrade a cluster.
I suggest you restore the last backup you took or reindex from the source.

3

How can i reindex from source?

4

Ok i were able to start elasticsearch 7.4.2 but i had to move the "broken" (not reindexed) indices out of the elasticsearch data location.

My plan now is to reindex the data within a docker 6.8 image:

docker run -it --rm -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms4096m -Xmx4096m" -v /opt/old_data:/usr/share/elasticsearch/data  elasticsearch:6.8.4

But when i start the container it shows me 0 shards / indices. Is there a way to "import" the old data to a fresh elasticsearch container and do the reindex and then move the data back to my 7.4 cluster?

5

The same way you indexed your data initially.

6

I think you already started Elasticsearch on this data dir using a 7.x version. So I said, you can not downgrade later to another version.
You need to start a 6.x version and restore the backup you took.

7

ok, thanks! we started to re-import / index the data from source.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.