Cannot upgrade to 8.x; cannot find incompatible indices

Elastic Stack Elasticsearch
1

I'm not able to upgrade my cluster to 8.x. The Upgrade Assistant (via Kibana) does not identify any indices that need to be migrated (I additionally clicked the "migrate button," regardless). I'm stuck.

After upgrading [via yum] a single node, the node fails to start with these error messages (basically saying that I have older indices). All nodes are running "7.17.2" HELP?!

[2022-04-04T09:07:21,944][INFO ][o.e.e.NodeEnvironment    ] [my_node] upgrading legacy data folders: [/data/elasticsearch]
[2022-04-04T09:07:22,032][INFO ][o.e.e.NodeEnvironment    ] [my_node] oldest index version recorded in NodeMetadata 6.5.3
[2022-04-04T09:07:22,033][ERROR][o.e.b.Bootstrap          ] [my_node] Exception
java.lang.IllegalStateException: cannot upgrade node because incompatible indices created with version [6.5.3] exist, while the minimum compatible index version is [7.0.0]. Upgrade your older indices by reindexing them in version [7.17.0] first.
2

Please note that I wrote a bit of Python to fetch all of the index versions... and the oldest I find is 7.6. I just don't understand/see what that error message is telling me.

For reference, I'm looking at:

idx_dict[idx][u'settings'][u'index'][u'version'][u'created_string']

3

Is that index dictionary also containing closed and system indices or just the ones your application has created?

If you have Kibana up and running, I highly recommend taking a look at the Upgrade Assistant

4

"The Upgrade Assistant (via Kibana) does not identify any indices that need to be migrated."

By recursively grep'ing for the reported index version 6.5.3, I found the "allegedly offending" index. Kibana/UpgradeAssistant doesn't report it. As I said, it's only when I attempt to actually upgrade that I see the error (i.e., the elasticsearch process/service fails to start).

This seems like a bug... or at least a "feature enhancement request."

I'm in the long process of using the reindex API to attempt to migrate those data to a new index alias (ILM). When this completes, I'll drop the original index (if the reindex API doesn't do that automatically -- time will tell), and then re-attempt my upgrade. If required, I'll then/next explicitly remove that index from the filesystem.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.