Cluster will not start after upgrade from 8.x to 9.x

[kernelpanic (FreeBSD user)]

Hello all,

I’ve upgraded from Elasticsearch 8.19 to 9.2.1. Since doing so none of the cluster nodes will start, the error message in the cluster log:

java.lang.IllegalStateException: The index [.reporting-2021-12-12/EzaQTB1HR8OZ5oH7RhbL0Q] created in version [7.16.1] with current compatibility version [7.16.1] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.2.1.

I did use the Upgrade Assistant to remediate any problematic indexes prior to the upgrade, however I don’t remember it mentioning any .reporting-* indexes.

As the cluster will not start I can’t do anything with these indexes - is there anyway to force the cluster to start so I can simply delete these indexes?

[RainTown (Kevin Maguire)]

There old indices that need upgrading are easy to miss when looking at Upgrade Assistant. I hit exactly same issue, just missed that I had to actually press a button. A couple of other people have also reported same.

I think best option is quickly re-install to 8.19.whatever, and check the Upgrade Assistant again.

[kernelpanic (FreeBSD user)]

Thanks for getting back to me; a quick search shows that Elasticsearch does not support downgrades - is it really a case of just removing the 9.x packages and installing the 8.x ones? According to the AI response on Google:
”The data path contains version information that prevents direct rollback.”

[RainTown (Kevin Maguire)]

none of your cluster nodes started on 9.x, so its not a downgrade? That was my understanding.

[kernelpanic (FreeBSD user)]

Thanks - yes they were on 8.19, the Elasticsearch service was shutdown on all nodes and then the packages were upgraded to the latest 9.x after which the cluster would not start

[RainTown (Kevin Maguire)]

I did a fresh install on test system of 7.16.2, and created 2 test indices.

Then upgraded to 7.latest, did whatever the Upgrade Assistant told me to do.

Then upgraded to 8.latest, and see I need effective do 2 steps

• migrate system indices
• re-index (one option from 3, I could have deleted them or made read-only) 6 other indices, 2 of which were from data streams. I chose reindex for all of them

Then upgrade assistant was happy and I could upgrade to 9.latest