Cluster will not start after upgrade from 8.x to 9.x

Elastic Stack Elasticsearch
1

Hello all,

I’ve upgraded from Elasticsearch 8.19 to 9.2.1. Since doing so none of the cluster nodes will start, the error message in the cluster log:

java.lang.IllegalStateException: The index [.reporting-2021-12-12/EzaQTB1HR8OZ5oH7RhbL0Q] created in version [7.16.1] with current compatibility version [7.16.1] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.2.1.

I did use the Upgrade Assistant to remediate any problematic indexes prior to the upgrade, however I don’t remember it mentioning any .reporting-* indexes.

As the cluster will not start I can’t do anything with these indexes - is there anyway to force the cluster to start so I can simply delete these indexes?

Problem with upgrading to 9.0.0, index compability
2

There old indices that need upgrading are easy to miss when looking at Upgrade Assistant. I hit exactly same issue, just missed that I had to actually press a button. A couple of other people have also reported same.

I think best option is quickly re-install to 8.19.whatever, and check the Upgrade Assistant again.

3

Thanks for getting back to me; a quick search shows that Elasticsearch does not support downgrades - is it really a case of just removing the 9.x packages and installing the 8.x ones? According to the AI response on Google:
”The data path contains version information that prevents direct rollback.”

4

none of your cluster nodes started on 9.x, so its not a downgrade? That was my understanding.

5

Thanks - yes they were on 8.19, the Elasticsearch service was shutdown on all nodes and then the packages were upgraded to the latest 9.x after which the cluster would not start

6

I did a fresh install on test system of 7.16.2, and created 2 test indices.

Then upgraded to 7.latest, did whatever the Upgrade Assistant told me to do.

Then upgraded to 8.latest, and see I need effective do 2 steps

  • migrate system indices
  • re-index (one option from 3, I could have deleted them or made read-only) 6 other indices, 2 of which were from data streams. I chose reindex for all of them

Then upgrade assistant was happy and I could upgrade to 9.latest

Screenshot 2025-11-26 at 19.12.35
Screenshot 2025-11-26 at 19.12.351603×649 149 KB

Screenshot 2025-11-26 at 19.13.02
Screenshot 2025-11-26 at 19.13.02551×735 91 KB

Screenshot 2025-11-26 at 19.15.40
Screenshot 2025-11-26 at 19.15.401619×675 158 KB

Screenshot 2025-11-26 at 19.18.23
Screenshot 2025-11-26 at 19.18.231460×656 93.5 KB

1 Like
7

Thanks for your efforts Kevin - I suppose its just reverting the packages from 9.x to 8.x that worries me - yes the cluster doesn’t start (well it does for a few seconds until it sees indexes that it doesn’t like) but presumably the upgrade process to 9.x will have updated metadata and things like that, so when I go back down to 8.x it may not like it.

All I can do is snapshot the VMs and give it a go I suppose.

8

I did exactly that. Someone else on here who had same issue did same thing.

Yep. Good luck. I am pretty sure you will be fine. With a full VM snapshot you have little to lose. Obviously the Upgrade Assistant did prompt you (and me!) to make a snapshot backup before you embarked on the 8.x → 9.x upgrade, its in the screenshots :wink:

9

Yeah understand about the snapshots - I did snapshot one node as part of upgrading the OS but afterwards VMware said the snapshot size was 34 TB! As I have 5 nodes I balked at doing any more snapshots.

It is a strange that something like an index compatibility is enough to stop the cluster from starting; surely it would have been better for 9.x to just treat incompatible indexes as read-only and still allow you to start the cluster and then remediate these indexes.

Anyway will report back here on the reversion to 8.x

10

Here I meant elasticsearch snapshots. Not VM snapshots. These snapshot, well whatever you ask them to snapshot, but typically it’s dominated by the indices themselves.

Well, someone from elastic can maybe comment on that. But I really want (as a minimum) software to be consistent with its own documentation, and this aspect is documented.

11

Hello again - I’ve managed to downgrade to 8.19 and after fixing a few file ownership issues, I was able to start the cluster again.

Below is what the Upgrade Assistant is identifying as problematic indexes:

image
image1647×804 136 KB

As I thought, it’s not saying anything about the .reporting-* indexes that were stopping the 9.x version from starting - is there something I’ve missed?

12

@kernelpanic

Glad you got back up and running.

Can you back up and show this screen?

Screenshot 2025-11-27 at 8.20.45 AM
Screenshot 2025-11-27 at 8.20.45 AM2660×980 223 KB

1 Like
13

Sure:

image
image955×688 46.3 KB

14

Did you migrate the system indicies?
And of course did you create a snapshot?

15

System indices migration - no I hadn’t and as I’m typing this i’m seeing the “This is only required during major version upgrades. Any hidden indices that need to be reindexed are shown in the next step.” - and this obviously is a major version upgrade :man_facepalming:

16

This one liner was what I used, likely easier way, and you need set shell variable EUSER/EPASS/EHOST/EPORT

curl -sk -u "${EUSER}:${EPASS}" "https://${EHOST}:${EPORT}" --request-target '_all/_settings?expand_wildcards=all' -X GET | jq -r 'to_entries[] | "\(.value.settings.index.creation_date) \(.value.settings.index.version.created) \(.key)"' | sort -k1nr -k2nr | while read f1 f2 f3 ; do echo $f2 $f1 $(date --utc --iso=seconds -d @$(( $f1  / 1000 )) ) $f3 ; done
17

Hey @RainTown You lost me a bit.... :sweat_smile:

what does that command do / what is the intention ? .... Looks like it gets the versions for all the indices? Then what?

Might help if you expound a bit :slight_smile:

18

Sure, sorry, Just to identify which indices were created on which versions, with human readable timestamps.

e.g. the little test cluster I use for absolutely nothing , except occasionally testing stuff prompted by this forum, but runs 9.2.1, has this:

9039001 1764237869856 2025-11-27T10:04:29+00:00 .monitoring-es-7-2025.11.27
9039001 1763683202012 2025-11-21T00:00:02+00:00 .monitoring-es-7-2025.11.21
9039001 1763596800736 2025-11-20T00:00:00+00:00 .monitoring-es-7-2025.11.20
9039001 1763510402318 2025-11-19T00:00:02+00:00 .monitoring-es-7-2025.11.19
9039001 1763424000589 2025-11-18T00:00:00+00:00 .monitoring-es-7-2025.11.18
9039001 1762357859784 2025-11-05T15:50:59+00:00 .ds-ilm-history-7-2025.11.05-000002
9039001 1762270265668 2025-11-04T15:31:05+00:00 .ds-.logs-elasticsearch.deprecation-default-2025.11.04-000001
8512000 1761752987894 2025-10-29T15:49:47+00:00 .ds-.logs-deprecation.elasticsearch-default-2025.10.29-000001
8512000 1761750162394 2025-10-29T15:02:42+00:00 .security-7

I hit the same issue as @kernelpanic at some point, I made a somewhat similar oversight, and was a bit confused as I’d thought my cluster did not exist on 7.x.

I ran that once before clicking through all the Migration Assistant steps, then again after, and compared. Trust, but verify :slight_smile:

HTH

19

Thanks for contributing guys - I think the problem here as me not following the Upgrade Assistant guidance on migrating system indices (which presumably includes the .reporting-* ones) - I just jumped straight to the indexes with warnings next to them and didn’t follow previous steps.

Apologies for the noise.

2 Likes
20

Since I made same, or similar, mistake, as have others on the forum, don’t be too hard on yourself. It’s what the forum is for really.

Glad you sorted it.:ok_hand:

2 Likes