# Cluster will not start after upgrade from 8.x to 9.x **URL:** \ **Category:** Elasticsearch\ **Created:** [November 26, 2025, 3:22pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690 "2025-11-26T15:22:37Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 26, 2025, 3:22pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/1 "2025-11-26T15:22:37Z")
Hello all, I’ve upgraded from Elasticsearch 8.19 to 9.2.1. Since doing so none of the cluster nodes will start, the error message in the cluster log: > java.lang.IllegalStateException: The index [.reporting-2021-12-12/EzaQTB1HR8OZ5oH7RhbL0Q] created in version [7.16.1] with current compatibility version [7.16.1] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.2.1. I did use the Upgrade Assistant to remediate any problematic indexes prior to the upgrade, however I don’t remember it mentioning any .reporting-\* indexes. As the cluster will not start I can’t do anything with these indexes - is there anyway to force the cluster to start so I can simply delete these indexes? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 26, 2025, 4:35pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/2 "2025-11-26T16:35:02Z")
There old indices that need upgrading are easy to miss when looking at Upgrade Assistant. I hit exactly same issue, just missed that I had to actually press a button. A couple of other people have also reported same. I think best option is quickly re-install to 8.19.whatever, and check the Upgrade Assistant again. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 26, 2025, 5:14pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/3 "2025-11-26T17:14:32Z")
Thanks for getting back to me; a quick search shows that Elasticsearch does not support downgrades - is it really a case of just removing the 9.x packages and installing the 8.x ones? According to the AI response on Google: ”The data path contains version information that prevents direct rollback.” ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 26, 2025, 5:37pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/4 "2025-11-26T17:37:41Z")
none of your cluster nodes started on 9.x, so its not a downgrade? That was my understanding. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 26, 2025, 5:41pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/5 "2025-11-26T17:41:46Z")
Thanks - yes they were on 8.19, the Elasticsearch service was shutdown on all nodes and then the packages were upgraded to the latest 9.x after which the cluster would not start ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 26, 2025, 6:24pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/6 "2025-11-26T18:24:09Z")
I did a fresh install on test system of 7.16.2, and created 2 test indices. Then upgraded to 7.latest, did whatever the Upgrade Assistant told me to do. Then upgraded to 8.latest, and see I need effective do 2 steps - migrate system indices - re-index (one option from 3, I could have deleted them or made read-only) 6 other indices, 2 of which were from data streams. I chose reindex for all of them Then upgrade assistant was happy and I could upgrade to 9.latest ![Screenshot 2025-11-26 at 19.12.35](https://us1.discourse-cdn.com/elastic/original/3X/6/0/6091172fdb58ce88eb7b8e7742c9d3fbe36f60bf.jpeg) ![Screenshot 2025-11-26 at 19.13.02](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af0b9932e6a360352528a460c799cb849031fbc7.jpeg) ![Screenshot 2025-11-26 at 19.15.40](https://us1.discourse-cdn.com/elastic/original/3X/b/e/beea9c9e35cf0ede3af21a4a5b5ffdfa6453d6d6.jpeg) ![Screenshot 2025-11-26 at 19.18.23](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0afa0f9dd7dba2871ade75bee45d48dded9d46a.jpeg) ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 10:51am UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/7 "2025-11-27T10:51:10Z")
Thanks for your efforts Kevin - I suppose its just reverting the packages from 9.x to 8.x that worries me - yes the cluster doesn’t start (well it does for a few seconds until it sees indexes that it doesn’t like) but presumably the upgrade process to 9.x will have updated metadata and things like that, so when I go back down to 8.x it may not like it. All I can do is snapshot the VMs and give it a go I suppose. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 27, 2025, 11:12am UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/8 "2025-11-27T11:12:32Z")
> [@kernelpanic](#): > > I suppose its just reverting the packages from 9.x to 8.x that worries me I did exactly that. Someone else on here who had same issue did same thing. > [@kernelpanic](#): > > All I can do is snapshot the VMs and give it a go I suppose. Yep. Good luck. I am pretty sure you will be fine. With a full VM snapshot you have little to lose. Obviously the Upgrade Assistant did prompt you (and me!) to make a snapshot backup before you embarked on the 8.x → 9.x upgrade, its in the screenshots 😉 ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 11:36am UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/9 "2025-11-27T11:36:36Z")
Yeah understand about the snapshots - I did snapshot one node as part of upgrading the OS but afterwards VMware said the snapshot size was 34 TB! As I have 5 nodes I balked at doing any more snapshots. It is a strange that something like an index compatibility is enough to stop the cluster from starting; surely it would have been better for 9.x to just treat incompatible indexes as read-only and still allow you to start the cluster and then remediate these indexes. Anyway will report back here on the reversion to 8.x ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 27, 2025, 11:41am UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/10 "2025-11-27T11:41:33Z")
> [@kernelpanic](#): > > I did snapshot one node Here I meant _elasticsearch_ snapshots. Not VM snapshots. These snapshot, well whatever you ask them to snapshot, but typically it’s dominated by the indices themselves. > [@kernelpanic](#): > > It is a strange that something like an index compatibility is enough to stop the cluster from starting; surely it would have been better for 9.x to just treat incompatible indexes as read-only and still allow you to start the cluster and then remediate these indexes. Well, someone from elastic can maybe comment on that. But I really want (as a minimum) software to be consistent with its own documentation, and this aspect is documented. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 3:47pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/11 "2025-11-27T15:47:29Z")
Hello again - I’ve managed to downgrade to 8.19 and after fixing a few file ownership issues, I was able to start the cluster again. Below is what the Upgrade Assistant is identifying as problematic indexes: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c40a9982815dcc8ffda9afca0fc41d2ddd9cddf3.png) As I thought, it’s not saying anything about the .reporting-\* indexes that were stopping the 9.x version from starting - is there something I’ve missed? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 27, 2025, 4:23pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/12 "2025-11-27T16:23:11Z")
@kernelpanic Glad you got back up and running. Can you back up and show this screen? ![Screenshot 2025-11-27 at 8.20.45 AM](https://us1.discourse-cdn.com/elastic/original/3X/3/8/3835e1ca9e095ebc6d56c2dc67927a4da6dc7ce0.png) ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 4:24pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/13 "2025-11-27T16:24:22Z")
Sure: ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/617c1570144d7b2c80887519beba50a96276a33b.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 27, 2025, 4:25pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/14 "2025-11-27T16:25:18Z")
Did you migrate the system indicies? And of course did you create a snapshot? ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 4:29pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/15 "2025-11-27T16:29:46Z")
System indices migration - no I hadn’t and as I’m typing this i’m seeing the “_This is only required during major version upgrades. Any hidden indices that need to be reindexed are shown in the next step.”_ - and this obviously is a major version upgrade 🤦‍♂️ ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 27, 2025, 4:30pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/16 "2025-11-27T16:30:02Z")
This one liner was what I used, likely easier way, and you need set shell variable EUSER/EPASS/EHOST/EPORT ```auto curl -sk -u "${EUSER}:${EPASS}" "https://${EHOST}:${EPORT}" --request-target '_all/_settings?expand_wildcards=all' -X GET | jq -r 'to_entries[] | "\(.value.settings.index.creation_date) \(.value.settings.index.version.created) \(.key)"' | sort -k1nr -k2nr | while read f1 f2 f3 ; do echo $f2 $f1 $(date --utc --iso=seconds -d @$(( $f1 / 1000 )) ) $f3 ; done ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 27, 2025, 4:35pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/17 "2025-11-27T16:35:14Z")
Hey @RainTown You lost me a bit.... 😅 what does that command do / what is the intention ? .... Looks like it gets the versions for all the indices? Then what? Might help if you expound a bit 🙂 ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 27, 2025, 4:42pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/18 "2025-11-27T16:42:53Z")
Sure, sorry, Just to identify which indices were created on which versions, with human readable timestamps. e.g. the little test cluster I use for absolutely nothing , except occasionally testing stuff prompted by this forum, but runs 9.2.1, has this: ```auto 9039001 1764237869856 2025-11-27T10:04:29+00:00 .monitoring-es-7-2025.11.27 9039001 1763683202012 2025-11-21T00:00:02+00:00 .monitoring-es-7-2025.11.21 9039001 1763596800736 2025-11-20T00:00:00+00:00 .monitoring-es-7-2025.11.20 9039001 1763510402318 2025-11-19T00:00:02+00:00 .monitoring-es-7-2025.11.19 9039001 1763424000589 2025-11-18T00:00:00+00:00 .monitoring-es-7-2025.11.18 9039001 1762357859784 2025-11-05T15:50:59+00:00 .ds-ilm-history-7-2025.11.05-000002 9039001 1762270265668 2025-11-04T15:31:05+00:00 .ds-.logs-elasticsearch.deprecation-default-2025.11.04-000001 8512000 1761752987894 2025-10-29T15:49:47+00:00 .ds-.logs-deprecation.elasticsearch-default-2025.10.29-000001 8512000 1761750162394 2025-10-29T15:02:42+00:00 .security-7 ``` I hit the same issue as @kernelpanic at some point, I made a somewhat similar oversight, and was a bit confused as I’d thought my cluster did not exist on 7.x. I ran that once before clicking through all the Migration Assistant steps, then again after, and compared. Trust, but verify 🙂 HTH ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [November 27, 2025, 4:45pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/19 "2025-11-27T16:45:23Z")
Thanks for contributing guys - I think the problem here as me not following the Upgrade Assistant guidance on migrating system indices (which presumably includes the .reporting-\* ones) - I just jumped straight to the indexes with warnings next to them and didn’t follow previous steps. Apologies for the noise. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [November 27, 2025, 5:03pm UTC](https://discuss.elastic.co/t/cluster-will-not-start-after-upgrade-from-8-x-to-9-x/383690/20 "2025-11-27T17:03:11Z")
> [@RainTown](#): > > indices that need upgrading are _easy to miss_ when looking at Upgrade Assistant > [@kernelpanic](#): > > Apologies for the noise. Since I made same, or similar, mistake, as have others on the forum, don’t be too hard on yourself. It’s what the forum is for really. Glad you sorted it.👌