Fleet indices prevent upgrade to Elasticsearch 9

Elastic Stack Elasticsearch
1

I am trying to upgrade our test ELK stack from 8.x to 9.x.

All of the components have been upgraded to 8.18.2 fine and have no issues. The Upgrade Assistant listed no critical issues and a handful of warnings (read-only indices and deprecated configuration settings).

I then decided to try and install 9.0.1 and it's not gone super well.

On first reboot, it flagged multiple indices that weren't flagged by the Upgrade Assistant, but each service restart set the 'index.blocks.write' value to true and moved on to another (in total I had around 20~ other indices to mark as read-only (or do something else with).

I'm now stuck wondering what to do with the following indices

  • .fleet-actions-7
  • .fleet-agents-7
  • .fleet-enrollment-api-keys-7
  • .fleet-policies-7
  • .fleet-policies-leader-7
  • .fleet-servers-7
  • .transform-internal-007

Even with the superuser role, I can't reindex (to -reindexed) or move the alias from these indices with _alias

So now I appear to be stuck on 8.18.2

2

FYI this is maybe 3rd thread on similar topic, old indices stopping upgrade, in last few weeks.

i.e. since release of 9.x - maybe upgrade documentation needs a review?

3

Interesting, I clearly suck at searching for similar issues, I did have a look (albeit via Google), but didn't find any about upgrading to v9...

Anyhoo... turns out this was entirely of my own making, largely through not doing the obvious task of "Migrate System Indices". Once it's complete and not an issue, it's pretty obvious...

{F82363A4-05F2-4E59-8F8C-762BE352810A}

{EAD75EBE-4773-4DD8-A1CE-9EBCB9F40493}
{EAD75EBE-4773-4DD8-A1CE-9EBCB9F40493}1201×168 17.1 KB

I had to pinch the image from google, but definitely in the dark mode, it doesn't look like it's a particularly mandatory step that you need to complete. I skipped taking a snapshot, because I don't really care about any of the data in my test environment.

I think that if any improvements were to be proposed, then making "Migrate system indices" behave more similarly to "Review deprecated settings and resolve issues" and have a clear numeric indicators of how many indices need migrating etc, then that feels like a decent improvement.

I did run into an IllegalStateException error for .async-search which was because I had a legacy index template (from v6 or so) to stretch the number of shards for indices like .kibana and .security across all the cluster nodes and this was preventing the system indices from doing their upgrades.

[2025-06-10T07:48:59,375][WARN ][o.e.p.PersistentTasksClusterService] [LOG-T-ESM01] persistent task upgrade-system-indices failed
java.lang.IllegalStateException: unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"stretch_system_templates":{"order":1000,"index_patterns":[".*"],"settings":{"index":{"auto_expand_replicas":"0-1"}},"mappings":{},"aliases":{}}}]]

but it's updated now I've paid more attention to the upgrade assistant

 "number" : "9.0.1"
4

It wasn't a criticism of you, "why didn't you search:" sort of thing. Rather the opposite. To give an example

had a not dis-similar story.

I think thats maybe a little unfair on yourself.

IMO thats a great idea.

Yes, I dont think many people would have done that, so it might make things more complex.

Thanks for the feedback and great that you resolved your issue.