Upgrade from 8.19.1 to 9.2.3 failed because of .security-7 index

Elastic Stack Elasticsearch
1

Hi, I upgrade the cluster from 7.16.1 to 8.19.1 as per the upgrade guide. The kibana ugprade assistant did’t report anything. Now I’m upgrading to version 9.2.3 where getting the below issue for security index.

The index .security-7 created in version \[7.16.1\] with current compatibility version \[7.16.1\] must be marked as read-only using the setting \[index.blocks.write\] set to \[true\] before upgrading to 9.2.3

Any correct procedure to migrate this .security-7 index to new one? Setting it to true didn't help.

Thanks

2

This has come upon in different guises a few times.

You should really have gone to 8.19.latest, which is 8.19.10 as of right now. 8.19.1 was a strange choice.

And using kibana 8.19.10s Upgrade Assistant, and earlier 8.19.x versions, would surely have shown you this index (and maybe others) was not 9.x compatible going forward. Please re-check this and share the screenshot.

The remediation you need to take is done via kibana while on 8.19.x, and since you are not the first to miss it (I did too!) it's not maybe as obvious as it might be.

1 Like
3

Thanks for the reply. This is my test cluster.. So I tried taking taking backup security-backup.json and then DELETE and created index which worked fine and were able to migrate.

But for my other cluster I already see the .security-7-reindexed-for-9 which is updated part of my migration to 8.19.1. So I think will not face this issue in these clusters. I yet to upgrade to 9.2.3 here.

 ".security-7-reindexed-for-9": {
    "aliases": {
      ".security": {
        "is_hidden": true
      },
      ".security-7": {
        "is_hidden": true
      }
    },
    "mappings": {
      "dynamic": "strict",
      "_meta": {
        "managed_index_mappings_version": 3,
        "security-version": "8.14.0"
4

When migrating between major version, you should be on the last patch version available of the previous version and upgrade to the last patch version available of the current version.

So you should migrate from 8.19.10 to 9.2.4.

If I'm not wrong, this is the expected migration flow.

5

Doesn't really matter for what purpose, if you follow the correct processes and use the Upgrade Assistant, let it reindex, you would not have had issues.

Yes, but it looks like you are following different upgrade processes on the different clusters? Follow different processes, get different results.
Follow documented processes, get best results.

I've shared this one liner before, it will show you the index.version.created value for all indices in your cluster, as well as index creation time and name. Replace EUSER/EPASS/EHOST/EPORT appropriately

curl -sk -u "${EUSER}:${EPASS}" "https://${EHOST}:${EPORT}" --request-target '_all/_settings?expand_wildcards=all' -X GET | jq -r 'to_entries[] | "\(.value.settings.index.creation_date) \(.value.settings.index.version.created) \(.key)"' | sort -k1nr -k2nr | while read f1 f2 f3 ; do echo $f2 $(date --utc --iso=seconds -d @$(( $f1  / 1000 )) ) $f3 ; done
6

Thanks for the reply. I could see few depreciated index only, remaining all index are with version 8. Anyway I will upgrade first to 8.19.10 and then to 9.2.4

7160199 2025-02-20T20:31:08 .ds-.logs-deprecation.elasticsearch-default-2025.02.20-000084
7160199 2025-02-06T20:21:08 .ds-.logs-deprecation.elasticsearch-default-2025.02.06-000083
7160199 2025-01-23T20:11:08 .ds-.logs-deprecation.elasticsearch-default-2025.01.23-000082
7160199 2025-01-09T20:01:08 .ds-.logs-deprecation.elasticsearch-default-2025.01.09-000081
7160199 2024-12-26T19:51:08 .ds-.logs-deprecation.elasticsearch-default-2024.12.26-000080
7160199 2024-12-12T19:41:08 .ds-.logs-deprecation.elasticsearch-default-2024.12.12-000079
7160199 2024-11-28T19:40:20 .ds-.logs-deprecation.elasticsearch-default-2024.11.28-000078
7160199 2024-11-14T19:30:19 .ds-.logs-deprecation.elasticsearch-default-2024.11.14-000077
7160199 2024-10-31T19:20:19 .ds-.logs-deprecation.elasticsearch-default-2024.10.31-000076
7160199 2024-10-17T19:10:19 .ds-.logs-deprecation.elasticsearch-default-2024.10.17-000075
7160199 2024-10-03T19:00:19 .ds-.logs-deprecation.elasticsearch-default-2024.10.03-000074
7160199 2024-09-19T18:51:31 .ds-.logs-deprecation.elasticsearch-default-2024.09.19-000073