Issues with upgrade from 8.18.1 to 9.0.1

Elastic Stack Elasticsearch
21

I have quite many of them, for example:

7080099 1.60217E+12 2020-10-08T13:50:43+00:00 .reporting-2020.10.04
7080099 1.60261E+12 2020-10-13T16:13:04+00:00 .reporting-2020.10.11
7080099 1.60312E+12 2020-10-19T14:18:39+00:00 .reporting-2020.10.18
7080099 1.60364E+12 2020-10-25T16:47:36+00:00 .reporting-2020.10.25
7080099 1.60433E+12 2020-11-02T16:19:48+00:00 .reporting-2020.11.01
7080099 1.60493E+12 2020-11-09T14:56:13+00:00 .reporting-2020.11.08
7080099 1.60548E+12 2020-11-15T21:41:17+00:00 .reporting-2020.11.15
7080099 1.60623E+12 2020-11-24T15:16:50+00:00 .reporting-2020.11.22

So I used for them:

PUT .reporting-2020*/_settings
{ "index": { "index.blocks.read_only" : true } }

and I think it should be fine,
but after restart I have error with that index

22

“That index”, the one I specifically referred to, was the .security-7 index !? I really don’t care about your 2020 indices, they are not a problem right now.

This back and forth is getting a little bit painful. I think I’ve done all I can here. Good luck.

23

Yes I know about that,
because rest I can manage to even delete,
My only one idea for that is reindex this index to new one, delete it, reset elastic account and reindex back to it.
I think that I cannot set read-only mode to security-7 index

I have a test environment for that (actually it is cloned production environment to isolated test environment) so I can try it

Honestly I don't see other option for it

24

Last try …

Did you actually try? What did you SPECIFICALLY try ? What was the actual response?

(these questions are typically answered via cut and paste)

25

Hi, I had the same issue a lot of indices like .security-7 prevented startup of elasticsearch after the upgrade (of a node) saying must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.0.1.

I did first upgrade the elasticsearch to version 8.18.2. That upgrade went fine. What I did NOT do was also upgrade Kibana... So I ran the Upgrade Assistent on Kibana version 8.17.3. That did not give any critical issues.. So I moved forward and ran into this issue about elasticsearch not starting up anymore.

I did as "told" by the error message and marked the problematic indices as read-only. Elasticsearch started again. But after upgrading Kibana to 9.0.1 logon is not possible, because of read-only status of the .security-7 index. I was lucky as this was a DEV log cluster. I will recreate it from scratch.

The solution to upgrade my other log clusters successfully:

  1. upgrade elasticsearch to 8.18.2
  2. upgrade kibana to 8.18.2 (VERY IMPORTANT)
  3. run the Upgrade assistent
  4. upgrade elasticsearch to 9.0.1
  5. upgrade other ELK components to 9.0.1

I hope this will help others who encounter this issue.

1 Like
26

We are running into the same issue version 7 indexes are not being restored to a version 9 elasticsearch cluster despite setting [index.blocks.write] set to [true] before upgrading to 9.0.1. index.blocks.write to true.

27

What do you mean by “restored” in this context ? Are you referring to snapshots ?

Probably helpful to give more details, but also worthy of a new thread IMHO.

28

Hi Broekie,
Thanks for that,
I don't know if this will solve my issue, because I'm running version 8.18.1 and I was failed during try of upgrade,

My main concern was also why was that happened if I identified my old indexes with issues, putted them in read-only mode and after restart with version 9 elastic told me that those indexes should be putted in read-only mode.
Is it possible that it didn't see that those all has been marked asd read-only mode ?
After restart and go back to version 8 I double-checked them and in fact they all were identified as read-only

I don't know if version 8.18.2 will solve my issue - this is change only on one version but it's worth to try.

Anyway thanks for helpful info about that

29

@dominbdg

I have a theory of the cause of your problem.

In the kibana Upgrade Assistant section, which you should check when on 8.18.2 at least, but almost certainly 8,.18.1 is the same, there is the suggestion to migrate system indices. (this will includes the crucial .security-7 index). You need to actually click it while on 8.18.2, it does NOT do it automatically in any way. It warns you it might cause brief interruption of service. It's maybe not as obvious as it could be that you need do this before upgrading - it's not in big red letters or something.

I installed 7.17.28, created an index on 7.x called snooker-world, upgraded to 8.18.2, checked the Upgrade Assistant, and ran through all the steps. Note at no point do I need to use the /index/_block/... endpoints or other API calls manually, the Upgrade Assistant does all the work.

My script (above) showed this for the indices before I ran the update

8525000 1749588450216 2025-06-10T20:47:30+00:00 .security-profile-8
8525000 1749588371224 2025-06-10T20:46:11+00:00 .ds-ilm-history-7-2025.06.10-000001
8525000 1749588365744 2025-06-10T20:46:05+00:00 .slo-observability.summary-v3.4.temp
8525000 1749588365413 2025-06-10T20:46:05+00:00 .ds-.kibana-event-log-ds-2025.06.10-000001
8525000 1749588365134 2025-06-10T20:46:05+00:00 .slo-observability.summary-v3.4
8525000 1749588365022 2025-06-10T20:46:05+00:00 .internal.alerts-observability.slo.alerts-default-000001
8525000 1749588364945 2025-06-10T20:46:04+00:00 .internal.alerts-security.alerts-default-000001
8525000 1749588364853 2025-06-10T20:46:04+00:00 .internal.alerts-ml.anomaly-detection-health.alerts-default-000001
8525000 1749588364783 2025-06-10T20:46:04+00:00 .internal.alerts-observability.uptime.alerts-default-000001
8525000 1749588364712 2025-06-10T20:46:04+00:00 .internal.alerts-observability.logs.alerts-default-000001
8525000 1749588364667 2025-06-10T20:46:04+00:00 .internal.alerts-ml.anomaly-detection.alerts-default-000001
8525000 1749588364602 2025-06-10T20:46:04+00:00 .internal.alerts-observability.threshold.alerts-default-000001
8525000 1749588364512 2025-06-10T20:46:04+00:00 .internal.alerts-stack.alerts-default-000001
8525000 1749588364429 2025-06-10T20:46:04+00:00 .internal.alerts-observability.metrics.alerts-default-000001
8525000 1749588364372 2025-06-10T20:46:04+00:00 .internal.alerts-transform.health.alerts-default-000001
8525000 1749588364331 2025-06-10T20:46:04+00:00 .internal.alerts-observability.apm.alerts-default-000001
8525000 1749588364132 2025-06-10T20:46:04+00:00 .kibana-siem-rule-migrations-integrations
8525000 1749588363840 2025-06-10T20:46:03+00:00 .internal.alerts-default.alerts-default-000001
8525000 1749588363772 2025-06-10T20:46:03+00:00 .slo-observability.sli-v3.4
8525000 1749588363503 2025-06-10T20:46:03+00:00 .kibana-siem-rule-migrations-prebuiltrules
8525000 1749588363166 2025-06-10T20:46:03+00:00 .apm-source-map
8525000 1749588362661 2025-06-10T20:46:02+00:00 .kibana_entities-definitions-1
8525000 1749588359570 2025-06-10T20:45:59+00:00 .kibana_analytics_8.18.2_001
8525000 1749588359529 2025-06-10T20:45:59+00:00 .kibana_ingest_8.18.2_001
8525000 1749588359495 2025-06-10T20:45:59+00:00 .kibana_security_solution_8.18.2_001
8525000 1749588359420 2025-06-10T20:45:59+00:00 .kibana_8.18.2_001
8525000 1749588359376 2025-06-10T20:45:59+00:00 .kibana_alerting_cases_8.18.2_001
8525000 1749588358985 2025-06-10T20:45:58+00:00 .tasks
8525000 1749588357739 2025-06-10T20:45:57+00:00 .kibana_usage_counters_8.18.2_001
7172899 1749587352161 2025-06-10T20:29:12+00:00 .async-search
7172899 1749587336209 2025-06-10T20:28:56+00:00 snooker-world
7172899 1749587241066 2025-06-10T20:27:21+00:00 .kibana-event-log-7.17.28-000001
7172899 1749587240364 2025-06-10T20:27:20+00:00 .kibana_security_session_1
7172899 1749587240140 2025-06-10T20:27:20+00:00 .apm-agent-configuration
7172899 1749587239745 2025-06-10T20:27:19+00:00 .apm-custom-link
7172899 1749587237828 2025-06-10T20:27:17+00:00 .kibana_7.17.28_001
7172899 1749587237756 2025-06-10T20:27:17+00:00 .kibana_task_manager_7.17.28_001
7172899 1749586377180 2025-06-10T20:12:57+00:00 .security-7
7172899 1749586063561 2025-06-10T20:07:43+00:00 .ds-ilm-history-5-2025.06.10-000001
7172899 1749586058443 2025-06-10T20:07:38+00:00 .ds-.logs-deprecation.elasticsearch-default-2025.06.10-000001

After performing all steps asked by the 8.18.2 Upgrade Assistant (and I let it set the 7.x created snooker-world index to read-only) the same script gives:

8525000 1749588843181 2025-06-10T20:54:03+00:00 .ds-.logs-deprecation.elasticsearch-default-2025.06.10-000002
8525000 1749588831035 2025-06-10T20:53:51+00:00 .ds-ilm-history-5-2025.06.10-000002
8525000 1749588787801 2025-06-10T20:53:07+00:00 .reindexed-v8-kibana-event-log-7.17.28-000001
8525000 1749588731666 2025-06-10T20:52:11+00:00 .security-7-reindexed-for-9
8525000 1749588731150 2025-06-10T20:52:11+00:00 .kibana_task_manager_7.17.28_001-reindexed-for-9
8525000 1749588730763 2025-06-10T20:52:10+00:00 .kibana_security_session_1-reindexed-for-9
8525000 1749588730420 2025-06-10T20:52:10+00:00 .kibana_7.17.28_001-reindexed-for-9
8525000 1749588730051 2025-06-10T20:52:10+00:00 .apm-custom-link-reindexed-for-9
8525000 1749588729676 2025-06-10T20:52:09+00:00 .apm-agent-configuration-reindexed-for-9
8525000 1749588728266 2025-06-10T20:52:08+00:00 .async-search-reindexed-for-9
8525000 1749588450216 2025-06-10T20:47:30+00:00 .security-profile-8
8525000 1749588371224 2025-06-10T20:46:11+00:00 .ds-ilm-history-7-2025.06.10-000001
8525000 1749588365744 2025-06-10T20:46:05+00:00 .slo-observability.summary-v3.4.temp
8525000 1749588365413 2025-06-10T20:46:05+00:00 .ds-.kibana-event-log-ds-2025.06.10-000001
8525000 1749588365134 2025-06-10T20:46:05+00:00 .slo-observability.summary-v3.4
8525000 1749588365022 2025-06-10T20:46:05+00:00 .internal.alerts-observability.slo.alerts-default-000001
8525000 1749588364945 2025-06-10T20:46:04+00:00 .internal.alerts-security.alerts-default-000001
8525000 1749588364853 2025-06-10T20:46:04+00:00 .internal.alerts-ml.anomaly-detection-health.alerts-default-000001
8525000 1749588364783 2025-06-10T20:46:04+00:00 .internal.alerts-observability.uptime.alerts-default-000001
8525000 1749588364712 2025-06-10T20:46:04+00:00 .internal.alerts-observability.logs.alerts-default-000001
8525000 1749588364667 2025-06-10T20:46:04+00:00 .internal.alerts-ml.anomaly-detection.alerts-default-000001
8525000 1749588364602 2025-06-10T20:46:04+00:00 .internal.alerts-observability.threshold.alerts-default-000001
8525000 1749588364512 2025-06-10T20:46:04+00:00 .internal.alerts-stack.alerts-default-000001
8525000 1749588364429 2025-06-10T20:46:04+00:00 .internal.alerts-observability.metrics.alerts-default-000001
8525000 1749588364372 2025-06-10T20:46:04+00:00 .internal.alerts-transform.health.alerts-default-000001
8525000 1749588364331 2025-06-10T20:46:04+00:00 .internal.alerts-observability.apm.alerts-default-000001
8525000 1749588364132 2025-06-10T20:46:04+00:00 .kibana-siem-rule-migrations-integrations
8525000 1749588363840 2025-06-10T20:46:03+00:00 .internal.alerts-default.alerts-default-000001
8525000 1749588363772 2025-06-10T20:46:03+00:00 .slo-observability.sli-v3.4
8525000 1749588363503 2025-06-10T20:46:03+00:00 .kibana-siem-rule-migrations-prebuiltrules
8525000 1749588363166 2025-06-10T20:46:03+00:00 .apm-source-map
8525000 1749588362661 2025-06-10T20:46:02+00:00 .kibana_entities-definitions-1
8525000 1749588359570 2025-06-10T20:45:59+00:00 .kibana_analytics_8.18.2_001
8525000 1749588359529 2025-06-10T20:45:59+00:00 .kibana_ingest_8.18.2_001
8525000 1749588359495 2025-06-10T20:45:59+00:00 .kibana_security_solution_8.18.2_001
8525000 1749588359420 2025-06-10T20:45:59+00:00 .kibana_8.18.2_001
8525000 1749588359376 2025-06-10T20:45:59+00:00 .kibana_alerting_cases_8.18.2_001
8525000 1749588358985 2025-06-10T20:45:58+00:00 .tasks
8525000 1749588357739 2025-06-10T20:45:57+00:00 .kibana_usage_counters_8.18.2_001
7172899 1749587336209 2025-06-10T20:28:56+00:00 snooker-world
8525000 1749586063561 2025-06-10T20:07:43+00:00 .migrated-ds-ilm-history-5-2025.06.10-000001
8525000 1749586058443 2025-06-10T20:07:38+00:00 .migrated-ds-.logs-deprecation.elasticsearch-default-2025.06.10-000001

Note now the only index with a 7172899 version is that snooker-world index, which is now read-only as I decided to not reindex it.

The crucial .security-7 index is now called .security-7-reindexed-for-9, this is before I did the upgrade to 9.0.2.

That upgrade to 9.0.2 then worked fine. In fact 9.0.2 gave me (before doing any real work) just 2 more/new 9xxxxxx indices:

9009000 1749589198248 2025-06-10T20:59:58+00:00 .ds-.logs-elasticsearch.deprecation-default-2025.06.10-000001
9009000 1749589246497 2025-06-10T21:00:46+00:00 .ds-.edr-workflow-insights-default-2025.06.10-000001

If anyone who might know got this far, maybe they can explain to me why

7172899 is 7.17.28 (yes, I see a pattern emerging)
but 8525000 is apparently 8.18.2 ??
and 9009000 is apparently 9.0.2 ??

Are these mappings documented somewhere ? Numbers taken from output of a GET to _all/_settings?expand_wildcards=all

1 Like