Issue Upgrading from Elasticsearch 8.19.4 to 9.1.4 – Indices Not Read-Only Despite Settings

Elastic Stack Elasticsearch
1

Hi everyone,

We're in the process of upgrading our Elasticsearch cluster from version 8.12.0 to 9.1.4, following the recommended upgrade path. Here's what we've done so far:

  1. Upgraded from 8.12.0 → 8.19.4 (latest 8.x version).

  2. Set all indices to read-only, especially those originally created in version 7.x.

  3. Elasticsearch is running in a Docker container, behind Nginx.

However, during the upgrade to 9.1.4, the logs indicate that some indices are not read-only, which blocks the upgrade. This is confusing because we explicitly set them to read-only using:

PUT /_all/_settings
{
  "settings": {
    "index.blocks.write": true
  }
}

We’ve verified the settings via:

GET /_all/_settings?include_defaults=true

And the indices appear to have "index.blocks.write": true.

Has anyone encountered this issue during a similar upgrade? Could this be related to:

  • Docker volume persistence?

  • Nginx proxy interfering with API calls?

  • Indices from version 7.x needing additional steps?

Any insights or suggestions would be greatly appreciated!

Thanks in advance,

Rui

2

Could anyone please help on this issue?

3

Which documentation are you following? Can you share it?

I don't think you need to set all indices to read-only, never had to do this, you may need to set older indices only, the ones created before version 8.X.

Can you share the message you are seeing? It is complaining about older system indices?

I don't think setting all indices to read only is part of the upgrade path, never had to do this.

4

I recall hitting something similar and later found I'd misunderstood the Kibana Upgrade Assistant UI around 8.19.0 / 9.0.0 release.

It told me I'd need to migrate indices or set them to read-only. They were old indices so I didn't really care to write to them. But I failed to notice there was a button to do the index version migration, which I had not hit, and instead I was messing around with read-only options which didn't work.

1 Like
5

But I failed to notice there was a button to do the index version migration, which I had not hit, and instead I was messing around with read-only options which didn't work.

I think that this is the answer. It is sometimes easy to miss some of the buttons in the upgrade assistant. That button sets the indices to read-only in a special way that indicates “this is read only and I am OK with it being read-only forever”. This way we don’t accidentally upgrade to 9.x with a 7.x index that you had previously marked as read-only with the intention of making it writable again in the future – because once you get to 9.x that decision is not reversible.

6

@RuiCosta78 have the suggestions helped?

7

Hello.

This is the error message that appears when trying to start the docker container:

{"@timestamp":"2025-10-16T10:15:42.324Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"87089138fb17","elasticsearch.cluster.name":"docker-elasticsearch","error.type":"java.lang.IllegalStateException","error.message":"The index [.ds-.logs-deprecation.elasticsearch-default-2022.04.26-000010/CLfSJWgRTzytCO-Dxw-WQQ] created in version [7.16.2] with current compatibility version [7.16.2] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.1.4.","error.stack_trace":"java.lang.IllegalStateException: The index [.ds-.logs-deprecation.elasticsearch-default-2022.04.26-000010/CLfSJWgRTzytCO-Dxw-WQQ] created in version [7.16.2] with current compatibility version [7.16.2] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.1.4.\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.cluster.metadata.IndexMetadataVerifier.isReadOnlySupportedVersion(IndexMetadataVerifier.java:180)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:126)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:98)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.upgradeProjectMetadata(GatewayMetaState.java:318)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:299)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:286)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.createOnDiskPersistedState(GatewayMetaState.java:194)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.createPersistedState(GatewayMetaState.java:148)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:106)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.node.Node.start(Node.java:315)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:620)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:420)\n\tat org.elasticsearch.server@9.1.4/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:100)\n"}

8

As a follow up to the previous answer, I don’t have Kibana so the upgrade assistant solution does not apply to me. Instead I have installed Grafana.

9

Well, I very strongly advise you to install a kibana instance. Grafana is a great tool for 101 things, but kibana is tightly integrated with elasticsearch.

The upgrade is documented here. Its very explicit, I am just going to quote:

The Upgrade Assistant identifies deprecated settings in your configuration and guides you through resolving issues that could prevent a successful upgrade. The Upgrade Assistant also helps resolve issues with older indices created before version 8.0.0, providing the option to reindex older indices or mark them as read-only. To prevent upgrade failures, we strongly recommend you do not skip this step.

1 Like
10

Even if you do not use Kibana you should have an instance configured, the Upgrade Assistant is a Kibana feature and when upgrading major version is strongly recommend to check for issues on it and solve them.

Besides that, Kibana also makes the management part of Elasticsearch a lot more easier, I would recommend that you spin-up a Kibana instance in your cluster.

I'm not sure how you recover from that now, is your cluster up? Do you have a snapshot?

11

Thanks for the replies.

Will try to run a Kibana service and upgrade from that. Will tell you the results afterwards.