Migration from ES 8.19.20 to 9.5.2 blocked by internal index: .transform-internal-007

Elastic Search
1

Hello,

I have an Elasticsearch 8.19.20 self-hosted basic/free cluster used for logging and Kibana that I am trying to upgrade to 9.5.2. in Kibana, under Management > Stack management I see no Migration Assistant so I am using the /_migration/deprecations API to identify critical issues before the upgrade.

GET /_migration/deprecations
{
  "cluster_settings": [],
  "node_settings": [
    {
      "level": "warning",
      "message": "Specifying multiple data paths is deprecated",
      "url": "https://ela.st/es-deprecation-7-multiple-paths",
      "details": "The [path.data] setting contains a list of paths. Specify a single path as a string. Use RAID or other system level features to utilize multiple disks. If multiple data paths are configured, the node will fail to start in 8.0. (nodes impacted: [log03, log04, log02, log06, log07, log01, log05])",
      "resolve_during_rolling_upgrade": false
    },
    {
      "level": "warning",
      "message": "Multiple data paths are not supported",
      "url": "https://ela.st/es-deprecation-7-multiple-paths",
      "details": "The [path.data] setting contains a list of paths. Specify a single path as a string. Use RAID or other system level features to utilize multiple disks. If multiple data paths are configured, the node will fail to start in 8.0. (nodes impacted: [log03, log04, log02, log06, log07, log01, log05])",
      "resolve_during_rolling_upgrade": false
    }
  ],
  "ml_settings": [],
  "index_settings": {},
  "data_streams": {},
  "ilm_policies": {},
  "templates": {}
}

Multiple data paths is a known issue with my configuration which I will resolve later by re-installing the nodes one by one and migrating shards.

The bigger issue is that /_migration/deprecations does not flag the index that will fail the migration:

{"@timestamp":"$DATE","log.level":"ERROR","message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"log01","elasticsearch.cluster.name":"logcluster","error.type":"java.lang.IllegalStateException","error.message":"The index [.transform-internal-007/Xl2lAg3STEq-huvgPC0QcA] created in version [7.14.0] with current compatibility version [7.14.0] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.5.2.","error.stack_trace":"java.lang.IllegalStateException: The index [.transform-internal-007/Xl2lAg3STEq-huvgPC0QcA] created in version [7.14.0] with current compatibility version [7.14.0] must be marked as read-only using the setting [index.blocks.write] set to [true]

It seems that even a user with superuser role cannot change this setting:

curl -sk -X PUT "http://localhost:9200/.transform-internal-007/_block/write"
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/block/add] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.transform-internal-007], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/block/add] is unauthorized for user [elastic] with effective roles [superuser] on restricted indices [.transform-internal-007], this action is granted by the index privileges [manage,all]"},"status":403}

Questions:

  1. Why did /_migrations/deprecations not flag internal indices as blocking the major version upgrade?
  2. Can I resolve this without either:
    1. Wiping the Elasticsearch data directory (losing all my data) and starting over with version 9.5.x?
    2. Dumping all of the indices in my 8.19.20 cluster and restoring them to a new 9.5.x cluster?

Unfortunately, AI was not able to come up with any alternative solutions that worked, so I present my question to meatware for consideration.

image
image1610×877 70.9 KB

And yes, I have read Issues with upgrade from 8.18.1 to 9.0.1

2

Why did /_migrations/deprecations not flag internal indices as blocking the major version upgrade?

Those are checked by _migration/system_features. The upgrade assistant runs that for you before running _migration/deprecations. You can theoretically do all of the checks that the upgrade assistant does for you using the APIs directly, but I highly recommend using the upgrade assistant. It will also walk you through what your options are for these older indices.

Can I resolve this without either:

The safest thing to do is to restore 8.19.20 from a snapshot and start over. It's possible that there are other ways, but they aren't tested.

3

@Keith_Massey thanks for the reply.

This was my mistake. The role we use to access Kibana has kibana_admin,kibana_admin_manage_index_templates,kibana_user,logs-role but not manage which is apparently required to run the upgrade assistant. I logged into Kibana using the elastic user which has superuser and I am able to see the Upgrade Assistant now:

Screenshot 2026-09-01 20.43.46
Screenshot 2026-09-01 20.43.461908×1202 97 KB

I was not able to find any references to /_migration/system_features in the Elastic documentation on upgrading. Search engine results for this path are also very few:

image
image834×846 96.5 KB

I understand that the vast majority of users will use the Upgrade Assistant, as this is the recommended course. If I may suggest two small documentation improvements:

  1. For the Upgrade Assistant to be visible in the UI, the Kibana user must have the manage, all or superuser role. This is somewhat obvious from the API response from the /api/upgrade_assistant/status but is not obvious to a UI user lacking those roles. Even just having a static page under Upgrade Assistant that explains the need for a higher privileged account would be an extremely helpful hint.
  2. Include /_migration/system_features along with /_migration/deprecations in the Prepare to upgrade documentation.
4

By the way, since you mention upgrading to 9.5.2, you probably ought to check the known issues. A new one was added today that might steer you towards 9.4.x until 9.5.3 comes out: Elasticsearch known issues | Elasticsearch Reference .