Delete unsupported or deprecated static index settings created with a previous release?

How to delete static settings from an index if this setting is not supported/known anymore to the running ES version.

Indices created with ES 5.2.2 or 5.3.0 have been subject to shrinking with a hot-warm strategy in order to lower the number of shards. This shrinking created two static index settings shrink.source.name and shrink.source.uuid in the newly created index.

The new index works as expected since then.

In the meantime I upgraded to ES 6.8.1 and I am preparing the Elasticsearch cluster for ES 7.0 as indices created with older versions (< 6.0) are not supported anymore with ES 7.0.

Kibana offers a nice UI for the required reindexing but this fails due to these two unsupported settings.

As I have no need for these settings anyway (they are just informational for me) I want to delete them from the indices.

Deleting a static setting from an index requires the follwing steps:

  1. close the index
  2. set the setting to null
  3. reopen the index

Unfortunately this does not work with settings which are not supported anymore with the current version of ES 6.8.1.

curl -X PUT "elk29:9200/logstash-20160915/_settings?pretty" -H 'Content-Type: application/json' -d' { "index" : {   "shrink.source.uuid" : null }}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "remote_transport_exception",
        "reason" : "[elk24][10.21.15.24:9300][indices:admin/settings/update]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unknown setting [index.shrink.source.uuid] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
  },
  "status" : 400
}

I expected that the setting is simply removed.

Oviously ES emulates the removal of a setting by setting the value of a setting to null . Unfortunately this only works with explicitly supported settings but not with outdated unsupported settings.

The question remains how to remove index settings which are not supported anymore with the current version of ES?

It sounds like you are expecting to be able to upgrade to 7.0 while there are still indices created in 5.x in your cluster. This will not be possible: Elasticsearch 7 is not compatible with indices from such an old version. You will need to reindex them in 6.x first, which will lose the settings you're concerned about.

Note that shrinking an index does not count as "creating" the new index. If you shrink a 5.x index then the resulting index also counts as a 5.x index even if you performed the shrink in 6.x.

Yes, I try to prepare a 6.8.1 ES cluster for upgrade to ES 7.0.

Yes, I have still 5.x indices which prevent me from an upgrade currently.

Reindexing old 5.x indices in preparation for the upgrade to 7.0 works with most indices just fine!

But I have also a significant amount of indices which got created two years ago with ES 5.x which happen to be created by a shrinking procedure. This shrinking procedure was executed on a 5.x cluster and created static settings in these indices.

These static settings in question look like

"index.shrink.source.name": "logstash-2016.12.05",
"index.shrink.source.uuid": "ACahUQ1RQK232StJOThQ7Q",

These settings got unsupported and cannot be removed (setting to 'null' on a closed index). I guess this is due to the fact that removing procedure actually does not remove the setting but sets it to default. The later is not possible for unsupported settings.

curl -X PUT "localhost:9200/logstash-20160915/_settings?pretty" -H 'Content-Type: application/json' -d' { "index" : { "shrink.source.uuid" : null }}' { "error" : { "root_cause" : [ { "type" : "remote_transport_exception", "reason" : "[elk24][10.21.15.24:9300][indices:admin/settings/update]" } ], "type" : "illegal_argument_exception", "reason" : "unknown setting [index.shrink.source.uuid] please check that any required plugins are installed, or check the breaking changes documentation for removed settings" }, "status" : 400 }

Yes, I think that's what I'd expect. But I'm having trouble understanding why you want to adjust or remove these settings. You're going to have to delete these indices before you can upgrade.