Resetting Cluster from Read-Only

[Trevor_Mills]

I have a single node cluster that previously had XPack installed. I have since removed XPack.

Today, my disk usage reached the flood water mark and so Elasticsearch put all blocks into read_only_allow_delete.

I cleared diskspace, and attempted to reset all blocks by running this in Kibana:

curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

I received an error though:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.xpack.watcher.template.version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "unknown setting [index.xpack.watcher.template.version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
  },
  "status": 400
}

I don't know how to get around this. It looks like something is still trying to use an xpack setting. Does anyone have any ideas?

Thanks,
Trevor

[warkolm (Mark Walkom)]

Ping @spinscale for an idea on what this might be?

[TimV (Tim Vernum)]

What version of Elasticsearch are you running?

[spinscale (Alexander Reelsen)]

This stems from the watcher history indices, which have this settings set. Due to problems like this from 6.3.0 onwards the watcher history is using metadata instead of a dedicated setting for this, see https://github.com/elastic/elasticsearch/pull/30832

If you don't need the watcher history indices anymore, you could try to delete them first. Otherwise you would need to reinstall x-pack again, but disable all the features (using xpack.security.enabled: false and so forth.

Hope this helps!

[Trevor_Mills]

I'm on 6.2

[Trevor_Mills]

That was it. I did DELETE .watcher-history-* in Kibana and now the settings call succeeds. Thanks very much.