Resetting Cluster from Read-Only

I have a single node cluster that previously had XPack installed. I have since removed XPack.

Today, my disk usage reached the flood water mark and so Elasticsearch put all blocks into read_only_allow_delete.

I cleared diskspace, and attempted to reset all blocks by running this in Kibana:

curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

I received an error though:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.xpack.watcher.template.version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "unknown setting [index.xpack.watcher.template.version] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
  },
  "status": 400
}

I don't know how to get around this. It looks like something is still trying to use an xpack setting. Does anyone have any ideas?

Thanks,
Trevor

Ping @spinscale for an idea on what this might be?

What version of Elasticsearch are you running?

This stems from the watcher history indices, which have this settings set. Due to problems like this from 6.3.0 onwards the watcher history is using metadata instead of a dedicated setting for this, see https://github.com/elastic/elasticsearch/pull/30832

If you don't need the watcher history indices anymore, you could try to delete them first. Otherwise you would need to reinstall x-pack again, but disable all the features (using xpack.security.enabled: false and so forth.

Hope this helps!

3 Likes

I'm on 6.2

That was it. I did DELETE .watcher-history-* in Kibana and now the settings call succeeds. Thanks very much.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.