ElasticSearch - DiskThresholdMonitor - Indexes read only

Elastic Stack Elasticsearch
1

Hi ,

I am receiving the following exception:

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1406)

This means that at least for an instance the free disk space was higher than 95% (default settings). I am unlocking the indexes like that: curl -u a_username:******* -H 'Content-Type: application/json' -XPUT 'http://>:9200/_all/_settings' -d '{"index.blocks.read_only_allow_delete": null}'

I would like to know:

  1. If there is an easy way to know if ANY index is currently locked.
  2. I have only one node. If I disable the locking mechanism by setting cluster.routing.allocation.disk.threshold_enabled to false (Disk-based Shard Allocation) am I risking anything except from the obvious fact that if the disk is full then I may loose data.

Thank you

2

GET _settings will show you the settings for all indices. So curl -s http://localhost:9200/_settings/ | grep read_only_allow_delete is a quick-and-dirty way to see if that setting appears anywhere. It's more robust to parse the appropriate setting out of the JSON of course.

Since 7.4.0 this block is removed automatically, so maybe upgrade and just monitor your disk space directly instead.

Elasticsearch may fail a shard on a write error, including disk full, so that you cannot search it any more. It may also require some disk space to recover shards after they have failed too. This means that a full disk may cause multiple entire shards go offline, at least until you can free up enough space for Elasticsearch to recover them.

Also filesystems tend to perform worse when they approach 100% full. For instance they cannot so easily allocate large files as contiguous blocks. This means you may find that performance degrades on a full disk.