Index READONLY

adwaitjoshi · April 28, 2019, 2:21pm

I am having a weird issue where my index keeps going into a readonly status. I understand that when the disk space is low, the index does this as a safety measure. The disk was getting full up to a certain point and then we cleared a lot of things and now it has more than 1.8TB of free space, the index is still doing this! Any ideas why?

dbakti7 · April 28, 2019, 2:28pm

github.com/elastic/kibana

Kibana stays read only when ES high disk watermark has been exceeded and later gone beneath the limit

opened 09:22AM - 24 Aug 17 UTC

closed 04:25PM - 07 Feb 18 UTC

algestam

Pioneer Program Team:Operations

**Kibana version**: 6.0.0-beta1 **Elasticsearch version**: 6.0.0-beta1 **Server OS version**: Ubuntu 16.04.2 LTS **Browser version**: Chrome 60.0.3112.90 **Browser OS version**: Windows 10 **Original install method (e.g. download page, yum, from source, etc.)**: Official tar.gz packages **Description of the problem including expected versus actual behavior**: I'm running a single node Elasticsearch instance, logstash and Kibana. Everything runs on the same host in separate docker containers. If the high disk watermark is exceeded on the ES host, the following is logged in the elasticsearch log: ``` [2017-08-24T07:45:11,757][INFO ][o.e.c.r.a.DiskThresholdMonitor] [CSOifAr] rerouting shards: [high disk watermark exceeded on one or more nodes] [2017-08-24T07:45:41,760][WARN ][o.e.c.r.a.DiskThresholdMonitor] [CSOifAr] flood stage disk watermark [95%] exceeded on [CSOifArqQK-7PBZM_keNoA][CSOifAr][/data/elasticsearch/nodes/0] free: 693.8mb[2.1%], all indice s on this node will marked read-only ``` When this has occured, changes to the `.kibana` index will of course fail as the index cannot be written to. This can be observed by trying to change any setting under _Management_->_Advanced Settings_ where a change to i.e. _search:queryLanguage_ fails with the message `Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];` ![index_read_only](https://user-images.githubusercontent.com/133108/29657724-d598f60c-88b8-11e7-8e0c-b647dfe5c101.png) If more disk space now is made available, ES will log that the node has gone under the high watermark: ``` [2017-08-24T07:47:11,774][INFO ][o.e.c.r.a.DiskThresholdMonitor] [CSOifAr] rerouting shards: [one or more nodes has gone under the high or low watermark] ``` One would now assume that it would be possible to make changes to Kibana settings but trying to make a settings change still fails with the error message: `Config: Error 403 Forbidden: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];` **Steps to reproduce**: 1. Make sure that setting changes can be performed without errors 2. Fill up the elasticsearch data disk so that the high disk watermark is exceeded (I used `fallocate -l9G largefile`) 3. Verify in the ES log that the high disk watermark has been exceeded and the indices has been marked read-only 4. Perform a setting change and verify that it fails since writes are prohibited 5. Resolve the high disk watermark condition (which I did with `rm largefile`) 6. Verify that the ES log states that the node has gone under the high disk watermark (and thus should be possible to write to?) 7. Perform a setting change and it will fail when it actually should succeed.

try this..

curl -XPUT -H "Content-Type: application/json"
https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d
'{"index.blocks.read_only_allow_delete": null}'

system · May 26, 2019, 2:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.