[FORBIDDEN/12/index read-only

Elastic Stack Elasticsearch
1

I am getting the following message from Logstash:

[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

It happened after I purged indices that I don't need anymore and I didn't run out of disk space.

I have already ran the commands to disable read-only mode
PUT _all/_settings{ "index.blocks.read_only_allow_delete": null}

Additionally, I tried to execute the same command when all indices were closed and I opened them up again.

Here is my configuration and health info:

{
"cluster_name" : "cluster01",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 3,
"active_primary_shards" : 1003,
"active_shards" : 2006,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

The following command also shows that all the shards are green
/_cluster/health/_all?level=shards

So I ran out of option. What else I can check? I appreciate you help.

2

Not specifically related to your error message but you probably have too many shards per node.

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

3

There will be messages in the Elasticsearch logs indicating why this block was put in place:

blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

They will indicate that you did at least temporarily exceed the flood stage watermark: although you didn't completely run out of disk space you got close. If your disk usage is still above this watermark then Elasticsearch will keep these blocks in place to protect itself.

If you think you have plenty of free space but Elasticsearch is still putting these blocks in place, please share the log messages so we can dig deeper.

4

Notice that the Elasticsearch stopped working on March 5th as you can see on the following screenshot

image

I have cluster of 3 node and only one of them have the following watewark warning:

Blockquote

2019-03-05T13:40:34,539][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] low disk watermark [85%] exceeded on [Ic4_iljLRlu6jmWSQvr0GQ][vplesdt01a][/data/elasticsearch/nodes/0] free: 14.6gb[14.6%], replicas will not be assigned to this node
[2019-03-05T13:40:34,539][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] low disk watermark [85%] exceeded on [hSIgIuyJTJycfcXO7c7Wow][vplesdt01b][/data/elasticsearch/nodes/0] free: 12.9gb[12.9%], replicas will not be assigned to this node
[2019-03-05T13:40:34,539][WARN ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] high disk watermark [90%] exceeded on [uKvFs32WSjS_MejyHRPfHQ][vplesdt01c][/data/elasticsearch/nodes/0] free: 8.7gb[8.7%], shards will be relocated away from this node
[2019-03-05T13:40:34,539][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.07/1PxZ-fnpSRm-M-w9iu1u4g] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.17/5TNJLkdoQ8SgQGO5qg905w] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.16/bv9dNet3SpqQz1ujc2oQwQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.30/_4_bNOSpSHmWEJ7tHHx4bQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.01/Xf7UPo8ATkOp2kKVptLTIQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.12/zNyU9cWxTnaCAXFIGtkeZw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.10/FjfFpeytTOWGsF3Kil41SQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.20/0B9CHyh_TXSWMl006lkOJQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.31/lltlb9sZQti_AyTkxM_bdQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.22/Y8v27k76TrCeFhQdqfjmPw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.06/-qmeuEVMQxGzGcsHJLdfBw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.04/6D-N4sHWSSKfnrN1vmbI4w] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.21/-urLAwKBTFe3a-UEqHcRpA] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.15/1WYjGNpdRcGrBythN6DCXA] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.14/EIjBole1SWSEUaaQyacVlQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.26/_bKbi36qRVmdCmqOyEX_Ow] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.02/EyCXFfKtSdSDB1unvBa67Q] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.27/eUYmJOK8QIKOE9UTFEk-4Q] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.11/O43ue5xHSf6EqccsHSvrZA] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.28/YtNge9T3QUSvKyt0ICRmGg] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.24/kyiKMjbARsK0fy0qqDnG3Q] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.03/hXqAaw81Sp-CXVlyzX1MuQ] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.08/SfaPN2w4QaS_rXy6V9bXlw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.18/maJEy6-sRUyhw4RU9ebf1Q] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.09/zGg7Lu75Q2mlNRnjLKt87g] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.29/08HljB4gSpSdA7c2XJbTGg] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.23/D5Kdepg2RlmcTj-3yZZ7jw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.19/ucNJOAIDQvmM-lDhYkdIrA] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.13/fDaUGAgNSFWI2FVAUjM30Q] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.05/Sz7jU41VRfCmuZd5PDUAEw] deleting index
[2019-03-05T13:40:41,034][INFO ][o.e.c.m.MetaDataDeleteIndexService] [vplesdt01a] [win-security-2019.01.25/4wFrUfZjRWGgHCeKKKp2zA] deleting index

Blockquote

5

Here is more logs

Blockquote

[2019-03-05T13:41:34,836][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] rerouting shards: [one or more nodes has gone under the high or low watermark]
[2019-03-05T13:42:35,165][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] rerouting shards: [one or more nodes has gone under the high or low watermark]
[2019-03-05T13:43:35,435][INFO ][o.e.c.r.a.DiskThresholdMonitor] [vplesdt01a] rerouting shards: [one or more nodes has gone under the high or low watermark]
[2019-03-05T13:45:54,767][INFO ][o.e.m.j.JvmGcMonitorService] [vplesdt01a] [gc][3961611] overhead, spent [275ms] collecting in the last [1s]
[2019-03-05T13:46:12,602][INFO ][o.e.m.j.JvmGcMonitorService] [vplesdt01a] [gc][old][3961621][5955] duration [7.9s], collections [1]/[8.8s], total [7.9s]/[6.7m], memory [4.9gb]->[1.8gb]/[7.9gb], all_pools {[young] [90.9mb]->[24.9mb]/[266.2mb]}{[survivor] [21.1mb]->[0b]/[33.2mb]}{[old] [4.8gb]->[1.8gb]/[7.6gb]}
[2019-03-05T13:46:12,602][WARN ][o.e.m.j.JvmGcMonitorService] [vplesdt01a] [gc][3961621] overhead, spent [8.1s] collecting in the last [8.8s]
[2019-03-05T13:47:32,491][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2019.03.05][0]] ...]).
[2019-03-05T14:00:05,747][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.05/fPX7fnaGS7iHszDVUzjl4Q] update_mapping [doc]
[2019-03-05T16:00:01,469][INFO ][o.e.c.m.MetaDataCreateIndexService] [vplesdt01a] [win-security-2019.03.06] creating index, cause [auto(bulk api)], templates , shards [5]/[1], mappings
[2019-03-05T16:00:02,842][INFO ][o.e.c.m.MetaDataCreateIndexService] [vplesdt01a] [.monitoring-es-6-2019.03.06] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2019-03-05T16:00:02,843][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] updating number_of_replicas to [1] for indices [.monitoring-es-6-2019.03.06]
[2019-03-05T16:00:03,546][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] create_mapping [doc]
[2019-03-05T16:00:03,551][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:00:03,922][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:00:04,000][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:00:04,327][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2019.03.06][0]] ...]).
[2019-03-05T16:00:05,402][INFO ][o.e.c.m.MetaDataCreateIndexService] [vplesdt01a] [micros-2019.03.06] creating index, cause [auto(bulk api)], templates , shards [5]/[1], mappings
[2019-03-05T16:00:06,690][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [micros-2019.03.06/1UvLazTtQNagk9J5bQ61yQ] create_mapping [doc]
[2019-03-05T16:00:06,693][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [micros-2019.03.06/1UvLazTtQNagk9J5bQ61yQ] update_mapping [doc]
[2019-03-05T16:00:06,695][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [micros-2019.03.06/1UvLazTtQNagk9J5bQ61yQ] update_mapping [doc]
[2019-03-05T16:00:06,974][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[micros-2019.03.06][3], [micros-2019.03.06][2]] ...]).
[2019-03-05T16:00:07,159][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:00:09,253][INFO ][o.e.c.m.MetaDataCreateIndexService] [vplesdt01a] [.monitoring-kibana-6-2019.03.06] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0], mappings [doc]
[2019-03-05T16:00:09,253][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] updating number_of_replicas to [1] for indices [.monitoring-kibana-6-2019.03.06]
[2019-03-05T16:00:10,198][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-kibana-6-2019.03.06][0]] ...]).
[2019-03-05T16:01:31,981][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:09:18,216][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:20:03,471][INFO ][o.e.c.m.MetaDataCreateIndexService] [vplesdt01a] [ms-services-2019.03.06] creating index, cause [auto(bulk api)], templates , shards [5]/[1], mappings
[2019-03-05T16:20:05,045][INFO ][o.e.c.r.a.AllocationService] [vplesdt01a] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ms-services-2019.03.06][2]] ...]).
[2019-03-05T16:20:05,090][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [ms-services-2019.03.06/Y3uRrQVVQE-ozryv0P3j1Q] create_mapping [doc]
[2019-03-05T16:30:59,605][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:31:03,692][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:31:03,746][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T16:53:02,326][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [win-security-2019.03.06/QwwHZbujSqmcNL2QwO9Rtw] update_mapping [doc]
[2019-03-05T17:44:05,691][INFO ][o.e.c.m.MetaDataMappingService] [vplesdt01a] [micros-2019.03.06/1UvLazTtQNagk9J5bQ61yQ] update_mapping [doc]

Blockquote

6

From your logs it looks like one the nodes has a (nearly) full disk. Also 3 nodes with 1003 primary shards looks a bit on the high side.

7

I have now more the 80% of free space on all nodes and I reduced the number of shards to 350 by deleting a lot of indices, but I still cannot write to any index.

8

You must release the block manually. From the reference manual that I linked above:

The index block must be released manually once there is enough disk space available to allow indexing operations to continue.

It also contains an example of how to do this.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.