# Index.blocks.read\_only\_allow\_delete becomes true even if water mark is reached in one of the node in the cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [March 27, 2019, 6:08am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048 "2019-03-27T06:08:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [March 27, 2019, 6:08am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/1 "2019-03-27T06:08:44Z")
I have multiple data nodes in my cluster with uneven disk capacities. While the data is being continuously being indexed, some nodes reach the 90% watermark (other nodes in the cluster do have sufficient remaining disk space), causing the cluster to go into a blocked state. That is, the **index.blocks.read\_only\_allow\_delete** is set to true for all the indices and hence no more data gets indexed. How to avoid this and maximise the disk usage. Thank you ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 27, 2019, 8:03am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/2 "2019-03-27T08:03:23Z")
This is by design. If a node hits `cluster.routing.allocation.disk.watermark.flood_stage` (95% by default not 90%) then Elasticsearch must stop writing to all the indices on that node. It cannot carry on writing to some of the shard copies on other nodes and just avoid the ones on the full node, because all shard copies must contain the same data. You can avoid this by giving Elasticsearch more space so it does not need to protect itself from a full disk by blocking indexing. You can also postpone the problem by increasing the `flood_stage` watermark. ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [March 27, 2019, 9:03am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/3 "2019-03-27T09:03:09Z")
Thank you @DavidTurner, but my doubt is why was the shards not moved away to a node that has free disk when it reached the `cluster.routing.allocation.disk.watermark.high` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 27, 2019, 9:20am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/4 "2019-03-27T09:20:09Z")
Normally that would be the case, but if disk usage grows too quickly then there might not be time to relocate enough shards away before hitting the `flood_stage` watermark. ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [March 27, 2019, 11:05am UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/5 "2019-03-27T11:05:54Z")
Ohhh that makes sense, thank you @DavidTurner . Can you please help me with below 2 questions 1. Will the shard relocation still happen even if the `flood_stage` watermark is hit. 2. Is there ways to automatically set **index.blocks.read\_only\_allow\_delete** to none if the remaining free disk space is below watermark levels ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 27, 2019, 12:50pm UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/6 "2019-03-27T12:50:05Z")
> [@Hari\_Prasad](#): > > Will the shard relocation still happen even if the `flood_stage` watermark is hit Yes, I think so. > [@Hari\_Prasad](#): > > Is there ways to automatically set **index.blocks.read\_only\_allow\_delete** to none if the remaining free disk space is below watermark levels No, but [there is an open feature request for this](https://github.com/elastic/elasticsearch/issues/39334). ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [March 27, 2019, 1:09pm UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/7 "2019-03-27T13:09:43Z")
Thank you @DavidTurner. It was very helpful ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2019, 1:09pm UTC](https://discuss.elastic.co/t/index-blocks-read-only-allow-delete-becomes-true-even-if-water-mark-is-reached-in-one-of-the-node-in-the-cluster/174048/8 "2019-04-24T13:09:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.