# Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

**URL:** <https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357>\
**Category:** Elasticsearch\
**Created:** [January 17, 2023, 7:10pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357 "2023-01-17T19:10:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![pmono](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@pmono](https://discuss.elastic.co/u/pmono)\
**Post date:** [January 17, 2023, 7:10pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/1 "2023-01-17T19:10:20Z")

</div>

Hello,

I'm getting a weird error:  
[2023-01-17T18:18:32.398997+00:00] main.ERROR: Child process failed with message: Elasticsearch engine returned an error response. item id: 20. Error type: "cluster\_block\_exception", reason "index [product\_1\_v1] blocked by: [TOO\_MANY\_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];".

This is a local DEV machine that is running multiple services.

ES version:

```auto
test:/home/test/public_html$ curl -X GET "localhost:9200"
{
  "name" : "test-1",
  "cluster_name" : "magento-test",
  "cluster_uuid" : "b8-gIJwoQP6UijiKfUY2-g",
  "version" : {
    "number" : "7.17.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
    "build_date" : "2022-01-28T08:36:04.875279988Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

I have tried suggestions in this article:

> **[Error: disk usage exceeded flood-stage watermark, index has...](https://www.elastic.co/guide/en/elasticsearch/reference/master/disk-usage-exceeded.html)**

But I'm not able to set the settings:

```auto
test:/home/test/public_html$ curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "50GB",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB"
  }
}

```

Output is:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom], not recognized"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom], not recognized"
  },
  "status" : 400
}

```

Deleting the shards did not help.

Please let me know if there are any other things I should try.

Thanks!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 17, 2023, 7:35pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/2 "2023-01-17T19:35:34Z")

</div>

You are running version 7.17, this setting only exists from version 8.6.

For version 7.17 you need to follow this [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/fix-common-cluster-issues.html#_error_disk_usage_exceeded_flood_stage_watermark_index_has_read_only_allow_delete_block).

Try to increase the size of watermark, if this does not solve you will need to keep delete indices until there is enough free space.

---

<div class="post-metadata">

**Author:** ![pmono](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@pmono](https://discuss.elastic.co/u/pmono)\
**Post date:** [January 17, 2023, 8:14pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/3 "2023-01-17T20:14:42Z")

</div>

Thanks!

I have tried:

```auto
curl -X PUT "localhost:9200/_cluster/settings"
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "50gb",
    "cluster.routing.allocation.disk.watermark.high": "25gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
  }
}

```

And then:

```auto
curl -X PUT */_settings?expand_wildcards=all
{
  "index.blocks.read_only_allow_delete": null
}

```

Same result. I also deleted all shards again, but to no avail.

I noticed presence of new shards:

```auto
 curl -X GET "localhost:9200/_cat/shards?v=true&pretty"
index shard prirep state docs store ip node
.geoip_databases 0 p STARTED 40 37.7mb 127.0.0.1 test-1
.ds-.logs-deprecation.elasticsearch-default-2023.01.14-000007 0 p STARTED 127.0.0.1 test-1
test-amasty2_product_1_v1 0 p STARTED 0 226b 127.0.0.1 test-1
.ds-ilm-history-5-2023.01.14-000006 0 p STARTED 127.0.0.1 test-1

```

No idea what those GeoIP and Deprecation / Ilm History are.

---

<div class="post-metadata">

**Author:** ![pmono](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@pmono](https://discuss.elastic.co/u/pmono)\
**Post date:** [January 17, 2023, 8:19pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/4 "2023-01-17T20:19:46Z")

</div>

BTW, there is a TON of space on the hard drive:

```auto
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.2M 0 4.2M 0% /dev
tmpfs 17G 0 17G 0% /dev/shm
tmpfs 6.7G 9.5M 6.7G 1% /run
/dev/mapper/cs-root 76G 74G 1.5G 99% /
/dev/mapper/cs-home 407G 160G 248G 40% /home
/dev/nvme0n1p2 1.1G 402M 663M 38% /boot
/dev/nvme0n1p1 628M 7.9M 621M 2% /boot/efi
tmpfs 3.4G 0 3.4G 0% /run/user/1000

```

Most of the time the shards were not going beyond 300mb, so it cant be the size of the data.

I also rebooted the machine and restarted the ES service.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 17, 2023, 8:52pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/5 "2023-01-17T20:52:36Z")

</div>

> [@pmono](#):
>
> `/dev/mapper/cs-root 76G 74G 1.5G 99% /`

What is the `path.data` you are using?

If you are using the default, elasticsearch store its data on `/var/lib/elasticsearch`, which will be in the `/` mount.

The information you share shows that the `/` partition is `99%` full.

---

<div class="post-metadata">

**Author:** ![pmono](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@pmono](https://discuss.elastic.co/u/pmono)\
**Post date:** [January 17, 2023, 9:14pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/6 "2023-01-17T21:14:59Z")

</div>

Clearly, I'm blind, dumb and stupid 🙂

I missed that!

Thanks, its working now!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2023, 9:15pm UTC](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/7 "2023-02-14T21:15:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.