Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

Elastic Stack Elasticsearch
1

Hello,

I'm getting a weird error:
[2023-01-17T18:18:32.398997+00:00] main.ERROR: Child process failed with message: Elasticsearch engine returned an error response. item id: 20. Error type: "cluster_block_exception", reason "index [product_1_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];".

This is a local DEV machine that is running multiple services.

ES version:

test:/home/test/public_html$ curl -X GET "localhost:9200"
{
  "name" : "test-1",
  "cluster_name" : "magento-test",
  "cluster_uuid" : "b8-gIJwoQP6UijiKfUY2-g",
  "version" : {
    "number" : "7.17.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
    "build_date" : "2022-01-28T08:36:04.875279988Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I have tried suggestions in this article:

But I'm not able to set the settings:

test:/home/test/public_html$ curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "50GB",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB"
  }
}

Output is:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom], not recognized"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage.max_headroom], not recognized"
  },
  "status" : 400
}

Deleting the shards did not help.

Please let me know if there are any other things I should try.

Thanks!

2

You are running version 7.17, this setting only exists from version 8.6.

For version 7.17 you need to follow this documentation.

Try to increase the size of watermark, if this does not solve you will need to keep delete indices until there is enough free space.

1 Like
3

Thanks!

I have tried:

curl -X PUT "localhost:9200/_cluster/settings"
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "50gb",
    "cluster.routing.allocation.disk.watermark.high": "25gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
  }
}

And then:

curl -X PUT */_settings?expand_wildcards=all
{
  "index.blocks.read_only_allow_delete": null
}

Same result. I also deleted all shards again, but to no avail.

I noticed presence of new shards:

 curl -X GET "localhost:9200/_cat/shards?v=true&pretty"
index                                                         shard prirep state   docs  store ip        node
.geoip_databases                                              0     p      STARTED   40 37.7mb 127.0.0.1 test-1
.ds-.logs-deprecation.elasticsearch-default-2023.01.14-000007 0     p      STARTED             127.0.0.1 test-1
test-amasty2_product_1_v1                                 0     p      STARTED    0   226b 127.0.0.1 test-1
.ds-ilm-history-5-2023.01.14-000006                           0     p      STARTED             127.0.0.1 test-1

No idea what those GeoIP and Deprecation / Ilm History are.

4

BTW, there is a TON of space on the hard drive:

Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             4.2M     0  4.2M   0% /dev
tmpfs                 17G     0   17G   0% /dev/shm
tmpfs                6.7G  9.5M  6.7G   1% /run
/dev/mapper/cs-root   76G   74G  1.5G  99% /
/dev/mapper/cs-home  407G  160G  248G  40% /home
/dev/nvme0n1p2       1.1G  402M  663M  38% /boot
/dev/nvme0n1p1       628M  7.9M  621M   2% /boot/efi
tmpfs                3.4G     0  3.4G   0% /run/user/1000

Most of the time the shards were not going beyond 300mb, so it cant be the size of the data.

I also rebooted the machine and restarted the ES service.

5

What is the path.data you are using?

If you are using the default, elasticsearch store its data on /var/lib/elasticsearch, which will be in the / mount.

The information you share shows that the / partition is 99% full.

2 Likes
6

Clearly, I'm blind, dumb and stupid :slight_smile:

I missed that!

Thanks, its working now!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.