Blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]

[Baygon (Vincent T)]

Hi,

I reached disk usage of 90%.
I increased the disk space and now have 80% usage:

/dev/root 562G 445G 118G 80% /

I've tried to run the commands to remove the read only blocker:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

PUT */_settings?expand_wildcards=all
{
  "index.blocks.read_only_allow_delete": null
}

However ES still output the same error:

Feb  5 01:47:37 elastic1 logstash[1704888]: [2023-02-05T01:47:37,200][INFO ][logstash.outputs.elasticsearch][twitch_streams_in_progress][459ad5f58b8eabb83dd337c46ccfbb5d7908ed1b3d76624ce088d75da30b94b9] Retrying failed action {:status=>429, :action=>["update", {:_id=>"41290676350", :_index=>"stream_id", :routing=>nil, :_type=>"ts", :retry_on_conflict=>1}, {
.....
], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [sid] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

Any idea what could be the blocker?

This is a single node running ES 7.17.9

I have just upgraded to 7.17 from 7.13.
I tried to add a document to an index using Kibana dev tools and it worked, but logstash keeps having this issue and does not push a single document

[stephenb (Stephen Brown)]

Perhaps take a look at this post

Although I thought in 7.17 it automatically reset those settings..

[Baygon (Vincent T)]

Thanks, I've just did that and same.... until I look at logs timestamp.
What was happening is that logstash yml file had an error, and on each restart attempt of the service, the last error logs which had the message above are thrown again in the syslog output, making me mistakingly believe that there was still the read only issue, but the timerstamp of the log was the one 1 hour ago before I execute the commands.