Cluster_block_exception: index [.kibana_security_session_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]

Elastic Stack Kibana
1

Dears Support Team,
Today, I login to Elastic in Kibana but have error message

[2024-09-06T04:49:33.914+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "index [.security-profile-8] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}
],
"type": "cluster_block_exception",
"reason": "index [.security-profile-8] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
},
"status": 429
}.

I found these solutions for that:

  • Solution 1: free up disk space
$ curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
  • Solution 2: change the flood stage watermark setting
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
    "cluster.info.update.interval": "1m"
  }
}

But I cannot do that because of I cannot login to system
Can you help me any suggestion?
Ours Elastic version: v 8.12.2

BRs,
Tuan - VinFast

2

That does not free up disk space... it just unblocks read only... which will happen automatically if you actually free up disk space on 8.12 so you do not need to run that command

What you need to do is clean up disk space by cleaning up / deleting indices you no longer need.

From curl / command line...

You need to run the _cat/indices/?v endpoint to see all the indices...

Then you need to run DELETE to delete some indices to free up space... (don't change the watermarks) ... and then the read only indices will self heal

curl -k -X DELETE -u elastic https://[YOUR_ELASTICSEARCH_ENDPOINT]/myindexnametodelete

NOTE this will permanently delete the index... and will not be recoverable unless you have snapshots to recover from

You can also expand the disk to provide more space...

1 Like
4

Thank you for your support,
I have already deleted old indices and it worked fine

BRs,

1 Like