Cluster Block Exception retrying failed action with response code: 403

Elastic Stack Logstash
1

Hello Dear Community,

since a Couple of days out of the Blue i get more and more:

retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [Logs] blocked by: [FORBIDDEN/8/index write (api)];"})

in my Logstash Logs and data doesnt reach the Elasticsearch instance

On every Index that this occurs on is a index write block which i have to manually set to false every day or every couple hours and after that the Logs get handled as usual.

And even when i set the Block to false in my Index Template it still gets overriden.

I have more than enough Storage cause that is the Explanation there is in any discussion i found on this Topic.

My Elasticsearch version is 7.17.15 and my logstash version 7.9.3.

Ty in Advance for the Help

2

Hi,

Check your cluster settings to see if the cluster.routing.allocation.disk.watermark.low and cluster.routing.allocation.disk.watermark.high values are set appropriately.

Regards

3

Hi Yago,

yes they are Sized Appropriatly:

      "disk" : {
        "watermark" : {
          "low" : "95%",
          "flood_stage" : "98%",
          "high" : "96%"
        }

I have 30% Space left in my Cluster.

Regards

4

What does elasticsearch log at the point where it transitions the index to read-only?

5

Hello Badger,

unfortunatly i dont see any Logs regarding the Read Only State in Elasticsearch.

The only logs i see related to the Topic are the logstash logs at the beginning of this Thread

Regards

6

Hello, i get this Info tho in my Elasticsearch Logs

2024-06-26T02:51:48,958][INFO ][o.e.x.i.IndexLifecycleTransition] [Node1] moving index [Logs] from [{"phase":"warm","action":"forcemerge","name":"readonly"}] to [{"phase":"warm","action":"forcemerge","name":"forcemerge"}] in policy [Hot_warm]

It says in the message: name readonly

But in my ILM or Templates i nowhere define a read only phase.
I know you can check the Box but i checked and its unchecked.