Cluster block exception: Forbidden

Elastic Stack Logstash
1

I keep getting messages like the following and the drop in indexing rate correlates with these messages but the watermark is far from being breached.

[2019-07-08T08:50:29,228][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/8/index write (api)];"})

What else is causing the block?

2

This typically means you are running out of disk space in Elasticsearch.

3

@Christian_Dahlqvist you probably mistook FORBIDDEN/8 for FORBIDDEN/12. The latter is the case of running out of disk space.

Also, the case is happening when I have 40% disk utilized so it is not related to that.

1 Like
4

That may very well be the case. What is the specification and size of your cluster? Which version are you using? Is there anything in the logs around those messages?

Are you using ILM so it could be something like this?

5

I am using 6.6.1 stack. I could not find any other logs around it.

There is ILM which moves data from hot to warm nodes after 2 days.

"hot_warm_purge_policy" : {
    "version" : 1,
    "modified_date" : "2019-07-04T15:50:26.847Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : { }
        },
        "delete" : {
          "min_age" : "4d",
          "actions" : {
            "delete" : { }
          }
        },
        "warm" : {
          "min_age" : "2d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            }
          }
        }
      }
    }
 }

There is no setting for making the index read-only yet if I understand correctly.

6

Also the messages are not coming up only at day boundary to indicate that ILM is the problem.

7

Have a look at the settings of your indices. I suspect you will find that only indices located on the warm nodes that have been forcemerged are blocked. If that is the case it may be that you have some events coming in 2 days late and you see the error when Logstash tries to index thee into the warm indices, which could happen at any point during the day.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.