TOO_MANY_REQUESTS/12/index read-only / allow delete (api

Elastic Stack Logstash
1

Why i am getting below logs in Logstash logs?

[2020-05-29T03:21:20,840][INFO ][logstash.outputs.elasticsearch][main][220c76045b556cf67d246e54931c092b729b0e8dcc83cd8283a28d6d531fc2bd] retrying failed action with response code: 429 ({"type"=>"cluster_block_exception", "reason"=>"index [%{[@metadata][target_index]}] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];"})
[2020-05-29T03:21:20,840][INFO ][logstash.outputs.elasticsearch][main][220c76045b556cf67d246e54931c092b729b0e8dcc83cd8283a28d6d531fc2bd] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:couch_and_lamp:
2

I already checked 80% free space on each es node
Below is my logstash config file.

filter {
  if "ISSQHEART" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqheart" } }
  } else if "ISSQMETRIC" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqmetric"} }
  } else if "ISSQWIN" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqwin" } }
  } else if "ISSQFILE" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqfile" } }
  } else if "ISSQPACKET" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqpacket" } }
  } else if "SNMP" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqnet-%{+YYYY.MM.dd}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
  }
}
output {
  elasticsearch {
    hosts => ["http://x.x.x.x:9200", "http://x.x.x.x:9200", "http://x.x.x.x:9200" ]
    index => "%{[@metadata][target_index]}"
 }}
3

It looks like the logstash node is overloading your Elasticsearch cluster with a high amount of indexing. It would be best to take this question to the Elasticsearch channel in this Discuss forum, to talk about scaling up your cluster. (Or the Logstash channel, to talk about scaling down Logstash)

Cheers,
-Tim

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.