TOO_MANY_REQUESTS/12/index read-only / allow delete (api

[PraveenKT (Praveen Kumar Teegala)]

Why i am getting below logs in Logstash logs?

[2020-05-29T03:21:20,840][INFO ][logstash.outputs.elasticsearch][main][220c76045b556cf67d246e54931c092b729b0e8dcc83cd8283a28d6d531fc2bd] retrying failed action with response code: 429 ({"type"=>"cluster_block_exception", "reason"=>"index [%{[@metadata][target_index]}] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];"})
[2020-05-29T03:21:20,840][INFO ][logstash.outputs.elasticsearch][main][220c76045b556cf67d246e54931c092b729b0e8dcc83cd8283a28d6d531fc2bd] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:couch_and_lamp:

[PraveenKT (Praveen Kumar Teegala)]

I already checked 80% free space on each es node
Below is my logstash config file.

filter {
  if "ISSQHEART" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqheart" } }
  } else if "ISSQMETRIC" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqmetric"} }
  } else if "ISSQWIN" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqwin" } }
  } else if "ISSQFILE" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqfile" } }
  } else if "ISSQPACKET" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqpacket" } }
  } else if "SNMP" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "issqnet-%{+YYYY.MM.dd}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
  }
}
output {
  elasticsearch {
    hosts => ["http://x.x.x.x:9200", "http://x.x.x.x:9200", "http://x.x.x.x:9200" ]
    index => "%{[@metadata][target_index]}"
 }}

[tsullivan (Tim Sullivan)]

It looks like the logstash node is overloading your Elasticsearch cluster with a high amount of indexing. It would be best to take this question to the Elasticsearch channel in this Discuss forum, to talk about scaling up your cluster. (Or the Logstash channel, to talk about scaling down Logstash)

Cheers,
-Tim