Logstash trying to write to read-only index endlessly

[alexbde (Alex)]

Hello, we're using BELK-Stack with 5 Filebeats => 1 Logstash => 1 Elasticsearch <= 1 Kibana. During the last weeks we experienced some downtime of 2 Filebeat services (root cause doesn't matter) which led to some buffered log entries sent from Filebeat to Logstash "too late".

Too late means they belong to Elasticsearch indices that are already set to read_only. Because of this Logstash is not able to write to these indices and there are a lot of following entries in Logstash system log:

[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,525][INFO ][logstash.outputs.elasticsearch][main][635..dd5] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>97}

It looks like Logstash is trying to send them to Elasticsearch endlessly (which will never succeed).

Is there a way to configure maximum retries for Logstash? Best would be time-based of course (like stop retrying after 3 days) but count-based would also be fine.

Thank you very much in advance!

Here's our output config if it helps:

output {
  elasticsearch {
    hosts => [ "elasticsearch:9200" ]
    index => "%{[stack]}-%{+YYYY.MM.dd}"
    ilm_enabled => false
  }
}

[leandrojmp (Leandro Pereira)]

There is no such config, logstash will keep on trying to write when it gets an 403 error.

On this old post there is a suggestion to use a ruby filter in the pipeline to drop some events if they are older than some specified time.

Make Logstash drop documents on 403

Greetings Recently I started using forcemerge on my old indices. However, I found out that occasionally, Logstash writes into the older indices, increasing the segment count, so the curator has to merge them again on the next day. To prevent this, I now switch older indices to read-only just before merging. However, now when I look at the Logstash logs, there is a lot of entries like [2018-09-26T11:59:47,219][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 40…

[alexbde (Alex)]

That's an interesting design decision.

Thank you for the hint with the ruby solution, I found some workaround like this in my previous Google search, but it feels a bit dirty.

Is it possible to file a feature request somehow?

[Kurt_S (Kurt Schraeyen)]

To cancel old logs we sometimes use this ruby code for example:

#checking date and if it's to old drop it. Last number in caculation is number of days
ruby {
     code => "event.cancel if (Time.now.to_f - event.get('@timestamp').to_f) > (60 * 60 * 24 * 7)"
}

If you still want the logs indexed you need a system that always writes to the latest index. In that case you should consider using Index Lifecycle Management for this.