Logstash trying to write to read-only index endlessly

Elastic Stack Logstash
1

Hello, we're using BELK-Stack with 5 Filebeats => 1 Logstash => 1 Elasticsearch <= 1 Kibana. During the last weeks we experienced some downtime of 2 Filebeat services (root cause doesn't matter) which led to some buffered log entries sent from Filebeat to Logstash "too late".

Too late means they belong to Elasticsearch indices that are already set to read_only. Because of this Logstash is not able to write to these indices and there are a lot of following entries in Logstash system log:

[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,524][INFO ][logstash.outputs.elasticsearch][main][635..dd5] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [example-2020.12.07] blocked by: [FORBIDDEN/8/index write (api)];"})
[2021-01-04T15:07:01,525][INFO ][logstash.outputs.elasticsearch][main][635..dd5] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>97}

It looks like Logstash is trying to send them to Elasticsearch endlessly (which will never succeed).

Is there a way to configure maximum retries for Logstash? Best would be time-based of course (like stop retrying after 3 days) but count-based would also be fine.

Thank you very much in advance!

Here's our output config if it helps:

output {
  elasticsearch {
    hosts => [ "elasticsearch:9200" ]
    index => "%{[stack]}-%{+YYYY.MM.dd}"
    ilm_enabled => false
  }
}
2

There is no such config, logstash will keep on trying to write when it gets an 403 error.

On this old post there is a suggestion to use a ruby filter in the pipeline to drop some events if they are older than some specified time.

3

That's an interesting design decision.

Thank you for the hint with the ruby solution, I found some workaround like this in my previous Google search, but it feels a bit dirty.

Is it possible to file a feature request somehow?

4

To cancel old logs we sometimes use this ruby code for example:

#checking date and if it's to old drop it. Last number in caculation is number of days
ruby {
     code => "event.cancel if (Time.now.to_f - event.get('@timestamp').to_f) > (60 * 60 * 24 * 7)"
}

If you still want the logs indexed you need a system that always writes to the latest index. In that case you should consider using Index Lifecycle Management for this.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.