Logstash: retrying failed action with response code: 403; reason=> blocked by: FORBIDDEN/5/index read-only (api)

We have set up our production environment with a 5 ES node cluster (3 hot and 2 warm) with a curator executing cronjob to push older logs into the warm nodes. As per industry regulations, curator password expired and was not reset on time, hence older logs filled up the data storage in the hot nodes. Even after password reset and moving older logs into warm nodes, logstash is unable to push the logs into elasticsearch.
Attaching the screenshot of the error at logstash and elasticsearch.

Let me know if further details are required?


Please do not post screenshots. That screenshots is hard to read and does not contain the full exception as it is cut off.

Use markdown for proper formatting of snippets here.

My suspicion is, that you ran out of diskspace, where elasticsearch sets indices to readonly automatically. See https://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html for more info, there is also a mention how to unset this setting (which happens by default on 7.6, but not on some older releases).

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.