# Logstash Error: Retrying Failed Action With Response Code 403

**URL:** <https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864>\
**Category:** Elasticsearch\
**Created:** [April 1, 2019, 8:08pm UTC](https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864 "2019-04-01T20:08:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\
**Post date:** [April 1, 2019, 8:08pm UTC](https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864/1 "2019-04-01T20:08:19Z")

</div>

I'm receiving the following error in Logstash:

```
[2019-04-01T14:43:45,454][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,454][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,455][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,455][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,455][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,455][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-04-01T14:43:45,455][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>69}

```

The closest I can find is [this post](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067) and [this post](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/4), which seem to indicate that it might be related to a disk space issue. However, I'm only at ~80% consumption on the partition where I'm storing my ES data. Additionally, I don't appear to be getting any errors in my ES logs.

Additionally, I do not have SSL configured for ES (and made certain that I didn't accidentally specify it for my beats.conf file). This config worked fine up until sometime late on Mar. 28, and I haven't made any changes since then, so I really don't understand the 403/Forbiddent response code.

The only thing I can think of is that somehow **all** of my indices got set to read-only, and I have no idea how that would happen. Is there an easy way to iterate through them to unset the RO?

---

<div class="post-metadata">

**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\
**Post date:** [April 1, 2019, 8:20pm UTC](https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864/2 "2019-04-01T20:20:47Z")

</div>

Proceeding on the theory that all indices somehow got set to `read_only_allow_delete`, I executed the following command iterate through all indices and reset that flag:

> _Edit: Buggy script removed_

It did not resolve the issue.

## Update

OK...a little embarassing here. As I looked back through what I posted yesterday, I noticed that I had a typo in the `curl` statement. It read: `curl -s -X PUT -H "<header>" "<json>" -d "<url>"`.

I updated it as follows:

```
#!/bin/bash
for i in $(curl -s -X GET http://localhost:9200/_cat/indices | awk -F ' ' '{print $3}' | sort)
do
    echo Updating ${i}: $(curl -s -X PUT -H "Content-Type: application/json" \
                               -d '{"index.blocks.read_only_allow_delete": null}' \
                               "http://localhost:9200/${i}/_settings")
done

```

...which unlocked my indices as expected and allowed log entries to start flowing again. I'm still uncertain why this occurred, however, as I'm unable to see any errors on the ES side of things.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2019, 8:20pm UTC](https://discuss.elastic.co/t/logstash-error-retrying-failed-action-with-response-code-403/174864/3 "2019-04-29T20:20:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.