Logstash Error: Retrying Failed Action With Response Code 403

Elastic Stack Elasticsearch
2

Proceeding on the theory that all indices somehow got set to read_only_allow_delete, I executed the following command iterate through all indices and reset that flag:

Edit: Buggy script removed

It did not resolve the issue.

Update

OK...a little embarassing here. As I looked back through what I posted yesterday, I noticed that I had a typo in the curl statement. It read: curl -s -X PUT -H "<header>" "<json>" -d "<url>".

I updated it as follows:

#!/bin/bash
for i in $(curl -s -X GET http://localhost:9200/_cat/indices | awk -F ' ' '{print $3}' | sort)
do
    echo Updating ${i}: $(curl -s -X PUT -H "Content-Type: application/json" \
                               -d '{"index.blocks.read_only_allow_delete": null}' \
                               "http://localhost:9200/${i}/_settings")
done

...which unlocked my indices as expected and allowed log entries to start flowing again. I'm still uncertain why this occurred, however, as I'm unable to see any errors on the ES side of things.

5 Likes