Retrying individual bulk actions that failed or were rejected by the previous bulk request

m_5amy · July 3, 2018, 3:47pm

Hi all,

logstash logs return this error

retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

what should i do to fix this issue ?

m_5amy · July 4, 2018, 10:35am

could any one help me?

Bernt_Rostad · July 4, 2018, 10:54am

I've never had this issue myself but the explanation you get for the failed bulk operation seems pretty clear: The index you try to write to is read-only. The big question is why? I can only think of two possibilities:

The disc where Elasticsearch stored its index data has become read-only or
The index has been closed.

You should take a look at the discs in your cluster to rule out #1 and the Open / Close Index API for #2.

m_5amy · July 4, 2018, 10:58am

Hi @Bernt_Rostad ,
Thanks for your reply , after i checked logs for logstash and elasticsearch the result is as

[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}

and elasticsearch logs is

[2018-07-04T10:13:45,050][INFO ][o.e.c.r.a.AllocationService] [JkvjWj8] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[metricbeat-6.3.0-2018.06.28][0]] ...]).
[2018-07-04T10:14:04,213][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:14:04,214][INFO ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:14:34,246][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:15:04,298][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:15:04,298][INFO ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:15:34,327][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node
[2018-07-04T10:16:04,355][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node
[2018-07-04T10:16:04,355][INFO ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:16:34,383][WARN ][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node

is this cause of hard disk ?

Christian_Dahlqvist · July 4, 2018, 11:05am

As you can see in the logs, you are running out of disk space and have exceeded the high watermark, which is you are seeing these errors.

How large is your cluster? How much disk does each data node have? Which version are you running?

Bernt_Rostad · July 4, 2018, 11:07am

Yes, that could be the cause. The warning tells you what's happening:

One of more node in your cluster has passed the high disk watermark which means more than 90% of the disk is full. When that happens Elasticsearch will try to move shards away from the node to free up space, but only if it can find another node with enough space.

You need to add more disk space, either on each node or by adding more nodes to the cluster to let Elasticsearch spread the load.

m_5amy · July 4, 2018, 11:28am

@Christian_Dahlqvist @Bernt_Rostad
After i expanding my instance hard disk and become more than 150G free disk
The discover still return the same

so what should i do after increase hard disk ?

Christian_Dahlqvist · July 4, 2018, 11:30am

Did you reset the read-only index block as described in the docs I linked to?

m_5amy · July 4, 2018, 11:34am

@Christian_Dahlqvist ok but how to know my current index name to put it ??

m_5amy · July 4, 2018, 11:35am

@Christian_Dahlqvist what is your opinion on using this fix ?
https://benjaminknofe.com/blog/2017/12/23/forbidden-12-index-read-only-allow-delete-api-read-only-elasticsearch-indices/

Christian_Dahlqvist · July 4, 2018, 11:37am

That is what the docs I linked to tells you to do.

m_5amy · July 4, 2018, 11:39am

@Christian_Dahlqvist very very thanks you works fine

m_5amy · July 4, 2018, 11:43am

@Christian_Dahlqvist I have another question could you help me in it ?

I have multiple remote filebeats and all files logs into a single ELK server but logstash get only logs for single server and every restart for logstash service it get logs from another server from the remote filebeats
what should i do to make logstash get logs from the all filebeats same time?

Christian_Dahlqvist · July 4, 2018, 11:47am

It should be able to get data from multiple sources, so I would recommend opening a new issue under the Logstash category and share all your configs.

system · August 1, 2018, 11:47am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash Error: Retrying Failed Action With Response Code 403 Elasticsearch	2	8986	April 29, 2019
Cluster Block Exception retrying failed action with response code: 403 Logstash	5	172	June 26, 2024
Logstash-Elastic problems Elasticsearch	1	676	March 14, 2018
I've been getting "retrying failed action with response code: 403" can someone help me fix this? Elasticsearch	2	6147	September 4, 2019
FORBIDDEN/12/index read-only / allow delete (api) Elasticsearch	12	58312	April 27, 2018

Retrying individual bulk actions that failed or were rejected by the previous bulk request

Related topics