# Retrying individual bulk actions that failed or were rejected by the previous bulk request
**URL:** https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419
**Category:** Elasticsearch
**Created:** [July 3, 2018, 3:47pm UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419 "2018-07-03T15:47:42Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 3, 2018, 3:47pm UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/1 "2018-07-03T15:47:42Z")
Hi all,
logstash logs return this error
```auto
retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
```
what should i do to fix this issue ?
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 10:35am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/2 "2018-07-04T10:35:19Z")
could any one help me?
---
### Author:  [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)
#### Post date: [July 4, 2018, 10:54am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/3 "2018-07-04T10:54:57Z")
> [@m\_5amy](#):
>
> "reason"=\>"blocked by: [FORBIDDEN/12/index read-only
I've never had this issue myself but the explanation you get for the failed bulk operation seems pretty clear: The index you try to write to is read-only. The big question is why? I can only think of two possibilities:
1. The disc where Elasticsearch stored its index data has become read-only or
2. The index has been [closed](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-open-close.html).
You should take a look at the discs in your cluster to rule out #1 and the Open / Close Index API for #2.
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 10:58am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/4 "2018-07-04T10:58:52Z")
Hi @Bernt_Rostad ,
Thanks for your reply , after i checked logs for logstash and elasticsearch the result is as
```auto
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,715][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/1
[2018-07-04T10:16:33,716][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}
```
and elasticsearch logs is
```auto
[2018-07-04T10:13:45,050][INFO][o.e.c.r.a.AllocationService] [JkvjWj8] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[metricbeat-6.3.0-2018.06.28][0]] ...]).
[2018-07-04T10:14:04,213][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:14:04,214][INFO][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:14:34,246][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:15:04,298][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.2%], shards will be relocated away from this node
[2018-07-04T10:15:04,298][INFO][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:15:34,327][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node
[2018-07-04T10:16:04,355][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node
[2018-07-04T10:16:04,355][INFO][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-07-04T10:16:34,383][WARN][o.e.c.r.a.DiskThresholdMonitor] [JkvjWj8] high disk watermark [90%] exceeded on [JkvjWj8PQmGioaUsLTrYIw][JkvjWj8][/var/lib/elasticsearch/nodes/0] free: 4.4gb[9.1%], shards will be relocated away from this node
```
is this cause of hard disk ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 4, 2018, 11:05am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/5 "2018-07-04T11:05:57Z")
As you can see in the logs, you are running out of disk space and have exceeded the [high watermark](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/disk-allocator.html), which is you are seeing these errors.
How large is your cluster? How much disk does each data node have? Which version are you running?
---
### Author:  [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)
#### Post date: [July 4, 2018, 11:07am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/6 "2018-07-04T11:07:14Z")
Yes, that could be the cause. The warning tells you what's happening:
> [@m\_5amy](#):
>
> high disk watermark exceeded on one or more nodes
One of more node in your cluster has passed the [high disk watermark](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) which means more than 90% of the disk is full. When that happens Elasticsearch will try to move shards away from the node to free up space, but only if it can find another node with enough space.
You need to add more disk space, either on each node or by adding more nodes to the cluster to let Elasticsearch spread the load.
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 11:28am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/7 "2018-07-04T11:28:14Z")
@Christian_Dahlqvist @Bernt_Rostad
After i expanding my instance hard disk and become more than 150G free disk
The discover still return the same
so what should i do after increase hard disk ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 4, 2018, 11:30am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/8 "2018-07-04T11:30:25Z")
Did you reset the read-only index block as described in the docs I linked to?
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 11:34am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/9 "2018-07-04T11:34:22Z")
@Christian_Dahlqvist ok but how to know my current index name to put it ??
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 11:35am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/10 "2018-07-04T11:35:58Z")
@Christian_Dahlqvist what is your opinion on using this fix ?
[https://benjaminknofe.com/blog/2017/12/23/forbidden-12-index-read-only-allow-delete-api-read-only-elasticsearch-indices/](https://benjaminknofe.com/blog/2017/12/23/forbidden-12-index-read-only-allow-delete-api-read-only-elasticsearch-indices/)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 4, 2018, 11:37am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/11 "2018-07-04T11:37:08Z")
That is what the docs I linked to tells you to do.
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 11:39am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/12 "2018-07-04T11:39:24Z")
@Christian_Dahlqvist very very thanks you works fine
---
### Author:  [@m\_5amy](https://discuss.elastic.co/u/m_5amy)
#### Post date: [July 4, 2018, 11:43am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/13 "2018-07-04T11:43:47Z")
@Christian_Dahlqvist I have another question could you help me in it ?
I have multiple remote filebeats and all files logs into a single ELK server but logstash get only logs for single server and every restart for logstash service it get logs from another server from the remote filebeats
what should i do to make logstash get logs from the all filebeats same time?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 4, 2018, 11:47am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/14 "2018-07-04T11:47:28Z")
It should be able to get data from multiple sources, so I would recommend opening a new issue under the Logstash category and share all your configs.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 1, 2018, 11:47am UTC](https://discuss.elastic.co/t/retrying-individual-bulk-actions-that-failed-or-were-rejected-by-the-previous-bulk-request/138419/15 "2018-08-01T11:47:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.