# Error FORBIDDEN/8/index write (api)\] and Elasticsearch stop to receiving logstash bulk requests
**URL:** https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028
**Category:** Logstash
**Created:** [January 14, 2020, 6:05pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028 "2020-01-14T18:05:57Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ALEXANDRE\_BUNN1](https://discuss.elastic.co/u/ALEXANDRE_BUNN1)
#### Post date: [January 14, 2020, 6:05pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/1 "2020-01-14T18:05:58Z")
I have a server running Logstash, Kibana and ElasticSearch 6.8, all in the same server.
I was setting up the ILM for the indexes, just to set a retention and executing force\_merge for the old indices (older than 1 day) and "suddenly" I've started to receive the error below at logstash log.
[2020-01-14T13:20:55,327][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/8/index write (api)];"})
[2020-01-14T13:20:55,327][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>80}
I've already removed the ILM from all the indexes, but still receiving this error.
POST \_all/\_ilm/remove
I'm not running out of disk space
[2020-01-14T13:23:54,864][INFO][o.e.e.NodeEnvironment] [I8x9STN] using [1] data paths, mounts [[/var (/dev/mapper/vg00-var)]], net usable\_space [196.4gb], net total\_space [399.9gb], types [ext4]
See below the \_cluster/health output:
{
"cluster\_name" : "elasticsearch",
"status" : "yellow",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 3583,
"active\_shards" : 3583,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 115,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 96.89021092482423
}
The server has 64GB of ram, logstash has setup 2GB of heap and elasticsearch with 16GB of heap.
I'm able to insert data into a teste indice with kibana.
The server has 751 indices and 3698 shards, but it was much more, I'was setting the ILM and Templates for each indice prefix to have the properly number\_of\_shards and retention to keep the server organized and avoid disk space issues.
Following below the json of one ILM Policies, and I'm not using rollover since I have just one server.
{
"policy": {
"phases": {
"hot": {
"min\_age": "0ms",
"actions": {
"set\_priority": {
"priority": 100
}
}
},
"warm": {
"min\_age": "2d",
"actions": {
"forcemerge": {
"max\_num\_segments": 1
},
"set\_priority": {
"priority": 50
}
}
},
"delete": {
"min\_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
Thanks
---
### Author:  [@ALEXANDRE\_BUNN1](https://discuss.elastic.co/u/ALEXANDRE_BUNN1)
#### Post date: [January 15, 2020, 4:55pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/2 "2020-01-15T16:55:46Z")
As a workaroud I've executed the command below.
curl -X PUT -H "Content-Type: application/json" '[http://localhost:9200/\_all/\_settings](http://localhost:9200/_all/_settings)' -d '{ "index": { "blocks": { "write": "false" } } }
I found this at the topic below
> [@FORBIDDEN/8/index write (api) - During date change on indexes](https://discuss.elastic.co/t/forbidden-8-index-write-api-during-date-change-on-indexes/179319/7):
>
> Nope, no prerequisite for that. I don't think either of those would cause anything related to the index being read-only. Is it possible that your node filled up with disk space and the index was marked read only because of that?
The doubt that I have is, why it happen? Because of the configuration in the warm phase, without rollover, I've setup force merge and "Timing for warm phase" 1 day from index creation.
Thanks
---
### Author:  [@logger](https://discuss.elastic.co/u/logger)
#### Post date: [January 15, 2020, 5:15pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/3 "2020-01-15T17:15:16Z")
Hi,
how old are those Indices? If they are all older than 2 days, maybe elasticsearch tried to merge all at once. While a forcemerge ist active the index size is rising. If there is nothing to merge it will double.
As far as I know from experience.
If all of them start at once your size of indices reaches the limit.
> net usable\_space [196.4gb], net total\_space [399.9gb]
After a threshold elasticsearch locks all indices.
Maybe someone can confirm my theory.
---
### Author:  [@ALEXANDRE\_BUNN1](https://discuss.elastic.co/u/ALEXANDRE_BUNN1)
#### Post date: [January 15, 2020, 5:38pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/4 "2020-01-15T17:38:49Z")
Good afternoon
There are indices up to 90 days. ILM would performe a force merge in a indice that it have already performed this action?
Or the best aproach in my case is to set the warm phase "Timing for warm phase" for a greater threshold and gradually decrease it till reach the desired valeu?
Thanks
---
### Author:  [@logger](https://discuss.elastic.co/u/logger)
#### Post date: [January 15, 2020, 5:47pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/5 "2020-01-15T17:47:46Z")
Oh there is maybe another and more likely explanation.
Forcemerge is only executed on **read only** Indices [forcemerge docu](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-forcemerge.html)
maybe your logstash ist trying to write to an "forcemerged" and therefor "read only" index.
Maybe you have to check the indices if they are still used for writing.
---
### Author:  [@ALEXANDRE\_BUNN1](https://discuss.elastic.co/u/ALEXANDRE_BUNN1)
#### Post date: [January 15, 2020, 6:50pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/6 "2020-01-15T18:50:56Z")
I didn't use forcemerge directly, I've setup it through ILM.
My point is how to avoid to have this problem again.
Thanks
---
### Author:  [@logger](https://discuss.elastic.co/u/logger)
#### Post date: [January 15, 2020, 8:57pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/7 "2020-01-15T20:57:04Z")
you can avoid this by not writing into an forcemerged index.
I dont know how your indices look like.
It looks like logstash tries to write into an "old" index. Do you have daily indices like "logstash-2020.01.01"? Or do you use aliases?
---
### Author:  [@ALEXANDRE\_BUNN1](https://discuss.elastic.co/u/ALEXANDRE_BUNN1)
#### Post date: [January 15, 2020, 10:16pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/8 "2020-01-15T22:16:55Z")
I have only daily indexes, and logstash receives data from filebeat instances... That's what sounds weird the fact that after setup ILM for all indexes, new indices stop to receive writes too.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 12, 2020, 10:16pm UTC](https://discuss.elastic.co/t/error-forbidden-8-index-write-api-and-elasticsearch-stop-to-receiving-logstash-bulk-requests/215028/9 "2020-02-12T22:16:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.