# FORBIDDEN/12/index read-only / allow delete (api) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2018, 9:58am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067 "2018-03-29T09:58:37Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 9:58am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/1 "2018-03-29T09:58:38Z")
Hi I am getting below error in Logstash again and again. [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T09:56:20,280][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>125} ELASTIC SEARCH LOG: h/nodes/0] free: 546.9mb[6.5%], shards will be relocated away from this node [2018-03-29T09:45:43,970][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.9mb[6.5%], shards will be relocated away from this node [2018-03-29T09:45:43,970][INFO][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] rerouting shards: [high disk watermark exceeded on one or more nodes] [2018-03-29T09:46:14,835][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.8mb[6.5%], shards will be relocated away from this node [2018-03-29T09:46:45,684][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.7mb[6.5%], shards will be relocated away from this node [2018-03-29T09:46:45,685][INFO][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] rerouting shards: [high disk watermark exceeded on one or more nodes] [2018-03-29T09:47:16,594][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.7mb[6.5%], shards will be relocated away from this node [2018-03-29T09:47:47,719][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 357.7mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:48:18,631][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 354.5mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:48:49,616][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.9mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:49:20,507][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.8mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:49:51,458][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.8mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:50:22,466][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.7mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:50:53,390][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.6mb[4.2%], all indices on this node will marked read-only [2018-03-29T09:51:24,307][WARN][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.5mb[4.2%], all indices on this node will marked read-only Can someone tell me how to resolve this issue permanently? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 29, 2018, 10:01am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/2 "2018-03-29T10:01:43Z")
You are running out of disk space, which is causing these errors. I think the error messages are quite clear. You need to delete data, add storage or expand your cluster. ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 10:02am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/3 "2018-03-29T10:02:40Z")
how can i check the disk space? [root@wqtest09 logstash]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/centos-root 8609412 7638212 510812 94% / devtmpfs 3994096 0 3994096 0% /dev tmpfs 4005044 0 4005044 0% /dev/shm tmpfs 4005044 443704 3561340 12% /run tmpfs 4005044 0 4005044 0% /sys/fs/cgroup /dev/vda1 487634 192214 265724 42% /boot /dev/mapper/data-appl 5029504 20472 4992648 1% /appl /dev/mapper/data-root 201128232 61464 201050384 1% /data tmpfs 801012 0 801012 0% /run/user/0 tmpfs 801012 0 801012 0% /run/user/1003 [root@wqtest09 logstash]# How can i find the mount point of elastic search? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 29, 2018, 10:04am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/4 "2018-03-29T10:04:26Z")
> [@Monica1](#): > > /dev/mapper/centos-root 8609412 7638212 510812 94% / Based on the error message it seems this is where your data might be stored. Check the `elasticsearch.yml` file to check the config. ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 10:05am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/5 "2018-03-29T10:05:29Z")
# # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory\_lock: true "elasticsearch.yml" 88L, 2869C Can you please tell me which line i have to check in elasticsearch.yml ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 29, 2018, 10:08am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/6 "2018-03-29T10:08:06Z")
> [@Monica1](#): > > path.data: /var/lib/elasticsearch This is where your data is stored. ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 10:19am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/7 "2018-03-29T10:19:05Z")
thanks Christian. I will increase the disk space and check .. One more clarification. Do we need to follow any sequence while restarting ELK components. Like elastic search first, then logstash ,kibana? kindly confirm on this ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 10:28am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/8 "2018-03-29T10:28:00Z")
And can you please tell me how much disk space is recommended for ELK stack implementation? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 29, 2018, 10:31am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/9 "2018-03-29T10:31:04Z")
This only affects Elasticsearch. > [@Monica1](#): > > And can you please tell me how much disk space is recommended for ELK stack implementation? That depends entirely on how much data you intend to store in the cluster. ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 10:47am UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/10 "2018-03-29T10:47:46Z")
ok fine. Is there any way to change the default installation path of ELK components? By default it is installed in /etc/ directory. Is there any way to change it to other directory? and data is currently stored in /var/lib/elasticsearch/ path. Instead can we make it store the data indexed in /data folder? ---
**Author:** ![Monica1](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Monica1](https://discuss.elastic.co/u/Monica1)\ **Post date:** [March 29, 2018, 12:16pm UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/11 "2018-03-29T12:16:48Z")
I have changed the data path of elasticsearch from /var/lib/elasticsearch to /data/elasticsearch but still i am getting the same error. Can you please help by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,518][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}) [2018-03-29T12:15:36,519][INFO][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=\>125 ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [March 30, 2018, 5:48pm UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/12 "2018-03-30T17:48:16Z")
Once you hit the floodstage watermark (95% by default), you must manually release the index lock — see [https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html). The following will do the trick, which you need to run for every affected index: ``` PUT /your-index/_settings { "index.blocks.read_only_allow_delete": null } ``` PS: Please properly format your code; it's pretty hard to read otherwise. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2018, 5:49pm UTC](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/126067/13 "2018-04-27T17:49:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.