FORBIDDEN/12/index read-only / allow delete (api)

Elastic Stack Elasticsearch
1

Hi I am getting below error in Logstash again and again.
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T09:56:20,280][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125}

ELASTIC SEARCH LOG:

h/nodes/0] free: 546.9mb[6.5%], shards will be relocated away from this node
[2018-03-29T09:45:43,970][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.9mb[6.5%], shards will be relocated away from this node
[2018-03-29T09:45:43,970][INFO ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-03-29T09:46:14,835][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.8mb[6.5%], shards will be relocated away from this node
[2018-03-29T09:46:45,684][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.7mb[6.5%], shards will be relocated away from this node
[2018-03-29T09:46:45,685][INFO ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-03-29T09:47:16,594][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] high disk watermark [90%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 546.7mb[6.5%], shards will be relocated away from this node
[2018-03-29T09:47:47,719][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 357.7mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:48:18,631][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 354.5mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:48:49,616][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.9mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:49:20,507][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.8mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:49:51,458][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.8mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:50:22,466][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.7mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:50:53,390][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.6mb[4.2%], all indices on this node will marked read-only
[2018-03-29T09:51:24,307][WARN ][o.e.c.r.a.DiskThresholdMonitor] [t2VzJSg] flood stage disk watermark [95%] exceeded on [t2VzJSglTre0M-6mFRiOhw][t2VzJSg][/var/lib/elasticsearch/nodes/0] free: 353.5mb[4.2%], all indices on this node will marked read-only

Can someone tell me how to resolve this issue permanently?

2 Likes
2

You are running out of disk space, which is causing these errors. I think the error messages are quite clear.

You need to delete data, add storage or expand your cluster.

3

how can i check the disk space?

[root@wqtest09 logstash]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 8609412 7638212 510812 94% /
devtmpfs 3994096 0 3994096 0% /dev
tmpfs 4005044 0 4005044 0% /dev/shm
tmpfs 4005044 443704 3561340 12% /run
tmpfs 4005044 0 4005044 0% /sys/fs/cgroup
/dev/vda1 487634 192214 265724 42% /boot
/dev/mapper/data-appl 5029504 20472 4992648 1% /appl
/dev/mapper/data-root 201128232 61464 201050384 1% /data
tmpfs 801012 0 801012 0% /run/user/0
tmpfs 801012 0 801012 0% /run/user/1003
[root@wqtest09 logstash]#

How can i find the mount point of elastic search?

4

Based on the error message it seems this is where your data might be stored. Check the elasticsearch.yml file to check the config.

5

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true
"elasticsearch.yml" 88L, 2869C

Can you please tell me which line i have to check in elasticsearch.yml

6

This is where your data is stored.

7

thanks Christian. I will increase the disk space and check ..

One more clarification. Do we need to follow any sequence while restarting ELK components. Like elastic search first, then logstash ,kibana?

kindly confirm on this

8

And can you please tell me how much disk space is recommended for ELK stack implementation?

9

This only affects Elasticsearch.

That depends entirely on how much data you intend to store in the cluster.

10

ok fine. Is there any way to change the default installation path of ELK components?

By default it is installed in /etc/ directory. Is there any way to change it to other directory?

and data is currently stored in /var/lib/elasticsearch/ path. Instead can we make it store the data indexed in /data folder?

11

I have changed the data path of elasticsearch from /var/lib/elasticsearch to /data/elasticsearch
but still i am getting the same error.

Can you please help
by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,518][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2018-03-29T12:15:36,519][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>125

1 Like
12

Once you hit the floodstage watermark (95% by default), you must manually release the index lock — see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html. The following will do the trick, which you need to run for every affected index:

PUT /your-index/_settings
{
  "index.blocks.read_only_allow_delete": null
}

PS: Please properly format your code; it's pretty hard to read otherwise.

8 Likes
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.