Questions about [FORBIDDEN/12/index read-only

Elastic Stack Elasticsearch
1

Hi,

I have a small dev cluster running in kubernetes.
Since I have quite limited diskspace elasticsearch is often throwing the following error to indexing applications like logstash or kibana:

... blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

I have following questions about it:

  1. how can I configure different watermarks?
  2. why are my cluster and indices shown as green in monitoring tab although my indices are currently read_only?
  3. what is the easiest way to check write ability of indices? I don't want to import all logs of logstash, kibana, elasticsearch if not needed.

Thanks, Andreas

2

Hi,

You can change the watermark settings with

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.disk.watermark.flood_stage": "98%",
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "96%"
  }
}

That the index has been put into read-only/allow delete mode suggests that You've reached the flood_stage.

The index / cluster health is an indication if all data is redundant / available to Elasticsearch. When the configured redundancy level is reached, Elasticsearch health is "green". If all data is available, but the redundancy is degraded, it is "yellow" and if some index shards are not available it is "red".

When You reach a watermark, Elasticsearch puts a log entry in the logfiles:

[o.e.c.r.a.DiskThresholdMonitor] [xxxxdb05.yyy.zz] low disk watermark [95%] exceeded on [tNXqbk9YTOq9MeNhW5eS6g][xxxxdb04.yyy.zz][/var/lib/elasticsearch/nodes/0] free: 23.4gb[4.5%], replicas will not be assigned to this node

Regards,
Alex

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.