Elasticsearch index stopped

Elastic Stack Elasticsearch
1

have graylog 4.x on ubuntu 20.04. standard installation.
I can log in to my gralog but all data is one week old.
services says running but : graylog-server.service: Failed with result ‘timeout’.
in graylog log i have warning
<
WARN [MessagesAdapterES6] Failed to index message: index=<graylog_0> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>

in syslog i have error :
2021-11-18T23:25:02.093084+00:00 rsyslog systemd[1]: motd-news.service: Failed with result ‘exit-code’.
2021-11-18T23:25:02.094389+00:00 rsyslog systemd[1]: Failed to start Message of the Day.

my Elasticsearch is :
~# curl localhost:9200
{
“name” : “peoR6Gc”,
“cluster_name” : “graylog”,
“cluster_uuid” : “o-clJDFLSz2BcV-fK5ld_Q”,
“version” : {
"number" : "6.8.20" ,
“build_flavor” : “oss”,
“build_type” : “deb”,
“build_hash” : “c859302”,
“build_date” : “2021-10-07T22:00:24.085009Z”,
“build_snapshot” : false,
“lucene_version” : “7.7.3”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”

and graylog version 4.2

logfile /var/log/elas…/graylog say
[2021-11-19T22:23:35,163][WARN ][o.e.c.r.a.DiskThresholdMonitor] [peoR6Gc] high disk watermark [90%] exceeded on [peoR6GcRQpqhJZlebPSo5g][peoR6Gc][/var/lib/Elasticsearch/nodes/0] free: 1.4gb[9.3%], shards will be relocated away from this node
..
...
…
[2021-11-20T03:45:38,447][INFO ][o.e.c.r.a.DiskThresholdMonitor] [peoR6Gc] low disk watermark [85%] exceeded on [peoR6GcRQpqhJZlebPSo5g][peoR6Gc][/var/lib/Elasticsearch/nodes/0] free: 2.3gb[14.8%], replicas will not be assigned to this node

/>

2

An index goes into read-only mode when the flood stage is hit for a node.

As you can see :

The index block is automatically released when the disk utilization falls below the high watermark post ES version 7.4

but as your ES version is lower than 7 , you will need to clear the usage first so that it goes below 90% then change the index setting manually for your ES version to fix the issue.

PUT your_index_name/_settings
{
 "index": {
   "blocks": {
     "read_only_allow_delete": "false"
    }
  }
}
1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.