Cluster going read only, trying to understand why

Elastic Stack Elasticsearch
1

Hi,
I have a really simple setup with a cluster of 2 nodes, and a really small number of documents (<20k). But it's the second time my cluster goes to read only mode and I can't understand why.
I've plenty of storage left (way more than the watermark).

Here are the cluster stats:

{
  "_nodes" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "timestamp" : 1650020067217,
  "status" : "green",
  "indices" : {
    "count" : 17,
    "shards" : {
      "total" : 162,
      "primaries" : 81,
      "replication" : 1.0,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 10,
          "avg" : 9.529411764705882
        },
        "primaries" : {
          "min" : 1,
          "max" : 5,
          "avg" : 4.764705882352941
        },
        "replication" : {
          "min" : 1.0,
          "max" : 1.0,
          "avg" : 1.0
        }
      }
    },
    "docs" : {
      "count" : 15521,
      "deleted" : 685
    },
    "store" : {
      "size_in_bytes" : 29705299
    },
    "fielddata" : {
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size_in_bytes" : 0,
      "total_count" : 0,
      "hit_count" : 0,
      "miss_count" : 0,
      "cache_size" : 0,
      "cache_count" : 0,
      "evictions" : 0
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 690,
      "memory_in_bytes" : 1472320,
      "terms_memory_in_bytes" : 810112,
      "stored_fields_memory_in_bytes" : 216528,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 91904,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 353776,
      "index_writer_memory_in_bytes" : 0,
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set_memory_in_bytes" : 3360,
      "max_unsafe_auto_id_timestamp" : -1,
      "file_sizes" : { }
    }
  },
  "nodes" : {
    "count" : {
      "total" : 2,
      "data" : 2,
      "coordinating_only" : 0,
      "master" : 2,
      "ingest" : 2
    },
    "versions" : [ "7.1.1" ],
    "os" : {
      "available_processors" : 2,
      "allocated_processors" : 2,
      "names" : [ {
        "count" : 2
      } ],
      "pretty_names" : [ {
        "count" : 2
      } ],
      "mem" : {
        "total_in_bytes" : 4160036864,
        "free_in_bytes" : 786558976,
        "used_in_bytes" : 3373477888,
        "free_percent" : 19,
        "used_percent" : 81
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 1
      },
      "open_file_descriptors" : {
        "min" : 2006,
        "max" : 2399,
        "avg" : 2202
      }
    },
    "jvm" : {
      "max_uptime_in_millis" : 3274059514,
      "mem" : {
        "heap_used_in_bytes" : 1230129400,
        "heap_max_in_bytes" : 2130051072
      },
      "threads" : 262
    },
    "fs" : {
      "total_in_bytes" : 20869324800,
      "free_in_bytes" : 20743815168,
      "available_in_bytes" : 19636518912
    },
    "network_types" : {
      "transport_types" : {
        "netty4" : 2
      },
      "http_types" : {
        "filter-jetty" : 2
      }
    },
    "discovery_types" : {
      "zen" : 2
    }
  }

Did someone even experience this? Did you find a solution?

Thanks.

2

What exactly do you mean by "goes to read only mode"? As in, what errors or other messages are you seeing that indicate that anything is wrong?

This version is ancient, it went EOL well over a year ago. You should upgrade as a matter of urgency.

3

Hi, when trying to index new documents, I'm getting "forbidden/6/cluster read-only". I had to update the cluster settings to remove the read only mode.

Thanks, will update asap.
Still this is weird.

4

You should check the master node logs the figure out the exact cause for the cluster going to read only mode.

Also in your version of ES , once it goes in read only state , you need to change it manually after reducing the usage . In the newer version this is taken care automatically ( once the usage comes below watermark level).

5

This block is nothing to do with disk space, it means that something external to Elasticsearch deliberately set cluster.blocks.read_only: true. You can clear the block by setting it back to null.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.