# Cluster going read only, trying to understand why

**URL:** <https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502>\
**Category:** Elasticsearch\
**Created:** [April 15, 2022, 11:24am UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502 "2022-04-15T11:24:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![graphman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/graphman/32/104423_2.png) [@graphman](https://discuss.elastic.co/u/graphman)\
**Post date:** [April 15, 2022, 11:24am UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/1 "2022-04-15T11:24:09Z")

</div>

Hi,  
I have a really simple setup with a cluster of 2 nodes, and a really small number of documents (\<20k). But it's the second time my cluster goes to read only mode and I can't understand why.  
I've plenty of storage left (way more than the watermark).

Here are the cluster stats:

```auto
{
  "_nodes" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "timestamp" : 1650020067217,
  "status" : "green",
  "indices" : {
    "count" : 17,
    "shards" : {
      "total" : 162,
      "primaries" : 81,
      "replication" : 1.0,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 10,
          "avg" : 9.529411764705882
        },
        "primaries" : {
          "min" : 1,
          "max" : 5,
          "avg" : 4.764705882352941
        },
        "replication" : {
          "min" : 1.0,
          "max" : 1.0,
          "avg" : 1.0
        }
      }
    },
    "docs" : {
      "count" : 15521,
      "deleted" : 685
    },
    "store" : {
      "size_in_bytes" : 29705299
    },
    "fielddata" : {
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size_in_bytes" : 0,
      "total_count" : 0,
      "hit_count" : 0,
      "miss_count" : 0,
      "cache_size" : 0,
      "cache_count" : 0,
      "evictions" : 0
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 690,
      "memory_in_bytes" : 1472320,
      "terms_memory_in_bytes" : 810112,
      "stored_fields_memory_in_bytes" : 216528,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 91904,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 353776,
      "index_writer_memory_in_bytes" : 0,
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set_memory_in_bytes" : 3360,
      "max_unsafe_auto_id_timestamp" : -1,
      "file_sizes" : { }
    }
  },
  "nodes" : {
    "count" : {
      "total" : 2,
      "data" : 2,
      "coordinating_only" : 0,
      "master" : 2,
      "ingest" : 2
    },
    "versions" : ["7.1.1"],
    "os" : {
      "available_processors" : 2,
      "allocated_processors" : 2,
      "names" : [ {
        "count" : 2
      } ],
      "pretty_names" : [ {
        "count" : 2
      } ],
      "mem" : {
        "total_in_bytes" : 4160036864,
        "free_in_bytes" : 786558976,
        "used_in_bytes" : 3373477888,
        "free_percent" : 19,
        "used_percent" : 81
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 1
      },
      "open_file_descriptors" : {
        "min" : 2006,
        "max" : 2399,
        "avg" : 2202
      }
    },
    "jvm" : {
      "max_uptime_in_millis" : 3274059514,
      "mem" : {
        "heap_used_in_bytes" : 1230129400,
        "heap_max_in_bytes" : 2130051072
      },
      "threads" : 262
    },
    "fs" : {
      "total_in_bytes" : 20869324800,
      "free_in_bytes" : 20743815168,
      "available_in_bytes" : 19636518912
    },
    "network_types" : {
      "transport_types" : {
        "netty4" : 2
      },
      "http_types" : {
        "filter-jetty" : 2
      }
    },
    "discovery_types" : {
      "zen" : 2
    }
  }

```

Did someone even experience this? Did you find a solution?

Thanks.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 15, 2022, 11:34am UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/2 "2022-04-15T11:34:32Z")

</div>

What exactly do you mean by "goes to read only mode"? As in, what errors or other messages are you seeing that indicate that anything is wrong?

> [@graphman](#):
>
> ```auto
> "versions" : ["7.1.1"],
> 
> ```

This version is ancient, it went [EOL](https://www.elastic.co/support/eol) well over a year ago. You should upgrade as a matter of urgency.

---

<div class="post-metadata">

**Author:** ![graphman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/graphman/32/104423_2.png) [@graphman](https://discuss.elastic.co/u/graphman)\
**Post date:** [April 15, 2022, 2:12pm UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/3 "2022-04-15T14:12:12Z")

</div>

Hi, when trying to index new documents, I'm getting "forbidden/6/cluster read-only". I had to update the cluster settings to remove the read only mode.

Thanks, will update asap.  
Still this is weird.

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [April 15, 2022, 8:22pm UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/4 "2022-04-15T20:22:13Z")

</div>

You should check the master node logs the figure out the exact cause for the cluster going to read only mode.

Also in your version of ES , once it goes in read only state , you need to change it manually after reducing the usage . In the newer version this is taken care automatically ( once the usage comes below watermark level).

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 16, 2022, 5:40am UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/5 "2022-04-16T05:40:10Z")

</div>

> [@graphman](#):
>
> I'm getting "forbidden/6/cluster read-only"

This block is nothing to do with disk space, it means that something external to Elasticsearch deliberately set `cluster.blocks.read_only: true`. You can clear the block by setting it back to `null`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2022, 5:40am UTC](https://discuss.elastic.co/t/cluster-going-read-only-trying-to-understand-why/302502/6 "2022-05-14T05:40:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.