# Elasticsearch 7.10.1 new test cluster - cannot bring up - state not recovered

**URL:** <https://discuss.elastic.co/t/elasticsearch-7-10-1-new-test-cluster-cannot-bring-up-state-not-recovered/259520>\
**Category:** Elasticsearch\
**Created:** [December 23, 2020, 6:08pm UTC](https://discuss.elastic.co/t/elasticsearch-7-10-1-new-test-cluster-cannot-bring-up-state-not-recovered/259520 "2020-12-23T18:08:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lakshmi-github](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshmi-github/32/81281_2.png) [@lakshmi-github](https://discuss.elastic.co/u/lakshmi-github)\
**Post date:** [December 23, 2020, 6:08pm UTC](https://discuss.elastic.co/t/elasticsearch-7-10-1-new-test-cluster-cannot-bring-up-state-not-recovered/259520/1 "2020-12-23T18:08:15Z")

</div>

Hello,

I am trying to bring up a new ES 7.10.1 cluster for testing as follows:

- node1 - master (node.roles: ['master'])
- node2 - master (node.roles: ['master'])
- node3 - master (node.roles: ['master'])
- node4 - data (node.roles: ['data'])

These are independent docker containers that I brought up in this order:

1. start node1
2. start node2
3. start node3
4. start node4

Next I try setting elastic user's password as follows:

1. create a temp local file-based super user on node1 via: `/usr/share/elasticsearch/bin/elasticsearch-users useradd elastic-tmp -p password -r superuser`
2. use this temp super user to set elastic user's password via `PUT /_xpack/security/user/elastic/_password`

However when I execute step 2, it fails:

```auto
[node1 ~]# curl -k -uelastic-tmp:welcome1 -XPUT https://localhost:9201/_xpack/security/user/elastic/_password?pretty -H 'Content-Type: application/json' -d'{"password": "password" }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "status_exception",
        "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index"
      }
    ],
    "type" : "status_exception",
    "reason" : "Cluster state has not been recovered yet, cannot write to the [null] index"
  },
  "status" : 503
}

```

I can list nodes, view the current elected master, via `/_cat/nodes?v`

```auto
[node1 ~]# curl -sk -uelastic-tmp:welcome1 -XGET https://localhost:9201/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
<ip1> 2 7 0 0.03 0.07 0.07 m * node1-master
<ip2> 2 4 0 0.12 0.13 0.09 m - node2-master
<ip3> 2 7 0 0.03 0.04 0.01 d - node4-data1
<ip4> 1 7 0 0.03 0.04 0.01 m - node3-master

```

but can not view indexes or shards - these fail with "cluster block exception, service unavailable, state not recovered" errors, which is likely why the `PUT` fails.

```auto
[node1 ~]# curl -sk -uelastic-tmp:welcome1 -XGET https://localhost:9201/_cat/indices?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_block_exception",
        "reason" : "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : "ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
    "caused_by" : {
      "type" : "cluster_block_exception",
      "reason" : "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
    }
  },
  "status" : 503
}

```

The cluster health is red:

```auto
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1608676569 22:36:09 es7-test-cluster red 4 1 0 0 0 0 0 0 - NaN%

```

Any help would be appreciated.

Thanks.

---

<div class="post-metadata">

**Author:** ![lakshmi-github](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshmi-github/32/81281_2.png) [@lakshmi-github](https://discuss.elastic.co/u/lakshmi-github)\
**Post date:** [December 26, 2020, 11:36pm UTC](https://discuss.elastic.co/t/elasticsearch-7-10-1-new-test-cluster-cannot-bring-up-state-not-recovered/259520/2 "2020-12-26T23:36:53Z")

</div>

Resolved.

This property: `gateway.recover_after_data_nodes: 3` apparently caused the "Cluster state has not been recovered yet" exceptions since there was only one data node whereas it was waiting for 3 data nodes.

Once I started 2 more data nodes for a total of 3, everything worked.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 23, 2021, 11:37pm UTC](https://discuss.elastic.co/t/elasticsearch-7-10-1-new-test-cluster-cannot-bring-up-state-not-recovered/259520/3 "2021-01-23T23:37:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.