Cluster RED, unassigned shards

Elastic Stack Elasticsearch
1

After complete cluster restart have all the indices shards unassigned.

662 indices
820 shards in summary
3 master nodes
3 data nodes

Master and data nodes are up, and it says "can_allocate" : "yes", however shards stay UNASSIGNED.

http://cluster:9200/_cluster/allocation/explain?pretty

{
  "index" : "INDEX-NAME-2020.09.11-000101",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2020-09-17T08:16:44.145Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "yes",
  "allocate_explanation" : "can allocate the shard",
  "target_node" : {
    "id" : "ByPaeoenTvOjuF74CDn2Kw",
    "name" : "NNODE02",
    "transport_address" : "SNIP:9300",
    "attributes" : {
      "ml.machine_memory" : "33567047680",
      "ml.max_open_jobs" : "20",
      "xpack.installed" : "true",
      "transform.node" : "true"
    }
  },
  "allocation_id" : "_3xOIvAKSAamMBOblElQqw",
  "node_allocation_decisions" : [
    {
      "node_id" : "ByPaeoenTvOjuF74CDn2Kw",
      "node_name" : "NODE01",
      "transport_address" : "SNIP:9300",
      "node_attributes" : {
        "ml.machine_memory" : "33567047680",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "yes",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "_3xOIvAKSAamMBOblElQqw"
      }
    },
    {
      "node_id" : "YAL9edThQwKT5DntaYNcow",
      "node_name" : "NODE03",
      "transport_address" : "SNIP:9300",
      "node_attributes" : {
        "ml.machine_memory" : "33567047680",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "ZsEiXQ0jSgu6GHWFg17KPQ",
      "node_name" : "NODE03",
      "transport_address" : "SNIP:9300",
      "node_attributes" : {
        "ml.machine_memory" : "33567039488",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    }
  ]
}

All shards are unassigned (just a snip below):
http://cluster:9200/_cat/shards

INDEX-NAME-1-2020.09.14-000005           0 p UNASSIGNED    
INDEX-NAME-2-2020.09.10-000059 0 p UNASSIGNED    
INDEX-NAME-3-2020.08.22-000126      0 p UNASSIGNED

And stores are allocated but not used (just a snip again):
http://elasticsearch.hq.trayport.com:9200/_shard_stores?pretty

{
  "indices" : {
    "INDEX-NAME-1-2020.07.15-000088" : {
      "shards" : {
        "0" : {
          "stores" : [
            {
              "ByPaeoenTvOjuF74CDn2Kw" : {
                "name" : "NODE02.hq.trayport.com",
                "ephemeral_id" : "rySOjmTgTe-myLmCmq0ZaQ",
                "transport_address" : "SNIP:9300",
                "attributes" : {
                  "ml.machine_memory" : "33567047680",
                  "ml.max_open_jobs" : "20",
                  "xpack.installed" : "true",
                  "transform.node" : "true"
                }
              },
              "allocation_id" : "t1EuiFuqQh6I0A0KfgVl6Q",
              "allocation" : "unused"
            }
          ]
        }
      }
    },
2

This issue has been resolved after doing full custer restart three times :frowning:

3

What version are you on?

4

7.8.0

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.