After complete cluster restart have all the indices shards unassigned.
662 indices
820 shards in summary
3 master nodes
3 data nodes
Master and data nodes are up, and it says "can_allocate" : "yes", however shards stay UNASSIGNED.
http://cluster:9200/_cluster/allocation/explain?pretty
{
"index" : "INDEX-NAME-2020.09.11-000101",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2020-09-17T08:16:44.145Z",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "yes",
"allocate_explanation" : "can allocate the shard",
"target_node" : {
"id" : "ByPaeoenTvOjuF74CDn2Kw",
"name" : "NNODE02",
"transport_address" : "SNIP:9300",
"attributes" : {
"ml.machine_memory" : "33567047680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"transform.node" : "true"
}
},
"allocation_id" : "_3xOIvAKSAamMBOblElQqw",
"node_allocation_decisions" : [
{
"node_id" : "ByPaeoenTvOjuF74CDn2Kw",
"node_name" : "NODE01",
"transport_address" : "SNIP:9300",
"node_attributes" : {
"ml.machine_memory" : "33567047680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"transform.node" : "true"
},
"node_decision" : "yes",
"store" : {
"in_sync" : true,
"allocation_id" : "_3xOIvAKSAamMBOblElQqw"
}
},
{
"node_id" : "YAL9edThQwKT5DntaYNcow",
"node_name" : "NODE03",
"transport_address" : "SNIP:9300",
"node_attributes" : {
"ml.machine_memory" : "33567047680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"transform.node" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "ZsEiXQ0jSgu6GHWFg17KPQ",
"node_name" : "NODE03",
"transport_address" : "SNIP:9300",
"node_attributes" : {
"ml.machine_memory" : "33567039488",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"transform.node" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
}
]
}
All shards are unassigned (just a snip below):
http://cluster:9200/_cat/shards
INDEX-NAME-1-2020.09.14-000005 0 p UNASSIGNED
INDEX-NAME-2-2020.09.10-000059 0 p UNASSIGNED
INDEX-NAME-3-2020.08.22-000126 0 p UNASSIGNED
And stores are allocated but not used (just a snip again):
http://elasticsearch.hq.trayport.com:9200/_shard_stores?pretty
{
"indices" : {
"INDEX-NAME-1-2020.07.15-000088" : {
"shards" : {
"0" : {
"stores" : [
{
"ByPaeoenTvOjuF74CDn2Kw" : {
"name" : "NODE02.hq.trayport.com",
"ephemeral_id" : "rySOjmTgTe-myLmCmq0ZaQ",
"transport_address" : "SNIP:9300",
"attributes" : {
"ml.machine_memory" : "33567047680",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"transform.node" : "true"
}
},
"allocation_id" : "t1EuiFuqQh6I0A0KfgVl6Q",
"allocation" : "unused"
}
]
}
}
},