Thanks for the tips! Here's the explanation output:
First server is the one having no data, second & third - each having the full dataset.
[rihad@eldey ~]$ curl -X GET "amiata.local:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
"index": "my-index",
"shard": 0,
"primary": false,
"current_node": "amiata.example.com"
}
'
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unable to find a replica shard assigned to node [amiata.example.com]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unable to find a replica shard assigned to node [amiata.example.com]"
},
"status" : 400
}
[rihad@eldey ~]$ curl -X GET "eldey.local:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
"index": "my-index",
"shard": 0,
"primary": false,
"current_node": "eldey.example.com"
}
'
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unable to find a replica shard assigned to node [eldey.example.com]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unable to find a replica shard assigned to node [eldey.example.com]"
},
"status" : 400
}
[rihad@eldey ~]$ curl -X GET "pico.local:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
"index": "my-index",
"shard": 0,
"primary": false,
"current_node": "pico.example.com"
}
'
{
"index" : "my-index",
"shard" : 0,
"primary" : false,
"current_state" : "started",
"current_node" : {
"id" : "qj963ExLRI-bceFokseNTQ",
"name" : "pico.example.com",
"transport_address" : "172.16.1.11:9300",
"attributes" : {
"xpack.installed" : "true",
"transform.node" : "true"
},
"weight_ranking" : 1
},
"can_remain_on_current_node" : "yes",
"can_rebalance_cluster" : "yes",
"can_rebalance_to_other_node" : "no",
"rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
"node_allocation_decisions" : [
{
"node_id" : "jxM3gIeDQAu8ex9ghdjshg",
"node_name" : "eldey.example.com",
"transport_address" : "172.16.1.8:9300",
"node_attributes" : {
"xpack.installed" : "true",
"transform.node" : "true"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[my-index][0], node[jxM3gIeDQAu8ex9ghdjshg], [P], s[STARTED], a[id=gh3V_k79Rg6KW2OP02ijTw]]"
}
]
},
{
"node_id" : "pXeE9Ij_T-64jFUDMhJ34w",
"node_name" : "amiata.example.com",
"transport_address" : "172.16.1.6:9300",
"node_attributes" : {
"xpack.installed" : "true",
"transform.node" : "true"
},
"node_decision" : "worse_balance",
"weight_ranking" : 1
}
]
}
Allocation explain really is a useful API in this context. It's unlikely to relate to