ElasticSearch Service: 2 of 15 shards failed

[AndresL (Andres)]

Hi,
In a ESS cluster, from kibana dashboard i can see the error "
2 of 15 shards failed

how can i debuig this problem? i have been checking in system logs like .monitoring or .kibana, and i cant find the reason.
Version is 7.3

[Alex_Piggott]

I'm moving this to the Elasticsearch forum since it doesn't seem cloud specific - note that Elasticsearch Service doesn't have a discuss forum, instead we ask that people request support here: https://cloud.elastic.co/help)

I would suggest _cat/shards and _cluster/allocation/explain as 2 good APIs to provide information about why shards are not allocating properly. The elasticsearch logs (accessible from ES) are also useful

[AndresL (Andres)]

Hi Alex,
using _cat/shards i can see all the index in status "Started" (system index, filebeat, etc).

Using _cluster/allocation/explain for one specific index, i got this answer:

Summary

"index" : "filebeat-development-7.3.0-2019.10.28",
"shard" : 0,
"primary" : true,
"current_state" : "started",
"current_node" : {
"id" : "roBcUoF0SJGixpfEwq2PrQ",
"name" : "instance-0000000001",
"transport_address" : "10.46.24.5:19095",
"attributes" : {
"logical_availability_zone" : "zone-1",
"server_name" : "instance-0000000001.509b35a803234a588f5da5e49f9a1e3c",
"availability_zone" : "westeurope-1",
"xpack.installed" : "true",
"region" : "unknown-region",
"instance_configuration" : "azure.data.highio.l32sv2"
},
"weight_ranking" : 3
},
"can_remain_on_current_node" : "yes",
"can_rebalance_cluster" : "yes",
"can_rebalance_to_other_node" : "no",
"rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance",
"node_allocation_decisions" : [
{
"node_id" : "KWBh3OBrQWWKWAGSmnU8cg",
"node_name" : "instance-0000000005",
"transport_address" : "10.46.24.45:19337",
"node_attributes" : {
"logical_availability_zone" : "zone-1",
"server_name" : "instance-0000000005.509b35a803234a588f5da5e49f9a1e3c",
"availability_zone" : "westeurope-1",
"xpack.installed" : "true",
"region" : "unknown-region",
"instance_configuration" : "azure.data.highstorage.e16sv3"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "filter",
"decision" : "NO",
"explanation" : """node does not match index setting [index.routing.allocation.include] filters [instance_configuration:"azure.data.highio.l32sv2"]"""
}
]
},
{
"node_id" : "8eUTEMX_Q0aoHJ1nyQfKJQ",
"node_name" : "instance-0000000006",
"transport_address" : "10.46.24.52:19824",
"node_attributes" : {
"logical_availability_zone" : "zone-0",
"server_name" : "instance-0000000006.509b35a803234a588f5da5e49f9a1e3c",
"availability_zone" : "westeurope-3",
"xpack.installed" : "true",
"region" : "unknown-region",
"instance_configuration" : "azure.data.highstorage.e16sv3"
},
"node_decision" : "no",
"weight_ranking" : 2,
"deciders" : [
{
"decider" : "filter",
"decision" : "NO",
"explanation" : """node does not match index setting [index.routing.allocation.include] filters [instance_configuration:"azure.data.highio.l32sv2"]"""
}
]
},
{
"node_id" : "O_1rItnVQW-sbRsjWmRdFw",
"node_name" : "instance-0000000007",
"transport_address" : "10.46.24.17:19878",
"node_attributes" : {
"logical_availability_zone" : "zone-0",
"server_name" : "instance-0000000007.509b35a803234a588f5da5e49f9a1e3c",
"availability_zone" : "westeurope-3",
"xpack.installed" : "true",
"region" : "unknown-region",
"instance_configuration" : "azure.data.highio.l32sv2"
},
"node_decision" : "no",
"weight_ranking" : 3,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-development-7.3.0-2019.10.28][0], node[O_1rItnVQW-sbRsjWmRdFw], [R], s[STARTED], a[id=BdNfo8jrSO-CqqbUHGzG7Q]]"
}
]
}
]
}