ELK Red status

Elastic Stack Elasticsearch
1

Hi,

Every day i am facing an issue on my ELK cluster with red status. Some Indexes everyday are red so i am deleting them and they are recreating and after this the ELK Status is becoming green again.

Here is the log of
GET /_cluster/allocation/explain?pretty

{
"index" : "metricbeat-2019.12.31",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "PRIMARY_FAILED",
"at" : "2020-01-22T08:28:18.081Z",
"details" : "primary failed while replica initializing",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "46jaKQPSRC2drwy-bAfhxw",
"node_name" : "xh-gr-elastic-2",
"transport_address" : "10.159.166.9:9300",
"node_attributes" : {
"ml.machine_memory" : "269930561536",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},
{
"node_id" : "CBiU9s_BSai2oBfVeZjwiQ",
"node_name" : "xh-fr-elastic-1",
"transport_address" : "135.238.239.48:9300",
"node_attributes" : {
"ml.machine_memory" : "16654974976",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},
{
"node_id" : "EmyFlH3zS3KkvJptV02pMQ",
"node_name" : "xh-gr-elastic-1",
"transport_address" : "10.158.67.175:9300",
"node_attributes" : {
"ml.machine_memory" : "16654729216",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},
{
"node_id" : "WNfbhnzgTIi4y3PWM4Yg0A",
"node_name" : "xh-it-elastic-2",
"transport_address" : "151.98.17.61:9300",
"node_attributes" : {
"ml.machine_memory" : "269939691520",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},

Do you have any idea what is wrong??

Best Regards,
Thanos
"node_id" : "bLV-av6fSAWBTep4FKAqWA",
"node_name" : "xh-gr-elastic-3",
"transport_address" : "10.158.67.107:9300",
"node_attributes" : {
"ml.machine_memory" : "17179332608",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},
{
"node_id" : "g9M0MIndTBW62pSrEdktzA",
"node_name" : "xh-it-elastic-1",
"transport_address" : "151.98.17.60:9300",
"node_attributes" : {
"ml.machine_memory" : "34359738368",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
},
{
"node_id" : "obT4-IMdTU6kOAHKVFNxzA",
"node_name" : "xh-fr-elastic-2",
"transport_address" : "135.238.239.132:9300",
"node_attributes" : {
"ml.machine_memory" : "269930713088",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "replica_after_primary_active",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
},
{
"decider" : "throttling",
"decision" : "NO",
"explanation" : "primary shard for this replica is not yet active"
}
]
}
]
}

2

Here is a blog post about explaining allocation:

Without further instructions, the allocation explain API tries to explain a random unassigned shard, and it's picking a replica. In your case you want to understand why a specific shard is assigned (i.e. the corresponding primary) so you need to ask it a more specific question:

GET /_cluster/allocation/explain 
{ 
   "index": "metricbeat-2019.12.31", 
   "shard": 0, 
   "primary": true
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.