Hello,
We have a ELK cluster setup with 35 nodes (3 masters and 32 data nodes).
New indices get created everyday once.
Approximately, 280 indices are created everyday.
Number of shards per index is 24 with replication factor of 1.
The disk size of each node is 15TB and RAM is 125GB.
The problem is that, everyday after the index creation, the cluster goes to yellow state with too many unassigned indices and takes a lot of time to recover. Following is the state of the cluster after 4.5 hours of new index creation.
{
"cluster_name" : <cluster_name>,
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 35,
"number_of_data_nodes" : 32,
"active_primary_shards" : 48589,
"active_shards" : 89434,
"relocating_shards" : 0,
"initializing_shards" : 30,
"unassigned_shards" : 7760,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1936,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 2036819,
"active_shards_percent_as_number" : 91.98757508434132
}
The reasons for shards being unassigned are the following:
INDEX_CREATED - on newly created indices
NODE_LEFT - on older indices
The following is the output from _cat/allocation/explain api
.
.
.
.
{
"node_id" : "XZAy1ITXSjmO_a-8yulfig",
"node_name" : "node1",
"transport_address" : ":9300",
"node_attributes" : {
"ml.max_open_jobs" : "10",
"ml.enabled" : "true"
},
"node_decision" : "throttled",
"deciders" : [
{
"decider" : "throttling",
"decision" : "THROTTLE",
"explanation" : "reached the limit of incoming shard recoveries [10], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=10] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
}
]
},
{
"node_id" : "dnB6TExcSOaI_poL3d3t9g",
"node_name" : "node2",
"transport_address" : ":9300",
"node_attributes" : {
"ml.max_open_jobs" : "10",
"ml.enabled" : "true"
},
"node_decision" : "throttled",
"store" : {
"matching_sync_id" : true
},
"deciders" : [
{
"decider" : "throttling",
"decision" : "THROTTLE",
"explanation" : "reached the limit of incoming shard recoveries [10], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=10] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
}
]
},
{
"node_id" : "BrIKoXg4TkuJ2Ekn8YKuPQ",
"node_name" : "node3",
"transport_address" : ":9300",
"node_attributes" : {
"ml.max_open_jobs" : "10",
"ml.enabled" : "true"
},
"node_decision" : "no",
"store" : {
"matching_sync_id" : true
},
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-hadoop_axonitered-hdfsproxy-audit-s3-2017.12.15][18], node[BrIKoXg4TkuJ2Ekn8YKuPQ], [P], s[STARTED], a[id=OzSdAIySQFWKj7PuzUHbjQ]]"
}
]
}
We don't want the cluster to go into yellow state everyday where thousands of shards go unassigned and takes time to recover. We want the index creation, shard distribution of both primary and replica to be smooth.Would increasing the number of shards per index help. If yes, by how much?
Also, what does NODE_LEFT mean? I do not see any services getting restarted on nodes.
