Hi,
Day by day i am facing red status on the Elasticsearch due to some indices that are also red.
On my ELK we have 7 nodes and 44 primary, 44 replica shards.
I am keeping 7 indices for each beat (filebeat - winloagbeat - metricbeat - heartbeat). The format is filebeat-(date) so every day is creating a new index for each beat. So 7xfilebeat indices, 7xmetricbeat indices, 7xwinlogbeat indices, 7xheartbeat indices.
The problem is that some days some indices of them are red..they don't have data or disk space and the only way to make my ELK green again is to delete them. Then the deleted indices are creating again and my ELK become from red to yellow and the green.
I am facing this issue day by day. Here is also the log:
GET /_cluster/allocation/explain?pretty
{
"index" : "metricbeat-2019.12.23",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2019-12-23T19:14:11.090Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [g9M0MIndTBW62pSrEdktzA]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[metricbeat-2019.12.23][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
"node_allocation_decisions" : [
{
"node_id" : "46jaKQPSRC2drwy-bAfhxw",
"node_name" : "xh-gr-elastic-2",
"transport_address" : "10.159.166.9:9300",
"node_attributes" : {
"ml.machine_memory" : "269930561536",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : false,
"allocation_id" : "VOlwBoGhRo6LcB-sSytWSg"
}
},
{
"node_id" : "CBiU9s_BSai2oBfVeZjwiQ",
"node_name" : "xh-fr-elastic-1",
"transport_address" : "135.238.239.48:9300",
"node_attributes" : {
"ml.machine_memory" : "16654974976",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "EmyFlH3zS3KkvJptV02pMQ",
"node_name" : "xh-gr-elastic-1",
"transport_address" : "10.158.67.175:9300",
"node_attributes" : {
"ml.machine_memory" : "16654729216",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "WNfbhnzgTIi4y3PWM4Yg0A",
"node_name" : "xh-it-elastic-2",
"transport_address" : "151.98.17.61:9300",
"node_attributes" : {
"ml.machine_memory" : "269931491328",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "bLV-av6fSAWBTep4FKAqWA",
"node_name" : "xh-gr-elastic-3",
"transport_address" : "10.158.67.107:9300",
"node_attributes" : {
"ml.machine_memory" : "17179332608",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"found" : false
}
},
{
"node_id" : "g9M0MIndTBW62pSrEdktzA",
"node_name" : "xh-it-elastic-1",
"transport_address" : "151.98.17.60:9300",
"node_attributes" : {
"ml.machine_memory" : "34359738368",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "woCBGp9PQ8SvxXttV_x8qg"
},
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-12-23T19:14:11.090Z], failed_attempts[5], failed_nodes[[g9M0MIndTBW62pSrEdktzA]], delayed=false, details[failed shard on node [g9M0MIndTBW62pSrEdktzA]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[metricbeat-2019.12.23][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], allocation_status[deciders_no]]]"
}
]
},
{
"node_id" : "obT4-IMdTU6kOAHKVFNxzA",
"node_name" : "xh-fr-elastic-2",
"transport_address" : "135.238.239.132:9300",
"node_attributes" : {
"ml.machine_memory" : "269930713088",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"store" : {
"in_sync" : false,
"allocation_id" : "6bNctu7wQnmtQYml4SICPg",
"store_exception" : {
"type" : "file_not_found_exception",
"reason" : "no segments* file found in SimpleFSDirectory@/data/elasticsearch/nodes/0/indices/1MkRLbVgSPGEy2LkH0AK1g/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3ec3d7e7: files: [write.lock]"
}
}
}
]
}
Merry Christmas
Best Regards,
Thanos
