Hi, I'm having a single node cluster architecture where my ELK is in a server (say servA) and Filebeat is in remote server.
there are 2 remote servers say
filebeat-A(fA) in serverB(sB) and filebeat-B(fB) in serverC(sC)
these two filebeats pushes respective logs to my serverA..
it was working fine untill filebeatA pushing logs to ELK of servA..(running fine for 10+ days)
as soon as I pushed logs from filebeatB with include_lines:['regex'] (this settings is not in filebeatA), every index turns RED..
then I stopped both fA and fB.. and checked index & cluster health.. everything become red.. I dont know whether that settings had the impact or any other problem
I tried GET /_cluster/health/mdcp_contact
result is
> {
"index": "mdcp_contact", "shard": 3, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "ALLOCATION_FAILED", "at": "2019-11-07T12:30:02.711Z", "failed_allocation_attempts": 5, "details": "failed shard on node [Bewr4jriQziexcfUXZSfdg]: failed recovery, failure RecoveryFailedException[[mdcp_contact][3]: Recovery failed on {Bewr4jr}{Bewr4jriQziexcfUXZSfdg}{Rqi93PhcTnaC4gegILxBww}{ip}{ip:9300}{ml.machine_memory=67368890368, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: FileSystemException[/opt/jboss/elk/elasticsearch-6.4.2/data/nodes/0/indices/60sjgq9vSZKv7CbTcUi6_Q/3/index/_5wu.nvd: Too many open files]; ", "last_allocation_status": "no" }, "can_allocate": "no", "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy", "node_allocation_decisions": [ { "node_id": "Bewr4jriQziexcfUXZSfdg", "node_name": "Bewr4jr", "transport_address": "ip:9300", "node_attributes": { "ml.machine_memory": "67368890368", "xpack.installed": "true", "ml.max_open_jobs": "20", "ml.enabled": "true" }, "node_decision": "no", "store": { "in_sync": true, "allocation_id": "U4BmrwmxTNKGPPMqCMKEpA" }, "deciders": [ { "decider": "max_retry", "decision": "NO", "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-07T12:30:02.711Z], failed_attempts[5], delayed=false, details[failed shard on node [Bewr4jriQziexcfUXZSfdg]: failed recovery, failure RecoveryFailedException[[mdcp_contact][3]: Recovery failed on {Bewr4jr}{Bewr4jriQziexcfUXZSfdg}{Rqi93PhcTnaC4gegILxBww}{ip}{ip:9300}{ml.machine_memory=67368890368, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: FileSystemException[/opt/jboss/elk/elasticsearch-6.4.2/data/nodes/0/indices/60sjgq9vSZKv7CbTcUi6_Q/3/index/_5wu.nvd: Too many open files]; ], allocation_status[deciders_no]]]" } ] } ] }
So unassigned shards and tried reallocating and rerouting apis.. no help
Any suggestions or workaround?
