Hello all, I was draining each node in my cluster one at a time. All of a sudden the draining paused. I received the error below:
$ curl -s -H 'content-type: application/json' -XGET http://localhost:9200/_cluster/allocation/explain
{"index":"filebeat-buyside-api-lax1-production-2022.12.24","shard":4,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"ALLOCATION_FAILED","at":"2022-12-27T16:39:11.745Z","failed_allocation_attempts":1,"details":"failed shard on node [IvDy1whBQ4KHQqV3bAhcuQ]: shard failure, reason [translog trimming failed], failure NoSuchFileException[/var/lib/elasticsearch/13/nodes/0/indices/fmsf-FvITSWZyg4LS4dcRQ/4/translog/translog-1.tlog]","last_allocation_status":"no_attempt"},"can_allocate":"yes","allocate_explanation":"can allocate the shard","target_node":{"id":"2bDL_C1aRUO7s_TafFohUg","name":"localhost-14","transport_address":"10.1.113.235:9300","attributes":{"xpack.installed":"true"}},"node_allocation_decisions":[{"node_id":"2bDL_C1aRUO7s_TafFohUg","node_name":"localhost","transport_address":"10.1.11.25:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"yes","weight_ranking":3},{"node_id":"QAiO5ycBTsiEEgDwRk8jGA","node_name":""transport_address":node2:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"yes","weight_ranking":4},{"node_id":"msFULE-uQw2-Sm8IMbqBaw","node_name":"node3"transport_address":"10.1.12.21:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"yes","weight_ranking":5},{"node_id":"IvDy1whBQ4KHQqV3bAhcuQ","node_name":"node4"transport_address":"10.1.11.1:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"throttled","weight_ranking":2,"deciders":[{"decider":"throttling","decision":"THROTTLE","explanation":"reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"}]},{"node_id":"U5sOFYZOSxigf3Xi46EnMg","node_name":"localhost","transport_address":"10.3.1.4:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"no","weight_ranking":1,"deciders":[{"decider":"filter","decision":"NO","explanation":"node matches cluster setting [cluster.routing.allocation.exclude] filters [_ip:"10.3.1.5"]"}]},{"node_id":"cj0Xy-88SMCrdTSJaMnhEQ","node_name":"node6","transport_address":"10.1.1.6:9300","node_attributes":{"xpack.installed":"true"},"node_decision":"no","weight_ranking":6,"deciders":[{"decider":"same_shard","decision":"NO","explanation":"the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-buyside-api-lax1-production-2022.12.24][4], node[cj0Xy-88SMCrdTSJaMnhEQ], [P], s[STARTED], a[id=XSMacJ01QCqU2h7W_IdqwA]]"}]}]}