# ALLOCATION\_FAILED while draining nodes

**URL:** <https://discuss.elastic.co/t/allocation-failed-while-draining-nodes/322048>\
**Category:** Elasticsearch\
**Created:** [December 27, 2022, 6:24pm UTC](https://discuss.elastic.co/t/allocation-failed-while-draining-nodes/322048 "2022-12-27T18:24:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Reg\_Dancy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reg_dancy/32/98580_2.png) [@Reg\_Dancy](https://discuss.elastic.co/u/Reg_Dancy)\
**Post date:** [December 27, 2022, 6:24pm UTC](https://discuss.elastic.co/t/allocation-failed-while-draining-nodes/322048/1 "2022-12-27T18:24:07Z")

</div>

Hello all, I was draining each node in my cluster one at a time. All of a sudden the draining paused. I received the error below:

$ curl -s -H 'content-type: application/json' -XGET [http://localhost:9200/\_cluster/allocation/explain](http://localhost:9200/_cluster/allocation/explain)  
{"index":"filebeat-buyside-api-lax1-production-2022.12.24","shard":4,"primary":false,"current\_state":"unassigned","unassigned\_info":{"reason":"ALLOCATION\_FAILED","at":"2022-12-27T16:39:11.745Z","failed\_allocation\_attempts":1,"details":"failed shard on node [IvDy1whBQ4KHQqV3bAhcuQ]: shard failure, reason [translog trimming failed], failure NoSuchFileException[/var/lib/elasticsearch/13/nodes/0/indices/fmsf-FvITSWZyg4LS4dcRQ/4/translog/translog-1.tlog]","last\_allocation\_status":"no\_attempt"},"can\_allocate":"yes","allocate\_explanation":"can allocate the shard","target\_node":{"id":"2bDL\_C1aRUO7s\_TafFohUg","name":"localhost-14","transport\_address":"10.1.113.235:9300","attributes":{"xpack.installed":"true"}},"node\_allocation\_decisions":[{"node\_id":"2bDL\_C1aRUO7s\_TafFohUg","node\_name":"localhost","transport\_address":"10.1.11.25:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"yes","weight\_ranking":3},{"node\_id":"QAiO5ycBTsiEEgDwRk8jGA","node\_name":""transport\_address":node2:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"yes","weight\_ranking":4},{"node\_id":"msFULE-uQw2-Sm8IMbqBaw","node\_name":"node3"transport\_address":"10.1.12.21:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"yes","weight\_ranking":5},{"node\_id":"IvDy1whBQ4KHQqV3bAhcuQ","node\_name":"node4"transport\_address":"10.1.11.1:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"throttled","weight\_ranking":2,"deciders":[{"decider":"throttling","decision":"THROTTLE","explanation":"reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node\_concurrent\_incoming\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])"}]},{"node\_id":"U5sOFYZOSxigf3Xi46EnMg","node\_name":"localhost","transport\_address":"10.3.1.4:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"no","weight\_ranking":1,"deciders":[{"decider":"filter","decision":"NO","explanation":"node matches cluster setting [cluster.routing.allocation.exclude] filters [\_ip:"10.3.1.5"]"}]},{"node\_id":"cj0Xy-88SMCrdTSJaMnhEQ","node\_name":"node6","transport\_address":"10.1.1.6:9300","node\_attributes":{"xpack.installed":"true"},"node\_decision":"no","weight\_ranking":6,"deciders":[{"decider":"same\_shard","decision":"NO","explanation":"the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-buyside-api-lax1-production-2022.12.24][4], node[cj0Xy-88SMCrdTSJaMnhEQ], [P], s[STARTED], a[id=XSMacJ01QCqU2h7W\_IdqwA]]"}]}]}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 24, 2023, 6:24pm UTC](https://discuss.elastic.co/t/allocation-failed-while-draining-nodes/322048/2 "2023-01-24T18:24:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.