### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 1, 2019, 8:10am UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/2 "2019-03-01T08:10:17Z")
Please don't post images of text, it's impossible to search them or quote them in responses and they can't be read at all by those of us that use screenreaders. Use the `` button to format code-like text more neatly instead.
How many shards do you currently have in this cluster?
> [@redcho](#):
>
> It says no on max\_retry condition but shouldn't it be yes?
The retry counter is reset _after_ trying to perform any commands. TIL. I opened an issue, and documented a workaround here:
>
>
> We respect allocation deciders, including the \`MaxRetryAllocationDecider\`, when …executing reroute commands. If you specify \`?retry\_failed=true\` then the retry counter is reset, but not until after trying to execute the reroute commands. This means that if an allocation has repeatedly failed, but you want to take control and assign a shard to a particular node to work around the repeated failures, you cannot just do this:
>
> \`\`\`
> POST /\_cluster/reroute?retry\_failed=true
> {
> "commands": \[
> {
> "allocate\_replica": {
> "index": "blahblah",
> "shard": 2,
> "node": "node-4"
> }
> }
> \]
> }
> \`\`\`
>
> ~The command is rejected by the \`MaxRetryAllocationDecider\`, then the retry counter is reset, then Elasticsearch tries in vain to allocate the shard to a node of its choosing, hitting the retry limit again.~ Correction: The command is rejected by the \`MaxRetryAllocationDecider\` which stops the whole reroute process.
>
> The workaround today is:
>
> \- disable allocation
> \- reset the retry counter with a bare \`POST /\_cluster/reroute?retry\_failed=true\`
> \- execute the desired reroute command with another call to \`POST /\_cluster/reroute\`
> \- re-enable allocation
>
> I think we should reset the retry counter before executing the reroute commands here:
>
> https://github.com/elastic/elasticsearch/blob/4ca241b78a80f92d5443caf6c943243b166515f5/server/src/main/java/org/elasticsearch/cluster/routing/allocation/AllocationService.java#L338-L348
>
> cf https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396
---
### Author:  [@redcho](https://discuss.elastic.co/u/redcho)
#### Post date: [March 2, 2019, 12:52pm UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/3 "2019-03-02T12:52:56Z")
Thank you for your reply. I will edit my post as soon as I have access to cluster.
Currently cluster has 915 shards in the cluster. There are 9 Data nodes with 16GB memory (Elasticsearch has 8GB memory).
How can I be sure that if counter is reset ? From your issue I understand that after executing the command, the counter is reset and Elasticsearch again hits the limit. But looking at the first image above, unassigned shard says it tried at 25 February which is 2 days before I executed the reroute commands. Shouldn't it be updated if it retried to allocate ?
I will try the workaround and keep here updated.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 2, 2019, 1:55pm UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/4 "2019-03-02T13:55:09Z")
> [@redcho](#):
>
> How can I be sure that if counter is reset ?
The simplest way to tell is that the shard is now allocated.
> [@redcho](#):
>
> But looking at the first image above, unassigned shard says it tried at 25 February which is 2 days before I executed the reroute commands. Shouldn't it be updated if it retried to allocate ?
The message was generated before the counter was reset. If you tried again, I think you would have seen a more recent message.
---
### Author:  [@redcho](https://discuss.elastic.co/u/redcho)
#### Post date: [March 4, 2019, 8:13am UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/5 "2019-03-04T08:13:13Z")
> [@DavidTurner](#):
>
> The message was generated before the counter was reset. If you tried again, I think you would have seen a more recent message.
Actually, I tried more than once to run this command and nothing updated at that response and shards are still unassigned.
Also, just in case, I still check breaker stats on nodes and I believe there is enough room. (Elasticsearch was complaining for CircuitBreakingException[parent] and when I check the parent breaker, at least 4GB available.)
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 4, 2019, 10:21am UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/6 "2019-03-04T10:21:12Z")
> [@redcho](#):
>
> Actually, I tried more than once to run this command and nothing updated at that response and shards are still unassigned.
I looked at this again and you're right, if you have an allocation blocked by the `max_retries` condition and try and run a reroute command then it will do nothing, even if you set `?retry_failed=true`, so the message will remain unchanged. The workaround and fix remain the same.
---
### Author:  [@redcho](https://discuss.elastic.co/u/redcho)
#### Post date: [March 5, 2019, 5:52am UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/7 "2019-03-05T05:52:26Z")
> [@DavidTurner](#):
>
> The workaround and fix remain the same.
Thanks, David. I simply disabled the allocation, reset the counter, re-enabled the allocation and all of them is assigned correctly.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 2, 2019, 5:52am UTC](https://discuss.elastic.co/t/shard-allocation-says-max-retry-but-fails-to-allocate-on-retry-failed-true/170396/8 "2019-04-02T05:52:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.