# ILM Retry API not honoring timeouts

**URL:** <https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [August 18, 2021, 6:32am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772 "2021-08-18T06:32:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Himanshu\_Mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/himanshu_mishra/32/49860_2.png) [@Himanshu\_Mishra](https://discuss.elastic.co/u/Himanshu_Mishra)\
**Post date:** [August 18, 2021, 6:32am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/1 "2021-08-18T06:32:47Z")

</div>

I'm try to retry ILM steps for all indexes, using following command, however

```auto
POST */_ilm/retry
{
  "timeout": "30m",
  "master_timeout": "30m"
}

```

Still following error is thrown. Is there another param to override `process_cluster_event_timeout_exception`?

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "process_cluster_event_timeout_exception",
        "reason" : "failed to process cluster event (ilm-re-run) within 30s"
      }
    ],
    "type" : "process_cluster_event_timeout_exception",
    "reason" : "failed to process cluster event (ilm-re-run) within 30s"
  },
  "status" : 503
}

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 18, 2021, 7:06am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/2 "2021-08-18T07:06:02Z")

</div>

The parameters go in the URL: `POST */_ilm/retry?master_timeout=30m&timeout=30m`. However if it's taking more than 30s to do this then there's something very wrong with your cluster, your master is far too busy doing other things. You should investigate (e.g. look at [cluster pending tasks](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-pending.html)) and fix that, not just pile more work onto an already-overloaded master.

---

<div class="post-metadata">

**Author:** ![Himanshu\_Mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/himanshu_mishra/32/49860_2.png) [@Himanshu\_Mishra](https://discuss.elastic.co/u/Himanshu_Mishra)\
**Post date:** [August 18, 2021, 9:01am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/3 "2021-08-18T09:01:32Z")

</div>

Ouch 😅 Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Himanshu\_Mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/himanshu_mishra/32/49860_2.png) [@Himanshu\_Mishra](https://discuss.elastic.co/u/Himanshu_Mishra)\
**Post date:** [August 18, 2021, 9:07am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/4 "2021-08-18T09:07:07Z")

</div>

A quick noob question, should our logstash point to master nodes, or data nodes? We've pointed it to master node with the assumption that master decides on which shard should the data be kept.

If we send it to data node, can we simply point it to dns of pool of data nodes?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 18, 2021, 9:43am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/5 "2021-08-18T09:43:41Z")

</div>

You should always prefer to send traffic to non-master nodes.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 19, 2021, 7:10am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/6 "2021-08-19T07:10:57Z")

</div>

I opened a bug report since I think it would have helped to reject the request rather than silently ignore the bits we didn't expect:

> <https://github.com/elastic/elasticsearch/issues/76694>
>
> In https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772 a use…r reported confusion over the way that Elasticsearch ignored the parameters in this request:
> 
> \`\`\`
> POST \*/\_ilm/retry
> {
> "timeout": "30m",
> "master\_timeout": "30m"
> }
> \`\`\`
> 
> The immediate solution was to move the parameters into the URL where they belong, but the question highlights a broader problem: if an API endpoint expects a request body then we parse the body fairly strictly and reject requests that contain unexpected things, but if the endpoint doesn't expect a body at all then we seem to leniently accept requests with a body anyway. Silently ignoring input like this is confusing, and I think we should have rejected this request with a \`400 Bad request\` instead.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 16, 2021, 7:11am UTC](https://discuss.elastic.co/t/ilm-retry-api-not-honoring-timeouts/281772/7 "2021-09-16T07:11:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.