# Possible bug: Elasticsearch not honouring index.unassigned.node\_left.delayed\_timeout

**URL:** https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393
**Category:** Elasticsearch
**Created:** [March 25, 2025, 4:42pm UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393 "2025-03-25T16:42:46Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)
#### Post date: [March 25, 2025, 4:42pm UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/1 "2025-03-25T16:42:46Z")

</div>

In Elasticsearch latest (8.17.3) running in Kubernetes ECK (2.16.0) **will wait for 5 minutes to assign an Unassigned shard** no matter the value of `index.unassigned.node_left.delayed_timeout` in the index configuration.

Please help me to verify this issue before opening a case.

To reproduce the issue we will create a test 6 node cluster, configure an index template with custom settings for number replicas (3), delayed\_timeout (5 seconds), index an event and terminate a pod.  
Once the pod leaves, the expected result is Elasticsearch to assign the unassigned shard after ~5 seconds **but it tries to wait for 5 minutes** [contradicting the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/delayed-allocation.html).

Via Explain command (full output at the end of the reproduction steps) we can see that the new setting of 5 seconds (5000 milliseconds) is in place but Elasticsearch intents to wait for 5 minutes (300,000 milliseconds):

```auto
  "can_allocate" : "allocation_delayed",
  "allocate_explanation" : "The node containing this shard copy recently left the cluster. Elasticsearch is waiting for it to return. If the node does not return within [4.9m] then Elasticsearch will allocate this shard to another node. Please wait.",
  "configured_delay_in_millis" : 5000,
  "remaining_delay_in_millis" : 295241,

```

In this example the pod comes back quickly and the recovery does not reach the 5 minutes mark but this behaviour matches what we have seen in more complex situations in production. Elasticsearch will wait.

To reproduce:

- Create test cluster

```auto
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.17.3
  nodeSets:
  - name: default
    count: 6
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 4Gi
              cpu: 1
            limits:
              memory: 4Gi
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
EOF

```

- Setup env and validate

```auto
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}') && k port-forward service/quickstart-es-default 9200 &

% curl -u "elastic:$PASSWORD" -k "https://localhost:9200/_cat/health?v"
epoch timestamp cluster status node.total node.data shards pri relo init unassign unassign.pri pending_tasks max_task_wait_time active_shards_percent
1742917976 15:52:56 quickstart green 6 6 4 2 0 0 0 0 0 - 100.0%

```

- Create index template with our custom settings

```auto
curl -u "elastic:$PASSWORD" -k -XPUT "https://localhost:9200/_component_template/test_default" -H 'Content-Type: application/json' -d'
{
  "template" : {
    "settings" : {
      "index.number_of_shards" : "3",
      "index.unassigned.node_left.delayed_timeout": "5s"
    }
  }
}
';echo

```

```auto
curl -u "elastic:$PASSWORD" -k -XPUT "https://localhost:9200/_index_template/my-index" -H 'Content-Type: application/json' -d'
{
    "index_patterns": ["my-index-*"],
    "priority": 500,
    "composed_of": ["test_default"]
}
';echo

```

- Create an event

```auto
curl -u "elastic:$PASSWORD" -k -XPOST "https://localhost:9200/my-index-000001/_doc/" -H 'Content-Type: application/json' -d'
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "GET /search HTTP/1.1 200 1070000",
  "user": {
    "id": "kimchy"
  }
}
';echo

```

- Validate index template and presence of settings

```auto
% curl -u "elastic:$PASSWORD" -sk "https://localhost:9200/my-index-000001?pretty"
{
  "my-index-000001" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "user" : {
          "properties" : {
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "3",
        "provided_name" : "my-index-000001",
        "creation_date" : "1742918205375",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "5s"
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "bK9d6lGXRBeiS9qOj_UMbg",
        "version" : {
          "created" : "8521000"
        }
      }
    }
  }
}

```

_Please note delayed\_timeout = 5 seconds_

- Terminate a pod (the third in this example)

```auto
kubectl delete pod quickstart-es-default-2 

```

- Explain command output

```auto
% curl -s -u "elastic:$PASSWORD" -k "https://localhost:9200/_cluster/allocation/explain?pretty"
{
  "note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API. See https://www.elastic.co/guide/en/elasticsearch/reference/8.17/cluster-allocation-explain.html for more information.",
  "index" : "my-index-000001",
  "shard" : 1,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_RESTARTING",
    "at" : "2025-03-25T16:05:39.835Z",
    "details" : "node_left [_dMiTTrdTg-NaqSFut2K2Q]",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "allocation_delayed",
  "allocate_explanation" : "The node containing this shard copy recently left the cluster. Elasticsearch is waiting for it to return. If the node does not return within [4.9m] then Elasticsearch will allocate this shard to another node. Please wait.",
  "configured_delay_in_millis" : 5000,
  "remaining_delay_in_millis" : 295241,
  "node_allocation_decisions" : [
    {
      "node_id" : "F5apKdxnRDSzgKKB6MkFNw",
      "node_name" : "quickstart-es-default-4",
      "transport_address" : "10.77.99.79:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "ml.machine_memory" : "4294967296",
        "ml.allocated_processors_double" : "80.0",
        "k8s_node_name" : "xxx11542",
        "transform.config_version" : "10.0.0",
        "xpack.installed" : "true",
        "ml.allocated_processors" : "80",
        "ml.max_jvm_size" : "2147483648"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "yes"
    },
    {
      "node_id" : "iAyiHv3AS7WMd-KRVusFKg",
      "node_name" : "quickstart-es-default-5",
      "transport_address" : "10.77.104.205:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "ml.machine_memory" : "4294967296",
        "ml.allocated_processors_double" : "80.0",
        "k8s_node_name" : "xxx11981",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0",
        "ml.allocated_processors" : "80",
        "ml.max_jvm_size" : "2147483648"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "yes"
    },
    {
      "node_id" : "oyI-TM8ZQD6i1jP33gjrWA",
      "node_name" : "quickstart-es-default-3",
      "transport_address" : "10.77.68.85:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "ml.machine_memory" : "4294967296",
        "ml.allocated_processors_double" : "128.0",
        "k8s_node_name" : "xxx15210",
        "transform.config_version" : "10.0.0",
        "xpack.installed" : "true",
        "ml.allocated_processors" : "128",
        "ml.max_jvm_size" : "2147483648"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "yes"
    },
    {
      "node_id" : "qFaFBK3YTVCH7MnSjQk9PQ",
      "node_name" : "quickstart-es-default-1",
      "transport_address" : "10.77.92.75:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "ml.max_jvm_size" : "2147483648",
        "ml.allocated_processors" : "80",
        "k8s_node_name" : "xxx12200",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0",
        "ml.allocated_processors_double" : "80.0",
        "ml.machine_memory" : "4294967296"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "yes"
    },
    {
      "node_id" : "h1aC0fU8SfGZ2lq0LJ2DRg",
      "node_name" : "quickstart-es-default-0",
      "transport_address" : "10.77.29.80:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "ml.allocated_processors_double" : "128.0",
        "ml.machine_memory" : "4294967296",
        "k8s_node_name" : "xxx16412",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0",
        "ml.max_jvm_size" : "2147483648",
        "ml.allocated_processors" : "128"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "no",
      "store" : {
        "matching_size_in_bytes" : 6061
      },
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[my-index-000001][1], node[h1aC0fU8SfGZ2lq0LJ2DRg], [P], s[STARTED], a[id=cEcC5IhdSbujh2EFh5PpNQ], failed_attempts[0]]"
        },
        {
          "decider" : "awareness",
          "decision" : "NO",
          "explanation" : "there are [2] copies of this shard and [5] values for attribute [k8s_node_name] ([xxx11542, xxx11981, xxx12200, xxx15210, xxx16412] from nodes in the cluster and no forced awareness) so there may be at most [1] copies of this shard allocated to nodes with each value, but (including this copy) there would be [2] copies allocated to nodes with [node.attr.k8s_node_name: xxx16412]"
        }
      ]
    }
  ]
}

```

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 26, 2025, 7:57am UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/2 "2025-03-26T07:57:21Z")

</div>

The behaviour you describe is deliberate: ECK may set the `allocation_delay` parameter when marking the node for shutdown using the [put-shutdown API](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-shutdown.html#put-shutdown-api-request-body), and the docs say:

> If you specify both a restart allocation delay and an index-level allocation delay, the longer of the two is used.

Note that this only applies when ECK knows that a node is restarting, and in this case it's much more efficient to wait for the node to return to the cluster instead of rushing to allocate shards to the remaining nodes which can cause a rebalancing storm.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 26, 2025, 8:01am UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/3 "2025-03-26T08:01:47Z")

</div>

Actually I see a [small docs bug](https://github.com/elastic/elasticsearch/issues/125643) here, the docs should mention that the default `allocation_delay` is `5m` too, so that's what happens if ECK does not set the `allocation_delay` parameter.

---

<div class="post-metadata">

### Author: ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)
#### Post date: [March 26, 2025, 9:44am UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/4 "2025-03-26T09:44:25Z")

</div>

Thank you David for your quick answer.  
But I'd like to add a couple of additional issues.

- Users looking for information about how Elasticsearch manages shards allocation in the case of a failure, will be directed to [Delaying allocation when a node leaves](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/delayed-allocation.html) documentation page and the existence of this cluster default is not mentioned.
- The documentation indicates that the way to change `delayed_timeout` is via `_all/_settings` index configuration but the fact that this setting will be ignore when it is lower than the cluster default is not mentioned on the above page.
- The documentation indicates that the default value of `delayed_timeout` is 1 minute which is misleading.

Besides those items that can be improved expanding the documentation, this opens the question: How can I adjust the value of `index.unassigned.node_left.delayed_timeout` below 5 minutes?

Thanks.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 26, 2025, 10:04am UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/5 "2025-03-26T10:04:36Z")

</div>

> [@juan.domenech](#):
>
> the existence of this cluster default is not mentioned

Yep fair point I added that to the [docs bug report](https://github.com/elastic/elasticsearch/issues/125643).

> [@juan.domenech](#):
>
> this setting will be ignore when it is lower than the cluster default is not mentioned on the above page

This setting is only ignored during a graceful node restart (i.e. `kubectl delete pod`). If the node is not expected to come back after a restart, or it falls out of the cluster unexpectedly, then `index.unassigned.node_left.delayed_timeout` will apply.

The value in use is under ECK's control since it's ECK that calls the put-shutdown API. I don't know if it's configurable there, you would need to ask some ECK experts about that.

Can you share a little more context? Why do you want to force faster replica allocations (and a potential rebalancing storm) when doing a graceful node restart?

---

<div class="post-metadata">

### Author: ![juan.domenech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan.domenech/32/142257_2.png) [@juan.domenech](https://discuss.elastic.co/u/juan.domenech)
#### Post date: [March 28, 2025, 1:16pm UTC](https://discuss.elastic.co/t/possible-bug-elasticsearch-not-honouring-index-unassigned-node-left-delayed-timeout/376393/6 "2025-03-28T13:16:20Z")

</div>

Thank you David for your interest.

I'm chatting with my colleagues to articulate a proper response (I'm working on an Elasticsearch migration to K8 but I'm not a Kubernetes expert).