# Cluster takes too long to apply cluster state
**URL:** https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407
**Category:** Elasticsearch
**Created:** [March 24, 2023, 1:49am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407 "2023-03-24T01:49:01Z")
**Posts on this page:** 8
**Page:** 2
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [May 25, 2023, 2:20am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/21 "2023-05-25T02:20:39Z")
Hi guys,
We've added a few nodes to spread the disk load. Cluster stayed yellow but we still got 4 nodes disconnect from the cluster during the index deletion:
```auto
insertOrder timeInQueue priority source
7316 21.3s IMMEDIATE node-left
7317 21.3s IMMEDIATE node-left
7318 21.3s IMMEDIATE node-left
7319 21.3s IMMEDIATE node-left
7325 19.3s URGENT node-join
7321 21.3s HIGH shard-failed
7322 21.3s HIGH shard-failed
7323 20.8s HIGH shard-failed
7324 20.8s HIGH shard-failed
7320 21.3s HIGH shard-failed
[2023-05-25T00:47:35,421][WARN][o.e.c.c.LagDetector] [esm04] node [{esd02}{nIoZq1ZWRiKgPBz3x6uJAg}{BbJGSC4zRv2ID0hEfXghGw}{x.x.x.x:9300}{cdfhstw}{xpack.installed=true, transform.node=true}] is lagging at cluster state version [13093], although publication of cluster state version [13094] completed [1.5m] ago
[2023-05-25T00:47:35,422][WARN][o.e.c.c.LagDetector] [esm04] node [{esd03}{mDYiwqFkS-Sj7A9YcyLmrA}{L2ZdyuXhTr6Mh9vbk8Acjg}{x.x.x.x:9300}{cdfhstw}{xpack.installed=true, transform.node=true}] is lagging at cluster state version [13093], although publication of cluster state version [13094] completed [1.5m] ago
[2023-05-25T00:47:35,422][WARN][o.e.c.c.LagDetector] [esm04] node [{esd08}{T83ju1TKQhyZUd2LI4Atlw}{R7tciuZQQQadTA3TFIgWCA}{x.x.x.x:9300}{cdfhstw}{xpack.installed=true, transform.node=true}] is lagging at cluster state version [13093], although publication of cluster state version [13094] completed [1.5m] ago
[2023-05-25T00:47:35,423][WARN][o.e.c.c.LagDetector] [esm04] node [{esd06}{ReFWrVXVSf-a1ould6uIEg}{TpJGLw5XQQe3Cen3lIVxIQ}{x.x.x.x:9300}{cdfhstw}{xpack.installed=true, transform.node=true}] is lagging at cluster state version [13093], although publication of cluster state version [13094] completed [1.5m] ago
```
Nodes immediately rejoined but we got a bunch of UNASSIGNED & INITIALIZING shards in between and YELLOW cluster state, which can go to RED if removed nodes would take out enough shards to cause outage.
Is it safe to bump node\_left.delayed\_timeout to ~5 minutes to prevent master kicking them out during the deletion operation? I realize that getting faster drives/more instances can speed up the process but we might not have this option.
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [May 25, 2023, 3:00am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/22 "2023-05-25T03:00:06Z")
One more question, is this SENT\_APPLY\_COMMIT operation asynchronous? What happens if we wait for a couple of nodes to apply cluster state? Do they still can accept writes or only reads? Or they aren't servicing the data until they reported the last cluster state change successfully?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [May 25, 2023, 7:10am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/23 "2023-05-25T07:10:06Z")
> [@Vadym](#):
>
> Is it safe to bump node\_left.delayed\_timeout to ~5 minutes to prevent master kicking them out during the deletion operation?
It is safe to increase this timeout indeed, but this parameter will not prevent the master from removing nodes that are lagging so badly.
> [@Vadym](#):
>
> is this SENT\_APPLY\_COMMIT operation asynchronous?
Yes.
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [May 25, 2023, 4:55pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/24 "2023-05-25T16:55:59Z")
So this timeout has no effect on whether master will kick node out of the cluster? I thought since deletion is asynchronous other writes/reads to ES shouldn't be affected when it's done?
What would be a usual suspect for this type of lag? Our ssds still can take a lot of writes/reads in parallel, sometimes this lag is triggered when deleted index is dropping only 1-2 15Gb shards from each data node.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [May 25, 2023, 5:07pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/25 "2023-05-25T17:07:46Z")
Usually it's either infrastructure problems or a bug. The [troubleshooting docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-fault-detection.html#cluster-fault-detection-troubleshooting) (linked [previously](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/4)) will help you collect the information needed to distinguish these cases.
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [June 6, 2023, 5:53pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/26 "2023-06-06T17:53:25Z")
Hi David,
But isn't it also expected that deleting 1Tb+ of data can take ~2 minutes? Can we speed up the deletion somehow by changing ? Is it possible that we are missing some throttle settings, which actually make the deletion slower, not the disk speed?
Is it expected for the long index deletion to block updating cluster state at all?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 6, 2023, 6:14pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/27 "2023-06-06T18:14:51Z")
Yes it might take some time if there's a lot of data to delete. But minutes to delete 1-2 small shards (as per your earlier post) seems surprising. I cannot recommend changing any settings without seeing the results of the troubleshooting I linked previously.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 4, 2023, 6:15pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/28 "2023-07-04T18:15:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
[Previous page](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407.md?page=1)