# Cluster takes too long to apply cluster state
**URL:** https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407
**Category:** Elasticsearch
**Created:** [March 24, 2023, 1:49am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407 "2023-03-24T01:49:01Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [March 24, 2023, 1:49am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/1 "2023-03-24T01:49:01Z")
Hi guys,
We have some 1Tb+ indices and it takes more than a minute to drop these indices when we rotate them. During the deletion cluster takes too long to apply cluster state and master nodes start to kick data nodes out of the cluster ending up in RED state with some primary shards disconnected.
We can tolerate delete operation to be long and put some queries in queue but want to avoid nodes being kicked out of the cluster.
Which settings can we tweak to avoid nodes to be kicked out when it takes too long to apply cluster state and nodes are busy with deleting the data?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 24, 2023, 3:21am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/2 "2023-03-24T03:21:36Z")
What do you mean by "drop"?
What is the output from the [`_cluster/stats?pretty&human` API](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/cluster-stats.html)?
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [March 24, 2023, 6:24pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/3 "2023-03-24T18:24:42Z")
Drop = Delete, I.e. curl -X DELETE "{endpoint/index}"
```auto
{
"_nodes" : {
"total" : 23,
"successful" : 23,
"failed" : 0
},
"cluster_name" : "es_cluster",
"cluster_uuid" : "0PWha9VWQBCLhgBwryfg2Q",
"timestamp" : 1679682126440,
"status" : "green",
"indices" : {
"count" : 10,
"shards" : {
"total" : 196,
"primaries" : 53,
"replication" : 2.69811320754717,
"index" : {
"shards" : {
"min" : 2,
"max" : 120,
"avg" : 19.6
},
"primaries" : {
"min" : 1,
"max" : 30,
"avg" : 5.3
},
"replication" : {
"min" : 1.0,
"max" : 3.0,
"avg" : 1.4
}
}
},
"docs" : {
"count" : 134566730,
"deleted" : 50773306
},
"store" : {
"size" : "1.5tb",
"size_in_bytes" : 1731111345325,
"total_data_set_size" : "1.5tb",
"total_data_set_size_in_bytes" : 1731111345325,
"reserved" : "0b",
"reserved_in_bytes" : 0
},
"fielddata" : {
"memory_size" : "6.8gb",
"memory_size_in_bytes" : 7406660464,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "1.2gb",
"memory_size_in_bytes" : 1378531580,
"total_count" : 39295360141,
"hit_count" : 6646100796,
"miss_count" : 32649259345,
"cache_size" : 1058237,
"cache_count" : 78144276,
"evictions" : 77086039
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 5186,
"memory" : "190.4mb",
"memory_in_bytes" : 199650108,
"terms_memory" : "118.7mb",
"terms_memory_in_bytes" : 124494296,
"stored_fields_memory" : "6.6mb",
"stored_fields_memory_in_bytes" : 6952672,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "12.1mb",
"norms_memory_in_bytes" : 12784256,
"points_memory" : "0b",
"points_memory_in_bytes" : 0,
"doc_values_memory" : "52.8mb",
"doc_values_memory_in_bytes" : 55418884,
"index_writer_memory" : "712mb",
"index_writer_memory_in_bytes" : 746672836,
"version_map_memory" : "10mb",
"version_map_memory_in_bytes" : 10505100,
"fixed_bit_set" : "177.1mb",
"fixed_bit_set_memory_in_bytes" : 185788680,
"max_unsafe_auto_id_timestamp" : 1679098436506,
"file_sizes" : { }
},
"mappings" : {
"field_types" : [
{
"name" : "boolean",
"count" : 23,
"index_count" : 7,
"script_count" : 0
},
{
"name" : "constant_keyword",
"count" : 6,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "date",
"count" : 77,
"index_count" : 8,
"script_count" : 0
},
{
"name" : "double",
"count" : 3,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "integer",
"count" : 27,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "ip",
"count" : 2,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "keyword",
"count" : 202,
"index_count" : 10,
"script_count" : 0
},
{
"name" : "long",
"count" : 66,
"index_count" : 7,
"script_count" : 0
},
{
"name" : "nested",
"count" : 11,
"index_count" : 2,
"script_count" : 0
},
{
"name" : "object",
"count" : 106,
"index_count" : 8,
"script_count" : 0
},
{
"name" : "text",
"count" : 137,
"index_count" : 10,
"script_count" : 0
}
],
"runtime_field_types" : []
},
"analysis" : {
"char_filter_types" : [],
"tokenizer_types" : [],
"filter_types" : [],
"analyzer_types" : [
{
"name" : "custom",
"count" : 2,
"index_count" : 2
}
],
"built_in_char_filters" : [],
"built_in_tokenizers" : [
{
"name" : "keyword",
"count" : 2,
"index_count" : 2
}
],
"built_in_filters" : [
{
"name" : "lowercase",
"count" : 2,
"index_count" : 2
}
],
"built_in_analyzers" : [
{
"name" : "standard",
"count" : 2,
"index_count" : 1
}
]
},
"versions" : [
{
"version" : "7.5.0",
"index_count" : 1,
"primary_shard_count" : 1,
"total_primary_size" : "52.5kb",
"total_primary_bytes" : 53830
},
{
"version" : "7.10.0",
"index_count" : 2,
"primary_shard_count" : 16,
"total_primary_size" : "2.6mb",
"total_primary_bytes" : 2791344
},
{
"version" : "7.17.7",
"index_count" : 7,
"primary_shard_count" : 36,
"total_primary_size" : "403.2gb",
"total_primary_bytes" : 432985967198
}
]
},
"nodes" : {
"count" : {
"total" : 23,
"coordinating_only" : 3,
"data" : 15,
"data_cold" : 15,
"data_content" : 15,
"data_frozen" : 15,
"data_hot" : 15,
"data_warm" : 15,
"ingest" : 0,
"master" : 5,
"ml" : 0,
"remote_cluster_client" : 0,
"transform" : 15,
"voting_only" : 0
},
"versions" : [
"7.17.7"
],
"os" : {
"available_processors" : 540,
"allocated_processors" : 540,
"names" : [
{
"name" : "Linux",
"count" : 23
}
],
"pretty_names" : [
{
"pretty_name" : "Red Hat Enterprise Linux",
"count" : 23
}
],
"architectures" : [
{
"arch" : "amd64",
"count" : 23
}
],
"mem" : {
"total" : "11.3tb",
"total_in_bytes" : 12436988469248,
"free" : "4tb",
"free_in_bytes" : 4470949167104,
"used" : "7.2tb",
"used_in_bytes" : 7966039302144,
"free_percent" : 36,
"used_percent" : 64
}
},
"process" : {
"cpu" : {
"percent" : 439
},
"open_file_descriptors" : {
"min" : 830,
"max" : 1998,
"avg" : 1249
}
},
"jvm" : {
"max_uptime" : "64.1d",
"max_uptime_in_millis" : 5538397280,
"versions" : [
{
"version" : "19",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "19+36-2238",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : true,
"count" : 23
}
],
"mem" : {
"heap_used" : "261.5gb",
"heap_used_in_bytes" : 280788898664,
"heap_max" : "690gb",
"heap_max_in_bytes" : 740881858560
},
"threads" : 2720
},
"fs" : {
"total" : "13.4tb",
"total_in_bytes" : 14810401996800,
"free" : "11.7tb",
"free_in_bytes" : 12867983949824,
"available" : "11.7tb",
"available_in_bytes" : 12867983949824
},
"plugins" : [],
"network_types" : {
"transport_types" : {
"security4" : 23
},
"http_types" : {
"security4" : 23
}
},
"discovery_types" : {
"zen" : 23
},
"packaging_types" : [
{
"flavor" : "default",
"type" : "rpm",
"count" : 23
}
],
"ingest" : {
"number_of_pipelines" : 2,
"processor_stats" : {
"gsub" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
},
"script" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time" : "0s",
"time_in_millis" : 0
}
}
}
}
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 24, 2023, 6:34pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/4 "2023-03-24T18:34:57Z")
> [@Vadym](#):
>
> We can tolerate delete operation to be long and put some queries in queue but want to avoid nodes being kicked out of the cluster.
Hmm this sounds like a possible bug, we shouldn't be blocking cluster state updates on deletes no matter how large the indices are. Could you follow the [troubleshooting guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-fault-detection.html#cluster-fault-detection-troubleshooting) to work out what extra information to collect to dig deeper?
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [March 24, 2023, 7:24pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/5 "2023-03-24T19:24:41Z")
Hi David,
Thanks for the link, I'll get through the guide as much as we can afford to interrupt that environment.
Most "heath checkers" show that everything is within green range.
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [March 24, 2023, 7:34pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/6 "2023-03-24T19:34:34Z")
In the interim, can you recommend what can we tune up from this list?
> **[Discovery and cluster formation settings | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-settings.html)**
I.e. how much we can safely bump cluster.publish.timeout ? Is cluster stuck until cluster state is updated from all nodes or it's a different behavior?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 24, 2023, 8:23pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/7 "2023-03-24T20:23:13Z")
Without seeing the results of the troubleshooting it's not possible for me to say what, if anything, can be done to work around the problem.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [March 27, 2023, 7:50am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/8 "2023-03-27T07:50:10Z")
Hi guys,
We had the same situation. Have 1Tb+ indices and we tried to remove every month them through DELETE /indexname\_202301\_\* and then 1-2 nodes could out of the cluster. Then we switched to delete indices one by one. We used 7.12 and thought that in the new version 7.17 it was fixed
The main reason is deleting this big indices takes a lot of time for cluster and makes big i/o load on the disk subsystem
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 27, 2023, 11:37am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/9 "2023-03-27T11:37:37Z")
> [@Denis\_Lamanov](#):
>
> The main reason is deleting this big indices takes a lot of time for cluster and makes big i/o load on the disk subsystem
That doesn't really explain the problem tho, at least not in enough detail for us to think about fixing it.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [March 27, 2023, 4:24pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/10 "2023-03-27T16:24:09Z")
We are going to add security to our cluster so we can reboot nodes/cluster as we want and can reproduce the issue. What should we enable in config in order to log all needed info?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 27, 2023, 5:05pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/11 "2023-03-27T17:05:22Z")
The [troubleshooting guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-fault-detection.html#cluster-fault-detection-troubleshooting) I linked [above](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/4) describes the steps to follow to capture what we need to understand the problem.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [March 28, 2023, 8:03am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/12 "2023-03-28T08:03:45Z")
We use Elasticsearch 7.17
Can we use the same as in 8.6?
logger.org.elasticsearch.cluster.coordination.LagDetector: DEBUG
logger.org.elasticsearch.transport.TcpTransport: DEBUG
logger.org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport: DEBUG
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 28, 2023, 8:08am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/13 "2023-03-28T08:08:39Z")
Yes those loggers are unchanged in 7.17.
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [March 28, 2023, 7:08pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/14 "2023-03-28T19:08:36Z")
Hi Denis,
Are there any issues with logging verbosity in DEBUG mode in your prod envs? I think I need to increase level too in order to debug this issue.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [March 29, 2023, 3:40pm UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/15 "2023-03-29T15:40:41Z")
Will inform you in one week. Are going to delete many big indices soon
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [April 19, 2023, 6:54am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/16 "2023-04-19T06:54:37Z")
Hi Denis,
Any updates from your side by chance?
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [April 19, 2023, 7:14am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/17 "2023-04-19T07:14:48Z")
Finally we have logs
>
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 19, 2023, 8:23am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/18 "2023-04-19T08:23:20Z")
Wow ok it looks like your disks are _really_ slow here. To some extent you're also affected by these issues:
>
>
> Today when allocating a replica the \`ReplicaShardAllocator\` asks every node in t…he cluster for details of their local copy of the shard. Any nodes currently holding a copy of the shard respond with a \`StoreFilesMetadata\` which includes a \`Store.MetadataSnapshot\` containing the checksum of every file in the latest index commit. These checksums are read from disk to satisfy these requests.
>
> If a node leaves the cluster then the \`ReplicaShardAllocator\` attempts to assign all of its now-unassigned shards at once, which may trigger several hundred concurrent attempts to read the checksum from every file of every remaining copy of the now-unassigned shards. These attempts run on the \`fetch\_shard\_store\` threadpool which in principle limits the concurrency per node. However, the default maximum size of this threadpool is \`2\*#processors\` which may permit more concurrent IO than a node can handle if its CPU-to-disk-bandwidth ratio is higher than expected.
>
> I'm opening this issue to discuss ways to reduce the impact of this mechanism. Some initial thoughts:
>
> \- Is the default maximum size of the \`fetch\_shard\_store\` appropriate?
>
> \- Nodes with poor disk bandwidth are typically used to hold indices that change rarely, if at all. Would it help to capture or compute the \`Store.MetadataSnapshot\` when the safe commit changes so we don't need to read the checksums from disk each time?
>
> \- Do running shard copies need to share their checksums at all? Today we ignore the checksums if the shard copy has a sufficiently recent peer recovery retention lease; can we rely on running shard copies to have such a lease?
>
> (with thanks to \`@tlrx\` for spotting this)
>
>
> When \`IndicesClusterStateService\` applies a new cluster state it does substantia…l amounts of work inline on the applier thread, including some IO (or blocking waits for IO on other threads) which can hold up cluster state updates and even cause nodes to leave the cluster. We should find ways to move this work off the applier thread.
>
> Relates #88055 which was a specific case of this, mitigated upstream in Lucene but not really fixed.
You can mitigate the first by setting `thread_pool.fetch_shard_store.max: 1`, and maybe that'll be enough to take the load off your disks. If it isn't, you'll need to improve your storage performance somehow.
---
### Author:  [@Denis\_Lamanov](https://discuss.elastic.co/u/Denis_Lamanov)
#### Post date: [April 20, 2023, 6:54am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/19 "2023-04-20T06:54:48Z")
Thanks, we will try to apply `thread_pool.fetch_shard_store.max: 1`
---
### Author:  [@Vadym](https://discuss.elastic.co/u/Vadym)
#### Post date: [May 11, 2023, 12:47am UTC](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407/20 "2023-05-11T00:47:45Z")
Hi Denis,
Did applying this parameter help?
[Next page](https://discuss.elastic.co/t/cluster-takes-too-long-to-apply-cluster-state/328407.md?page=2)