# Bulk indexing causes management threadpool queue to skyrocket

**URL:** https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852
**Category:** Elasticsearch
**Created:** [February 4, 2020, 4:58pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852 "2020-02-04T16:58:59Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 4, 2020, 4:58pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/1 "2020-02-04T16:58:59Z")

</div>

After bulk indexing for 10 minutes on a 7.4.1 cluster, it will suddenly become completely unresponsive to any cluster state requests.  
It still index's and still responds to queries, but the cluster state itself is unresponsive. After a while nodes will start to leave the cluster if it's left in this state, however, we find if we stop ingestion, it will recover.

This is the sudden increase in CPU we see across all data nodes in the cluster:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/4/64b2df2384e774994b2cc17bda6184d305b06970.jpeg)

This is the management thread pool queue count across the cluster. Note it skyrockets once this starts.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9eae169553a936cc66b9f4a13e8d14a942fc2679.png)

Once we stop ingestion, we find we can go the Data Node with the largest queue and restart ES on it and we are back in business.

Big questions: Why doesn't ES rate limit bulk ingestion? We get no rejections, no 429's, nothing on the write queues. We know these clusters can ingest a lot more, on ES 2 we are doing 50k docs per second on the same hardware and 7.4.1 seems to fall over at 25k per second.

How can we get these management cluster state queues to relax?

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 4, 2020, 4:59pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/2 "2020-02-04T16:59:48Z")

</div>

I should mention that the CPU/management queue drops are after we turn off all ingestion.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 4, 2020, 8:33pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/3 "2020-02-04T20:33:45Z")

</div>

As I mentioned in [another thread](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-issue/217830/7), it would be useful to see what these nodes are so busy doing. I'd try the [nodes hot threads API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html): `GET /_nodes/hot_threads?threads=99999`. However, this runs on the management threadpool, so if that threadpool is where the problem is then it won't work. If so, please use `jmap` to grab a thread dump directly from the JVM. Also, of course, if there are any messages in the logs from the time of the problem then they would be helpful.

When you say "nodes leave the cluster" this sounds very bad, but I can't yet see how that is related. We definitely need to see logs from at least (a) the elected master and (b) the node that left for a few minutes either side of its departure. Look for a log message from the `MasterService` containing the string `node-left`, and a log message from the `ClusterApplierService` saying `master node changed`.

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 4, 2020, 9:29pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/4 "2020-02-04T21:29:41Z")

</div>

Agreed about nodes leaving being bad 😄

Yeah, we looked at hot threads previously (in another thread on here somewhere I started). It all STATS, completion stats, segment stats, etc. I'm not sure why CompletionStats is even running, we have no "suggesters" or anything like that. We do run with the xpack monitoring on though.

Currently we are trying bumping `thread_pool.management.max: 8` to match the cores on the data nodes. We are finding that now they are averaging about 6 active during ingestion, 4 active during idle.

🤞

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 4, 2020, 10:14pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/5 "2020-02-04T22:14:16Z")

</div>

Our index rate ...

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2dfdf00f43e5843090b3d637c99d0b0540aac391.png)

And management thread\_pool queue:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/afe6310e562791b6fad6c7005819c8f169a85a60.png)

You can see it really blew up when we cross 30k docs per second

Here's an example of hot threads

```auto
93.3% (466.6ms out of 500ms) cpu usage by thread 'elasticsearch[esdata-03ea][management][T#2]'
     2/10 snapshots sharing following 16 elements
       app//org.elasticsearch.index.engine.Engine.fillSegmentStats(Engine.java:830)
       app//org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:812)
       app//org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:1028)
       app//org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:210) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:120) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:52) app//org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
       app//org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13/java.lang.Thread.run(Thread.java:830)
     6/10 snapshots sharing following 20 elements
       java.base@13/java.util.TreeMap.getEntry(TreeMap.java:357)
       java.base@13/java.util.TreeMap.get(TreeMap.java:277) app//org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:311)
       app//org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)
       app//org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:366)
       app//org.elasticsearch.index.engine.Engine.completionStats(Engine.java:197)
       app//org.elasticsearch.index.shard.IndexShard.completionStats(IndexShard.java:1048)
       app//org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:207) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:120) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:52) app//org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
       app//org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13/java.lang.Thread.run(Thread.java:830)
     2/10 snapshots sharing following 21 elements
       java.base@13/java.util.TreeMap.getEntry(TreeMap.java:357)
       java.base@13/java.util.TreeMap.get(TreeMap.java:277) app//org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.terms(BlockTreeTermsReader.java:304) app//org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:312)
       app//org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)
       app//org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:366)
       app//org.elasticsearch.index.engine.Engine.completionStats(Engine.java:197)
       app//org.elasticsearch.index.shard.IndexShard.completionStats(IndexShard.java:1048)
       app//org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:207) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:120) app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:52) app//org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
       app//org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13/java.lang.Thread.run(Thread.java:830)

```

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 4, 2020, 10:16pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/6 "2020-02-04T22:16:04Z")

</div>

I did see another post you made here @DavidTurner [Concurrent Indices stats requests cause cluster to go red](https://discuss.elastic.co/t/concurrent-indices-stats-requests-cause-cluster-to-go-red/161385)

Which gave us the hint to turn off `--es.indices` from the exporter. But we are still seeing all these `CompletionStats` ... is there a way to turn off anything to do with completion completely? (Beside staying on ES 2 😉)

---

<div class="post-metadata">

### Author: ![giorgio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/giorgio/32/61978_2.png) [@giorgio](https://discuss.elastic.co/u/giorgio)
#### Post date: [February 4, 2020, 11:26pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/7 "2020-02-04T23:26:53Z")

</div>

After switching to NVMe SSD storage my problem seems to be fixed. I had to allocate less JVM memory but still elasticsearch works flawlessly without any issues. I will be running this for another day to conclude it.

---

<div class="post-metadata">

### Author: ![giorgio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/giorgio/32/61978_2.png) [@giorgio](https://discuss.elastic.co/u/giorgio)
#### Post date: [February 5, 2020, 2:23am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/8 "2020-02-05T02:23:54Z")

</div>

whoops I meant to reply on my thread sorry I just noticed 😅

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 5, 2020, 3:11am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/9 "2020-02-05T03:11:28Z")

</div>

Yeah, over here still dealing with the `app//org.elasticsearch.index.engine.Engine.completionStats(Engine.java:197)` always running and sucking up my management threads ... going to attempt to turn off all monitoring tomorrow.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2020, 9:35am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/10 "2020-02-05T09:35:12Z")

</div>

As far as I can tell there's something outside your cluster that is hitting an overly-broad stats API and requesting all completion stats, and the fix is to stop it from doing so. Completion stats are expensive to compute, even if you are not using completion, and you should avoid requesting them if possible. The [indices stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html) lets clients select the metrics they need, so if you do not need completion stats then you should stop requesting them. I think it's something outside the cluster because Elasticsearch's own monitoring does the right thing here, only requesting the stats it needs:

> <https://github.com/elastic/elasticsearch/blob/e28509fbfea45bd381708a962ab4bdc04dda1a47/x-pack/plugin/monitoring/src/main/java/org/elasticsearch/xpack/monitoring/collector/indices/IndexStatsCollector.java#L59-L73>

I think this is trappy, btw, and since there's a few similar cases on the issue I opened [https://github.com/elastic/elasticsearch/issues/51915](https://github.com/elastic/elasticsearch/issues/51915) to discuss a possible improvement. I do also think that we are missing documentation that these stats are expensive to compute.

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 5, 2020, 12:00pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/11 "2020-02-05T12:00:22Z")

</div>

Thanks David. That's the conclusion I'm coming to as well. Big cluster means big stats. Hopefully the load balancer on the coordinator nodes points me at what is doing it ...

I agree with the issue you posted, this was unexpected as we don't use any suggesters and I've also seen a lot of cases out there. I'll keep this ticket posted

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 5, 2020, 3:52pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/12 "2020-02-05T15:52:23Z")

</div>

I'm hoping to wind up with 0 `completionStat` mentions showing in my hot\_threads. Currently I get about 250 occurrences `curl -XGET localhost:9200/_nodes/hot_threads | grep completionStat | wc -l`

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 5, 2020, 6:20pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/13 "2020-02-05T18:20:05Z")

</div>

Even with all of our monitoring turned off, we quickly see all hot\_threads become `completionStats` like this:

```auto
 java.base@13/java.util.TreeMap.getEntry(TreeMap.java:357)
       java.base@13/java.util.TreeMap.get(TreeMap.java:277)
       app//org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.terms(BlockTreeTermsReader.java:304)
       app//org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:312)
       app//org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)
       app//org.apache.lucene.index.FilterLeafReader.terms(FilterLeafReader.java:366)
       app//org.elasticsearch.index.engine.Engine.completionStats(Engine.java:197)
       app//org.elasticsearch.index.shard.IndexShard.completionStats(IndexShard.java:1048)
       app//org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:207)
       app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:120)
       app//org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:52)
       app//org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129)
       app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244)
       app//org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)
       app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
       app//org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13/java.lang.Thread.run(Thread.java:830)

```

Turning off xpack now

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 5, 2020, 6:36pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/14 "2020-02-05T18:36:14Z")

</div>

@DavidTurner Looking at this [https://github.com/elastic/elasticsearch/blob/v7.4.1/server/src/main/java/org/elasticsearch/action/admin/cluster/stats/TransportClusterStatsAction.java#L120](https://github.com/elastic/elasticsearch/blob/v7.4.1/server/src/main/java/org/elasticsearch/action/admin/cluster/stats/TransportClusterStatsAction.java#L120) (And my Latin isn't great ...) but isn't this calling completionStats for every shard every time there's a cluster stats call?

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2020, 8:08pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/15 "2020-02-05T20:08:50Z")

</div>

Oh good lord.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2020, 8:09pm UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/16 "2020-02-05T20:09:46Z")

</div>

Yes, you're quite right, I think this means cluster stats is also computing completion stats and there's no way to switch that off 😕

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 6, 2020, 12:56am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/17 "2020-02-06T00:56:49Z")

</div>

@DavidTurner saved the day! So once that was confirmed started watching for who was calling `/_cluster/stats` and found "NetData" hang out on all our AMI's that we'd forgotten about. It saw Elasticsearch running locally and began calling `_cluster/stats` every 5 seconds! Per data node, \* 24 data nodes. We removed the service and the CPU just started dropping. Thanks for watching the forums!

Cheers!

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [February 6, 2020, 2:25am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/18 "2020-02-06T02:25:35Z")

</div>

To give props where it's due: Elasticsearch 7.4.1 was dealing with a massive state query every 5 seconds from 32 different nodes and yet didn't stop ingesting data or responding to queries. Anything that didn't touch cluster state still responded. That's pretty awesome.

Yes, it would've been nice to have got rejection notices for cluster state, but at least it was doing it's core business.

Keep that in mind guys 😉 core business ...

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 6, 2020, 11:43am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/19 "2020-02-06T11:43:14Z")

</div>

Good catch, yes, that'd explain it. 5 cluster-wide stats calls per second is definitely on the abusive side. When you weren't indexing it looks like you had enough resources (particularly, IO bandwidth) to cope with the monitoring load, but it seems that the extra load from indexing pushed it over the edge.

I think we can generally do better here, optimisations of the completion stats calculation aside, so I opened [https://github.com/elastic/elasticsearch/issues/51992](https://github.com/elastic/elasticsearch/issues/51992) to discuss higher-level protection against this.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 7:43am UTC](https://discuss.elastic.co/t/bulk-indexing-causes-management-threadpool-queue-to-skyrocket/217852/20 "2022-11-04T07:43:30Z")

</div>


