# Urgent Help Needed Elastic Shutting down ingestion & All the write load is being transferred to one of the hot nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2026, 8:07pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083 "2026-07-09T20:07:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 9, 2026, 8:07pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/1 "2026-07-09T20:07:07Z")
I have issue with cluster all the time i can see is queue on one of the hot nodes at a time. node\_name name active queue rejected elastic-frozen2 write 0 0 0 elastic-hot1 write 2 0 0 elastic-hot5 write 32 3230 0 elastic-master1 write 0 0 0 elastic-hot6 write 13 0 0 elastic-master3 write 0 0 0 elastic-coordinating1 write 0 0 0 elastic-coordinating2 write 0 0 0 elastic-ml1 write 0 0 0 elastic-frozen1 write 0 0 0 elastic-hot4 write 25 0 0 elastic-hot3 write 5 0 0 elastic-ingest2 write 0 0 0 elastic-master2 write 0 0 0 elastic-ingest1 write 0 0 0 elastic-hot2 write 32 1 0 elastic-ingest3 write 0 0 0 elastic-ml2 write 0 0 0 it is going to sometime on of hot node high not equally diving it. node\_name name active queue rejected elastic-hot5 write 32 4505 0 elastic-hot6 generic 12 0 0 elastic-hot4 generic 12 0 0 elastic-hot5 generic 10 0 0 elastic-hot6 merge 9 0 0 elastic-hot2 generic 6 0 0 elastic-hot2 merge 6 0 0 elastic-hot3 merge 5 0 0 elastic-hot5 merge 4 0 0 elastic-hot1 generic 2 0 0 elastic-hot1 management 2 0 0 elastic-hot1 merge 2 0 0 elastic-hot5 flush 2 0 0 elastic-coordinating1 management 2 0 0 elastic-hot4 merge 2 0 0 elastic-hot3 generic 2 0 0 elastic-master2 inference\_utility 2 0 0 elastic-hot2 flush 2 0 0 elastic-frozen2 inference\_utility 1 0 0 elastic-frozen2 management 1 0 0 elastic-hot1 inference\_utility 1 0 0 elastic-hot1 refresh 1 0 0 elastic-hot1 system\_write 1 0 0 elastic-hot1 watcher 1 0 0 elastic-hot5 inference\_utility 1 0 0 elastic-hot5 management 1 0 0 elastic-hot5 refresh 1 0 0 elastic-master1 inference\_utility 1 0 0 elastic-master1 management 1 0 0 elastic-hot6 flush 1 0 0 elastic-hot6 inference\_utility 1 0 0 elastic-hot6 management 1 0 0 elastic-hot6 write 1 0 0 elastic-master3 inference\_utility 1 0 0 elastic-master3 management 1 0 0 elastic-coordinating1 inference\_utility 1 0 0 elastic-coordinating2 inference\_utility 1 0 0 elastic-coordinating2 management 1 0 0 elastic-ml1 inference\_utility 1 0 0 elastic-ml1 management 1 0 0 elastic-frozen1 inference\_utility 1 0 0 elastic-frozen1 management 1 0 0 elastic-hot4 inference\_utility 1 0 0 elastic-hot4 management 1 0 0 elastic-hot3 inference\_utility 1 0 0 elastic-hot3 management 1 0 0 elastic-ingest2 inference\_utility 1 0 0 elastic-ingest2 management 1 0 0 elastic-master2 management 1 0 0 elastic-ingest1 inference\_utility 1 0 0 elastic-ingest1 management 1 0 0 elastic-ingest1 write\_coordination 1 0 0 elastic-hot2 inference\_utility 1 0 0 elastic-hot2 management 1 0 0 elastic-ingest3 inference\_utility 1 0 0 elastic-ingest3 management 1 0 0 elastic-ml2 inference\_utility 1 0 0 elastic-ml2 management 1 0 0 elastic-frozen2 analyze 0 0 0 ---
**Author:** ![Rafa\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafa_silva/32/147814_2.png) [@Rafa\_Silva](https://discuss.elastic.co/u/Rafa_Silva)\ **Post date:** [July 9, 2026, 10:30pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/2 "2026-07-09T22:30:42Z")
Hi Kishor, Shutting down the ingest nodes will not redistribute the indexing load across the hot nodes. Ingest nodes process the pipelines, but the final write is handled by the hot data nodes that contain the target shards. In your output, elastic-hot5 has all 32 write threads active and thousands of queued operations. The fact that rejected is still 0 only means the queue has not reached its limit yet. This looks like write hot spotting, usually related to shard allocation, too few primary shards in the active index, custom routing, or a resource difference on that node. Could you share the Elasticsearch version and the output of: > GET \_cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used\_percent,load\_1m,allocated\_processors > > GET \_cat/allocation?v&s=node > > GET \_cat/shards/?v&s=index,shard,prirep,state,node > > GET \_nodes/elastic-hot5/hot\_threads I would not increase the write queue as the first action. The priority is to understand why the write workload or processing time is concentrated on hot5 [Hotspotting](https://www.elastic.co/docs/troubleshoot/elasticsearch/hotspotting) ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 10, 2026, 9:48am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/3 "2026-07-10T09:48:18Z")
> **** > > ```auto > GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors > > Summary > > name node.role cpu heap.percent disk.used_percent load_1m > elastic-coordinating1 - 1 48 6.82 0.08 > elastic-coordinating2 - 1 47 6.83 0.25 elastic-frozen1 f 1 57 86.17 0.19 elastic-frozen2 f 1 25 86.16 0.22 elastic-hot1 his 36 67 69.45 13.78 elastic-hot2 his 46 27 66.09 21.09 elastic-hot3 his 12 44 67.91 7.82 elastic-hot4 his 11 16 72.62 6.71 elastic-hot5 his 5 23 65.01 2.98 elastic-hot6 hs 14 61 73.09 7.24 elastic-ingest1 i 3 64 7.58 0.79 elastic-ingest2 i 3 59 7.32 0.75 elastic-ingest3 i 4 15 7.59 0.76 elastic-master1 m 7 11 7.11 0.13 elastic-master2 m 14 30 7.07 1.00 elastic-master3 m 1 50 6.92 0.00 elastic-ml1 lt 0 28 0.00 0.00 elastic-ml2 lt 0 9 0.00 0.08 > > ``` > **** > > ```auto > GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent > > Summary > > node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent > elastic-frozen1 533 0 0.0 7.3tb 0b 3.4tb 561.7gb 3.9tb 86 > elastic-frozen2 123 0 0.0 3.5tb 0b 3.4tb 562.1gb 3.9tb 86 > elastic-hot1 544 325 16.37320425123251 5.7tb 5.5tb 5.5tb 2.4tb 7.9tb 69 > elastic-hot2 316 0 23.767974006518322 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66 > elastic-hot3 438 30 9.503747177041182 5.6tb 5.3tb 5.4tb 2.5tb 7.9tb 67 > elastic-hot4 714 242 4.254057560988934 5.9tb 5.7tb 5.8tb 2.1tb 7.9tb 72 > elastic-hot5 278 33 4.385566225164219 5.3tb 5.1tb 5.2tb 2.7tb 7.9tb 65 > elastic-hot6 892 397 4.47431549184445 5.9tb 5.7tb 5.8tb 2.1tb 7.9tb 72 > > ``` > **** > ---
**Author:** ![Rafa\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafa_silva/32/147814_2.png) [@Rafa\_Silva](https://discuss.elastic.co/u/Rafa_Silva)\ **Post date:** [July 10, 2026, 11:56am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/4 "2026-07-10T11:56:12Z")
Thanks for sharing this. The interesting part is that hot5 is not the busiest node in this snapshot. Hot2 has higher CPU, load and forecasted write load, so the write pressure seems to move between nodes. That usually points more to the location of the active primary shards than to a problem with one specific node. I would check two things: 1. which node holds the primary shards of the index receiving the writes 2. whether the number of undesired shards is going down over time You currently have more than 1,000 undesired shards, so Elasticsearch is still trying to improve the balance. When the queue increases again, capture the shard distribution for the target index at the same time. That should make it easier to confirm whether one primary shard is receiving most of the traffic. I would not change the thread pool or manually move shards yet. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 10, 2026, 12:15pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/5 "2026-07-10T12:15:04Z")
> [@Rafa\_Silva](#): > > Thanks for sharing this. > > The interesting part is that hot5 is not the busiest node in this snapshot. Hot2 has higher CPU, load and forecasted write load, so the write pressure seems to move between nodes. > > That usually points more to the location of the active primary shards than to a problem with one specific node. > > I would check two things: > > 1. which node holds the primary shards of the index receiving the writes > 2. whether the number of undesired shards is going down over time > > You currently have more than 1,000 undesired shards, so Elasticsearch is still trying to improve the balance. > > When the queue increases again, capture the shard distribution for the target index at the same time. That should make it easier to confirm whether one primary shard is receiving most of the traffic. > > I would not change the thread pool or manually move shards yet. i have increased the number of primaries to 3 for few data-streams which were having highest ingestion. As current settings are to keep data for 7 days in cluster only for 6.5 tb per day primary data with 1 replica and have many small shards like mbs only. where to keep the policy was applied to rollover per day. but as smaller shards are increasing too much that was i think also impacting the cluster as well. so increasing the number of primaries has given something `node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent` `elastic-frozen1 521 0 0.0 7.4tb 0b 3.4tb 561.7gb 3.9tb 86` `elastic-frozen2 136 0 0.0 3.9tb 0b 3.4tb 562gb 3.9tb 86` `elastic-hot1 692 122 13.963806701068636 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66` `elastic-hot2 566 61 8.727707970013126 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65` `elastic-hot3 235 9 14.113464883152382 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65` `elastic-hot4 763 128 7.9550333410730065 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66` `elastic-hot5 640 36 9.707710075522193 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65` `elastic-hot6 250 9 8.160587552403719 5.4tb 5.2tb 5.2tb 2.7tb 7.9tb 66` what would you suggest and also i have increased the concurent\_shard Balancing to 4 and recoveries as well to 4. also identifying the smaller shards and making to hold till the meat the size. one more thing i have ingest role in the hot nodes as i have the dedicated ingest nodes, but when i wanted to remove the ingest role from all the apm ingestion stopped. fleet \>settings \> out has ingest nodes ip instead of coordinate or hot nods. but i have seen which nodes should be there as per elk suggestion in this scanario given below, as i need CPU i know and also we are going to have CCS, ML `name ram.max disk.total disk.avail disk.used node.role` `elastic-frozen2 31.3gb 3.9tb 562gb 3.4tb f` `elastic-hot1 62.7gb 7.9tb 2.6tb 5.3tb his` `elastic-hot5 62.7gb 7.9tb 2.7tb 5.2tb his` `elastic-master1 31.3gb 193.6gb 179.8gb 13.7gb m` `elastic-hot6 62.7gb 7.9tb 2.6tb 5.3tb hs` `elastic-master3 31.3gb 193.6gb 180.3gb 13.2gb m` `elastic-coordinating1 15.6gb 193.6gb 180.4gb 13.2gb -` `elastic-coordinating2 15.6gb 193.6gb 180.4gb 13.2gb -` `elastic-ml1 31.3gb 8192pb 8191.9pb 16.2tb lt` `elastic-frozen1 31.3gb 3.9tb 561.7gb 3.4tb f` `elastic-hot4 62.7gb 7.9tb 2.6tb 5.3tb his` `elastic-hot3 62.7gb 7.9tb 2.7tb 5.2tb his` `elastic-ingest2 31.3gb 193.6gb 179.4gb 14.2gb i` `elastic-master2 31.3gb 193.6gb 179.9gb 13.6gb m` `elastic-ingest1 62.7gb 193.6gb 178.9gb 14.6gb i` `elastic-hot2 62.7gb 7.9tb 2.7tb 5.2tb his` `elastic-ingest3 31.3gb 193.6gb 178.9gb 14.7gb i` `elastic-ml2 31.3gb 8192pb 8191.9pb 16.2tb lt` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 10, 2026, 1:42pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/6 "2026-07-10T13:42:57Z")
maybe me, but it seems you didn't attach output from: ```auto GET _cat/shards/?v&s=index,shard,prirep,state,node ``` or ```auto GET _nodes/elastic-hot5/hot_threads ``` Also, the formatting of the output on `/_cat/nodes` was a bit messed up on my browser at least, here's it tidied it up: ```auto name node.role cpu heap.percent disk.used_percent load_1m elastic-coordinating1 - 1 48 6.82 0.08 elastic-coordinating2 - 1 47 6.83 0.25 elastic-frozen1 f 1 57 86.17 0.19 elastic-frozen2 f 1 25 86.16 0.22 elastic-hot1 his 36 67 69.45 13.78 elastic-hot2 his 46 27 66.09 21.09 elastic-hot3 his 12 44 67.91 7.82 elastic-hot4 his 11 16 72.62 6.71 elastic-hot5 his 5 23 65.01 2.98 elastic-hot6 hs 14 61 73.09 7.24 elastic-ingest1 i 3 64 7.58 0.79 elastic-ingest2 i 3 59 7.32 0.75 elastic-ingest3 i 4 15 7.59 0.76 elastic-master1 m 7 11 7.11 0.13 elastic-master2 m 14 30 7.07 1.00 elastic-master3 m 1 50 6.92 0.00 elastic-ml1 lt 0 28 0.00 0.00 elastic-ml2 lt 0 9 0.00 0.08 ``` Did you change something in this cluster recently, eg add nodes, or have some cluster issues, like nodes leaving/joining, as the value of shards.undesired on some of the hot nodes is a bit surprising. What version are you running? There has been a few threads about clusters getting into a seemingly endless cycle of re-balancing shards. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 11, 2026, 2:13pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/7 "2026-07-11T14:13:37Z")
I haven't changed anything in the configuration yet, but one of our hot nodes has been hitting high CPU and restarting. When it drops, the write load transfers to the other hot nodes, causing a domino effect. I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams. Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs. _Note: I wanted to share the `_cat/shards` output, but the log is too large to paste and I don't have an upload option here._ ```auto GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors name node.role cpu heap.percent disk.used_percent load_1m elastic-coordinating1 - 1 77 6.92 0.52 elastic-coordinating2 - 2 18 6.93 0.40 elastic-frozen1 f 0 27 86.17 0.03 elastic-frozen2 f 0 47 86.16 0.10 elastic-hot1 his 25 40 77.63 22.51 elastic-hot2 his 21 43 73.76 41.42 elastic-hot3 his 21 57 73.20 17.42 elastic-hot4 his 18 28 74.42 16.90 elastic-hot5 his 32 72 74.08 20.31 elastic-hot6 hs 28 30 76.94 46.50 elastic-ingest1 i 3 50 7.71 1.28 elastic-ingest2 i 3 34 7.45 1.81 elastic-ingest3 i 3 53 7.72 1.74 elastic-master1 m 8 65 7.13 0.14 elastic-master2 m 13 41 7.10 1.02 elastic-master3 m 1 34 6.87 0.21 elastic-ml1 lt 0 56 0.00 0.00 elastic-ml2 lt 0 66 0.00 0.00 GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent elastic-frozen1 365 0 0.0 6.3tb 0b 3.4tb 561.8gb 3.9tb 86 elastic-frozen2 130 0 0.0 3.7tb 0b 3.4tb 562.1gb 3.9tb 86 elastic-hot1 492 276 10.537364188353848 6.4tb 6.1tb 6.2tb 1.7tb 7.9tb 77 elastic-hot2 263 13 9.999951303906215 6tb 5.7tb 5.9tb 2tb 7.9tb 73 elastic-hot3 542 189 11.715867722761827 6.1tb 5.7tb 5.8tb 2.1tb 7.9tb 73 elastic-hot4 308 44 8.74606137881083 6.1tb 5.8tb 5.9tb 2tb 7.9tb 73 elastic-hot5 453 132 10.990794145662559 6.1tb 5.8tb 5.9tb 2tb 7.9tb 74 elastic-hot6 752 401 10.358227671131372 6.3tb 6tb 6.1tb 1.8tb 7.9tb 77 ``` ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 11, 2026, 2:27pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/8 "2026-07-11T14:27:26Z")
ELASTIC VERSION: 9.3.4 I haven't changed anything in the configuration yet, but one of our hot nodes has been hitting high CPU and restarting. When it drops, the write load transfers to the other hot nodes, causing a domino effect. I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams. Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs. _Note: I wanted to share the `_cat/shards` output, but the log is too large to paste and I don't have an upload option here._ ```auto GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors name node.role cpu heap.percent disk.used_percent load_1m elastic-coordinating1 - 1 77 6.92 0.52 elastic-coordinating2 - 2 18 6.93 0.40 elastic-frozen1 f 0 27 86.17 0.03 elastic-frozen2 f 0 47 86.16 0.10 elastic-hot1 his 25 40 77.63 22.51 elastic-hot2 his 21 43 73.76 41.42 elastic-hot3 his 21 57 73.20 17.42 elastic-hot4 his 18 28 74.42 16.90 elastic-hot5 his 32 72 74.08 20.31 elastic-hot6 hs 28 30 76.94 46.50 elastic-ingest1 i 3 50 7.71 1.28 elastic-ingest2 i 3 34 7.45 1.81 elastic-ingest3 i 3 53 7.72 1.74 elastic-master1 m 8 65 7.13 0.14 elastic-master2 m 13 41 7.10 1.02 elastic-master3 m 1 34 6.87 0.21 elastic-ml1 lt 0 56 0.00 0.00 elastic-ml2 lt 0 66 0.00 0.00 GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent elastic-frozen1 365 0 0.0 6.3tb 0b 3.4tb 561.8gb 3.9tb 86 elastic-frozen2 130 0 0.0 3.7tb 0b 3.4tb 562.1gb 3.9tb 86 elastic-hot1 492 276 10.537364188353848 6.4tb 6.1tb 6.2tb 1.7tb 7.9tb 77 elastic-hot2 263 13 9.999951303906215 6tb 5.7tb 5.9tb 2tb 7.9tb 73 elastic-hot3 542 189 11.715867722761827 6.1tb 5.7tb 5.8tb 2.1tb 7.9tb 73 elastic-hot4 308 44 8.74606137881083 6.1tb 5.8tb 5.9tb 2tb 7.9tb 73 elastic-hot5 453 132 10.990794145662559 6.1tb 5.8tb 5.9tb 2tb 7.9tb 74 elastic-hot6 752 401 10.358227671131372 6.3tb 6tb 6.1tb 1.8tb 7.9tb 77 GET _nodes/elastic-hot6/hot_threads ::: {elastic-hot6}{tt83jQ_i2lS1fbjsijwR9w}{O7OPOSjsjjsjsIIA_Q}{elastic-hot6}{IP}{IP:9300}{hs}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} Hot threads at 2026-07-11T14:16:55.001Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=97.1%, other=2.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][search][T#37]' 7/10 snapshots sharing following 42 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader$FieldUsageTrackingStoredFieldsReader.document(FieldUsageTrackingDirectoryReader.java:258) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$$Lambda/0x0000000025c8db38.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$ReaderStoredFieldLoader.advanceTo(StoredFieldLoader.java:216) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider$Leaf.getSource(ConcurrentSegmentSourceProvider.java:101) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider.getSource(ConcurrentSegmentSourceProvider.java:72) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup.lambda$new$0(LeafSearchLookup.java:40) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup$$Lambda/0x0000000025b51970.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.lambda$static$0(AbstractFieldScript.java:70) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript$$Lambda/0x0000000024637cc0.apply(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.DynamicMap.get(DynamicMap.java:59) org.elasticsearch.painless.PainlessScript$Script.execute(def body = params._source?.http?.request?.body?.original; if (body != null) { emit(body.toString()); }:18) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.runForDoc(AbstractFieldScript.java:159) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:28) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:20) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractScriptFieldQuery$2.matches(AbstractScriptFieldQuery.java:99) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionScorer$TwoPhase.matches(DisjunctionScorer.java:153) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ConjunctionDISI$ConjunctionTwoPhaseIterator.matches(ConjunctionDISI.java:327) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$TwoPhaseFilterMatchingDisiWrapper.checkDocForMatch(FiltersAggregator.java:484) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$SingleFilterLeafCollector.collect(FiltersAggregator.java:383) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:87) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreIterator(Weight.java:298) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:280) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:53) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:466) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 47 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:192) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source) java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248) java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724) java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506) java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:210) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source) java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248) java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724) java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506) java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=90.7%, other=9.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#12]' 5/10 snapshots sharing following 33 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.advance(IndexedDISI.java:465) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.nextDoc(IndexedDISI.java:560) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$SparseNumericDocValues.nextDoc(Lucene90DocValuesProducer.java:499) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$22.nextDoc(Lucene90DocValuesProducer.java:1062) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$SortedDocValuesSub.nextDoc(DocValuesConsumer.java:626) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$Sub.nextMappedDoc(DocIDMerger.java:60) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$SortedDocIDMerger.next(DocIDMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$8.nextDoc(DocValuesConsumer.java:764) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer$2$1.nextDoc(Lucene90DocValuesConsumer.java:688) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SingletonSortedNumericDocValues.nextDoc(SingletonSortedNumericDocValues.java:53) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.writeValues(Lucene90DocValuesConsumer.java:368) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 5/10 snapshots sharing following 22 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=73.6%, other=26.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#10]' 7/10 snapshots sharing following 23 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:136) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 23 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:105) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.compress.LZ4.compressWithDictionary(LZ4.java:558) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.doCompress(LZ4WithPresetDictCompressionMode.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:184) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) ``` Load has been adjusted via that change i made by increasing number of primary shards. But let me share the metrics of today's ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 11, 2026, 2:41pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/9 "2026-07-11T14:41:02Z")
```auto GET _nodes/elastic-hot6/hot_threads ::: {elastic-hot6}{id}{ID_Q}{elastic-hot6}{ip}{ip:9300}{hs}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} Hot threads at 2026-07-11T14:16:55.001Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=97.1%, other=2.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][search][T#37]' 7/10 snapshots sharing following 42 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader$FieldUsageTrackingStoredFieldsReader.document(FieldUsageTrackingDirectoryReader.java:258) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$$Lambda/0x0000000025c8db38.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$ReaderStoredFieldLoader.advanceTo(StoredFieldLoader.java:216) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider$Leaf.getSource(ConcurrentSegmentSourceProvider.java:101) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider.getSource(ConcurrentSegmentSourceProvider.java:72) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup.lambda$new$0(LeafSearchLookup.java:40) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup$$Lambda/0x0000000025b51970.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.lambda$static$0(AbstractFieldScript.java:70) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript$$Lambda/0x0000000024637cc0.apply(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.DynamicMap.get(DynamicMap.java:59) org.elasticsearch.painless.PainlessScript$Script.execute(def body = params._source?.http?.request?.body?.original; if (body != null) { emit(body.toString()); }:18) app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.runForDoc(AbstractFieldScript.java:159) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:28) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:20) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractScriptFieldQuery$2.matches(AbstractScriptFieldQuery.java:99) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionScorer$TwoPhase.matches(DisjunctionScorer.java:153) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ConjunctionDISI$ConjunctionTwoPhaseIterator.matches(ConjunctionDISI.java:327) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$TwoPhaseFilterMatchingDisiWrapper.checkDocForMatch(FiltersAggregator.java:484) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$SingleFilterLeafCollector.collect(FiltersAggregator.java:383) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:87) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreIterator(Weight.java:298) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:280) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:53) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:466) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 47 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:192) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source) java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248) java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724) java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506) java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:210) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source) java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248) java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724) java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506) java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368) app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source) java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=90.7%, other=9.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#12]' 5/10 snapshots sharing following 33 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.advance(IndexedDISI.java:465) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.nextDoc(IndexedDISI.java:560) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$SparseNumericDocValues.nextDoc(Lucene90DocValuesProducer.java:499) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$22.nextDoc(Lucene90DocValuesProducer.java:1062) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$SortedDocValuesSub.nextDoc(DocValuesConsumer.java:626) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$Sub.nextMappedDoc(DocIDMerger.java:60) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$SortedDocIDMerger.next(DocIDMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$8.nextDoc(DocValuesConsumer.java:764) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer$2$1.nextDoc(Lucene90DocValuesConsumer.java:688) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SingletonSortedNumericDocValues.nextDoc(SingletonSortedNumericDocValues.java:53) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.writeValues(Lucene90DocValuesConsumer.java:368) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 5/10 snapshots sharing following 22 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=73.6%, other=26.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#10]' 7/10 snapshots sharing following 23 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:136) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 23 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:105) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.compress.LZ4.compressWithDictionary(LZ4.java:558) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.doCompress(LZ4WithPresetDictCompressionMode.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:184) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) ``` ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 11, 2026, 2:41pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/10 "2026-07-11T14:41:46Z")
```auto GET _nodes/elastic-hot3/hot_threads ::: {elastic-hot3}{9rkddfgdfgfgrwejkndksUJSDN4KSJ}{SDFSDFNAFBJSDJ}{elastic-hot3}{ip}{ip:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_b, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} Hot threads at 2026-07-11T14:30:26.607Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=51.2%, other=48.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][merge][T#33]' 9/10 snapshots sharing following 22 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a91090.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot java.base@26/sun.nio.ch.UnixFileDispatcherImpl.write0(Native Method) java.base@26/sun.nio.ch.UnixFileDispatcherImpl.write(UnixFileDispatcherImpl.java:65) java.base@26/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137) java.base@26/sun.nio.ch.IOUtil.write(IOUtil.java:102) java.base@26/sun.nio.ch.IOUtil.write(IOUtil.java:72) java.base@26/sun.nio.ch.FileChannelImpl.implWrite(FileChannelImpl.java:359) java.base@26/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:339) java.base@26/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:68) java.base@26/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:105) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:396) java.base@26/java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) java.base@26/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:123) java.base@26/java.io.BufferedOutputStream.write(BufferedOutputStream.java:188) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:59) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.store.FilterIndexOutput.writeBytes(FilterIndexOutput.java:49) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:60) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.ByteBuffersDataOutput.copyTo(ByteBuffersDataOutput.java:338) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:194) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a91090.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=12.0%, other=88.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][write][T#16]' 2/10 snapshots sharing following 38 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:136) app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:136) app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:129) app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.StringSorter.sort(StringSorter.java:54) app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.BytesRefHash.sort(BytesRefHash.java:273) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermsHashPerField.sortTerms(TermsHashPerField.java:114) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:97) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:342) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:391) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:413) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 8/10 snapshots sharing following 31 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:133) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:342) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:391) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:413) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=5.3%, other=94.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][flush][T#3]' 8/10 snapshots sharing following 40 elements java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method) java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57) java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284) java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202) java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174) java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:128) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:463) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3706) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4154) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4116) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:3131) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.flushHoldingLock(InternalEngine.java:2331) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.Engine.flush(Engine.java:1524) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$flush$15(IndexShard.java:1561) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019cb8c18.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1551) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1544) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$afterWriteOperation$63(IndexShard.java:4319) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019e12248.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 40 elements java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method) java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57) java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284) java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202) java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174) java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:109) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:87) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer$1$2.decompress(SortingStoredFieldsConsumer.java:66) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.document(Lucene90CompressingStoredFieldsReader.java:619) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:651) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3706) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4154) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4116) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:3131) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.flushHoldingLock(InternalEngine.java:2331) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.Engine.flush(Engine.java:1524) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$flush$15(IndexShard.java:1561) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019cb8c18.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1551) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1544) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$afterWriteOperation$63(IndexShard.java:4319) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019e12248.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) ``` ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 11, 2026, 2:49pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/11 "2026-07-11T14:49:17Z")
```auto ::: {elastic-hot5}{nnjgHS_ZsX9w}{09fkjA}{elastic-hot5}{IP}{IP:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} Hot threads at 2026-07-11T14:42:57.464Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=69.2%, other=30.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][write][T#19]' 3/10 snapshots sharing following 42 elements app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:157) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:103) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:114) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1101) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1026) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 17 elements app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 25 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 27 elements app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:157) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:103) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:114) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1101) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1026) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) unique snapshot app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.scanToTermNonLeaf(SegmentTermsEnumFrame.java:846) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.scanToTerm(SegmentTermsEnumFrame.java:551) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:513) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000033aed760.get(Unknown Source) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000033c8e5b8.apply(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=68.1%, other=31.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][merge][T#71]' 10/10 snapshots sharing following 21 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write(Lucene103BlockTreeTermsWriter.java:992) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write(Lucene103BlockTreeTermsWriter.java:393) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:95) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:195) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:222) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000033bb5b80.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:146) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000033bafc40.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=60.0%, other=40.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][write][T#18]' 3/10 snapshots sharing following 28 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:741) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:618) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 28 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:773) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:618) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 2/10 snapshots sharing following 17 elements app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 3/10 snapshots sharing following 34 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:189) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:511) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000033aed760.get(Unknown Source) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000033c8e5b8.apply(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 11, 2026, 3:37pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/12 "2026-07-11T15:37:56Z")
It looks like it is spending time writing and merging. What type of storage are you using for your hot nodes? Is it locally attached SSDs or some type of networked storage? What does the load on the storage look like, e.g. IOPS, utilisation and await? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 11, 2026, 4:14pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/13 "2026-07-11T16:14:12Z")
> [@kkumar123](#): > > Note: I wanted to share the `_cat/shards` output, but the log is too large to paste and I don't have an upload option her Yeah, can be an issue with lots of indices/shards. Your 2 posts that shared `_cat/allocation` both show around 1000 shards in the shards.undesired column. If a node is falling out of the cluster regularly, you can easily go round in circles here. Can you move some more of the data off the hot tier? Is there any data on the hot tier that you could actually just delete? Anything you can do currently to reduce load will help. > [@kkumar123](#): > > I am increasing the primary shard count to 3 and 6 for the high-traffic streams. Well, this might help. Can you add more hot nodes with more IO bandwidth? Whatever aggregated IO bandwidth you have, across the cluster, you want to use all of it. How many indices are being heavily written to simultaneously? The target is that all hot nodes should be close equally busy on indexing, unless your data is skewed somehow (e.g. custom routing) But _if_ you are also fighting against the cluster trying to rebalance itself, then. this might be difficult. In the `_cat/shards?v` output, do you often see shards relocating? ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 13, 2026, 10:48am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/14 "2026-07-13T10:48:38Z")
> [@Christian\_Dahlqvist](#): > > It looks like it is spending time writing and merging. What type of storage are you using for your hot nodes? Is it locally attached SSDs or some type of networked storage? What does the load on the storage look like, e.g. IOPS, utilisation and await? It's SSD, i am using reroute to route some of errors. i have to confirm the ssh i Yes bus not too many its normally relocating. Issue is at one time it keeps the load due to force merge before snapshot, and move to frozen. I can see errors for force merge in Elasticsearch server logs some time before breaking the ingestion. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 13, 2026, 12:37pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/15 "2026-07-13T12:37:35Z")
> [@kkumar123](#): > > Yes bus not too many its normally relocating. Watch output of `/_cluster/health` say every 10 minutes for a few hours. It will only relocate a limited number of shards at a time, there are controls for this, but if it's almost always relocating then I suspect it's chasing its own tail. Rebalancing on a not stable cluster is not helping here. > [@kkumar123](#): > > i am using reroute to route some of errors Can you be more precise as to what you mean here ? Is it possible to add more nodes to your cluster ? With your disk usage on all hot nodes at 70-something percent, and 6 nodes, that starts to get tight for space if you have only 5 nodes. and 5 nodes doing the work that 6 normally do is just more load per node, and you seem already constrained, so higher chance of problems. There would also an argument for a warm layer, as the force\_merge happens on the tier that has the data before frozen, in your case the hot nodes. This is not ideal if your hot nodes are already constrained. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 13, 2026, 2:45pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/16 "2026-07-13T14:45:06Z")
> [@RainTown](#): > > /\_cluster/health 1. ```auto "status": "green", "timed_out": false, "number_of_nodes": 18, "number_of_data_nodes": 8, "active_primary_shards": 1983, "active_shards": 3387, "relocating_shards": 4 ``` HOT THREADS: ```auto node_name name active queue rejected elastic-frozen2 write 0 0 0 elastic-hot1 write 0 0 0 elastic-hot5 write 2 0 0 elastic-master1 write 0 0 0 elastic-hot6 write 1 0 0 elastic-master3 write 0 0 0 elastic-coordinating1 write 0 0 0 elastic-coordinating2 write 0 0 0 elastic-ml1 write 0 0 0 elastic-frozen1 write 0 0 0 elastic-hot4 write 1 0 0 elastic-hot3 write 32 3026 0 elastic-ingest2 write 0 0 0 elastic-master2 write 0 0 0 elastic-ingest1 write 0 0 0 elastic-hot2 write 4 0 0 elastic-ingest3 write 0 0 0 elastic-ml2 write 0 0 0 ``` ```auto ::: {elastic-hot3}{}{}{elastic-hot3}{IP}{IP:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_b, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} Hot threads at 2026-07-13T14:42:46.829Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=91.2%, other=8.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][merge][T#35]' 10/10 snapshots sharing following 26 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.TrieReader.load(TrieReader.java:94) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.TrieReader.(TrieReader.java:82) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.FieldReader.newReader(FieldReader.java:87) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.FieldReader.iterator(FieldReader.java:141) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.MultiTerms.iterator(MultiTerms.java:185) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.MappedMultiFields$MappedMultiTerms.iterator(MappedMultiFields.java:65) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write(Lucene103BlockTreeTermsWriter.java:381) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:95) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:195) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:222) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a96000.merge(Unknown Source) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:146) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=58.8%, other=41.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][write][T#1]' 7/10 snapshots sharing following 34 elements app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:189) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:511) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000019b4c000.get(Unknown Source) app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bb0468.apply(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 3/10 snapshots sharing following 17 elements app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 100.0% [cpu=40.8%, other=59.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][refresh][T#9]' 7/10 snapshots sharing following 53 elements java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method) java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57) java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284) java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202) java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174) java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:128) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:463) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:580) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:382) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:356) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:346) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:49) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:28) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:505) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:485) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2138) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$maybeRefresh$9(InternalEngine.java:2096) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bcc698.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:2096) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$scheduledRefresh$64(IndexShard.java:4403) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019bcc000.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4384) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:1120) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1256) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:138) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) 3/10 snapshots sharing following 52 elements java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method) java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57) java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307) java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284) java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202) java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174) java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:127) app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:87) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer$1$2.decompress(SortingStoredFieldsConsumer.java:66) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.document(Lucene90CompressingStoredFieldsReader.java:619) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:651) app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:580) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:382) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:356) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:346) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:49) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:28) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:505) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:485) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2138) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$maybeRefresh$9(InternalEngine.java:2096) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bcc698.get(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:2096) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$scheduledRefresh$64(IndexShard.java:4403) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019bcc000.accept(Unknown Source) app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4384) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:1120) app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1256) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:138) app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) java.base@26/java.lang.Thread.runWith(Thread.java:1529) java.base@26/java.lang.Thread.run(Thread.java:1516) ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 13, 2026, 3:10pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/17 "2026-07-13T15:10:48Z")
Thanks for the output. Nothing contradicting that you cluster is stuck in a loop, or too close and sometimes over its limit. If you had more data nodes (and therefore more space) it could buy you some time, you could adjust the ILM policy so that the indices dont need to be force-merged and maybe the cluster would balance and stabilize. One thing I missed first time reading through: > [@kkumar123](#): > > Since we are paying for them under the ERU (Elastic Resource Unit) license ... Well, yes, probably so, 6 ingest does and 6 data nodes is an unusual ratio in my experience. But I dont know your data. You are an enterprise customer? Did you open a support case with Elastic? When I was an enterprise cluster, long time ago now, we had (very knowledgeable) Elastic support engineers joining us on support calls, with various stakeholders, shared screens, sharing API output close to live, and generally helping us resolve our issues. EDIT: 18 total nodes for a 6-hot-node / 2-frozen node cluster feels like it has been designed by a committee. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 13, 2026, 3:54pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/18 "2026-07-13T15:54:40Z")
> [@kkumar123](#): > > I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams. > > Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs. What are the specs for your nodes, specially the hot ones? And how are you indexing your data? Are you using the default `index.refresh_interval` for all your data? Is it coming from Logstash? Elastic Agent? Are you using hot-frozen in your ILM, it goes to frozen immediately after it rollover or it wait a couple of time? Is it traditional VMs or ECK/ECE? There are some things you may do to improve the performance. I would ditch the ingest nodes, leave the ingest role to all your hot nodes, there is one that is not an ingest node yet. Also, what is the reason for the coordinating nodes? ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 13, 2026, 8:18pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/19 "2026-07-13T20:18:09Z")
> [@RainTown](#): > > sharing The design was purposed by solution architect. I would have open that ticket but right now i don't access to support portal, will surely do that. but most problematic thing for me ingest nodes does not have that much load of CPU, i will try to remove these ingest nodes completely instead add hot nodes. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 13, 2026, 8:40pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/20 "2026-07-13T20:40:37Z")
Hi @leandrojmp , here are the spacs ```auto name ram.max disk.total disk.avail disk.used node.role elastic-frozen2 31.3gb 3.9tb 562.3gb 3.4tb f elastic-hot1 62.7gb 7.9tb 1.9tb 6tb his elastic-hot5 62.7gb 7.9tb 1.9tb 6tb his elastic-master1 31.3gb 193.6gb 179.8gb 13.7gb m elastic-hot6 62.7gb 7.9tb 1.9tb 6tb hs elastic-master3 31.3gb 193.6gb 180.3gb 13.3gb m elastic-coordinating1 15.6gb 193.6gb 180.4gb 13.2gb - elastic-coordinating2 15.6gb 193.6gb 180.3gb 13.2gb - elastic-ml1 31.3gb 8192pb 8191.9pb 19.2tb lt elastic-frozen1 31.3gb 3.9tb 561.1gb 3.4tb f elastic-hot4 62.7gb 7.9tb 1.9tb 6tb his elastic-hot3 62.7gb 7.9tb 1.9tb 6tb his elastic-ingest2 31.3gb 193.6gb 179.2gb 14.3gb i elastic-master2 31.3gb 193.6gb 179.8gb 13.7gb m elastic-ingest1 62.7gb 193.6gb 178.7gb 14.9gb i elastic-hot2 62.7gb 7.9tb 1.9tb 6tb his elastic-ingest3 31.3gb 193.6gb 178.7gb 14.8gb i elastic-ml2 31.3gb 8192pb 8191.9pb 19.2tb lt ``` > And how are you indexing your data? > Data is ingested via Elastic Agent , Logstash, APM, > Are you using the default `index.refresh_interval` for all your data? > We are are using our 30s refresh interval, in apm, kubernetes logs, and others but some of them have default. > Are you using hot-frozen in your ILM, it goes to frozen immediately after it rollover or it wait a couple of time? > > No after rollover it stays there for sometime then goes to the frozen, like apm data it rollover at 3 days or 30 gb as data is too much like 3 tb per day primary only, shard sizing we have kept 30 GB. > > - we are keeping short 7 days hot-frozen policy overall > - there is force merge applied. > Is it traditional VMs? > > yes its Oci VMs - Traditional Stand alone self hosted cluster > I would ditch the ingest nodes, leave the ingest role to all your hot nodes, there is one that is not an ingest node yet. > > I was about to do it as they are consumption the ERU only i don't see CPU consumption on those > Also, what is the reason for the coordinating nodes? > > CCS > In Kibana We have 300+ active users to do Reporting other > And ingestion rate is high like with 1 relplica 13 tb and also too free up master load of coordinating role from hot. [Next page](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083.md?page=2)