Urgent Help Needed Elastic Shutting down ingestion & All the write load is being transferred to one of the hot nodes

[kkumar123 (Kishor Kumar)]

I have issue with cluster all the time i can see is queue on one of the hot nodes at a time.

node_name name active queue rejected
elastic-frozen2 write 0 0 0
elastic-hot1 write 2 0 0
elastic-hot5 write 32 3230 0
elastic-master1 write 0 0 0
elastic-hot6 write 13 0 0
elastic-master3 write 0 0 0
elastic-coordinating1 write 0 0 0
elastic-coordinating2 write 0 0 0
elastic-ml1 write 0 0 0
elastic-frozen1 write 0 0 0
elastic-hot4 write 25 0 0
elastic-hot3 write 5 0 0
elastic-ingest2 write 0 0 0
elastic-master2 write 0 0 0
elastic-ingest1 write 0 0 0
elastic-hot2 write 32 1 0
elastic-ingest3 write 0 0 0
elastic-ml2 write 0 0 0

it is going to sometime on of hot node high not equally diving it.

node_name name active queue rejected
elastic-hot5 write 32 4505 0
elastic-hot6 generic 12 0 0
elastic-hot4 generic 12 0 0
elastic-hot5 generic 10 0 0
elastic-hot6 merge 9 0 0
elastic-hot2 generic 6 0 0
elastic-hot2 merge 6 0 0
elastic-hot3 merge 5 0 0
elastic-hot5 merge 4 0 0
elastic-hot1 generic 2 0 0
elastic-hot1 management 2 0 0
elastic-hot1 merge 2 0 0
elastic-hot5 flush 2 0 0
elastic-coordinating1 management 2 0 0
elastic-hot4 merge 2 0 0
elastic-hot3 generic 2 0 0
elastic-master2 inference_utility 2 0 0
elastic-hot2 flush 2 0 0
elastic-frozen2 inference_utility 1 0 0
elastic-frozen2 management 1 0 0
elastic-hot1 inference_utility 1 0 0
elastic-hot1 refresh 1 0 0
elastic-hot1 system_write 1 0 0
elastic-hot1 watcher 1 0 0
elastic-hot5 inference_utility 1 0 0
elastic-hot5 management 1 0 0
elastic-hot5 refresh 1 0 0
elastic-master1 inference_utility 1 0 0
elastic-master1 management 1 0 0
elastic-hot6 flush 1 0 0
elastic-hot6 inference_utility 1 0 0
elastic-hot6 management 1 0 0
elastic-hot6 write 1 0 0
elastic-master3 inference_utility 1 0 0
elastic-master3 management 1 0 0
elastic-coordinating1 inference_utility 1 0 0
elastic-coordinating2 inference_utility 1 0 0
elastic-coordinating2 management 1 0 0
elastic-ml1 inference_utility 1 0 0
elastic-ml1 management 1 0 0
elastic-frozen1 inference_utility 1 0 0
elastic-frozen1 management 1 0 0
elastic-hot4 inference_utility 1 0 0
elastic-hot4 management 1 0 0
elastic-hot3 inference_utility 1 0 0
elastic-hot3 management 1 0 0
elastic-ingest2 inference_utility 1 0 0
elastic-ingest2 management 1 0 0
elastic-master2 management 1 0 0
elastic-ingest1 inference_utility 1 0 0
elastic-ingest1 management 1 0 0
elastic-ingest1 write_coordination 1 0 0
elastic-hot2 inference_utility 1 0 0
elastic-hot2 management 1 0 0
elastic-ingest3 inference_utility 1 0 0
elastic-ingest3 management 1 0 0
elastic-ml2 inference_utility 1 0 0
elastic-ml2 management 1 0 0
elastic-frozen2 analyze 0 0 0

[Rafa_Silva]

Hi Kishor,

Shutting down the ingest nodes will not redistribute the indexing load across the hot nodes. Ingest nodes process the pipelines, but the final write is handled by the hot data nodes that contain the target shards.

In your output, elastic-hot5 has all 32 write threads active and thousands of queued operations. The fact that rejected is still 0 only means the queue has not reached its limit yet.

This looks like write hot spotting, usually related to shard allocation, too few primary shards in the active index, custom routing, or a resource difference on that node.

Could you share the Elasticsearch version and the output of:

GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

GET _cat/allocation?v&s=node

GET _cat/shards/?v&s=index,shard,prirep,state,node

GET _nodes/elastic-hot5/hot_threads

I would not increase the write queue as the first action. The priority is to understand why the write workload or processing time is concentrated on hot5

Hotspotting

[kkumar123 (Kishor Kumar)]

GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

Summary

name                   node.role cpu heap.percent disk.used_percent load_1m 
elastic-coordinating1 -           1           48              6.82    0.08 
elastic-coordinating2 -           1           47              6.83    0.25 elastic-frozen1       f           1           57             86.17    0.19 elastic-frozen2       f           1           25             86.16    0.22 elastic-hot1          his        36           67             69.45   13.78 elastic-hot2          his        46           27             66.09   21.09 elastic-hot3          his        12           44             67.91    7.82 elastic-hot4          his        11           16             72.62    6.71 elastic-hot5          his         5           23             65.01    2.98 elastic-hot6          hs         14           61             73.09    7.24 elastic-ingest1       i           3           64              7.58    0.79 elastic-ingest2       i           3           59              7.32    0.75 elastic-ingest3       i           4           15              7.59    0.76 elastic-master1       m           7           11              7.11    0.13 elastic-master2       m          14           30              7.07    1.00 elastic-master3       m           1           50              6.92    0.00 elastic-ml1           lt          0           28              0.00    0.00 elastic-ml2           lt          0            9              0.00    0.08

GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent

Summary

node                         shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1    533                0                 0.0                 7.3tb           0b     3.4tb    561.7gb      3.9tb           86
elastic-frozen2    123                0                 0.0                 3.5tb           0b     3.4tb    562.1gb      3.9tb           86
elastic-hot1       544              325   16.37320425123251                 5.7tb        5.5tb     5.5tb      2.4tb      7.9tb           69
elastic-hot2       316                0  23.767974006518322                 5.5tb        5.2tb     5.3tb      2.6tb      7.9tb           66
elastic-hot3       438               30   9.503747177041182                 5.6tb        5.3tb     5.4tb      2.5tb      7.9tb           67
elastic-hot4       714              242   4.254057560988934                 5.9tb        5.7tb     5.8tb      2.1tb      7.9tb           72
elastic-hot5       278               33   4.385566225164219                 5.3tb        5.1tb     5.2tb      2.7tb      7.9tb           65
elastic-hot6       892              397    4.47431549184445                 5.9tb        5.7tb     5.8tb      2.1tb      7.9tb           72

[Rafa_Silva]

Thanks for sharing this.

The interesting part is that hot5 is not the busiest node in this snapshot. Hot2 has higher CPU, load and forecasted write load, so the write pressure seems to move between nodes.

That usually points more to the location of the active primary shards than to a problem with one specific node.

I would check two things:

1. which node holds the primary shards of the index receiving the writes
2. whether the number of undesired shards is going down over time

You currently have more than 1,000 undesired shards, so Elasticsearch is still trying to improve the balance.

When the queue increases again, capture the shard distribution for the target index at the same time. That should make it easier to confirm whether one primary shard is receiving most of the traffic.

I would not change the thread pool or manually move shards yet.

[kkumar123 (Kishor Kumar)]

@Rafa_Silva

Thanks for sharing this.

The interesting part is that hot5 is not the busiest node in this snapshot. Hot2 has higher CPU, load and forecasted write load, so the write pressure seems to move between nodes.

That usually points more to the location of the active primary shards than to a problem with one specific node.

I would check two things:

1. which node holds the primary shards of the index receiving the writes
2. whether the number of undesired shards is going down over time

You currently have more than 1,000 undesired shards, so Elasticsearch is still trying to improve the balance.

When the queue increases again, capture the shard distribution for the target index at the same time. That should make it easier to confirm whether one primary shard is receiving most of the traffic.

I would not change the thread pool or manually move shards yet.

i have increased the number of primaries to 3 for few data-streams which were having highest ingestion.

As current settings are to keep data for 7 days in cluster only for 6.5 tb per day primary data with 1 replica and have many small shards like mbs only. where to keep the policy was applied to rollover per day.

but as smaller shards are increasing too much that was i think also impacting the cluster as well.

so increasing the number of primaries has given something

node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1 521 0 0.0 7.4tb 0b 3.4tb 561.7gb 3.9tb 86
elastic-frozen2 136 0 0.0 3.9tb 0b 3.4tb 562gb 3.9tb 86
elastic-hot1 692 122 13.963806701068636 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66
elastic-hot2 566 61 8.727707970013126 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot3 235 9 14.113464883152382 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot4 763 128 7.9550333410730065 5.5tb 5.2tb 5.3tb 2.6tb 7.9tb 66
elastic-hot5 640 36 9.707710075522193 5.4tb 5.1tb 5.2tb 2.7tb 7.9tb 65
elastic-hot6 250 9 8.160587552403719 5.4tb 5.2tb 5.2tb 2.7tb 7.9tb 66

what would you suggest and also i have increased the concurent_shard Balancing to 4 and recoveries as well to 4. also identifying the smaller shards and making to hold till the meat the size.

one more thing i have ingest role in the hot nodes as i have the dedicated ingest nodes, but when i wanted to remove the ingest role from all the apm ingestion stopped.

fleet >settings > out has ingest nodes ip instead of coordinate or hot nods.

but i have seen which nodes should be there as per elk suggestion in this scanario given below, as i need CPU i know and also we are going to have CCS, ML

name ram.max disk.total disk.avail disk.used node.role
elastic-frozen2 31.3gb 3.9tb 562gb 3.4tb f
elastic-hot1 62.7gb 7.9tb 2.6tb 5.3tb his
elastic-hot5 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-master1 31.3gb 193.6gb 179.8gb 13.7gb m
elastic-hot6 62.7gb 7.9tb 2.6tb 5.3tb hs
elastic-master3 31.3gb 193.6gb 180.3gb 13.2gb m
elastic-coordinating1 15.6gb 193.6gb 180.4gb 13.2gb -
elastic-coordinating2 15.6gb 193.6gb 180.4gb 13.2gb -
elastic-ml1 31.3gb 8192pb 8191.9pb 16.2tb lt
elastic-frozen1 31.3gb 3.9tb 561.7gb 3.4tb f
elastic-hot4 62.7gb 7.9tb 2.6tb 5.3tb his
elastic-hot3 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-ingest2 31.3gb 193.6gb 179.4gb 14.2gb i
elastic-master2 31.3gb 193.6gb 179.9gb 13.6gb m
elastic-ingest1 62.7gb 193.6gb 178.9gb 14.6gb i
elastic-hot2 62.7gb 7.9tb 2.7tb 5.2tb his
elastic-ingest3 31.3gb 193.6gb 178.9gb 14.7gb i
elastic-ml2 31.3gb 8192pb 8191.9pb 16.2tb lt

[RainTown (Kevin Maguire)]

maybe me, but it seems you didn't attach output from:

GET _cat/shards/?v&s=index,shard,prirep,state,node

or

GET _nodes/elastic-hot5/hot_threads

Also, the formatting of the output on /_cat/nodes was a bit messed up on my browser at least, here's it tidied it up:

name                   node.role cpu heap.percent disk.used_percent load_1m
elastic-coordinating1 -           1           48              6.82    0.08
elastic-coordinating2 -           1           47              6.83    0.25
elastic-frozen1       f           1           57             86.17    0.19
elastic-frozen2       f           1           25             86.16    0.22
elastic-hot1          his        36           67             69.45   13.78
elastic-hot2          his        46           27             66.09   21.09
elastic-hot3          his        12           44             67.91    7.82
elastic-hot4          his        11           16             72.62    6.71
elastic-hot5          his         5           23             65.01    2.98
elastic-hot6          hs         14           61             73.09    7.24
elastic-ingest1       i           3           64              7.58    0.79
elastic-ingest2       i           3           59              7.32    0.75
elastic-ingest3       i           4           15              7.59    0.76
elastic-master1       m           7           11              7.11    0.13
elastic-master2       m          14           30              7.07    1.00
elastic-master3       m           1           50              6.92    0.00
elastic-ml1           lt          0           28              0.00    0.00
elastic-ml2           lt          0            9              0.00    0.08

Did you change something in this cluster recently, eg add nodes, or have some cluster issues, like nodes leaving/joining, as the value of shards.undesired on some of the hot nodes is a bit surprising.

What version are you running? There has been a few threads about clusters getting into a seemingly endless cycle of re-balancing shards.

[kkumar123 (Kishor Kumar)]

I haven't changed anything in the configuration yet, but one of our hot nodes has been hitting high CPU and restarting. When it drops, the write load transfers to the other hot nodes, causing a domino effect.

I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams.

Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs.

Note: I wanted to share the _cat/shards output, but the log is too large to paste and I don't have an upload option here.

GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

name                               node.role cpu heap.percent disk.used_percent load_1m
elastic-coordinating1 -           1           77              6.92    0.52
elastic-coordinating2 -           2           18              6.93    0.40
elastic-frozen1       f           0           27             86.17    0.03
elastic-frozen2       f           0           47             86.16    0.10
elastic-hot1          his        25           40             77.63   22.51
elastic-hot2          his        21           43             73.76   41.42
elastic-hot3          his        21           57             73.20   17.42
elastic-hot4          his        18           28             74.42   16.90
elastic-hot5          his        32           72             74.08   20.31
elastic-hot6          hs         28           30             76.94   46.50
elastic-ingest1       i           3           50              7.71    1.28
elastic-ingest2       i           3           34              7.45    1.81
elastic-ingest3       i           3           53              7.72    1.74
elastic-master1       m           8           65              7.13    0.14
elastic-master2       m          13           41              7.10    1.02
elastic-master3       m           1           34              6.87    0.21
elastic-ml1           lt          0           56              0.00    0.00
elastic-ml2           lt          0           66              0.00    0.00


GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent

node                         shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1    365                0                 0.0                 6.3tb           0b     3.4tb    561.8gb      3.9tb           86
elastic-frozen2    130                0                 0.0                 3.7tb           0b     3.4tb    562.1gb      3.9tb           86
elastic-hot1       492              276  10.537364188353848                 6.4tb        6.1tb     6.2tb      1.7tb      7.9tb           77
elastic-hot2       263               13   9.999951303906215                   6tb        5.7tb     5.9tb        2tb      7.9tb           73
elastic-hot3       542              189  11.715867722761827                 6.1tb        5.7tb     5.8tb      2.1tb      7.9tb           73
elastic-hot4       308               44    8.74606137881083                 6.1tb        5.8tb     5.9tb        2tb      7.9tb           73
elastic-hot5       453              132  10.990794145662559                 6.1tb        5.8tb     5.9tb        2tb      7.9tb           74
elastic-hot6       752              401  10.358227671131372                 6.3tb          6tb     6.1tb      1.8tb      7.9tb           77

[kkumar123 (Kishor Kumar)]

ELASTIC VERSION: 9.3.4

I haven't changed anything in the configuration yet, but one of our hot nodes has been hitting high CPU and restarting. When it drops, the write load transfers to the other hot nodes, causing a domino effect.

I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams.

Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs.

Note: I wanted to share the _cat/shards output, but the log is too large to paste and I don't have an upload option here.

GET _cat/nodes?v&s=name&h=name,node.role,cpu,heap.percent,disk.used_percent,load_1m,allocated_processors

name                               node.role cpu heap.percent disk.used_percent load_1m
elastic-coordinating1 -           1           77              6.92    0.52
elastic-coordinating2 -           2           18              6.93    0.40
elastic-frozen1       f           0           27             86.17    0.03
elastic-frozen2       f           0           47             86.16    0.10
elastic-hot1          his        25           40             77.63   22.51
elastic-hot2          his        21           43             73.76   41.42
elastic-hot3          his        21           57             73.20   17.42
elastic-hot4          his        18           28             74.42   16.90
elastic-hot5          his        32           72             74.08   20.31
elastic-hot6          hs         28           30             76.94   46.50
elastic-ingest1       i           3           50              7.71    1.28
elastic-ingest2       i           3           34              7.45    1.81
elastic-ingest3       i           3           53              7.72    1.74
elastic-master1       m           8           65              7.13    0.14
elastic-master2       m          13           41              7.10    1.02
elastic-master3       m           1           34              6.87    0.21
elastic-ml1           lt          0           56              0.00    0.00
elastic-ml2           lt          0           66              0.00    0.00


GET _cat/allocation?v&s=node&h=node,node.node.role,shards,shards.undesired,write_load.forecast,disk.indices.forecast,disk.indices,disk.used,disk.avail,disk.total,disk.percent

node                         shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent
elastic-frozen1    365                0                 0.0                 6.3tb           0b     3.4tb    561.8gb      3.9tb           86
elastic-frozen2    130                0                 0.0                 3.7tb           0b     3.4tb    562.1gb      3.9tb           86
elastic-hot1       492              276  10.537364188353848                 6.4tb        6.1tb     6.2tb      1.7tb      7.9tb           77
elastic-hot2       263               13   9.999951303906215                   6tb        5.7tb     5.9tb        2tb      7.9tb           73
elastic-hot3       542              189  11.715867722761827                 6.1tb        5.7tb     5.8tb      2.1tb      7.9tb           73
elastic-hot4       308               44    8.74606137881083                 6.1tb        5.8tb     5.9tb        2tb      7.9tb           73
elastic-hot5       453              132  10.990794145662559                 6.1tb        5.8tb     5.9tb        2tb      7.9tb           74
elastic-hot6       752              401  10.358227671131372                 6.3tb          6tb     6.1tb      1.8tb      7.9tb           77


GET _nodes/elastic-hot6/hot_threads

::: {elastic-hot6}{tt83jQ_i2lS1fbjsijwR9w}{O7OPOSjsjjsjsIIA_Q}{elastic-hot6}{IP}{IP:9300}{hs}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
   Hot threads at 2026-07-11T14:16:55.001Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=97.1%, other=2.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][search][T#37]'
     7/10 snapshots sharing following 42 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader$FieldUsageTrackingStoredFieldsReader.document(FieldUsageTrackingDirectoryReader.java:258)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$$Lambda/0x0000000025c8db38.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$ReaderStoredFieldLoader.advanceTo(StoredFieldLoader.java:216)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider$Leaf.getSource(ConcurrentSegmentSourceProvider.java:101)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider.getSource(ConcurrentSegmentSourceProvider.java:72)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup.lambda$new$0(LeafSearchLookup.java:40)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup$$Lambda/0x0000000025b51970.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.lambda$static$0(AbstractFieldScript.java:70)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript$$Lambda/0x0000000024637cc0.apply(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.DynamicMap.get(DynamicMap.java:59)
       org.elasticsearch.painless.PainlessScript$Script.execute(def body = params._source?.http?.request?.body?.original; if (body != null) { emit(body.toString()); }:18)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.runForDoc(AbstractFieldScript.java:159)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:28)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:20)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractScriptFieldQuery$2.matches(AbstractScriptFieldQuery.java:99)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionScorer$TwoPhase.matches(DisjunctionScorer.java:153)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ConjunctionDISI$ConjunctionTwoPhaseIterator.matches(ConjunctionDISI.java:327)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$TwoPhaseFilterMatchingDisiWrapper.checkDocForMatch(FiltersAggregator.java:484)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$SingleFilterLeafCollector.collect(FiltersAggregator.java:383)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:87)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreIterator(Weight.java:298)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:280)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:53)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:466)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 47 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:192)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source)
       java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248)
       java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724)
       java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
       java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
       java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
       java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
       java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
       java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506)
       java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:210)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source)
       java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248)
       java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724)
       java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
       java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
       java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
       java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
       java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
       java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506)
       java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=90.7%, other=9.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#12]'
     5/10 snapshots sharing following 33 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.advance(IndexedDISI.java:465)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.nextDoc(IndexedDISI.java:560)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$SparseNumericDocValues.nextDoc(Lucene90DocValuesProducer.java:499)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$22.nextDoc(Lucene90DocValuesProducer.java:1062)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$SortedDocValuesSub.nextDoc(DocValuesConsumer.java:626)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$Sub.nextMappedDoc(DocIDMerger.java:60)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$SortedDocIDMerger.next(DocIDMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$8.nextDoc(DocValuesConsumer.java:764)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer$2$1.nextDoc(Lucene90DocValuesConsumer.java:688)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SingletonSortedNumericDocValues.nextDoc(SingletonSortedNumericDocValues.java:53)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.writeValues(Lucene90DocValuesConsumer.java:368)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     5/10 snapshots sharing following 22 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=73.6%, other=26.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#10]'
     7/10 snapshots sharing following 23 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:136)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 23 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:105)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.compress.LZ4.compressWithDictionary(LZ4.java:558)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.doCompress(LZ4WithPresetDictCompressionMode.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:184)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)

Load has been adjusted via that change i made by increasing number of primary shards.

But let me share the metrics of today's

[kkumar123 (Kishor Kumar)]

GET _nodes/elastic-hot6/hot_threads

::: {elastic-hot6}{id}{ID_Q}{elastic-hot6}{ip}{ip:9300}{hs}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
   Hot threads at 2026-07-11T14:16:55.001Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=97.1%, other=2.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][search][T#37]'
     7/10 snapshots sharing following 42 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader$FieldUsageTrackingStoredFieldsReader.document(FieldUsageTrackingDirectoryReader.java:258)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$$Lambda/0x0000000025c8db38.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.fieldvisitor.StoredFieldLoader$ReaderStoredFieldLoader.advanceTo(StoredFieldLoader.java:216)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider$Leaf.getSource(ConcurrentSegmentSourceProvider.java:101)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.ConcurrentSegmentSourceProvider.getSource(ConcurrentSegmentSourceProvider.java:72)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup.lambda$new$0(LeafSearchLookup.java:40)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.lookup.LeafSearchLookup$$Lambda/0x0000000025b51970.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.lambda$static$0(AbstractFieldScript.java:70)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript$$Lambda/0x0000000024637cc0.apply(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.DynamicMap.get(DynamicMap.java:59)
       org.elasticsearch.painless.PainlessScript$Script.execute(def body = params._source?.http?.request?.body?.original; if (body != null) { emit(body.toString()); }:18)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.script.AbstractFieldScript.runForDoc(AbstractFieldScript.java:159)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:28)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractStringScriptFieldQuery.matches(AbstractStringScriptFieldQuery.java:20)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.runtime.AbstractScriptFieldQuery$2.matches(AbstractScriptFieldQuery.java:99)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionScorer$TwoPhase.matches(DisjunctionScorer.java:153)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ConjunctionDISI$ConjunctionTwoPhaseIterator.matches(ConjunctionDISI.java:327)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$TwoPhaseFilterMatchingDisiWrapper.checkDocForMatch(FiltersAggregator.java:484)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$SingleFilterLeafCollector.collect(FiltersAggregator.java:383)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:87)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreIterator(Weight.java:298)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:280)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:53)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:466)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 47 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:192)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source)
       java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248)
       java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724)
       java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
       java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
       java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
       java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
       java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
       java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506)
       java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:210)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:442)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.SegmentTermsEnum$$Lambda/0x0000000025696ac0.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates.lambda$get$1(TermStates.java:199)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermStates$$Lambda/0x0000000025696ef8.get(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.getTermsEnum(TermQuery.java:137)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TermQuery$TermWeight$2.cost(TermQuery.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight$1.cost(DisjunctionMaxQuery.java:178)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier$$Lambda/0x0000000025a6b2e0.applyAsLong(Unknown Source)
       java.base@26/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:248)
       java.base@26/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1724)
       java.base@26/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
       java.base@26/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
       java.base@26/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
       java.base@26/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
       java.base@26/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
       java.base@26/java.util.stream.LongPipeline.reduce(LongPipeline.java:506)
       java.base@26/java.util.stream.LongPipeline.min(LongPipeline.java:464)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.computeCost(BooleanScorerSupplier.java:84)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.cost(BooleanScorerSupplier.java:110)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.getInternal(BooleanScorerSupplier.java:132)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.BooleanScorerSupplier.get(BooleanScorerSupplier.java:117)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.randomAccessScorer(QueryToFilterAdapter.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$Compatible.getLeafCollector(FiltersAggregator.java:296)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:228)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:88)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:92)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.aggregations.AggregatorCollector.getLeafCollector(AggregatorCollector.java:36)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:166)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:447)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:390)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x0000000025b22220.call(Unknown Source)
       java.base@26/java.util.concurrent.FutureTask.run(FutureTask.java:330)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.TaskExecutor$$Lambda/0x0000000025cea938.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=90.7%, other=9.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#12]'
     5/10 snapshots sharing following 33 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.advance(IndexedDISI.java:465)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.IndexedDISI.nextDoc(IndexedDISI.java:560)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$SparseNumericDocValues.nextDoc(Lucene90DocValuesProducer.java:499)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$22.nextDoc(Lucene90DocValuesProducer.java:1062)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$SortedDocValuesSub.nextDoc(DocValuesConsumer.java:626)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$Sub.nextMappedDoc(DocIDMerger.java:60)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocIDMerger$SortedDocIDMerger.next(DocIDMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer$8.nextDoc(DocValuesConsumer.java:764)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer$2$1.nextDoc(Lucene90DocValuesConsumer.java:688)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SingletonSortedNumericDocValues.nextDoc(SingletonSortedNumericDocValues.java:53)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.writeValues(Lucene90DocValuesConsumer.java:368)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     5/10 snapshots sharing following 22 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.doAddSortedField(Lucene90DocValuesConsumer.java:710)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.addSortedSetField(Lucene90DocValuesConsumer.java:935)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.mergeSortedSetField(DocValuesConsumer.java:856)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a85e38.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=73.6%, other=26.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot6][merge][T#10]'
     7/10 snapshots sharing following 23 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:136)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 23 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(LZ4WithPresetDictCompressionMode.java:105)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.compress.LZ4.compressWithDictionary(LZ4.java:558)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.doCompress(LZ4WithPresetDictCompressionMode.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:184)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000025a72438.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000025a6de00.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)

[kkumar123 (Kishor Kumar)]

GET _nodes/elastic-hot3/hot_threads

::: {elastic-hot3}{9rkddfgdfgfgrwejkndksUJSDN4KSJ}{SDFSDFNAFBJSDJ}{elastic-hot3}{ip}{ip:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_b, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
   Hot threads at 2026-07-11T14:30:26.607Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=51.2%, other=48.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][merge][T#33]'
     9/10 snapshots sharing following 22 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:524)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:514)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a91090.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.write0(Native Method)
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.write(UnixFileDispatcherImpl.java:65)
       java.base@26/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137)
       java.base@26/sun.nio.ch.IOUtil.write(IOUtil.java:102)
       java.base@26/sun.nio.ch.IOUtil.write(IOUtil.java:72)
       java.base@26/sun.nio.ch.FileChannelImpl.implWrite(FileChannelImpl.java:359)
       java.base@26/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:339)
       java.base@26/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:68)
       java.base@26/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:105)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:396)
       java.base@26/java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
       java.base@26/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:123)
       java.base@26/java.io.BufferedOutputStream.write(BufferedOutputStream.java:188)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:59)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.store.FilterIndexOutput.writeBytes(FilterIndexOutput.java:49)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:60)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.ByteBuffersDataOutput.copyTo(ByteBuffersDataOutput.java:338)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress(LZ4WithPresetDictCompressionMode.java:194)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:263)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:195)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.copyOneDoc(Lucene90CompressingStoredFieldsWriter.java:518)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.merge(Lucene90CompressingStoredFieldsWriter.java:636)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:248)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a91090.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:289)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=12.0%, other=88.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][write][T#16]'
     2/10 snapshots sharing following 38 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:136)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.radixSort(MSBRadixSorter.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:136)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.MSBRadixSorter.sort(MSBRadixSorter.java:129)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.StringSorter.sort(StringSorter.java:54)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.util.BytesRefHash.sort(BytesRefHash.java:273)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.TermsHashPerField.sortTerms(TermsHashPerField.java:114)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:97)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:342)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:391)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:413)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     8/10 snapshots sharing following 31 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:133)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:342)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:391)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:413)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=5.3%, other=94.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][flush][T#3]'
     8/10 snapshots sharing following 40 elements
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method)
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57)
       java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284)
       java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202)
       java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174)
       java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:128)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:463)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3706)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4154)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4116)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:3131)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.flushHoldingLock(InternalEngine.java:2331)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.Engine.flush(Engine.java:1524)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$flush$15(IndexShard.java:1561)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019cb8c18.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1551)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1544)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$afterWriteOperation$63(IndexShard.java:4319)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019e12248.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 40 elements
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method)
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57)
       java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284)
       java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202)
       java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174)
       java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:109)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:87)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer$1$2.decompress(SortingStoredFieldsConsumer.java:66)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.document(Lucene90CompressingStoredFieldsReader.java:619)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:651)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3706)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4154)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4116)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:3131)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.flushHoldingLock(InternalEngine.java:2331)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.Engine.flush(Engine.java:1524)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$flush$15(IndexShard.java:1561)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019cb8c18.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1551)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:1544)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$afterWriteOperation$63(IndexShard.java:4319)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019e12248.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)

[kkumar123 (Kishor Kumar)]

::: {elastic-hot5}{nnjgHS_ZsX9w}{09fkjA}{elastic-hot5}{IP}{IP:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_c, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
   Hot threads at 2026-07-11T14:42:57.464Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=69.2%, other=30.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][write][T#19]'
     3/10 snapshots sharing following 42 elements
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:157)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:103)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:114)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1101)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1026)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 17 elements
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 25 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 27 elements
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:157)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:103)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:114)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1101)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1026)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     unique snapshot
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.scanToTermNonLeaf(SegmentTermsEnumFrame.java:846)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.scanToTerm(SegmentTermsEnumFrame.java:551)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:513)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000033aed760.get(Unknown Source)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000033c8e5b8.apply(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=68.1%, other=31.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][merge][T#71]'
     10/10 snapshots sharing following 21 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write(Lucene103BlockTreeTermsWriter.java:992)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write(Lucene103BlockTreeTermsWriter.java:393)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:95)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:195)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:222)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000033bb5b80.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:146)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000033bafc40.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=60.0%, other=40.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot5][write][T#18]'
     3/10 snapshots sharing following 28 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:741)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:618)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 28 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:773)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:618)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1531)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1462)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1264)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     2/10 snapshots sharing following 17 elements
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000033b6e440.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     3/10 snapshots sharing following 34 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:189)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:511)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000033aed760.get(Unknown Source)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000033c8e5b8.apply(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)

[Christian_Dahlqvist]

It looks like it is spending time writing and merging. What type of storage are you using for your hot nodes? Is it locally attached SSDs or some type of networked storage? What does the load on the storage look like, e.g. IOPS, utilisation and await?

[RainTown (Kevin Maguire)]

@kkumar123

Note: I wanted to share the _cat/shards output, but the log is too large to paste and I don't have an upload option her

Yeah, can be an issue with lots of indices/shards.

Your 2 posts that shared _cat/allocation both show around 1000 shards in the shards.undesired column. If a node is falling out of the cluster regularly, you can easily go round in circles here. Can you move some more of the data off the hot tier? Is there any data on the hot tier that you could actually just delete? Anything you can do currently to reduce load will help.

@kkumar123

I am increasing the primary shard count to 3 and 6 for the high-traffic streams.

Well, this might help. Can you add more hot nodes with more IO bandwidth?

Whatever aggregated IO bandwidth you have, across the cluster, you want to use all of it. How many indices are being heavily written to simultaneously? The target is that all hot nodes should be close equally busy on indexing, unless your data is skewed somehow (e.g. custom routing)

But if you are also fighting against the cluster trying to rebalance itself, then. this might be difficult. In the _cat/shards?v output, do you often see shards relocating?

[kkumar123 (Kishor Kumar)]

@Christian_Dahlqvist

It looks like it is spending time writing and merging. What type of storage are you using for your hot nodes? Is it locally attached SSDs or some type of networked storage? What does the load on the storage look like, e.g. IOPS, utilisation and await?

It's SSD, i am using reroute to route some of errors. i have to confirm the ssh i

Yes bus not too many its normally relocating.

Issue is at one time it keeps the load due to force merge before snapshot, and move to frozen.

I can see errors for force merge in Elasticsearch server logs some time before breaking the ingestion.

[RainTown (Kevin Maguire)]

@kkumar123

Yes bus not too many its normally relocating.

Watch output of /_cluster/health say every 10 minutes for a few hours.

It will only relocate a limited number of shards at a time, there are controls for this, but if it's almost always relocating then I suspect it's chasing its own tail. Rebalancing on a not stable cluster is not helping here.

@kkumar123

i am using reroute to route some of errors

Can you be more precise as to what you mean here ?

Is it possible to add more nodes to your cluster ? With your disk usage on all hot nodes at 70-something percent, and 6 nodes, that starts to get tight for space if you have only 5 nodes. and 5 nodes doing the work that 6 normally do is just more load per node, and you seem already constrained, so higher chance of problems.

There would also an argument for a warm layer, as the force_merge happens on the tier that has the data before frozen, in your case the hot nodes. This is not ideal if your hot nodes are already constrained.

[kkumar123 (Kishor Kumar)]

@RainTown

/_cluster/health

1.   "status": "green",
     "timed_out": false,
     "number_of_nodes": 18,
     "number_of_data_nodes": 8,
     "active_primary_shards": 1983,
     "active_shards": 3387,
     "relocating_shards": 4
   

HOT THREADS:

node_name                          name  active queue rejected
elastic-frozen2       write      0     0        0
elastic-hot1          write      0     0        0
elastic-hot5          write      2     0        0
elastic-master1       write      0     0        0
elastic-hot6          write      1     0        0
elastic-master3       write      0     0        0
elastic-coordinating1 write      0     0        0
elastic-coordinating2 write      0     0        0
elastic-ml1           write      0     0        0
elastic-frozen1       write      0     0        0
elastic-hot4          write      1     0        0
elastic-hot3          write     32  3026        0
elastic-ingest2       write      0     0        0
elastic-master2       write      0     0        0
elastic-ingest1       write      0     0        0
elastic-hot2          write      4     0        0
elastic-ingest3       write      0     0        0
elastic-ml2           write      0     0        0

::: {elastic-hot3}{}{}{elastic-hot3}{IP}{IP:9300}{his}{9.3.4}{8000099-9060001}{nfs_group=hot_b, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
   Hot threads at 2026-07-13T14:42:46.829Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=91.2%, other=8.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][merge][T#35]'
     10/10 snapshots sharing following 26 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.TrieReader.load(TrieReader.java:94)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.TrieReader.<init>(TrieReader.java:82)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.FieldReader.newReader(FieldReader.java:87)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.FieldReader.iterator(FieldReader.java:141)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.MultiTerms.iterator(MultiTerms.java:185)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.MappedMultiFields$MappedMultiTerms.iterator(MappedMultiFields.java:65)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write(Lucene103BlockTreeTermsWriter.java:381)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:95)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:195)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:222)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger$$Lambda/0x0000000019a96000.merge(Unknown Source)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:146)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ThreadPoolMergeExecutorService$$Lambda/0x0000000019a88490.run(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=58.8%, other=41.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][write][T#1]'
     7/10 snapshots sharing following 34 elements
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.MemorySegmentIndexInput.readVLong(MemorySegmentIndexInput.java:292)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:189)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.lambda$prepareSeekExact$1(SegmentTermsEnum.java:511)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum$$Lambda/0x0000000019b4c000.get(Unknown Source)
       app/org.apache.lucene.backward_codecs@10.3.2/org.apache.lucene.backward_codecs.lucene90.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:599)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.getDocID(PerThreadIDVersionAndSeqNoLookup.java:161)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupVersion(PerThreadIDVersionAndSeqNoLookup.java:136)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.timeSeriesLoadDocIdAndVersion(VersionsAndSeqNoResolver.java:140)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$resolveDocVersion$7(InternalEngine.java:1080)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bb0468.apply(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.performActionWithDirectoryReader(InternalEngine.java:3694)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:1067)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1388)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1365)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1227)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1142)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1052)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:970)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:441)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:291)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:359)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:196)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     3/10 snapshots sharing following 17 elements
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:993)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:764)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:736)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$dispatchedShardOperationOnReplica$7(TransportShardBulkAction.java:688)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda/0x0000000019cb3270.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:686)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnReplica(TransportShardBulkAction.java:83)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.support.replication.TransportWriteAction$2.doRun(TransportWriteAction.java:273)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
   
   100.0% [cpu=40.8%, other=59.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elastic-hot3][refresh][T#9]'
     7/10 snapshots sharing following 53 elements
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method)
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57)
       java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284)
       java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202)
       java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174)
       java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:305)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:128)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.doReset(Lucene90CompressingStoredFieldsReader.java:463)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.reset(Lucene90CompressingStoredFieldsReader.java:448)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:648)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:580)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:382)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:356)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:346)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:49)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:28)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:505)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:485)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2138)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$maybeRefresh$9(InternalEngine.java:2096)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bcc698.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:2096)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$scheduledRefresh$64(IndexShard.java:4403)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019bcc000.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4384)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:1120)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1256)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:138)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)
     3/10 snapshots sharing following 52 elements
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread0(Native Method)
       java.base@26/sun.nio.ch.UnixFileDispatcherImpl.pread(UnixFileDispatcherImpl.java:57)
       java.base@26/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:339)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:307)
       java.base@26/sun.nio.ch.IOUtil.read(IOUtil.java:284)
       java.base@26/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:1202)
       java.base@26/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:1174)
       java.base@26/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:1156)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:182)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:127)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:87)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer$1$2.decompress(SortingStoredFieldsConsumer.java:66)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState.document(Lucene90CompressingStoredFieldsReader.java:619)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.serializedDocument(Lucene90CompressingStoredFieldsReader.java:651)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:669)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.SortingStoredFieldsConsumer.flush(SortingStoredFieldsConsumer.java:114)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:314)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:580)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:382)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:356)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:346)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:49)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:28)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:505)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:485)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
       app/org.apache.lucene.core@10.3.2/org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:2138)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.lambda$maybeRefresh$9(InternalEngine.java:2096)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine$$Lambda/0x0000000019bcc698.get(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:367)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:2096)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.lambda$scheduledRefresh$64(IndexShard.java:4403)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard$$Lambda/0x0000000019bcc000.accept(Unknown Source)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4384)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:1120)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1256)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:138)
       app/org.elasticsearch.server@9.3.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046)
       java.base@26/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
       java.base@26/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
       java.base@26/java.lang.Thread.runWith(Thread.java:1529)
       java.base@26/java.lang.Thread.run(Thread.java:1516)

[RainTown (Kevin Maguire)]

Thanks for the output. Nothing contradicting that you cluster is stuck in a loop, or too close and sometimes over its limit.

If you had more data nodes (and therefore more space) it could buy you some time, you could adjust the ILM policy so that the indices dont need to be force-merged and maybe the cluster would balance and stabilize.

One thing I missed first time reading through:

@kkumar123

Since we are paying for them under the ERU (Elastic Resource Unit) license ...

Well, yes, probably so, 6 ingest does and 6 data nodes is an unusual ratio in my experience. But I dont know your data. You are an enterprise customer? Did you open a support case with Elastic? When I was an enterprise cluster, long time ago now, we had (very knowledgeable) Elastic support engineers joining us on support calls, with various stakeholders, shared screens, sharing API output close to live, and generally helping us resolve our issues.

EDIT: 18 total nodes for a 6-hot-node / 2-frozen node cluster feels like it has been designed by a committee.

[leandrojmp (Leandro Pereira)]

@kkumar123

I suspect the root cause is a combination of small shard sizes, too many total shards, and heavy write traffic on specific data streams. To fix this and better distribute the write load across the hot nodes, I am increasing the primary shard count to 3 and 6 for the high-traffic streams.

Additionally, I noticed our ingest nodes have very low utilization. Since we are paying for them under the ERU (Elastic Resource Unit) license, they feel like wasted spend. I am considering dropping them entirely to save on licensing costs.

What are the specs for your nodes, specially the hot ones?

And how are you indexing your data? Are you using the default index.refresh_interval for all your data? Is it coming from Logstash? Elastic Agent?

Are you using hot-frozen in your ILM, it goes to frozen immediately after it rollover or it wait a couple of time?

Is it traditional VMs or ECK/ECE?

There are some things you may do to improve the performance.

I would ditch the ingest nodes, leave the ingest role to all your hot nodes, there is one that is not an ingest node yet.

Also, what is the reason for the coordinating nodes?

[kkumar123 (Kishor Kumar)]

@RainTown

sharing

The design was purposed by solution architect.

I would have open that ticket but right now i don't access to support portal, will surely do that.

but most problematic thing for me ingest nodes does not have that much load of CPU, i will try to remove these ingest nodes completely instead add hot nodes.

[kkumar123 (Kishor Kumar)]

Hi @leandrojmp ,

here are the spacs

name                               ram.max disk.total disk.avail disk.used node.role
elastic-frozen2        31.3gb      3.9tb    562.3gb     3.4tb f
elastic-hot1           62.7gb      7.9tb      1.9tb       6tb his
elastic-hot5           62.7gb      7.9tb      1.9tb       6tb his
elastic-master1        31.3gb    193.6gb    179.8gb    13.7gb m
elastic-hot6           62.7gb      7.9tb      1.9tb       6tb hs
elastic-master3        31.3gb    193.6gb    180.3gb    13.3gb m
elastic-coordinating1  15.6gb    193.6gb    180.4gb    13.2gb -
elastic-coordinating2  15.6gb    193.6gb    180.3gb    13.2gb -
elastic-ml1            31.3gb     8192pb   8191.9pb    19.2tb lt
elastic-frozen1        31.3gb      3.9tb    561.1gb     3.4tb f
elastic-hot4           62.7gb      7.9tb      1.9tb       6tb his
elastic-hot3           62.7gb      7.9tb      1.9tb       6tb his
elastic-ingest2        31.3gb    193.6gb    179.2gb    14.3gb i
elastic-master2        31.3gb    193.6gb    179.8gb    13.7gb m
elastic-ingest1        62.7gb    193.6gb    178.7gb    14.9gb i
elastic-hot2           62.7gb      7.9tb      1.9tb       6tb his
elastic-ingest3        31.3gb    193.6gb    178.7gb    14.8gb i
elastic-ml2            31.3gb     8192pb   8191.9pb    19.2tb lt

And how are you indexing your data?
Data is ingested via Elastic Agent , Logstash, APM,

Are you using the default index.refresh_interval for all your data?
We are are using our 30s refresh interval, in apm, kubernetes logs, and others but some of them have default.

Are you using hot-frozen in your ILM, it goes to frozen immediately after it rollover or it wait a couple of time?

No after rollover it stays there for sometime then goes to the frozen, like apm data it rollover at 3 days or 30 gb as data is too much like 3 tb per day primary only, shard sizing we have kept 30 GB.

• we are keeping short 7 days hot-frozen policy overall
• there is force merge applied.

Is it traditional VMs?

yes its Oci VMs - Traditional Stand alone self hosted cluster

I would ditch the ingest nodes, leave the ingest role to all your hot nodes, there is one that is not an ingest node yet.

I was about to do it as they are consumption the ERU only i don't see CPU consumption on those

Also, what is the reason for the coordinating nodes?

CCS
In Kibana We have 300+ active users to do Reporting other
And ingestion rate is high like with 1 relplica 13 tb

and also too free up master load of coordinating role from hot.