# Abnormally high CPU usage for specific queries/dashboards **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2024, 12:08pm UTC](https://discuss.elastic.co/t/abnormally-high-cpu-usage-for-specific-queries-dashboards/359543 "2024-05-15T12:08:38Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![benpi](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@benpi](https://discuss.elastic.co/u/benpi)\ **Post date:** [May 15, 2024, 12:08pm UTC](https://discuss.elastic.co/t/abnormally-high-cpu-usage-for-specific-queries-dashboards/359543/1 "2024-05-15T12:08:38Z")
Hi, i will start by sharing details regarding our infrastructure. We are running multiple Community Edition Elasticsearch clusters in parallel, all of which are connected via the "Remote Clusters" feature to a single instance of Kibana. Each cluster has 3 master nodes, and a differing amount of data nodes. Most of the clusters work well and as expected, but one of them (which i will be expanding upon) is having a lot of Load Average/CPU Usage issues when trying to process certain queries. Cluster details: Running self-managed on AWS (EC2 + EKS) Logstashes are running inside EKS 3 Master Nodes (m5n.large EC2) 32 Data Nodes (i3en.2xlarge EC2) Storage is ephemeral and hosted on SSD's residing on each Data Node. We have 5TB of Ephemeral storage on each node, which sums up for a total disk space of 160TB, we are currently using 80% of that storage space, and are adding nodes on a weekly/bi-weekly basis. Each data node has 64GB of RAM and 8 vCPU's - resulting in a total of 2048GB of memory (Heap size on each data node is set to 27GB), and 256 vCPU's Sharding - we went with the best practice of sizing shards up to 50GB, most of the shards are averaging between 35GB - 50GB. Indices - are being rolled over daily/weekly, depending on their size Our issue is this - when using certain queries/dashboards, Load Average increases immensely across all of our Data Nodes: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7cdef11dea161f5f38ed6f49b46f604279ee8ce6.png) In the image above you can see the Load Average of our nodes rapidly increasing due to running _ONE_ query. I sadly cannot share the query itself due to it containing sensitive data, but will provide some technical details regarding it: Query is searching across data from the last month, so at the time of writing this thread, 15 days. It has around 30~ search filters. Each document searched upon has on AVERAGE around a 100~ fields. We tried many, many different solutions to this issue, and have yet to find one that works. This causes many issues with time-outs on some of the queries, and our clients cannot use the cluster normally. We did not expect for a cluster with such a large amount of compute power to choke on a single query, and have exhausted most of our options, which is why we are turning to the forums for help with this issue. Please let me know what other information i can provide to assist with solving this issue, and thanks in advance 🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 15, 2024, 12:26pm UTC](https://discuss.elastic.co/t/abnormally-high-cpu-usage-for-specific-queries-dashboards/359543/2 "2024-05-15T12:26:50Z")
Which version of Elasticsearch are you using? What does the [hot threads API](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/cluster-nodes-hot-threads.html) show during the time the load on the cluster increases due to this query? ---
**Author:** ![benpi](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@benpi](https://discuss.elastic.co/u/benpi)\ **Post date:** [May 15, 2024, 12:43pm UTC](https://discuss.elastic.co/t/abnormally-high-cpu-usage-for-specific-queries-dashboards/359543/3 "2024-05-15T12:43:48Z")
Hi Christian, forgot to mention the version - it is v7.17.8 (Elasticsearch, Kibana & LogStash) Adding Hot Threads API output below: ```auto ::: {ip-10-6-35-105.ec2.internal}{equMatkVRzy9cusM6mhZ1w}{Zopk0v3uTeCnAZFfLkaohg}{10.6.35.105}{10.6.35.105:9300}{cdfhrstw}{xpack.installed=true, transform.node=true} Hot threads at 2024-05-15T12:38:28.045Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=74.5%, other=25.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-6-35-105.ec2.internal][search][T#13]' 5/10 snapshots sharing following 51 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:495) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 3/10 snapshots sharing following 51 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:524) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 2/10 snapshots sharing following 55 elements app//org.apache.lucene.codecs.blocktree.CompressionAlgorithm$3.read(CompressionAlgorithm.java:51) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:201) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame.loadNextFloorBlock(IntersectTermsEnumFrame.java:132) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.popPushNext(IntersectTermsEnum.java:320) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:523) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 100.0% [cpu=70.7%, other=29.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-6-35-105.ec2.internal][search][T#4]' 2/10 snapshots sharing following 51 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:523) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 7/10 snapshots sharing following 51 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:495) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) unique snapshot app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:495) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:135) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:162) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 100.0% [cpu=67.4%, other=32.6%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-6-35-105.ec2.internal][search][T#11]' 9/10 snapshots sharing following 50 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) unique snapshot app//org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame.decodeMetaData(IntersectTermsEnumFrame.java:348) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.postings(IntersectTermsEnum.java:219) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.postings(FilterLeafReader.java:219) app//org.elasticsearch.search.internal.FieldUsageTrackingDirectoryReader$FieldUsageTrackingLeafReader$FieldUsageTrackingTermsEnum.postings(FieldUsageTrackingDirectoryReader.java:291) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.postings(FilterLeafReader.java:219) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:188) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:704) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:866) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$6726/0x000000080212b1d8.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$6727/0x000000080212b8f0.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6701/0x000000080211fd10.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6702/0x0000000802120000.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6703/0x0000000802120210.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) ::: {ip-10-6-34-96.ec2.internal}{WHPxArD1TemP1LWNTNy3Qg}{WEdJiuJtQ0eOVmFPZqtf0w}{10.6.34.96}{10.6.34.96:9300}{cdfhrstw}{xpack.installed=true, transform.node=true} Hot threads at 2024-05-15T12:38:28.045Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 100.0% [cpu=66.8%, other=33.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-6-34-96.ec2.internal][search][T#7]' 9/10 snapshots sharing following 33 elements app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$7250/0x0000000802288658.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$7251/0x0000000802288d70.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6738/0x000000080218b1d8.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6740/0x000000080218b5f8.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6742/0x000000080218ba18.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) unique snapshot app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$DenseGlobalOrds.forEach(GlobalOrdinalsStringTermsAggregator.java:474) app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:602) app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:575) app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) app//org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:154) app//org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:67) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:104) app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$26(IndicesService.java:1523) app//org.elasticsearch.indices.IndicesService$$Lambda$7250/0x0000000802288658.accept(Unknown Source) app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$27(IndicesService.java:1589) app//org.elasticsearch.indices.IndicesService$$Lambda$7251/0x0000000802288d70.get(Unknown Source) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:178) app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:161) app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:419) app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:124) app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1595) app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1517) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:456) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:622) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) app//org.elasticsearch.search.SearchService$$Lambda$6738/0x000000080218b1d8.get(Unknown Source) app//org.elasticsearch.search.SearchService$$Lambda$6740/0x000000080218b5f8.get(Unknown Source) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) app//org.elasticsearch.action.ActionRunnable$$Lambda$6742/0x000000080218ba18.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base@19.0.1/java.lang.Thread.run(Thread.java:1589) 100.0% [cpu=66.3%, other=33.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-6-34-96.ec2.internal][search][T#9]' 3/10 snapshots sharing following 43 elements app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:495) app//org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:355) app//org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:194) app//org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:172) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:190) app//org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:234) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:148) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:755) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:788) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:344) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182) app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:869) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:244) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:191) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:255) app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:212) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:98) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:458) ``` Had to cut it a little short due to the reply being too big.