kunalyadav
October 2, 2024, 4:09pm
1
We have an Elasticsearch cluster where a data node randomly becomes hot which results in an increase in its CPU and reduction in its search rate (no. of queries processed per second by the node). There is no considerable change in the traffic when this happens.
The cluster is also fairly balanced in terms of no. of shards and disk usage.
For some work, I created a new index with 30 primary shards but then deleted it within 5 mins because of a typo in the index name. A few minutes later I noticed that the CPU on the hot node dropped and its search rate increased.
I compared the shards allocation of this node before and after creating this index and there was no change.
What could be the reason for this fluctuation in the CPU? Please advise
We are running this ES version on c7gd.16xlarge AWS EC2 nodes:
"version": {
"number": "8.6.1",
"build_flavor": "default",
"build_type": "rpm",
"build_hash": "180c9830da956993e59e2cd70eb32b5e383ea42c",
"build_date": "2023-01-24T21:35:11.506992272Z",
"build_snapshot": false,
"lucene_version": "9.4.2",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
Do you see any increase in disk I/O that could indicate that merging is going on? That is something that can be triggered anytime without any specific pattern.
If you can capture the hot threads API during the elevated CPU usage that would also give a clue.
kunalyadav
October 3, 2024, 9:48am
3
These are the read and write IOPS of the cluster during this time. There are a few spikes nothing elevated
These are the read and write IOPS of the hot node. Read IOPS seems to be increasing but there's no change in the write IOPS.
We tried the hot threads API while the node was hot and got these results. There were primarily search requests with high CPU.
::: {es-data-i-ff7}{XhIOCSQUTc2olJOhsoyvwQ}{Eyr2YghwQnu3si9pZaUWww}{es-data-i-ff7}{10.0.54.54}{10.0.54.54:9300}{d}{xpack.installed=true, aws_availability_zone=us-east-1d}
Hot threads at 2024-10-02T10:03:13.268Z, interval=5s, busiestThreads=10, ignoreIdleThreads=true:
62.2% [cpu=61.4%, other=0.8%] (3.1s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#75]'
10/10 snapshots sharing following 45 elements
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
47.5% [cpu=46.7%, other=0.8%] (2.3s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#28]'
3/10 snapshots sharing following 54 elements
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:542)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
3/10 snapshots sharing following 56 elements
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:192)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:202)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:530)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
4/10 snapshots sharing following 54 elements
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:543)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
47.3% [cpu=47.1%, other=0.2%] (2.3s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#31]'
2/10 snapshots sharing following 33 elements
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
6/10 snapshots sharing following 13 elements
java.base@19.0.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@19.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
/org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:152)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
46.2% [cpu=46.2%, other=0.0%] (2.3s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#46]'
10/10 snapshots sharing following 33 elements
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
kunalyadav
October 3, 2024, 9:49am
4
Other half of the hot threads API result
44.8% [cpu=44.0%, other=0.8%] (2.2s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#66]'
2/10 snapshots sharing following 24 elements
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
8/10 snapshots sharing following 13 elements
java.base@19.0.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@19.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
/org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:152)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
41.9% [cpu=40.9%, other=1.0%] (2s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#19]'
2/10 snapshots sharing following 40 elements
/org.apache.lucene.search.Boolean2ScorerSupplier.req(Boolean2ScorerSupplier.java:196)
/org.apache.lucene.search.Boolean2ScorerSupplier.getInternal(Boolean2ScorerSupplier.java:116)
/org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:96)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:503)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
7/10 snapshots sharing following 13 elements
java.base@19.0.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@19.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
/org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:152)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
unique snapshot
/org.apache.lucene.util.LSBRadixSorter.reorder(LSBRadixSorter.java:51)
/org.apache.lucene.util.LSBRadixSorter.sort(LSBRadixSorter.java:65)
/org.apache.lucene.util.LSBRadixSorter.sort(LSBRadixSorter.java:102)
/org.apache.lucene.util.DocIdSetBuilder.build(DocIdSetBuilder.java:275)
/org.apache.lucene.search.PointRangeQuery$1$4.get(PointRangeQuery.java:349)
/org.apache.lucene.search.Boolean2ScorerSupplier.req(Boolean2ScorerSupplier.java:196)
/org.apache.lucene.search.Boolean2ScorerSupplier.getInternal(Boolean2ScorerSupplier.java:116)
/org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:96)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:503)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:917)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
40.4% [cpu=39.6%, other=0.8%] (2s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#57]'
5/10 snapshots sharing following 54 elements
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:543)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
2/10 snapshots sharing following 57 elements
/org.apache.lucene.codecs.lucene90.blocktree.CompressionAlgorithm$3.read(CompressionAlgorithm.java:45)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:192)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:202)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:530)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
3/10 snapshots sharing following 56 elements
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.next(IntersectTermsEnumFrame.java:244)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.popPushNext(IntersectTermsEnum.java:351)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:542)
/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:372)
/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:228)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:137)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:169)
/org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.scorer(MultiTermQueryConstantScoreWrapper.java:242)
/org.apache.lucene.search.Weight.scorerSupplier(Weight.java:136)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
app/org.apache.lucene.join@9.4.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:144)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:798)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:159)
/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:533)
/org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:499)
/org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
/org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:395)
/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:932)
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
38.8% [cpu=38.5%, other=0.3%] (1.9s out of 5s) cpu usage by thread 'elasticsearch[es-data-i-ff7][search][T#64]'
2/10 snapshots sharing following 33 elements
/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:165)
/org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:241)
/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:193)
/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$29(IndicesService.java:1517)
/org.elasticsearch.indices.IndicesService$$Lambda$7870/0x00000008023a3c08.accept(Unknown Source)
/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$30(IndicesService.java:1583)
/org.elasticsearch.indices.IndicesService$$Lambda$7871/0x00000008023a84b8.get(Unknown Source)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174)
/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157)
/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:120)
/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1589)
/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1511)
/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:468)
/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:635)
/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
/org.elasticsearch.search.SearchService$$Lambda$7493/0x00000008023206a8.get(Unknown Source)
/org.elasticsearch.search.SearchService$$Lambda$7494/0x00000008023208b8.get(Unknown Source)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
8/10 snapshots sharing following 13 elements
java.base@19.0.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@19.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
/org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:152)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
kunalyadav
October 7, 2024, 9:31am
5
Our cluster has 30 data nodes and 25 indices. 18 indices have 30 primary and 30 replica shards while the rest 7 indices have 20 primary and 40 replica shards. This ensures that all the shards are evenly balanced among all the data nodes.
We noticed the CPU on one of the data nodes started to increase again and search rate on this node decreased.
We decided to disable Adaptive replica selection to see what will happen if all the data nodes receive search traffic in round-robin.
We noticed that CPU spiked even more on the hot node and another data node also became hot.
The search rate became even as expected.
We also noticed that the search queue started to fill on the hot node.
Any idea what might be wrong and what we should investigate next? @DavidTurner can you help here?
It's very rude to ping random volunteers to demand their attention like that. Please don't do it again. Sorry but I don't have the spare capacity to investigate this further.
1 Like
