Hi,
I am facing slowness in Kibana Production. Discover and dashboard is taking too much time to load and sometimes getting the error as shown in the first screenshot.
I checked the cluster status from the stack monitoring and found that one of my nodes had almost 100% CPU utilization throughout the day, I tried restarting the cluster but still it went back to 100% CPU utilization and can see still see the slowness.
Then I checked the cluster log and found below:
[2024-02-07T09:01:24,958][ERROR][o.e.x.w.Watcher ] [node-1] watch history could not be written [watch_id1_7a318241-b965-47d5-a80e-f4fe0d412739-2024-02-07T09:01:03.356624867Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,055][ERROR][o.e.x.w.Watcher ] [node-1] watch history could not be written [watch_id2__67cde037-f556-4a9f-a541-dc4df43bf787-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,103][ERROR][o.e.x.w.Watcher ] [node-1] watch history could not be written [watch_id3__8cfb38f3-ef5d-43e4-92c2-41477d16af6e-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,420][ERROR][o.e.x.w.Watcher ] [node-1] watch history could not be written [watch_id4__104ffd97-fc21-47f2-88ef-38656438b433-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
We currently have around 140 active watcher scripts running. Can you please guide us what exactly is causing the issue? Is there any correlation between the above watcher errors and slowness in Kibana/High CPU utilisation? Let me know if you need any additional information.
Try running the hot threads API when CPU is high and post the output here. This will show what CPU is being spent on.
Seemant_Bind:
ELK Version: 7.11.1
This is very old. I would recommend upgrading to at least 7.17.
What is the specification of the Elasticsearch nodes with respect to CPU and RAM? What type of storage are you using? Local SSDs? What does await and disk utilisation look like?
Are any of these using a lot of CPU intensive scripting? How often do they trigger?
Seemant_Bind:
Kibana Users: 154
Do your dashboard use a lot of runtime or scripted fields that could result in high CPU usage?
Try running the hot threads API when CPU is high and post the output here. This will show what CPU is being spent on.
Currently the CPU usage is normal, we will paste it once we see any high CPU usage
What is the specification of the Elasticsearch nodes with respect to CPU and RAM? What type of storage are you using? Local SSDs? What does await and disk utilisation look like?
Disk Space
Memory
Cores
Elasticsearch1
1024.4 GB
32.88 GB
8
Elasticsearch2
1024.4 GB
32.88 GB
8
Elasticsearch3
1024.4 GB
32.88 GB
8
Kibana
200.4 GB
32.88 GB
8
Logstash
200.4 GB
32.88 GB
8
Are any of these using a lot of CPU intensive scripting? How often do they trigger?
Watcher trigger varies from script to scripts, we have many scripts with the trigger interval of 1minute,5minute,30 min and there are some script with trigger of 1 week, 1 month.
Do your dashboard use a lot of runtime or scripted fields that could result in high CPU usage?
We have some scripted field but it has been running since last 2-3 years but never faced such issue with the same scripted fields
You did not mention what type of storage you are using, just the size. What does disk I/O and await look like when CPU usage is high?
That does not necessarily mean it is not the problem. Have the amount of data in the cluster increased over time? Have the number of Watches and users of Kibana grown over time?
You did not mention what type of storage you are using, just the size. What does disk I/O and await look like when CPU usage is high?
We are using AWS EC2 instance for the storage. Please see the below I/O operation rate for all the 3 nodes.
That does not necessarily mean it is not the problem. Have the amount of data in the cluster increased over time? Have the number of Watches and users of Kibana grown over time?
Yes, the data has been increased. We recently had to make some changes in the watcher scripts for that we had disabled around 200 scripts and created around 140 alert with the minor change in the scripts. There hasnt been significant growth in the users of Kibana
Which instance type are you using? Are you using EBS storage? If so, what type of EBS storage?
Which instance type are you using? Are you using EBS storage? If so, what type of EBS storage?
This information is with the client.
::: {node-2}{XhSgnpKfQC-R8Ie74MU2XA}{Ba11tLLtRgqVmcbgBlNMeg}{172.31.8.228}{172.31.8.228:9300}{cdfhilmrstw}{ml.machine_memory=33675792384, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=17179869184, transform.node=true}
Hot threads at 2024-02-08T11:15:16.021Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
81.5% (407.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#10]'
2/10 snapshots sharing following 51 elements
app//org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:275)
app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
app//org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:264)
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
4/10 snapshots sharing following 55 elements
java.base@16/java.nio.ByteBuffer.get(ByteBuffer.java:797)
java.base@16/java.nio.DirectByteBuffer.get(DirectByteBuffer.java:346)
app//org.apache.lucene.store.ByteBufferGuard.getBytes(ByteBufferGuard.java:94)
app//org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:99)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$TermsDict.next(Lucene80DocValuesProducer.java:1135)
app//org.apache.lucene.index.OrdinalMap$TermsEnumIndex.next(OrdinalMap.java:59)
app//org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:258)
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
4/10 snapshots sharing following 48 elements
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
79.5% (397.2ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#7]'
2/10 snapshots sharing following 49 elements
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.termsEnum(Lucene80DocValuesProducer.java:1049)
app//org.apache.lucene.index.SingletonSortedSetDocValues.termsEnum(SingletonSortedSetDocValues.java:107)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
2/10 snapshots sharing following 44 elements
app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182)
app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338)
app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:838)
app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:157)
app//org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.bulkScorer(QueryToFilterAdapter.java:269)
app//org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.collect(QueryToFilterAdapter.java:227)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:451)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
2/10 snapshots sharing following 43 elements
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
2/10 snapshots sharing following 45 elements
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
2/10 snapshots sharing following 60 elements
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$TermsDict.<init>(Lucene80DocValuesProducer.java:1107)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.termsEnum(Lucene80DocValuesProducer.java:1049)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.<init>(Lucene80DocValuesProducer.java:1022)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$22.<init>(Lucene80DocValuesProducer.java:978)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer.getSorted(Lucene80DocValuesProducer.java:978)
app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer.getSortedSet(Lucene80DocValuesProducer.java:1411)
app//org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.getSortedSet(PerFieldDocValuesFormat.java:344)
app//org.apache.lucene.index.SegmentDocValuesProducer.getSortedSet(SegmentDocValuesProducer.java:124)
app//org.apache.lucene.index.CodecReader.getSortedSetDocValues(CodecReader.java:178)
app//org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:404)
app//org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:404)
app//org.apache.lucene.index.DocValues.getSortedSet(DocValues.java:406)
app//org.elasticsearch.index.fielddata.plain.SortedSetBytesLeafFieldData.getOrdinalsValues(SortedSetBytesLeafFieldData.java:41)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
71.5% (357.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#8]'
3/10 snapshots sharing following 42 elements
app//org.elasticsearch.search.aggregations.InternalOrder.lambda$comparingKeys$0(InternalOrder.java:322)
app//org.elasticsearch.search.aggregations.InternalOrder$$Lambda$6613/0x0000000801b01920.compare(Unknown Source)
java.base@16/java.util.Collections$ReverseComparator2.compare(Collections.java:5365)
app//org.elasticsearch.search.aggregations.InternalOrder$SimpleOrder.lambda$partiallyBuiltBucketComparator$0(InternalOrder.java:255)
app//org.elasticsearch.search.aggregations.InternalOrder$SimpleOrder$$Lambda$6757/0x0000000801b3b860.compare(Unknown Source)
app//org.elasticsearch.search.aggregations.bucket.terms.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:25)
app//org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:275)
app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
app//org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:152)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy$1.accept(GlobalOrdinalsStringTermsAggregator.java:595)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$DenseGlobalOrds.forEach(GlobalOrdinalsStringTermsAggregator.java:462)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:584)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182)
app//org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143)
app//org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
7/10 snapshots sharing following 35 elements
app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
app//org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:152)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy$1.accept(GlobalOrdinalsStringTermsAggregator.java:595)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$DenseGlobalOrds.forEach(GlobalOrdinalsStringTermsAggregator.java:462)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:584)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557)
app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182)
app//org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143)
app//org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71)
app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145)
app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
java.base@16/java.lang.Thread.run(Thread.java:831)
Your cluster is overwhelmed by the read (search) iops. it can be multiple factors , primarily if data is large better to have dedicated warm nodes or add additional node could help. check if you change the instance disk type to increased performance disk type like flash based .
Is there any correlation between high CPU usage and the error we are getting in the cluster log mention above? Could that be affecting the cluster?
