# Slowness in Kibana and high CPU utilization usage

**URL:** <https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824>\
**Category:** Elasticsearch\
**Created:** [February 8, 2024, 7:04am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824 "2024-02-08T07:04:28Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seemant\_Bind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seemant_bind/32/107045_2.png) [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\
**Post date:** [February 8, 2024, 7:04am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/1 "2024-02-08T07:04:28Z")

</div>

Hi,

I am facing slowness in Kibana Production. Discover and dashboard is taking too much time to load and sometimes getting the error as shown in the first screenshot.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6b8b808fe8b4a469d324030d5642424050e4d49a.png)

I checked the cluster status from the stack monitoring and found that one of my nodes had almost 100% CPU utilization throughout the day, I tried restarting the cluster but still it went back to 100% CPU utilization and can see still see the slowness.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e6fb8bcbf826f1f5a16eee24603b8e97e437fff.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/d/dd7a983ccbdf33ac6f0dd3b341b83d63c595c811.jpeg)

Then I checked the cluster log and found below:

```auto
[2024-02-07T09:01:24,958][ERROR][o.e.x.w.Watcher] [node-1] watch history could not be written [watch_id1_7a318241-b965-47d5-a80e-f4fe0d412739-2024-02-07T09:01:03.356624867Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,055][ERROR][o.e.x.w.Watcher] [node-1] watch history could not be written [watch_id2__67cde037-f556-4a9f-a541-dc4df43bf787-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,103][ERROR][o.e.x.w.Watcher] [node-1] watch history could not be written [watch_id3__8cfb38f3-ef5d-43e4-92c2-41477d16af6e-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
[2024-02-07T09:02:04,420][ERROR][o.e.x.w.Watcher] [node-1] watch history could not be written [watch_id4__104ffd97-fc21-47f2-88ef-38656438b433-2024-02-07T09:02:03.390800241Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]

```

We currently have around 140 active watcher scripts running. Can you please guide us what exactly is causing the issue? Is there any correlation between the above watcher errors and slowness in Kibana/High CPU utilisation? Let me know if you need any additional information.  
Below is my ELK Stack Details -  
ELK Version: 7.11.1  
ES Node: 3  
Dedicated servers for Kibana and logstash  
Kibana Users: 154

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 8, 2024, 7:14am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/2 "2024-02-08T07:14:54Z")

</div>

Try running the [hot threads API](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/cluster-nodes-hot-threads.html) when CPU is high and post the output here. This will show what CPU is being spent on.

> [@Seemant\_Bind](#):
>
> ELK Version: 7.11.1

This is very old. I would recommend upgrading to at least 7.17.

What is the specification of the Elasticsearch nodes with respect to CPU and RAM? What type of storage are you using? Local SSDs? What does await and disk utilisation look like?

> [@Seemant\_Bind](#):
>
> We currently have around 140 active watcher scripts running.

Are any of these using a lot of CPU intensive scripting? How often do they trigger?

> [@Seemant\_Bind](#):
>
> Kibana Users: 154

Do your dashboard use a lot of runtime or scripted fields that could result in high CPU usage?

---

<div class="post-metadata">

**Author:** ![Seemant\_Bind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seemant_bind/32/107045_2.png) [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\
**Post date:** [February 8, 2024, 7:48am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/3 "2024-02-08T07:48:36Z")

</div>

_Try running the [hot threads API](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/cluster-nodes-hot-threads.html) when CPU is high and post the output here. This will show what CPU is being spent on._

Currently the CPU usage is normal, we will paste it once we see any high CPU usage

_What is the specification of the Elasticsearch nodes with respect to CPU and RAM? What type of storage are you using? Local SSDs? What does await and disk utilisation look like?_

| | Disk Space | Memory | Cores |
| --- | --- | --- | --- |
| Elasticsearch1 | 1024.4 GB | 32.88 GB | 8 |
| Elasticsearch2 | 1024.4 GB | 32.88 GB | 8 |
| Elasticsearch3 | 1024.4 GB | 32.88 GB | 8 |
| Kibana | 200.4 GB | 32.88 GB | 8 |
| Logstash | 200.4 GB | 32.88 GB | 8 |

_Are any of these using a lot of CPU intensive scripting? How often do they trigger?_

Watcher trigger varies from script to scripts, we have many scripts with the trigger interval of 1minute,5minute,30 min and there are some script with trigger of 1 week, 1 month.

_Do your dashboard use a lot of runtime or scripted fields that could result in high CPU usage?_

We have some scripted field but it has been running since last 2-3 years but never faced such issue with the same scripted fields

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 8, 2024, 8:17am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/4 "2024-02-08T08:17:56Z")

</div>

> [@Seemant\_Bind](#):
>
> What is the specification of the Elasticsearch nodes with respect to CPU and RAM? What type of storage are you using? Local SSDs? What does await and disk utilisation look like?

You did not mention what type of storage you are using, just the size. What does disk I/O and await look like when CPU usage is high?

> [@Seemant\_Bind](#):
>
> We have some scripted field but it has been running since last 2-3 years but never faced such issue with the same scripted fields

That does not necessarily mean it is not the problem. Have the amount of data in the cluster increased over time? Have the number of Watches and users of Kibana grown over time?

---

<div class="post-metadata">

**Author:** ![Seemant\_Bind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seemant_bind/32/107045_2.png) [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\
**Post date:** [February 8, 2024, 10:25am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/5 "2024-02-08T10:25:56Z")

</div>

> You did not mention what type of storage you are using, just the size. What does disk I/O and await look like when CPU usage is high?

We are using AWS EC2 instance for the storage. Please see the below I/O operation rate for all the 3 nodes.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/9/4923ba7409b07018f4e01f643d8778f1a8affcf8.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da293ad1266873a74a2e952e2d16b2f3f692bb87.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cfdc18d7b623e1912964025fa55d3cb324a49551.png)

> That does not necessarily mean it is not the problem. Have the amount of data in the cluster increased over time? Have the number of Watches and users of Kibana grown over time?

Yes, the data has been increased. We recently had to make some changes in the watcher scripts for that we had disabled around 200 scripts and created around 140 alert with the minor change in the scripts. There hasnt been significant growth in the users of Kibana

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 8, 2024, 11:03am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/6 "2024-02-08T11:03:42Z")

</div>

> [@Seemant\_Bind](#):
>
> We are using AWS EC2 instance for the storage.

Which instance type are you using? Are you using EBS storage? If so, what type of EBS storage?

---

<div class="post-metadata">

**Author:** ![Seemant\_Bind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seemant_bind/32/107045_2.png) [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\
**Post date:** [February 8, 2024, 11:18am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/7 "2024-02-08T11:18:51Z")

</div>

> Which instance type are you using? Are you using EBS storage? If so, what type of EBS storage?

This information is with the client.  
We have seen some high usage for node2 in past one hour and below is the output running the hot thread API

```auto
::: {node-2}{XhSgnpKfQC-R8Ie74MU2XA}{Ba11tLLtRgqVmcbgBlNMeg}{172.31.8.228}{172.31.8.228:9300}{cdfhilmrstw}{ml.machine_memory=33675792384, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=17179869184, transform.node=true}
   Hot threads at 2024-02-08T11:15:16.021Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   81.5% (407.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#10]'
     2/10 snapshots sharing following 51 elements
       app//org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:275)
       app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
       app//org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:264)
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
       app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
       app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
       app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     4/10 snapshots sharing following 55 elements
       java.base@16/java.nio.ByteBuffer.get(ByteBuffer.java:797)
       java.base@16/java.nio.DirectByteBuffer.get(DirectByteBuffer.java:346)
       app//org.apache.lucene.store.ByteBufferGuard.getBytes(ByteBufferGuard.java:94)
       app//org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:99)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$TermsDict.next(Lucene80DocValuesProducer.java:1135)
       app//org.apache.lucene.index.OrdinalMap$TermsEnumIndex.next(OrdinalMap.java:59)
       app//org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:258)
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
       app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
       app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
       app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     4/10 snapshots sharing following 48 elements
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168)
       app//org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:53)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:140)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalDirect(AbstractIndexOrdinalsFieldData.java:33)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:163)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$6721/0x0000000801b2f420.load(Unknown Source)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:160)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobalInternal(AbstractIndexOrdinalsFieldData.java:128)
       app//org.elasticsearch.index.fielddata.plain.AbstractIndexOrdinalsFieldData.loadGlobal(AbstractIndexOrdinalsFieldData.java:94)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:283)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:256)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:299)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.access$000(TermsAggregatorFactory.java:44)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$1.build(TermsAggregatorFactory.java:105)
       app//org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:249)
       app//org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:36)
       app//org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:63)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:187)
       app//org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:197)
       app//org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:35)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:138)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
   
   79.5% (397.2ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#7]'
     2/10 snapshots sharing following 49 elements
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.termsEnum(Lucene80DocValuesProducer.java:1049)
       app//org.apache.lucene.index.SingletonSortedSetDocValues.termsEnum(SingletonSortedSetDocValues.java:107)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
       app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     2/10 snapshots sharing following 44 elements
       app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:182)
       app//org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:338)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:838)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:157)
       app//org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.bulkScorer(QueryToFilterAdapter.java:269)
       app//org.elasticsearch.search.aggregations.bucket.filter.QueryToFilterAdapter.collect(QueryToFilterAdapter.java:227)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:451)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
       app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     2/10 snapshots sharing following 43 elements
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
       app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     2/10 snapshots sharing following 45 elements
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
       app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     2/10 snapshots sharing following 60 elements
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$TermsDict.<init>(Lucene80DocValuesProducer.java:1107)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.termsEnum(Lucene80DocValuesProducer.java:1049)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$BaseSortedDocValues.<init>(Lucene80DocValuesProducer.java:1022)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$22.<init>(Lucene80DocValuesProducer.java:978)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer.getSorted(Lucene80DocValuesProducer.java:978)
       app//org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer.getSortedSet(Lucene80DocValuesProducer.java:1411)
       app//org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.getSortedSet(PerFieldDocValuesFormat.java:344)
       app//org.apache.lucene.index.SegmentDocValuesProducer.getSortedSet(SegmentDocValuesProducer.java:124)
       app//org.apache.lucene.index.CodecReader.getSortedSetDocValues(CodecReader.java:178)
       app//org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:404)
       app//org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:404)
       app//org.apache.lucene.index.DocValues.getSortedSet(DocValues.java:406)
       app//org.elasticsearch.index.fielddata.plain.SortedSetBytesLeafFieldData.getOrdinalsValues(SortedSetBytesLeafFieldData.java:41)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.getOrLoadTermsEnums(GlobalOrdinalsIndexFieldData.java:155)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer.access$600(GlobalOrdinalsIndexFieldData.java:139)
       app//org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsIndexFieldData$Consumer$1.getOrdinalsValues(GlobalOrdinalsIndexFieldData.java:221)
       app//org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:285)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.getLeafCollector(GlobalOrdinalsStringTermsAggregator.java:111)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:206)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.collectSubs(FiltersAggregator.java:449)
       app//org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator$FilterByFilter.getLeafCollector(FiltersAggregator.java:396)
       app//org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207)
       app//org.elasticsearch.search.aggregations.AdaptingAggregator.getLeafCollector(AdaptingAggregator.java:81)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94)
       app//org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75)
       app//org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:175)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
   
   71.5% (357.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#8]'
     3/10 snapshots sharing following 42 elements
       app//org.elasticsearch.search.aggregations.InternalOrder.lambda$comparingKeys$0(InternalOrder.java:322)
       app//org.elasticsearch.search.aggregations.InternalOrder$$Lambda$6613/0x0000000801b01920.compare(Unknown Source)
       java.base@16/java.util.Collections$ReverseComparator2.compare(Collections.java:5365)
       app//org.elasticsearch.search.aggregations.InternalOrder$SimpleOrder.lambda$partiallyBuiltBucketComparator$0(InternalOrder.java:255)
       app//org.elasticsearch.search.aggregations.InternalOrder$SimpleOrder$$Lambda$6757/0x0000000801b3b860.compare(Unknown Source)
       app//org.elasticsearch.search.aggregations.bucket.terms.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:25)
       app//org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:275)
       app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
       app//org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:152)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy$1.accept(GlobalOrdinalsStringTermsAggregator.java:595)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$DenseGlobalOrds.forEach(GlobalOrdinalsStringTermsAggregator.java:462)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:584)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182)
       app//org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143)
       app//org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)
     7/10 snapshots sharing following 35 elements
       app//org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:202)
       app//org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:152)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy$1.accept(GlobalOrdinalsStringTermsAggregator.java:595)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$DenseGlobalOrds.forEach(GlobalOrdinalsStringTermsAggregator.java:462)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:584)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557)
       app//org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182)
       app//org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143)
       app//org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145)
       app//org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1412)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6847/0x0000000801b57198.accept(Unknown Source)
       app//org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1475)
       app//org.elasticsearch.indices.IndicesService$$Lambda$6850/0x0000000801b52738.get(Unknown Source)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
       app//org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
       app//org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:423)
       app//org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
       app//org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1481)
       app//org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1406)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394)
       app//org.elasticsearch.search.SearchService$$Lambda$6324/0x0000000801a73310.get(Unknown Source)
       app//org.elasticsearch.search.SearchService$$Lambda$6325/0x0000000801a73538.get(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$6139/0x0000000801a3e290.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@16/java.lang.Thread.run(Thread.java:831)

```

---

<div class="post-metadata">

**Author:** ![Vijayakumar\_Kannan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijayakumar_kannan/32/34873_2.png) [@Vijayakumar\_Kannan](https://discuss.elastic.co/u/Vijayakumar_Kannan)\
**Post date:** [February 8, 2024, 11:32am UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/8 "2024-02-08T11:32:46Z")

</div>

Your cluster is overwhelmed by the read (search) iops. it can be multiple factors , primarily if data is large better to have dedicated warm nodes or add additional node could help. check if you change the instance disk type to increased performance disk type like flash based .

---

<div class="post-metadata">

**Author:** ![Seemant\_Bind](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seemant_bind/32/107045_2.png) [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\
**Post date:** [February 8, 2024, 1:26pm UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/9 "2024-02-08T13:26:55Z")

</div>

Is there any correlation between high CPU usage and the error we are getting in the cluster log mention above? Could that be affecting the cluster?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2024, 1:27pm UTC](https://discuss.elastic.co/t/slowness-in-kibana-and-high-cpu-utilization-usage/352824/10 "2024-03-07T13:27:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.