# High CPU Usage on a few data nodes / Hotspotting of data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2025, 11:45am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954 "2025-12-10T11:45:07Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 11, 2025, 12:52pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/29 "2025-12-11T12:52:38Z")
We are using 63 data nodes, 3 master and 5 data nodes…sharing the machine specs for each category: - data node: 320gb ssd, 20 cores, 53gb memory per node - co-ordinator node: 4 cores, 53gb memory per node - master node: 4 cores, 24gb memory per node ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2025, 1:06pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/30 "2025-12-11T13:06:29Z")
> [@Lakshya\_Gupta](#): > > We are using 63 data nodes, 3 master and 5 data nodes…sharing the machine specs for each category: > > - data node: 320gb ssd, 20 cores, 53gb memory per node > - co-ordinator node: 4 cores, 53gb memory per node Odd sizes. Are these VMs? Is the SSD local or accessed via network? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 11, 2025, 1:07pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/31 "2025-12-11T13:07:43Z")
Since you wrote: > [@Lakshya\_Gupta](#): > > Tried increasing number of cores from 20 cores per machine to 32 cores per machine I presume we are talking here about either some VMs, or cloud instances, or similar, right? [\*Life was so much simpler when ppl could say "it's a Dell PowerServe 220, or a Hp ProLiant G36, ...".] > [@Lakshya\_Gupta](#): > > data node: 320gb ssd, 20 cores, 53gb memory per node That 320gb SSD, is it locally attached (in the very real sense) to the host and passed thru to the VM, or its some AWS "EBS storage type", or ... ? Or is it maybe "320gb from a large pool of SSD storage provisioned by the VMware team to the VMs", like happens in a typical corporate environments. The point isn't really to know the precise detail, but to get a sense. And you have access to run commands like iostat on the data nodes? if so, what do they show? btw, I am a details guy, and > [@Lakshya\_Gupta](#): > > `Hot threads at 2025-07-24T08:42:35.590Z,` Not exactly recent data, is it? 🙂 ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 11, 2025, 1:10pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/32 "2025-12-11T13:10:34Z")
Yes, these are VMs…we have the capability to configure machines of any specs. For local or network, let me get back on that. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 11, 2025, 1:14pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/33 "2025-12-11T13:14:50Z")
> [@RainTown](#): > > That 320gb SSD, is it locally attached (in the very real sense) to the host and passed thru to the VM, or its some AWS "EBS storage type", or ... ? Or is it maybe "320gb from a large pool of SSD storage provisioned by the VMware team to the VMs", like happens in a typical corporate environments. The point isn't really to know the precise detail, but to get a sense. > > And you have access to run commands like iostat on the data nodes? if so, what do they show? Yes, we are using VMs…let me get back with the exact details of the questions you’ve asked…and as for the hot threads part, yeah, it’s not recent, but it’s the same even now…I just shared the log which I had copied and pasted in my tracker. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2025, 1:18pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/34 "2025-12-11T13:18:22Z")
As @RainTown pointed out how you access your SSD storage is very important. On AWS there is several tiers of SSD backed EBS and the cheapest types do not provide anywhere near the performance you would see from a local SSD of good quality. If you can try to run iostat on the hot data nodes and check await, IOPS and other I/O related statistics. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 9:50am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/35 "2025-12-12T09:50:14Z")
The 320gb SSD is locally attached to the Elasticsearch process and yes, we have access to run the iostat command. Do you want to see the iostat response during the periods of CPU spike, or shall I provide you the current point in time information? ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 9:52am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/36 "2025-12-12T09:52:04Z")
Yes sure, I can do that…sharing the current point in time logs (currently cpu is around 30%). ```auto Linux 5.10.0-33-cloud-amd64 12/12/25 _x86_64_ (20 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 11.49 0.14 1.14 0.08 0.00 87.13 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd vda 8.01 1.79 42.89 15.08 30514728 730631874 256923132 vdb 249.06 149.36 4320.12 124.47 2544293144 73589851768 2120169144 ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 11:18am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/37 "2025-12-12T11:18:20Z")
249 tps is not a lot, but we dont know if it was under any stress for that period. Please run on the 15 "hot" nodes: `iostat -t -c -d /dev/vdb -x 10 360` This will take an hour. Hopefully it will include a period where the node is showing high (90%+) CPU. if it doesn't, then run it again for another hour. repeat until you get an hour where there such a period, and please share that hours data for the hosts impacted. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 11:41am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/38 "2025-12-12T11:41:54Z")
Sure, thanks a lot for the command, I’ll run it and share the results. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 2:02pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/39 "2025-12-12T14:02:55Z")
This is what we got at around a 60% cpu spike…I’ll try to share one for 90% as well, but seems like we didn’t get enough writes today haha. ```auto 12/12/25 19:04:30 avg-cpu: %user %nice %system %iowait %steal %idle 30.42 0.27 3.32 0.55 0.12 65.32 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 711.00 48579.20 0.00 0.00 0.23 68.33 533.30 7262.80 1280.80 70.60 0.14 13.62 0.00 0.00 0.00 0.00 0.00 0.00 246.40 0.08 0.26 54.52 12/12/25 19:04:40 avg-cpu: %user %nice %system %iowait %steal %idle 30.33 0.21 4.15 0.60 0.12 64.58 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 494.30 40285.60 0.00 0.00 0.24 81.50 832.10 7205.60 947.90 53.25 0.11 8.66 0.00 0.00 0.00 0.00 0.00 0.00 480.40 0.07 0.25 66.32 12/12/25 19:04:50 avg-cpu: %user %nice %system %iowait %steal %idle 25.46 0.00 3.21 0.84 0.14 70.35 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 370.80 32260.40 0.00 0.00 0.27 87.00 1736.60 42340.00 5236.80 75.10 0.13 24.38 0.00 0.00 0.00 0.00 0.00 0.00 869.30 0.08 0.39 65.16 12/12/25 19:05:00 avg-cpu: %user %nice %system %iowait %steal %idle 38.40 0.26 6.25 1.01 0.26 53.82 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 655.90 44006.00 0.00 0.00 0.36 67.09 1745.90 84306.80 14319.00 89.13 0.24 48.29 0.00 0.00 0.00 0.00 0.00 0.00 775.80 0.13 0.77 73.76 ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 2:58pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/40 "2025-12-12T14:58:42Z")
lets see what higher CPU load stats says, but already: You say "around a 60% cpu spike" but the logs show CPU, and highest I see is 38%, measured over 10 seconds. How long was the spike to "60%"? A couple of seconds? The last set shows %util at 73%. That means for 73% of that 10 second, there was an IO request being processed. r/s : w/s : f/s were 655.90 : 1745.90 : 775.80. Thats respectable, but nothing remarkable and it's already at 73% util. > [@Lakshya\_Gupta](#): > > our write rate is 10K+. Whats the refresh\_interval on the relevant index? Are you using refresh=true? ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 4:22pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/41 "2025-12-12T16:22:11Z")
The refresh interval is 30seconds ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 4:23pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/42 "2025-12-12T16:23:45Z")
If possible could you please explain a bit about what these terminologies mean and how do we determine if these can lead to a cpu spike or not 😅, that will help us in understanding elastic in more depth as well. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2025, 4:29pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/43 "2025-12-12T16:29:48Z")
> [@Lakshya\_Gupta](#): > > Batching the results before writing to Elastic Does this mean implementing bulk indexing requests as [outlined here](https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/indexing-speed#_use_bulk_requests)? How are you batching up the data? Are you using a message queue or some other method? > [@Lakshya\_Gupta](#): > > Added a de-duping layer to reduce writes by 90%. Can you provide some more details about what you did in this layer and how it works? Is it related to the comment I made about frequent updates to individual documents? ---
**Author:** ![Lakshya-Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya-gupta/32/146362_2.png) [@Lakshya-Gupta](https://discuss.elastic.co/u/Lakshya-Gupta)\ **Post date:** [December 12, 2025, 4:41pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/44 "2025-12-12T16:41:08Z")
We are batching the data in 2 ways - 1. First we have added a sort of delayed queue where we are creating a delay of 30 seconds before sending the requests to the downstream. This helps in batching duplicate results. 2. Secondly, when polling from kafka, we poll a batch of documents and upsert it in a single call. ---
**Author:** ![Lakshya-Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya-gupta/32/146362_2.png) [@Lakshya-Gupta](https://discuss.elastic.co/u/Lakshya-Gupta)\ **Post date:** [December 12, 2025, 4:43pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/45 "2025-12-12T16:43:10Z")
It’s a redis cache wherein we are using a key-value sort of data structure so that the data for 1 key doesn’t create another upsert request for another 30 seconds…this is because in our case whenever we experience a higher number of writes, 90% of the IDs are duplicate, therefore this key-value sort of data structure works. ---
**Author:** ![Lakshya-Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya-gupta/32/146362_2.png) [@Lakshya-Gupta](https://discuss.elastic.co/u/Lakshya-Gupta)\ **Post date:** [December 12, 2025, 5:28pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/46 "2025-12-12T17:28:57Z")
@RainTown as we disuccsed on the pm, can you please guide me how this can be an IOPS issue. I don’t have much knowledge about that, if you could please suggest me how to go in depth and figure out if it’s actually an IOPS issue. And if yes, then how can we aim to solve it? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 5:31pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/47 "2025-12-12T17:31:56Z")
> [@Lakshya-Gupta](#): > > And if yes, then how can we aim to solve it? That bit is easy. Get faster disks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2025, 5:39pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/48 "2025-12-12T17:39:44Z")
OK. Then it sounds like you have addressed the potential issue around frequent updates and implemented bulk requests. You mention that you have a high indexing load. What does the query load look like? How many concurrent queries do you need to support? What is your latency target/limit? How is the cluster performing from a query perspective now (for both hot and cold IDs)? [Previous page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=1) [Next page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=3)