# High CPU Usage on a few data nodes / Hotspotting of data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2025, 11:45am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954 "2025-12-10T11:45:07Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 5:41pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/49 "2025-12-12T17:41:42Z")
> [@RainTown](#): > > Whats the refresh\_interval on the relevant index? Are you using refresh=true? Haha, true that, but how do I ensure that it’s an IOPS issue 😛 ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 5:42pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/50 "2025-12-12T17:42:45Z")
> [@Lakshya-Gupta](#): > > as we discussed on the pm, can you please guide me how this can be an IOPS issue Well, if the initial claim: > [@Lakshya\_Gupta](#): > > The machines we are using are of top notch hardware **and IOPS is also not a bottleneck**. is not true then, automatically, IOPS **is** a bottleneck. The rest of my comments was more on my experience. Which is that often, in a corporate env specifically, or by a vendor, you get told things that are marketing or specs or ... but not actually true in reality. There seems to me evidence that you are in fact IO bound. You wrote in that PM exchange (slightly edited) "Increasing cores from 20 to 32 didn’t help at all, **we still capped at 10k upserts per second**. Although, reducing the writes load by 90%, we were able to reduce the duration of the cpu spike. Previously the cpu used to be spiked up for hours, but now it down to a couple of minutes, but for us, even those 5-10 minutes become critical." Maybe I am wrong, but this smells IO bound to me. The other big brains on the forum might have different view. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 5:43pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/51 "2025-12-12T17:43:26Z")
The query is an upsert query if you intend to ask that. We currently support 10k max writes per second, however, on reaching 10k, we hit 95% cpu on the hot shards/nodes, which causes a decrease in further writes along with a higher read latency. What we aim to achieve is 10k consistent writes with no-cpu spikes, similar to what we experience on the non-hot nodes. For hot nodes, 10k writes is the peak. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 5:44pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/52 "2025-12-12T17:44:58Z")
Sounds a bit right, but I have very less knowledge about IOPS at the moment to be frank 😅, I’ll need to study a bit on that for now. Can you meanwhile please help me with some debugging steps which I can perform on this cluster to guarantee that it’s an IOPS issue. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2025, 5:47pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/53 "2025-12-12T17:47:09Z")
Could you please answer my questions around the query load? I just want to check that the routing, which primarily is designed to help at query time, is actually necessary. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 5:52pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/54 "2025-12-12T17:52:50Z")
I didn’t completely understand your question, are you asking if we have specific usecase for this custom routing key? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2025, 6:06pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/55 "2025-12-12T18:06:02Z")
I am asking for details about how you query and use the data in the cluster, not how you are updating it. This goes back into diving a bit deeper into this issue that was raised early on (and as far as I can see might have been dropped). > [@RainTown](#): > > You don't mention the benefit you get from the custom routing, just the cost? Is it not worth reviewing the design choice? Actually, let's be straight - if you were designing the solution today, knowing what you know now, would you use the same approach? Routing is primarily used to make querying more efficient as only a limited number of shards need to be queried to serve each query request. As this was set up a long time ago I was wondering if there is still a strong argument supporting this design decision. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 12, 2025, 6:17pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/56 "2025-12-12T18:17:47Z")
We have composite read aggregation query use case and scroll / slice use case. If we don’t use custom routing, then the requests will go all the shards, that can lead to too many open scroll contexts per node, hence we require routing. ---
**Author:** ![Lakshya-Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya-gupta/32/146362_2.png) [@Lakshya-Gupta](https://discuss.elastic.co/u/Lakshya-Gupta)\ **Post date:** [December 12, 2025, 6:37pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/58 "2025-12-12T18:37:04Z")
> [@Lakshya\_Gupta](#): > > Sounds a bit right, but I have very less knowledge about IOPS at the moment to be frank 😅, I’ll need to study a bit on that for now. Can you meanwhile please help me with some debugging steps which I can perform on this cluster to guarantee that it’s an IOPS issue. @RainTown / @Christian_Dahlqvist could you please help me with this. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 6:48pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/59 "2025-12-12T18:48:00Z")
If there was a /usr/bin/do-i-have-an-iops-issue command, I'd have suggested you run it ages ago. And, even if it existed, it might return "No", or even "are you asking the right question?" ! I am waiting to see the iostat data for when you have "95% CPU usage". For a semi-extended period, like a couple of minutes. Not just @Christian_Dahlqvist , but others, are all welcome to weigh in with new/different/better ideas. ---
**Author:** ![Lakshya-Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya-gupta/32/146362_2.png) [@Lakshya-Gupta](https://discuss.elastic.co/u/Lakshya-Gupta)\ **Post date:** [December 12, 2025, 6:52pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/60 "2025-12-12T18:52:21Z")
Sure, I’ll add that iostat data as soon as I get the window. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2025, 6:52pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/61 "2025-12-12T18:52:23Z")
I personally think breaking out the hot IDs into a separate index with a good number of primary shards and no routing on a separate subset of nodes in the cluster would be the best way to resolve the issue. As you however have ruled that out I have a hard time seeing anything but storage performance being the bottleneck. Maybe others have some suggestions though. > [@RainTown](#): > > I am waiting to see the iostat data for when you have "95% CPU usage". For a semi-extended period, like a couple of minutes. The output of `iostat -x` during a period of heavy load when you are experiencing issues is exactly what we want to see. Please note though that every storage system and node has some limitation and if your storage is indeed very fast and performant there may not be an easy solution that avoids reachitecting the sharding there. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 9:45pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/62 "2025-12-12T21:45:33Z")
> [@Lakshya-Gupta](#): > > Sure, I’ll add that iostat data as soon as I get the window. Cool. While we wait you could also enquire about the _specifics_ of the IO that is showing as vdb in your linux VMs. There was this exchange: > [@Christian\_Dahlqvist](#): > > Is the SSD local or accessed via network? > [@Lakshya\_Gupta](#): > > For local or network, let me get back on that. One possibility is it's a virtual disk carved from a large/huge VMware storage pool, backed by a big brand storage vendor. Which is usually good. But maybe all 60 of your data node VMs all have a virtual disk from exactly _same_ storage pool (just speculation). Anyways, your 15 hot VMs are all writing to their "looks like a local disk" disk at pretty much the same time, the other 45 too but less so due to skew. Depending on what it is, this might be stressing the storage, or stressing something else. It'll certainly be "doing the best it can", but it's possible you need to know the specific details here. So, if there is a "Storage Management" team, or the "VMware Team", ask them for specifics, and tell them your cluster is very IOPS dependent. If they say 'yeah, we noticed!" that would tell us something in itself 🙂 EDIT: Now I recall that /dev/vda and /dev/vdb probably means it's using a paravirt device, which then maps to a native (local) device from the host. So this is all maybe moot. Output of `lshw -C disk` would be nice to know, if `lshw` is installed. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 13, 2025, 5:54am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/63 "2025-12-13T05:54:52Z")
> [@Christian\_Dahlqvist](#): > > he output of `iostat -x` during a period of heavy load when you are experiencing issues is exactly what we want to see. Please note though that every storage system and node has some limitation and if your storage is indeed very fast and performant there may not be an easy solution that avoids reachitecting the sharding there. Sure, I’ve re-ran the command, hopefully we get the spike this time. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 13, 2025, 5:56am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/64 "2025-12-13T05:56:53Z")
> [@RainTown](#): > > EDIT: Now I recall that /dev/vda and /dev/vdb probably means it's using a paravirt device, which then maps to a native (local) device from the host. So this is all maybe moot. Output of `lshw -C disk` would be nice to know, if `lshw` is installed. > \*-virtio2 > > ``` > description: Virtual I/O device > > physical id: 0 > > bus info: virtio@2 > > logical name: /dev/vda > > size: 10GiB (10GB) > > capabilities: gpt-1.00 partitioned partitioned:gpt > > configuration: driver=virtio_blk guid=3cf094a2-5168-9444-b870-5ddad3b6327a logicalsectorsize=512 sectorsize=4096 > > ``` > > \*-virtio3 > > ``` > description: Virtual I/O device > > physical id: 0 > > bus info: virtio@3 > > logical name: /dev/vdb > > size: 446GiB (478GB) > > capabilities: partitioned partitioned:dos > > configuration: driver=virtio_blk logicalsectorsize=4096 sectorsize=4096 > > ``` ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 13, 2025, 6:33am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/65 "2025-12-13T06:33:18Z")
@RainTown / @Christian_Dahlqvist this is the machine we use [https://download.semiconductor.samsung.com/resources/brochure/PM1733%20NVMe%20SSD.pdf](https://download.semiconductor.samsung.com/resources/brochure/PM1733%20NVMe%20SSD.pdf) ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 13, 2025, 6:35am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/66 "2025-12-13T06:35:24Z")
> [@RainTown](#): > > One possibility is it's a virtual disk carved from a large/huge VMware storage pool, backed by a big brand storage vendor. Which is usually good. But maybe all 60 of your data node VMs all have a virtual disk from exactly _same_ storage pool (just speculation). I might be wrong, but don’t you think if it would have been a “noisy neighbour” problem, then all 60 nodes would have experienced a cpu spike and not only the 15 nodes? ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 13, 2025, 6:50am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/67 "2025-12-13T06:50:33Z")
> [@Lakshya\_Gupta](#): > > [https://download.semiconductor.samsung.com/resources/brochure/PM1733%20NVMe%20SSD.pdf](https://download.semiconductor.samsung.com/resources/brochure/PM1733%20NVMe%20SSD.pdf) Since the write IOPS is approximately 135,000 for this instance type, and our total indexing rate is 50k at peak times, don’t you think we should be good wrt IOPS? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 13, 2025, 8:23am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/68 "2025-12-13T08:23:30Z")
Good morning > [@Lakshya\_Gupta](#): > > Since the write IOPS is approximately 135,000 for this instance type, and our total indexing rate is 50k at peak times, don’t you think we should be good wrt IOPS? should be good - yes. Are good - TBD > [@Lakshya\_Gupta](#): > > I might be wrong, but don’t you think if it would have been a “noisy neighbour” problem, then all 60 nodes would have experienced a cpu spike and not only the 15 nodes? Yes, but this depends on the skew. If 15 nodes need 10k IOPS and the other 45 nodes need 100 IOPS then the other 45 wont spike. But, that was based on a false conjecture, so it's moot now. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 13, 2025, 8:25am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/69 "2025-12-13T08:25:24Z")
> [@Lakshya\_Gupta](#): > > ```auto > description: Virtual I/O device > physical id: 0 > bus info: virtio@3 > logical name: /dev/vdb > size: 446GiB (478GB) > capabilities: partitioned partitioned:dos > configuration: driver=virtio_blk logicalsectorsize=4096 sectorsize > > ``` So it's not direct pass-thru to the actual device? I'm no VMware expert, but is this the best way (someone else might know)? [Previous page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=2) [Next page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=4)