# High CPU Usage on a few data nodes / Hotspotting of data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2025, 11:45am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954 "2025-12-10T11:45:07Z")\ **Posts on this page:** 20\ **Page:** 9
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 21, 2025, 1:00pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/170 "2025-12-21T13:00:51Z")
> [@Christian\_Dahlqvist](#): > > The issue with frequent updates will be caused by all types of updates, not just duplicate events. Of course, but "updating" same document \_id to be exactly the same (well, different \_version) is generally pointless. It's also easy-ish to remove, e.g. use the doc (or hash thereof) as a key. eg in redis Hash the document (e.g. SHA-256, ...) Use that hash as the redis key SET hashed\_doc NX PX 30000 If SET succeeds it is first occurrence in time window so doc can be sent to next layer (eg Kafka, which you can then pull from in decent bulk sizes knowing there are no dups) If SET fails → a duplicate in the time window, so just drop, increase a counter Obviously the width of time window is easy to tune and show be low-cost. If the doc is not always the same, rather just rapidly changing, it's a bit harder to define what "de-dup" would really mean, it's more sampling than de-deduplication. @Lakshya-Gupta showed his model, but I didn't quite follow, sorry again. > [@Lakshya-Gupta](#): > > Yeah, I’m trying to do that but somehow not able to generate duplicate events, I’ll see if there is some other way. Er, this is a bit like the "high CPU load" moments. If there are none in the real data, dont feel obliged to generate them. The pseudo code above I suggested is trivial to adapt to keep counts of how many dups you are removing - you really should be tracking this anyways. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 21, 2025, 1:11pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/171 "2025-12-21T13:11:14Z")
> [@Christian\_Dahlqvist](#): > > breaking out the hot IDs into a separate index I was also thinking along these lines. Special cases can be handled differently. if you have some small number of IDs that are very clearly "special", completely different access pattern, then don't just chuck them in same bucket as the zillions of other IDs. You have 5 people yelling in a library, and we are discussing the building's acoustic design ? 🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 21, 2025, 1:17pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/172 "2025-12-21T13:17:38Z")
> [@RainTown](#): > > I was also thinking along these lines. Special cases can be handled differently. if you have some small number of IDs that are very clearly "special", completely different access pattern, then don't just chuck them in same bucket as the zillions of other IDs. Indeed. The issue driving the selection of a number iof routing shards was if I recall correctly the number of open context. By separating the high volume IDs into a separate index it may be possible to use tighter routing, maybe only 1 shard per ID, for the current index while you could have a significantly higher number for the high volume IDs. Depending on the query patterns, this may overall reduce the number of open contexts while at the same time spread the load more evenly. Doing this would give more room for fine-tuning. > [@RainTown](#): > > You have 5 people yelling in a library, and we are discussing the building's acoustic design ? 🙂 We are interested in architecture, right? 😉 ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 21, 2025, 4:46pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/173 "2025-12-21T16:46:35Z")
> [@Christian\_Dahlqvist](#): > > We are interested in architecture, right? In a different sphere, it's well known you cannot architecture away anti-social behaviour, all you can do is try constrain it. A few "bad" people can ruin an area. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 23, 2025, 5:10am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/174 "2025-12-23T05:10:27Z")
I’ve got sick 😕 will come back in a day. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 26, 2025, 5:58am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/175 "2025-12-26T05:58:47Z")
> [@Christian\_Dahlqvist](#): > > I would therefore again recommend that you verify that it works as intended with respect to merging and deduplicating events and check that you are using reasonably large bulk requests. Yeah, figured out a strategy to test this…will share the results once tested (although you can consider that the deduplication logic should be working as we are expecting it to). > [@Christian\_Dahlqvist](#): > > breaking out the hot IDs into a separate index We though of this, but the issue is we have at least 10 different consumers firing read queries. If we create 2 indices, then managing it can become a bit of a challenge and moreover if another cold id becomes a hot id tomorrow, then we will end up with the same issue. > [@Christian\_Dahlqvist](#): > > Elasticsearch is a search engine and is optimised for retrieval of reasonably small result sets based on relevance. It is not optimised for highly concurrent retrieval of large bulk result sets. It is also, as you have noted, not optimised for high update rates. True 😕 ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [December 26, 2025, 6:00am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/176 "2025-12-26T06:00:41Z")
> [@RainTown](#): > > Of course, but "updating" same document \_id to be exactly the same (well, different \_version) is generally pointless. It's also easy-ish to remove, e.g. use the doc (or hash thereof) as a key. eg in redis Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 26, 2025, 9:45am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/177 "2025-12-26T09:45:59Z")
> [@Lakshya\_Gupta](#): > > Yeah, figured out a strategy to test this…will share the results once tested (although you can consider that the deduplication logic should be working as we are expecting it to). > [@Lakshya\_Gupta](#): > > Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go. I know it _ **should** _ work, but have a hard time seeing how merging could be an issue if each document is updated at most once per 30 seconds and you have a large refresh interval given the specification of the hardware you are using unless you are using some mapping features that make merging very expensive (not sure what that could be). Could you perhaps provide some additional details on exactly how you intend to test this? > [@Lakshya\_Gupta](#): > > We though of this, but the issue is we have at least 10 different consumers firing read queries. If we create 2 indices, then managing it can become a bit of a challenge and moreover if another cold id becomes a hot id tomorrow, then we will end up with the same issue. It would require a process to move specific routing IDs to the specialised index but this is something you may not need to use frequently as it likely will take time for new IDs to start causing severe problems. Estimating the probability and frequency of this requires knowledge of the data and use case so I will leave this to you. With many clients I would likely look to create a custom query proxy used by all clients that can rewrite and redirect queries where required. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 26, 2025, 1:10pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/178 "2025-12-26T13:10:55Z")
> [@Christian\_Dahlqvist](#): > > It would require a process to move specific routing IDs to the specialised index I stretch my metaphor, but if you have five anti-social neighbors, and find a solution for that problem, move them to a out-of-the-way location, and a sixth emerges, just use the same solution ? But, there's something not quite adding up here: > [@Christian\_Dahlqvist](#): > > ... have a hard time seeing how merging could be an issue if each document is updated at most once per 30 seconds Me too. You never did share the iostat output when under highest load, but it could be that there's another "load generator" we've not covered yet. At start of thread you wrote: > [@Lakshya\_Gupta](#): > > We have a custom routing field comprising of a single field….this field has **1,315,608** unique values out of which 5 of them are highly skewed. What are the doc counts for those 5 "special" values for this field ? How does that compare to say the next 100 field values in terms of count? What's the mapping type of that field? Without divulging stuff you dont want to divulge, what is that field? btw 1,315,608 = ( 2 \* 2 \* 2 \* 3 \* 7 \* 41 \* 191) was a very specific number, is there some meaning in it, or was just the count the time you checked it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 4, 2026, 6:27pm UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/179 "2026-01-04T18:27:18Z")
> [@Lakshya\_Gupta](#): > > Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go. This description is interesting. How are these updates batched together? Are they sent as a number of different individual updates in a single bulk request (this is my default interpretation of batched together) or are they actually combined into a single logical update request for that document? ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 9:18am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/180 "2026-01-05T09:18:06Z")
Hey, I’m back with some new data points. I created 2 virtual machines using the same number of cores (20) and memory (53G), but with different disk sizes, 320Gb and 640Gb, and established a single node Elastic Search cluster on top of it. Thereafter I created an index (similar on both the Clusters) with a single primary shard and no routing (because it’s 1 shard anyways) and then ran NFRs for 15minutes on both of them (the NFR was creating random documents and inserting them). I ran 2 commands in parallel to record the output 1. nohup sar -u ALL -P ALL 1 \> sar\_cpu.log 2\>&1 & 2. nohup iostat -ztcxd 15 \> iostat\_output.log 2\>&1 & Attaching the detailed metrics for both the machines. 320 GB - Iostat output - [Unique Download Link | WeTransfer](https://we.tl/t-eRB8pL2EO5) 320 GB - sar output - (Not able to publish due to Firewell, will post some content in next message itself) 640 GB - Iostate output - [Unique Download Link | WeTransfer](https://we.tl/t-B7WfaRJsFs) 640 GB - sar output - (Not able to publish due to Firewell, will post some content in next message itself) ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 9:19am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/181 "2026-01-05T09:19:50Z")
320 Gb SAR output for some duration ```auto 14:00:50 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:00:51 all 84.80 0.00 7.02 0.75 1.05 0.00 3.46 0.00 0.00 2.91 14:00:51 0 86.27 0.00 6.86 1.96 0.98 0.00 2.94 0.00 0.00 0.98 14:00:51 1 79.38 0.00 5.15 1.03 4.12 0.00 3.09 0.00 0.00 7.22 14:00:51 2 86.87 0.00 6.06 1.01 0.00 0.00 4.04 0.00 0.00 2.02 14:00:51 3 86.27 0.00 4.90 0.00 0.98 0.00 4.90 0.00 0.00 2.94 14:00:51 4 86.00 0.00 6.00 0.00 1.00 0.00 4.00 0.00 0.00 3.00 14:00:51 5 88.12 0.00 4.95 0.99 0.99 0.00 1.98 0.00 0.00 2.97 14:00:51 6 89.11 0.00 6.93 0.99 0.00 0.00 0.99 0.00 0.00 1.98 14:00:51 7 73.00 0.00 18.00 1.00 1.00 0.00 4.00 0.00 0.00 3.00 14:00:51 8 80.61 0.00 7.14 1.02 2.04 0.00 6.12 0.00 0.00 3.06 14:00:51 9 94.00 0.00 3.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 14:00:51 10 90.82 0.00 6.12 0.00 0.00 0.00 1.02 0.00 0.00 2.04 14:00:51 11 87.13 0.00 7.92 0.00 0.00 0.00 2.97 0.00 0.00 1.98 14:00:51 12 75.00 0.00 12.00 1.00 3.00 0.00 5.00 0.00 0.00 4.00 14:00:51 13 73.74 0.00 11.11 1.01 1.01 0.00 6.06 0.00 0.00 7.07 14:00:51 14 91.09 0.00 4.95 0.99 0.99 0.00 1.98 0.00 0.00 0.00 14:00:51 15 79.00 0.00 8.00 1.00 2.00 0.00 7.00 0.00 0.00 3.00 14:00:51 16 81.82 0.00 7.07 2.02 1.01 0.00 4.04 0.00 0.00 4.04 14:00:51 17 95.88 0.00 2.06 0.00 0.00 0.00 1.03 0.00 0.00 1.03 14:00:51 18 86.60 0.00 7.22 0.00 1.03 0.00 2.06 0.00 0.00 3.09 14:00:51 19 85.15 0.00 4.95 0.99 0.99 0.00 2.97 0.00 0.00 4.95 14:00:51 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:00:52 all 76.02 0.00 15.33 0.05 0.66 0.00 3.95 0.00 0.00 4.00 14:00:52 0 90.82 0.00 4.08 0.00 0.00 0.00 2.04 0.00 0.00 3.06 14:00:52 1 83.00 0.00 7.00 0.00 1.00 0.00 5.00 0.00 0.00 4.00 14:00:52 2 61.00 0.00 30.00 0.00 0.00 0.00 4.00 0.00 0.00 5.00 14:00:52 3 82.83 0.00 8.08 0.00 0.00 0.00 3.03 0.00 0.00 6.06 14:00:52 4 85.86 0.00 11.11 0.00 0.00 0.00 1.01 0.00 0.00 2.02 14:00:52 5 81.82 0.00 13.13 0.00 0.00 0.00 1.01 0.00 0.00 4.04 14:00:52 6 49.00 0.00 48.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00 14:00:52 7 72.45 0.00 21.43 0.00 0.00 0.00 2.04 0.00 0.00 4.08 14:00:52 8 86.73 0.00 5.10 0.00 0.00 0.00 4.08 0.00 0.00 4.08 14:00:52 9 80.61 0.00 7.14 0.00 0.00 0.00 6.12 0.00 0.00 6.12 14:00:52 10 42.00 0.00 41.00 0.00 7.00 0.00 7.00 0.00 0.00 3.00 14:00:52 11 85.71 0.00 8.16 0.00 0.00 0.00 4.08 0.00 0.00 2.04 14:00:52 12 91.92 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 1.01 14:00:52 13 81.63 0.00 8.16 0.00 1.02 0.00 3.06 0.00 0.00 6.12 14:00:52 14 65.66 0.00 22.22 0.00 1.01 0.00 6.06 0.00 0.00 5.05 14:00:52 15 86.87 0.00 7.07 0.00 0.00 0.00 3.03 0.00 0.00 3.03 14:00:52 16 66.67 0.00 16.16 0.00 1.01 0.00 8.08 0.00 0.00 8.08 14:00:52 17 56.57 0.00 25.25 1.01 1.01 0.00 10.10 0.00 0.00 6.06 14:00:52 18 86.87 0.00 6.06 0.00 1.01 0.00 3.03 0.00 0.00 3.03 14:00:52 19 83.67 0.00 11.22 0.00 0.00 0.00 3.06 0.00 0.00 2.04 14:00:52 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:00:53 all 76.14 0.00 14.19 0.10 0.90 0.00 3.76 0.00 0.00 4.91 14:00:53 0 59.41 0.00 27.72 0.00 0.99 0.00 4.95 0.00 0.00 6.93 14:00:53 1 70.71 0.00 10.10 0.00 2.02 0.00 6.06 0.00 0.00 11.11 14:00:53 2 78.00 0.00 8.00 0.00 0.00 0.00 10.00 0.00 0.00 4.00 14:00:53 3 78.12 0.00 8.33 0.00 1.04 0.00 6.25 0.00 0.00 6.25 14:00:53 4 79.59 0.00 7.14 0.00 1.02 0.00 5.10 0.00 0.00 7.14 14:00:53 5 82.65 0.00 7.14 0.00 0.00 0.00 7.14 0.00 0.00 3.06 14:00:53 6 64.00 0.00 23.00 0.00 1.00 0.00 5.00 0.00 0.00 7.00 14:00:53 7 72.73 0.00 21.21 0.00 0.00 0.00 1.01 0.00 0.00 5.05 14:00:53 8 76.24 0.00 12.87 0.00 0.99 0.00 4.95 0.00 0.00 4.95 14:00:53 9 90.00 0.00 4.00 0.00 1.00 0.00 1.00 0.00 0.00 4.00 14:00:53 10 50.50 0.00 33.66 0.99 6.93 0.00 2.97 0.00 0.00 4.95 14:00:53 11 80.00 0.00 8.00 0.00 1.00 0.00 3.00 0.00 0.00 8.00 14:00:53 12 86.14 0.00 4.95 0.00 0.99 0.00 1.98 0.00 0.00 5.94 14:00:53 13 90.10 0.00 3.96 0.99 0.00 0.00 2.97 0.00 0.00 1.98 14:00:53 14 83.67 0.00 7.14 0.00 0.00 0.00 5.10 0.00 0.00 4.08 14:00:53 15 92.16 0.00 3.92 0.00 0.00 0.00 0.00 0.00 0.00 3.92 14:00:53 16 60.61 0.00 37.37 0.00 0.00 0.00 1.01 0.00 0.00 1.01 14:00:53 17 63.73 0.00 32.35 0.00 0.00 0.00 2.94 0.00 0.00 0.98 14:00:53 18 90.00 0.00 2.00 0.00 1.00 0.00 3.00 0.00 0.00 4.00 14:00:53 19 74.75 0.00 20.20 0.00 0.00 0.00 1.01 0.00 0.00 4.04 14:00:53 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:00:54 all 75.14 0.00 15.34 0.15 0.80 0.00 3.61 0.00 0.00 4.96 14:00:54 0 74.00 0.00 19.00 0.00 1.00 0.00 3.00 0.00 0.00 3.00 14:00:54 1 68.32 0.00 21.78 0.00 0.00 0.00 2.97 0.00 0.00 6.93 14:00:54 2 86.00 0.00 8.00 0.00 0.00 0.00 2.00 0.00 0.00 4.00 14:00:54 3 85.15 0.00 4.95 0.00 0.99 0.00 3.96 0.00 0.00 4.95 14:00:54 4 89.11 0.00 5.94 0.00 0.99 0.00 1.98 0.00 0.00 1.98 14:00:54 5 87.76 0.00 4.08 0.00 0.00 0.00 3.06 0.00 0.00 5.10 14:00:54 6 76.47 0.00 15.69 0.00 0.98 0.00 3.92 0.00 0.00 2.94 14:00:54 7 42.00 0.00 53.00 1.00 2.00 0.00 1.00 0.00 0.00 1.00 14:00:54 8 76.77 0.00 10.10 0.00 2.02 0.00 5.05 0.00 0.00 6.06 14:00:54 9 68.69 0.00 12.12 1.01 1.01 0.00 7.07 0.00 0.00 10.10 14:00:54 10 89.80 0.00 6.12 0.00 1.02 0.00 0.00 0.00 0.00 3.06 14:00:54 11 79.59 0.00 8.16 1.02 1.02 0.00 3.06 0.00 0.00 7.14 14:00:54 12 67.33 0.00 18.81 0.00 0.99 0.00 6.93 0.00 0.00 5.94 14:00:54 13 78.35 0.00 12.37 0.00 1.03 0.00 2.06 0.00 0.00 6.19 14:00:54 14 78.57 0.00 7.14 0.00 1.02 0.00 7.14 0.00 0.00 6.12 14:00:54 15 62.63 0.00 29.29 0.00 0.00 0.00 4.04 0.00 0.00 4.04 14:00:54 16 81.00 0.00 12.00 0.00 1.00 0.00 2.00 0.00 0.00 4.00 14:00:54 17 90.10 0.00 4.95 0.00 0.00 0.00 1.98 0.00 0.00 2.97 14:00:54 18 75.73 0.00 10.68 0.00 0.97 0.00 5.83 0.00 0.00 6.80 14:00:54 19 45.45 0.00 42.42 0.00 0.00 0.00 5.05 0.00 0.00 7.07 14:00:54 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:00:55 all 67.80 0.00 15.82 0.05 1.02 0.00 3.97 0.00 0.00 11.34 14:00:55 0 65.66 0.00 18.18 1.01 0.00 0.00 4.04 0.00 0.00 11.11 14:00:55 1 65.62 0.00 8.33 0.00 4.17 0.00 5.21 0.00 0.00 16.67 14:00:55 2 47.47 0.00 41.41 0.00 1.01 0.00 4.04 0.00 0.00 6.06 14:00:55 3 68.32 0.00 15.84 0.00 0.00 0.00 4.95 0.00 0.00 10.89 14:00:55 4 68.00 0.00 10.00 0.00 1.00 0.00 6.00 0.00 0.00 15.00 14:00:55 5 89.00 0.00 6.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00 14:00:55 6 78.79 0.00 8.08 0.00 0.00 0.00 3.03 0.00 0.00 10.10 14:00:55 7 72.45 0.00 7.14 0.00 1.02 0.00 2.04 0.00 0.00 17.35 14:00:55 8 80.61 0.00 3.06 0.00 0.00 0.00 5.10 0.00 0.00 11.22 14:00:55 9 82.18 0.00 7.92 0.00 0.99 0.00 3.96 0.00 0.00 4.95 14:00:55 10 74.23 0.00 7.22 0.00 0.00 0.00 3.09 0.00 0.00 15.46 14:00:55 11 71.88 0.00 9.38 0.00 0.00 0.00 2.08 0.00 0.00 16.67 14:00:55 12 70.41 0.00 14.29 0.00 2.04 0.00 4.08 0.00 0.00 9.18 14:00:55 13 79.80 0.00 6.06 0.00 0.00 0.00 1.01 0.00 0.00 13.13 14:00:55 14 75.00 0.00 9.38 0.00 0.00 0.00 6.25 0.00 0.00 9.38 14:00:55 15 67.01 0.00 10.31 0.00 0.00 0.00 6.19 0.00 0.00 16.49 14:00:55 16 65.31 0.00 20.41 0.00 1.02 0.00 3.06 0.00 0.00 10.20 14:00:55 17 40.00 0.00 36.84 0.00 3.16 0.00 5.26 0.00 0.00 14.74 14:00:55 18 57.58 0.00 35.35 0.00 1.01 0.00 3.03 0.00 0.00 3.03 14:00:55 19 36.00 0.00 41.00 0.00 5.00 0.00 4.00 0.00 0.00 14.00 ``` ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 9:21am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/182 "2026-01-05T09:21:13Z")
640 Gb SAR output for some duration ```auto 14:22:59 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:00 all 87.17 0.00 6.12 0.05 0.35 0.00 2.71 0.00 0.00 3.61 14:23:00 0 84.85 0.00 6.06 0.00 0.00 0.00 3.03 0.00 0.00 6.06 14:23:00 1 87.13 0.00 3.96 0.00 0.00 0.00 2.97 0.00 0.00 5.94 14:23:00 2 65.69 0.00 27.45 0.00 0.00 0.00 4.90 0.00 0.00 1.96 14:23:00 3 93.00 0.00 3.00 0.00 1.00 0.00 1.00 0.00 0.00 2.00 14:23:00 4 93.94 0.00 2.02 0.00 0.00 0.00 3.03 0.00 0.00 1.01 14:23:00 5 85.15 0.00 6.93 0.00 0.99 0.00 1.98 0.00 0.00 4.95 14:23:00 6 86.00 0.00 4.00 0.00 0.00 0.00 5.00 0.00 0.00 5.00 14:23:00 7 85.00 0.00 7.00 0.00 1.00 0.00 4.00 0.00 0.00 3.00 14:23:00 8 74.00 0.00 15.00 0.00 0.00 0.00 3.00 0.00 0.00 8.00 14:23:00 9 77.08 0.00 8.33 0.00 2.08 0.00 3.12 0.00 0.00 9.38 14:23:00 10 88.78 0.00 6.12 0.00 0.00 0.00 3.06 0.00 0.00 2.04 14:23:00 11 91.00 0.00 4.00 0.00 1.00 0.00 3.00 0.00 0.00 1.00 14:23:00 12 88.89 0.00 6.06 0.00 0.00 0.00 2.02 0.00 0.00 3.03 14:23:00 13 95.00 0.00 2.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00 14:23:00 14 84.00 0.00 5.00 0.00 0.00 0.00 4.00 0.00 0.00 7.00 14:23:00 15 88.24 0.00 3.92 0.98 0.98 0.00 1.96 0.00 0.00 3.92 14:23:00 16 97.98 0.00 1.01 0.00 0.00 0.00 1.01 0.00 0.00 0.00 14:23:00 17 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 14:23:00 18 91.00 0.00 4.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00 14:23:00 19 87.88 0.00 5.05 0.00 0.00 0.00 4.04 0.00 0.00 3.03 14:23:00 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:01 all 84.80 0.00 8.20 0.40 0.55 0.00 2.32 0.00 0.00 3.72 14:23:01 0 85.71 0.00 6.12 2.04 0.00 0.00 4.08 0.00 0.00 2.04 14:23:01 1 87.76 0.00 5.10 0.00 1.02 0.00 2.04 0.00 0.00 4.08 14:23:01 2 71.72 0.00 25.25 0.00 1.01 0.00 1.01 0.00 0.00 1.01 14:23:01 3 88.00 0.00 6.00 2.00 1.00 0.00 2.00 0.00 0.00 1.00 14:23:01 4 87.38 0.00 6.80 0.00 0.00 0.00 1.94 0.00 0.00 3.88 14:23:01 5 87.00 0.00 4.00 0.00 1.00 0.00 2.00 0.00 0.00 6.00 14:23:01 6 82.83 0.00 8.08 1.01 1.01 0.00 2.02 0.00 0.00 5.05 14:23:01 7 71.13 0.00 9.28 2.06 2.06 0.00 3.09 0.00 0.00 12.37 14:23:01 8 80.81 0.00 9.09 1.01 1.01 0.00 2.02 0.00 0.00 6.06 14:23:01 9 81.00 0.00 9.00 0.00 1.00 0.00 3.00 0.00 0.00 6.00 14:23:01 10 87.76 0.00 8.16 0.00 0.00 0.00 2.04 0.00 0.00 2.04 14:23:01 11 90.91 0.00 5.05 0.00 0.00 0.00 1.01 0.00 0.00 3.03 14:23:01 12 86.14 0.00 8.91 0.00 0.00 0.00 0.99 0.00 0.00 3.96 14:23:01 13 87.88 0.00 8.08 0.00 0.00 0.00 2.02 0.00 0.00 2.02 14:23:01 14 77.78 0.00 9.09 0.00 0.00 0.00 7.07 0.00 0.00 6.06 14:23:01 15 80.81 0.00 7.07 0.00 1.01 0.00 6.06 0.00 0.00 5.05 14:23:01 16 90.10 0.00 6.93 0.00 0.00 0.00 0.99 0.00 0.00 1.98 14:23:01 17 89.80 0.00 9.18 0.00 0.00 0.00 1.02 0.00 0.00 0.00 14:23:01 18 91.92 0.00 6.06 0.00 0.00 0.00 1.01 0.00 0.00 1.01 14:23:01 19 89.11 0.00 6.93 0.00 0.99 0.00 0.99 0.00 0.00 1.98 14:23:01 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:02 all 83.22 0.00 8.29 0.00 0.66 0.00 2.68 0.00 0.00 5.16 14:23:02 0 95.00 0.00 4.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 14:23:02 1 78.00 0.00 8.00 0.00 2.00 0.00 4.00 0.00 0.00 8.00 14:23:02 2 82.83 0.00 14.14 0.00 1.01 0.00 1.01 0.00 0.00 1.01 14:23:02 3 65.00 0.00 17.00 0.00 4.00 0.00 2.00 0.00 0.00 12.00 14:23:02 4 91.92 0.00 6.06 0.00 0.00 0.00 2.02 0.00 0.00 0.00 14:23:02 5 92.93 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 0.00 14:23:02 6 76.34 0.00 5.38 0.00 0.00 0.00 6.45 0.00 0.00 11.83 14:23:02 7 66.33 0.00 8.16 0.00 3.06 0.00 9.18 0.00 0.00 13.27 14:23:02 8 72.16 0.00 12.37 0.00 1.03 0.00 5.15 0.00 0.00 9.28 14:23:02 9 82.29 0.00 9.38 0.00 0.00 0.00 2.08 0.00 0.00 6.25 14:23:02 10 84.00 0.00 11.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00 14:23:02 11 86.87 0.00 4.04 0.00 0.00 0.00 2.02 0.00 0.00 7.07 14:23:02 12 82.00 0.00 11.00 0.00 1.00 0.00 1.00 0.00 0.00 5.00 14:23:02 13 88.78 0.00 4.08 0.00 0.00 0.00 1.02 0.00 0.00 6.12 14:23:02 14 79.41 0.00 18.63 0.00 0.98 0.00 0.98 0.00 0.00 0.00 14:23:02 15 85.86 0.00 5.05 0.00 0.00 0.00 3.03 0.00 0.00 6.06 14:23:02 16 84.85 0.00 11.11 0.00 0.00 0.00 2.02 0.00 0.00 2.02 14:23:02 17 89.00 0.00 4.00 0.00 0.00 0.00 2.00 0.00 0.00 5.00 14:23:02 18 96.04 0.00 1.98 0.00 0.00 0.00 1.98 0.00 0.00 0.00 14:23:02 19 83.84 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 9.09 14:23:02 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:03 all 89.22 0.20 4.53 0.05 0.35 0.00 2.92 0.00 0.00 2.72 14:23:03 0 87.13 0.99 3.96 0.00 0.99 0.00 3.96 0.00 0.00 2.97 14:23:03 1 92.86 0.00 5.10 0.00 0.00 0.00 1.02 0.00 0.00 1.02 14:23:03 2 96.00 1.00 2.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 14:23:03 3 90.91 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 2.02 14:23:03 4 98.99 0.00 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 14:23:03 5 93.07 0.00 2.97 0.00 0.00 0.00 1.98 0.00 0.00 1.98 14:23:03 6 84.85 0.00 7.07 0.00 1.01 0.00 2.02 0.00 0.00 5.05 14:23:03 7 82.35 0.98 5.88 0.00 0.00 0.00 4.90 0.00 0.00 5.88 14:23:03 8 77.55 1.02 11.22 0.00 1.02 0.00 4.08 0.00 0.00 5.10 14:23:03 9 74.23 0.00 5.15 0.00 1.03 0.00 10.31 0.00 0.00 9.28 14:23:03 10 93.94 0.00 3.03 0.00 1.01 0.00 1.01 0.00 0.00 1.01 14:23:03 11 90.00 0.00 5.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00 14:23:03 12 92.86 0.00 4.08 0.00 0.00 0.00 3.06 0.00 0.00 0.00 14:23:03 13 84.00 0.00 6.00 1.00 0.00 0.00 6.00 0.00 0.00 3.00 14:23:03 14 88.78 0.00 3.06 0.00 0.00 0.00 5.10 0.00 0.00 3.06 14:23:03 15 92.08 0.00 4.95 0.00 0.00 0.00 1.98 0.00 0.00 0.99 14:23:03 16 88.89 0.00 5.05 0.00 1.01 0.00 2.02 0.00 0.00 3.03 14:23:03 17 97.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 14:23:03 18 90.00 0.00 4.00 0.00 0.00 0.00 4.00 0.00 0.00 2.00 14:23:03 19 88.66 0.00 3.09 0.00 1.03 0.00 2.06 0.00 0.00 5.15 14:23:03 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:04 all 88.13 0.25 6.39 0.00 0.25 0.00 3.12 0.00 0.00 1.86 14:23:04 0 88.00 0.00 7.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00 14:23:04 1 94.00 0.00 3.00 0.00 0.00 0.00 2.00 0.00 0.00 1.00 14:23:04 2 88.89 0.00 6.06 0.00 0.00 0.00 4.04 0.00 0.00 1.01 14:23:04 3 79.59 1.02 7.14 0.00 0.00 0.00 7.14 0.00 0.00 5.10 14:23:04 4 81.00 0.00 15.00 0.00 0.00 0.00 4.00 0.00 0.00 0.00 14:23:04 5 90.00 0.00 7.00 0.00 1.00 0.00 1.00 0.00 0.00 1.00 14:23:04 6 91.00 1.00 5.00 0.00 0.00 0.00 2.00 0.00 0.00 1.00 14:23:04 7 80.41 0.00 11.34 0.00 0.00 0.00 5.15 0.00 0.00 3.09 14:23:04 8 84.85 0.00 7.07 0.00 0.00 0.00 6.06 0.00 0.00 2.02 14:23:04 9 87.00 1.00 7.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00 14:23:04 10 94.95 0.00 3.03 0.00 0.00 0.00 2.02 0.00 0.00 0.00 14:23:04 11 90.91 1.01 3.03 0.00 0.00 0.00 4.04 0.00 0.00 1.01 14:23:04 12 91.92 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 1.01 14:23:04 13 90.10 0.00 5.94 0.00 0.99 0.00 2.97 0.00 0.00 0.00 14:23:04 14 91.92 0.00 5.05 0.00 0.00 0.00 1.01 0.00 0.00 2.02 14:23:04 15 87.76 0.00 6.12 0.00 0.00 0.00 2.04 0.00 0.00 4.08 14:23:04 16 86.14 0.00 7.92 0.00 0.00 0.00 2.97 0.00 0.00 2.97 14:23:04 17 97.03 0.00 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 14:23:04 18 84.69 0.00 6.12 0.00 0.00 0.00 5.10 0.00 0.00 4.08 14:23:04 19 82.18 0.99 7.92 0.00 1.98 0.00 3.96 0.00 0.00 2.97 14:23:04 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle 14:23:05 all 83.08 0.15 8.43 0.05 0.51 0.00 3.18 0.00 0.00 4.60 14:23:05 0 86.87 0.00 4.04 0.00 0.00 0.00 3.03 0.00 0.00 6.06 14:23:05 1 92.78 0.00 2.06 0.00 0.00 0.00 1.03 0.00 0.00 4.12 14:23:05 2 78.79 0.00 7.07 1.01 1.01 0.00 3.03 0.00 0.00 9.09 14:23:05 3 64.58 0.00 13.54 0.00 2.08 0.00 7.29 0.00 0.00 12.50 14:23:05 4 67.00 0.00 25.00 0.00 1.00 0.00 3.00 0.00 0.00 4.00 14:23:05 5 82.00 1.00 7.00 0.00 4.00 0.00 1.00 0.00 0.00 5.00 14:23:05 6 87.63 0.00 6.19 0.00 0.00 0.00 3.09 0.00 0.00 3.09 14:23:05 7 88.00 0.00 5.00 0.00 0.00 0.00 4.00 0.00 0.00 3.00 14:23:05 8 82.83 0.00 5.05 0.00 0.00 0.00 6.06 0.00 0.00 6.06 14:23:05 9 78.79 0.00 8.08 0.00 1.01 0.00 5.05 0.00 0.00 7.07 14:23:05 10 82.83 1.01 6.06 0.00 0.00 0.00 4.04 0.00 0.00 6.06 14:23:05 11 80.81 0.00 8.08 0.00 1.01 0.00 4.04 0.00 0.00 6.06 14:23:05 12 90.10 0.00 2.97 0.00 0.00 0.00 3.96 0.00 0.00 2.97 14:23:05 13 90.91 1.01 3.03 0.00 0.00 0.00 3.03 0.00 0.00 2.02 14:23:05 14 87.13 0.00 7.92 0.00 0.00 0.00 1.98 0.00 0.00 2.97 14:23:05 15 78.79 0.00 18.18 0.00 0.00 0.00 1.01 0.00 0.00 2.02 14:23:05 16 84.85 0.00 9.09 0.00 0.00 0.00 3.03 0.00 0.00 3.03 14:23:05 17 72.00 0.00 22.00 0.00 0.00 0.00 3.00 0.00 0.00 3.00 14:23:05 18 91.84 0.00 5.10 0.00 0.00 0.00 2.04 0.00 0.00 1.02 14:23:05 19 92.93 0.00 3.03 0.00 0.00 0.00 1.01 0.00 0.00 3.03 ``` ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 9:31am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/183 "2026-01-05T09:31:23Z")
> [@RainTown](#): > > btw 1,315,608 = ( 2 \* 2 \* 2 \* 3 \* 7 \* 41 \* 191) was a very specific number, is there some meaning in it, or was just the count the time you checked it. Nope, not any special number, just the count. > [@Christian\_Dahlqvist](#): > > How are these updates batched together? Are they sent as a number of different individual updates in a single bulk request (this is my default interpretation of batched together) or are they actually combined into a single logical update request for that document? So let’s say there are 10 different updates for a single document in a 30 second window, then all of these updates are combined together in our application and upserted as a single document, meaning that all updates for a single document are combined into a single logical request for that document. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [January 5, 2026, 10:02am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/184 "2026-01-05T10:02:23Z")
OK to save others wetransfer, here's the sdb disk data, taken at 15 second intervals in both cases. ```auto ## iostat_320GB_2026-01-05_1409.log Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 0.03 3.56 0.00 0.27 0.67 105.53 5.73 393.90 52.54 90.17 1.21 68.75 0.00 406.25 0.00 0.00 1.82 230509.24 1.45 0.65 0.01 0.34 vdb 0.00 0.00 0.00 0.00 0.00 0.00 10.73 198.93 38.33 78.12 0.77 18.53 0.00 0.00 0.00 0.00 0.00 0.00 3.80 0.74 0.01 0.85 vdb 0.00 0.00 0.00 0.00 0.00 0.00 360.93 115226.93 14029.33 97.49 1.38 319.25 0.00 0.00 0.00 0.00 0.00 0.00 37.07 0.31 0.51 12.29 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1219.40 371660.80 63065.00 98.10 0.97 304.79 0.00 0.00 0.00 0.00 0.00 0.00 151.07 0.14 1.21 36.91 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1155.60 414278.40 74973.07 98.48 1.09 358.50 0.00 0.00 0.00 0.00 0.00 0.00 133.60 0.23 1.29 39.65 vdb 0.00 0.00 0.00 0.00 0.00 0.00 979.27 351356.53 62913.47 98.47 0.81 358.80 0.00 0.00 0.00 0.00 0.00 0.00 110.60 0.19 0.82 35.81 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1048.27 347034.40 62437.87 98.35 1.06 331.06 0.00 0.00 0.00 0.00 0.00 0.00 138.53 0.16 1.13 37.47 vdb 0.00 0.00 0.00 0.00 0.00 0.00 797.93 306260.53 55359.47 98.58 0.88 383.82 0.00 0.00 0.00 0.00 0.00 0.00 93.33 0.13 0.71 31.84 vdb 0.00 0.00 0.00 0.00 0.00 0.00 782.61 298732.84 54074.22 98.57 1.14 381.71 0.00 0.00 0.00 0.00 0.00 0.00 77.95 0.21 0.90 31.87 vdb 0.00 0.00 0.00 0.00 0.00 0.00 847.63 313932.76 60229.89 98.61 0.85 370.36 0.00 0.00 0.00 0.00 0.00 0.00 111.54 0.12 0.73 35.49 vdb 0.00 0.00 0.00 0.00 0.00 0.00 781.15 240481.55 41002.20 98.13 0.89 307.86 0.00 0.00 0.00 0.00 0.00 0.00 81.41 0.16 0.70 28.49 vdb 0.00 0.00 0.00 0.00 0.00 0.00 823.00 263872.80 48211.13 98.32 0.88 320.62 0.00 0.00 0.00 0.00 0.00 0.00 110.67 0.27 0.75 35.04 vdb 0.00 0.00 0.00 0.00 0.00 0.00 702.87 248756.80 45463.00 98.48 1.02 353.92 0.00 0.00 0.00 0.00 0.00 0.00 74.87 0.17 0.73 31.36 vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.80 254354.40 46731.47 98.63 1.17 393.25 0.00 0.00 0.00 0.00 0.00 0.00 72.47 0.16 0.77 31.31 vdb 0.00 0.00 0.00 0.00 0.00 0.00 640.13 177248.00 29948.80 97.91 0.76 276.89 0.00 0.00 0.00 0.00 0.00 0.00 95.00 0.11 0.49 24.48 vdb 0.27 8.80 1.93 87.88 0.50 33.00 796.33 364409.07 77641.20 98.98 1.28 457.61 0.00 0.00 0.00 0.00 0.00 0.00 96.60 0.28 1.04 41.07 vdb 0.00 0.00 0.00 0.00 0.00 0.00 729.27 328409.33 67852.13 98.94 1.13 450.33 0.00 0.00 0.00 0.00 0.00 0.00 72.27 0.13 0.84 37.92 vdb 0.00 0.00 0.00 0.00 0.00 0.00 590.40 242202.93 46730.13 98.75 0.84 410.24 0.00 0.00 0.00 0.00 0.00 0.00 71.73 0.12 0.51 28.77 vdb 0.00 0.00 0.00 0.00 0.00 0.00 604.33 235453.07 44514.33 98.66 0.94 389.61 0.00 0.00 0.00 0.00 0.00 0.00 73.27 0.21 0.58 27.89 vdb 0.00 0.00 0.00 0.00 0.00 0.00 573.67 187122.40 33198.87 98.30 0.81 326.19 0.00 0.00 0.00 0.00 0.00 0.00 82.60 0.14 0.48 24.08 vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.20 220578.67 41140.60 98.45 0.90 341.35 0.00 0.00 0.00 0.00 0.00 0.00 105.13 0.19 0.60 29.65 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1038.93 395617.60 75117.13 98.64 1.02 380.79 0.00 0.00 0.00 0.00 0.00 0.00 123.80 0.13 1.08 38.16 vdb 0.60 2.40 0.00 0.00 0.33 4.00 1073.80 376683.20 70257.67 98.49 1.30 350.79 0.00 0.00 0.00 0.00 0.00 0.00 144.47 0.16 1.42 38.21 vdb 0.13 0.53 0.00 0.00 0.50 4.00 559.93 264561.60 56500.87 99.02 1.25 472.49 0.00 0.00 0.00 0.00 0.00 0.00 88.67 0.14 0.71 21.55 vdb 382.93 38029.07 0.00 0.00 0.28 99.31 515.00 215640.27 37577.93 98.65 1.33 418.72 0.00 0.00 0.00 0.00 0.00 0.00 77.60 0.22 0.81 24.83 vdb 30.87 2789.07 0.00 0.00 0.41 90.36 1242.13 621722.93 132224.80 99.07 1.29 500.53 0.00 0.00 0.00 0.00 0.00 0.00 120.53 0.24 1.64 56.05 vdb 35.33 3567.73 0.00 0.00 0.28 100.97 1220.80 464354.13 93318.13 98.71 0.82 380.37 0.00 0.00 0.00 0.00 0.00 0.00 176.33 0.12 1.04 45.17 vdb 7.27 382.93 0.00 0.00 0.46 52.70 1010.80 330330.67 54465.47 98.18 0.78 326.80 0.00 0.00 0.00 0.00 0.00 0.00 133.27 0.13 0.81 34.32 vdb 8.00 404.27 0.00 0.00 0.44 50.53 1129.47 432742.13 85700.67 98.70 1.36 383.14 0.00 0.00 0.00 0.00 0.00 0.00 153.73 0.18 1.57 42.61 vdb 78.40 8293.33 0.00 0.00 0.62 105.78 1141.40 430768.27 83246.93 98.65 1.36 377.40 0.00 0.00 0.00 0.00 0.00 0.00 172.60 0.30 1.65 45.33 vdb 363.13 43784.27 0.40 0.11 0.70 120.57 979.27 338804.80 61585.33 98.43 1.68 345.98 0.00 0.00 0.00 0.00 0.00 0.00 129.13 0.30 1.94 44.99 vdb 20.00 2042.93 0.00 0.00 0.31 102.15 1011.60 410913.60 83870.13 98.81 1.05 406.20 0.00 0.00 0.00 0.00 0.00 0.00 146.27 0.24 1.11 42.91 vdb 39.27 4570.40 0.00 0.00 0.31 116.39 880.80 290303.20 50768.80 98.29 0.83 329.59 0.00 0.00 0.00 0.00 0.00 0.00 124.67 0.14 0.76 31.65 vdb 27.07 2176.00 0.00 0.00 0.79 80.39 881.53 385717.87 76757.87 98.86 1.03 437.55 0.00 0.00 0.00 0.00 0.00 0.00 103.53 0.20 0.95 39.25 vdb 0.53 2.13 0.00 0.00 0.12 4.00 960.80 346204.53 65026.07 98.54 1.11 360.33 0.00 0.00 0.00 0.00 0.00 0.00 98.47 0.17 1.08 36.75 vdb 4.00 16.00 0.00 0.00 0.25 4.00 848.93 321556.27 62153.60 98.65 1.78 378.78 0.00 0.00 0.00 0.00 0.00 0.00 100.53 0.54 1.57 39.55 vdb 528.40 38596.53 0.00 0.00 1.00 73.04 891.87 354328.00 70124.87 98.74 2.31 397.29 0.00 0.00 0.00 0.00 0.00 0.00 101.93 0.34 2.62 51.04 vdb 352.53 18881.87 0.00 0.00 0.67 53.56 1011.67 551422.67 119157.00 99.16 1.52 545.06 0.00 0.00 0.00 0.00 0.00 0.00 85.67 0.34 1.81 58.85 vdb 18.47 1226.40 0.00 0.00 0.23 66.41 799.13 306472.27 57286.53 98.62 0.82 383.51 0.00 0.00 0.00 0.00 0.00 0.00 98.27 0.12 0.67 33.55 vdb 2.47 10.40 0.13 5.13 0.30 4.22 898.60 340812.00 67165.73 98.68 0.86 379.27 0.00 0.00 0.00 0.00 0.00 0.00 98.20 0.10 0.78 35.33 vdb 1.07 12.27 2.00 65.22 0.31 11.50 785.93 273817.33 49750.87 98.44 0.90 348.40 0.00 0.00 0.00 0.00 0.00 0.00 97.60 0.25 0.73 32.08 vdb 3.07 12.27 0.00 0.00 0.37 4.00 814.33 341875.20 67947.60 98.82 1.14 419.82 0.00 0.00 0.00 0.00 0.00 0.00 98.20 0.24 0.95 39.57 vdb 1.33 36.53 0.00 0.00 0.30 27.40 751.80 241671.47 41905.73 98.24 0.77 321.46 0.00 0.00 0.00 0.00 0.00 0.00 97.20 0.15 0.59 29.36 vdb 2.13 10.40 0.47 17.95 0.69 4.88 869.20 354629.60 71582.40 98.80 2.02 408.00 0.00 0.00 0.00 0.00 0.00 0.00 97.87 0.24 1.78 38.85 vdb 613.40 52583.20 0.00 0.00 0.77 85.72 719.60 277639.47 52604.00 98.65 2.22 385.82 0.00 0.00 0.00 0.00 0.00 0.00 96.60 0.80 2.15 53.89 vdb 7.60 648.00 0.00 0.00 1.04 85.26 758.93 297681.07 58154.47 98.71 1.59 392.24 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.27 1.24 36.56 vdb 29.00 3484.80 0.00 0.00 0.63 120.17 680.33 274476.53 50714.73 98.68 1.00 403.44 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.25 0.72 32.05 vdb 2.13 156.53 0.00 0.00 0.25 73.38 667.53 225117.87 41565.67 98.42 1.01 337.24 0.00 0.00 0.00 0.00 0.00 0.00 78.00 0.14 0.68 28.59 vdb 0.07 0.27 0.00 0.00 1.00 4.00 905.47 383426.40 81464.80 98.90 1.21 423.46 0.00 0.00 0.00 0.00 0.00 0.00 113.07 0.24 1.12 42.19 vdb 0.20 0.80 0.00 0.00 0.33 4.00 607.33 186962.93 30445.13 98.04 0.85 307.84 0.00 0.00 0.00 0.00 0.00 0.00 71.87 0.11 0.53 23.39 vdb 17.40 1910.67 0.00 0.00 0.25 109.81 604.53 247407.47 46532.53 98.72 0.88 409.25 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.19 0.55 28.77 vdb 1077.20 107264.00 0.00 0.00 0.75 99.58 843.60 296176.27 61161.33 98.64 3.12 351.09 0.00 0.00 0.00 0.00 0.00 0.00 117.33 0.92 3.55 76.51 vdb 401.07 25366.67 0.00 0.00 0.98 63.25 777.40 411032.27 87238.33 99.12 2.67 528.73 0.00 0.00 0.00 0.00 0.00 0.00 66.07 0.62 2.51 59.95 vdb 43.80 4234.40 2.27 4.92 0.72 96.68 853.53 492707.73 110831.60 99.24 1.72 577.26 0.00 0.00 0.00 0.00 0.00 0.00 64.27 0.37 1.53 53.49 vdb 67.27 7241.87 0.93 1.37 0.68 107.66 692.67 279851.73 55327.80 98.76 1.45 404.02 0.00 0.00 0.00 0.00 0.00 0.00 71.13 0.21 1.07 37.97 vdb 225.53 28188.53 0.00 0.00 0.87 124.99 573.13 228129.60 43783.13 98.71 1.86 398.04 0.00 0.00 0.00 0.00 0.00 0.00 69.67 0.65 1.30 39.47 vdb 28.07 3540.27 0.00 0.00 0.42 126.14 620.73 197980.53 35245.00 98.27 0.82 318.95 0.00 0.00 0.00 0.00 0.00 0.00 79.20 0.15 0.53 26.29 vdb 116.93 14670.93 0.00 0.00 0.33 125.46 604.07 244633.07 47383.80 98.74 0.95 404.98 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.11 0.62 33.65 vdb 25.73 2795.47 0.00 0.00 0.47 108.63 674.87 215106.13 42645.33 98.44 1.23 318.74 0.00 0.00 0.00 0.00 0.00 0.00 95.20 0.14 0.85 29.17 vdb 1.47 70.40 0.00 0.00 0.32 48.00 862.13 352412.53 71098.80 98.80 1.32 408.77 0.00 0.00 0.00 0.00 0.00 0.00 109.33 0.17 1.15 38.03 vdb 309.40 38766.67 0.00 0.00 0.90 125.30 833.67 287650.67 51645.13 98.41 1.99 345.04 0.00 0.00 0.00 0.00 0.00 0.00 110.80 0.51 2.00 41.25 vdb 118.20 14795.47 0.00 0.00 1.70 125.17 602.20 248080.53 50837.47 98.83 2.74 411.96 0.00 0.00 0.00 0.00 0.00 0.00 70.93 0.63 1.89 36.27 vdb 191.27 24146.40 0.00 0.00 1.16 126.24 562.67 186333.07 33426.87 98.34 2.23 331.16 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.51 1.52 33.33 vdb 2.07 88.80 0.00 0.00 0.29 42.97 562.47 191215.73 33472.73 98.35 0.96 339.96 0.00 0.00 0.00 0.00 0.00 0.00 71.60 0.21 0.56 25.17 vdb 0.20 0.80 0.00 0.00 0.33 4.00 644.93 277751.47 57686.60 98.89 1.51 430.67 0.00 0.00 0.00 0.00 0.00 0.00 69.20 0.39 1.00 34.83 vdb 31.47 3969.07 0.00 0.00 0.45 126.14 557.60 210196.00 41199.67 98.66 1.04 376.97 0.00 0.00 0.00 0.00 0.00 0.00 73.20 0.15 0.61 28.40 vdb 0.00 0.00 0.00 0.00 0.00 0.00 530.93 181512.53 33837.40 98.46 1.50 341.87 0.00 0.00 0.00 0.00 0.00 0.00 69.00 0.31 0.82 26.05 vdb 36.67 4111.20 0.00 0.00 0.52 112.12 560.80 208841.07 41924.87 98.68 1.14 372.40 0.00 0.00 0.00 0.00 0.00 0.00 78.93 0.16 0.67 30.35 vdb 1.20 22.93 0.00 0.00 0.28 19.11 709.13 249012.27 49458.73 98.59 0.97 351.15 0.00 0.00 0.00 0.00 0.00 0.00 97.53 0.19 0.71 32.19 ## iostat_640GB_2026-01-05_1432.log Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util vdb 0.01 1.61 0.00 0.02 0.52 118.16 6.43 327.39 33.05 83.72 0.94 50.95 0.00 830.31 0.00 0.00 1.13 175039.68 1.49 0.49 0.01 0.29 vdb 0.00 0.00 0.00 0.00 0.00 0.00 12.67 291.73 60.67 82.73 0.57 23.03 0.00 0.00 0.00 0.00 0.00 0.00 4.47 0.48 0.01 0.85 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1230.80 371603.47 56267.60 97.86 2.11 301.92 0.00 0.00 0.00 0.00 0.00 0.00 134.53 0.47 2.66 34.93 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1417.47 483401.33 86083.73 98.38 0.96 341.03 0.00 0.00 0.00 0.00 0.00 0.00 170.67 0.13 1.38 40.67 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1159.73 422145.60 75298.27 98.48 1.58 364.00 0.00 0.00 0.00 0.00 0.00 0.00 149.27 0.15 1.85 37.89 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1134.07 407770.93 74040.67 98.49 1.73 359.57 0.00 0.00 0.00 0.00 0.00 0.00 129.40 0.27 2.00 37.65 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1068.60 398468.00 72778.53 98.55 1.71 372.89 0.00 0.00 0.00 0.00 0.00 0.00 121.07 0.45 1.88 39.39 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1042.33 390344.00 73901.60 98.61 1.20 374.49 0.00 0.00 0.00 0.00 0.00 0.00 122.20 0.17 1.27 38.37 vdb 0.00 0.00 0.00 0.00 0.00 0.00 788.07 289938.67 50836.80 98.47 1.27 367.91 0.00 0.00 0.00 0.00 0.00 0.00 101.20 0.30 1.03 32.21 vdb 0.00 0.00 0.00 0.00 0.00 0.00 995.47 353162.67 64973.67 98.49 1.28 354.77 0.00 0.00 0.00 0.00 0.00 0.00 125.93 0.13 1.29 37.20 vdb 0.00 0.00 0.00 0.00 0.00 0.00 916.07 317304.53 57496.20 98.43 1.13 346.38 0.00 0.00 0.00 0.00 0.00 0.00 125.47 0.11 1.05 33.87 vdb 0.00 0.00 0.00 0.00 0.00 0.00 926.47 310887.20 56513.67 98.39 2.38 335.56 0.00 0.00 0.00 0.00 0.00 0.00 129.87 0.22 2.23 35.31 vdb 0.00 0.00 0.00 0.00 0.00 0.00 866.67 360320.00 72203.13 98.81 1.35 415.75 0.00 0.00 0.00 0.00 0.00 0.00 104.67 0.13 1.18 39.07 vdb 0.00 0.00 0.00 0.00 0.00 0.00 807.27 326996.53 61410.60 98.70 0.97 405.07 0.00 0.00 0.00 0.00 0.00 0.00 91.20 0.14 0.80 34.72 vdb 5.60 516.80 0.00 0.00 0.85 92.29 946.13 427712.27 89597.53 98.96 0.99 452.06 0.00 0.00 0.00 0.00 0.00 0.00 127.40 0.19 0.96 42.69 vdb 65.40 7930.93 1.27 1.90 1.17 121.27 713.33 258136.27 47356.07 98.52 1.25 361.87 0.00 0.00 0.00 0.00 0.00 0.00 83.00 0.34 0.99 35.52 vdb 2.87 190.67 0.00 0.00 0.28 66.51 759.80 286863.20 56985.40 98.68 1.28 377.55 0.00 0.00 0.00 0.00 0.00 0.00 107.00 0.38 1.01 36.27 vdb 11.67 1036.27 0.00 0.00 0.31 88.82 611.07 205282.93 34224.73 98.25 2.00 335.94 0.00 0.00 0.00 0.00 0.00 0.00 80.47 0.20 1.24 26.61 vdb 3.00 56.80 0.27 8.16 0.56 18.93 719.87 259650.67 50056.80 98.58 1.07 360.69 0.00 0.00 0.00 0.00 0.00 0.00 102.73 0.32 0.80 32.11 vdb 48.07 5618.67 0.00 0.00 1.67 116.89 699.87 259702.67 47204.40 98.54 2.05 371.07 0.00 0.00 0.00 0.00 0.00 0.00 80.80 0.44 1.55 33.63 vdb 308.20 38897.87 0.00 0.00 0.98 126.21 1171.67 424724.00 83513.13 98.62 2.20 362.50 0.00 0.00 0.00 0.00 0.00 0.00 163.73 0.66 2.98 55.15 vdb 0.07 0.27 0.00 0.00 0.00 4.00 1222.20 472392.53 90645.80 98.67 1.57 386.51 0.00 0.00 0.00 0.00 0.00 0.00 156.53 0.26 1.96 42.11 vdb 891.47 109743.73 0.00 0.00 0.58 123.10 892.00 531333.87 118240.27 99.25 3.24 595.67 0.00 0.00 0.00 0.00 0.00 0.00 76.00 0.84 3.47 55.39 vdb 45.00 4273.60 0.00 0.00 0.34 94.97 240.27 140334.13 30744.60 99.22 1.88 584.08 0.00 0.00 0.00 0.00 0.00 0.00 47.33 0.07 0.47 12.00 vdb 45.67 5248.27 0.00 0.00 0.27 114.93 740.80 277701.33 46834.40 98.44 1.58 374.87 0.00 0.00 0.00 0.00 0.00 0.00 95.60 0.20 1.20 24.96 vdb 307.33 38948.27 0.00 0.00 0.76 126.73 1079.40 422967.20 75879.60 98.60 1.63 391.85 0.00 0.00 0.00 0.00 0.00 0.00 136.13 0.32 2.03 43.73 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1291.67 500264.80 92085.20 98.62 1.06 387.30 0.00 0.00 0.00 0.00 0.00 0.00 144.53 0.19 1.40 40.45 vdb 39.20 4896.80 0.00 0.00 0.35 124.92 1245.60 462864.80 86248.93 98.58 0.83 371.60 0.00 0.00 0.00 0.00 0.00 0.00 163.20 0.12 1.06 40.35 vdb 36.87 4097.60 0.00 0.00 0.40 111.15 1338.27 541657.33 107145.40 98.77 1.57 404.75 0.00 0.00 0.00 0.00 0.00 0.00 162.67 0.12 2.13 47.89 vdb 349.27 44054.93 0.07 0.02 0.48 126.14 1268.93 509679.47 98026.80 98.72 1.80 401.66 0.00 0.00 0.00 0.00 0.00 0.00 156.93 0.42 2.52 54.56 vdb 155.33 19547.73 0.00 0.00 1.15 125.84 1435.13 644687.47 133423.40 98.94 2.42 449.22 0.00 0.00 0.00 0.00 0.00 0.00 144.33 0.48 3.72 58.27 vdb 12.13 1458.67 0.00 0.00 0.45 120.22 1254.47 545881.07 106459.27 98.84 1.32 435.15 0.00 0.00 0.00 0.00 0.00 0.00 128.73 0.16 1.68 44.35 vdb 0.07 0.27 0.00 0.00 0.00 4.00 1131.87 446450.67 86985.73 98.72 1.54 394.44 0.00 0.00 0.00 0.00 0.00 0.00 132.07 0.48 1.81 43.47 vdb 175.20 16444.53 0.00 0.00 0.89 93.86 1110.13 413014.67 80836.60 98.65 2.06 372.04 0.00 0.00 0.00 0.00 0.00 0.00 145.07 0.34 2.50 45.25 vdb 90.07 10244.80 0.00 0.00 0.65 113.75 1108.47 424598.13 84175.73 98.70 3.24 383.05 0.00 0.00 0.00 0.00 0.00 0.00 131.67 0.30 3.69 45.65 vdb 39.40 4760.27 0.00 0.00 0.37 120.82 931.53 343535.73 61653.60 98.51 1.12 368.79 0.00 0.00 0.00 0.00 0.00 0.00 108.53 0.18 1.07 35.31 vdb 153.87 18186.40 1.80 1.16 0.87 118.20 1066.73 417408.00 85193.27 98.76 2.35 391.30 0.00 0.00 0.00 0.00 0.00 0.00 123.33 0.27 2.68 48.35 vdb 191.20 23293.07 0.00 0.00 0.81 121.83 986.33 368774.93 69115.67 98.59 2.38 373.88 0.00 0.00 0.00 0.00 0.00 0.00 118.53 0.26 2.53 41.84 vdb 3.47 17.87 0.00 0.00 1.12 5.15 971.40 399100.80 78724.93 98.78 1.52 410.85 0.00 0.00 0.00 0.00 0.00 0.00 102.47 0.19 1.50 40.43 vdb 0.00 0.00 0.00 0.00 0.00 0.00 823.27 284509.60 49577.00 98.37 1.11 345.59 0.00 0.00 0.00 0.00 0.00 0.00 105.67 0.21 0.93 31.65 vdb 0.27 1.07 0.00 0.00 0.25 4.00 1038.07 384037.60 77545.20 98.68 1.45 369.95 0.00 0.00 0.00 0.00 0.00 0.00 121.80 0.26 1.54 41.41 vdb 1.07 4.80 0.00 0.00 0.62 4.50 842.33 309266.67 56807.33 98.54 1.31 367.15 0.00 0.00 0.00 0.00 0.00 0.00 106.27 0.18 1.12 33.76 vdb 674.80 77366.93 0.13 0.02 0.97 114.65 966.33 375238.40 73028.60 98.69 2.82 388.31 0.00 0.00 0.00 0.00 0.00 0.00 95.93 0.74 3.46 64.13 vdb 0.00 0.00 0.00 0.00 0.00 0.00 910.80 432904.53 89790.40 99.00 1.75 475.30 0.00 0.00 0.00 0.00 0.00 0.00 93.53 0.21 1.61 45.52 vdb 0.13 0.53 0.00 0.00 0.00 4.00 934.47 443489.33 92719.67 99.00 1.14 474.59 0.00 0.00 0.00 0.00 0.00 0.00 93.87 0.19 1.08 44.53 vdb 66.20 4892.27 0.00 0.00 0.63 73.90 838.07 290399.73 51988.00 98.41 1.52 346.51 0.00 0.00 0.00 0.00 0.00 0.00 103.60 0.18 1.33 33.89 vdb 0.20 0.80 0.00 0.00 0.33 4.00 849.20 357271.73 69929.93 98.80 1.53 420.72 0.00 0.00 0.00 0.00 0.00 0.00 96.40 0.27 1.32 36.83 vdb 0.47 1.87 0.00 0.00 1.57 4.00 807.67 280924.27 50872.53 98.44 1.24 347.82 0.00 0.00 0.00 0.00 0.00 0.00 100.27 0.18 1.02 30.85 vdb 0.87 19.73 0.00 0.00 0.69 22.77 786.93 359258.13 72853.20 98.93 1.83 456.53 0.00 0.00 0.00 0.00 0.00 0.00 71.20 0.36 1.47 41.79 vdb 0.27 1.07 0.00 0.00 0.25 4.00 737.07 237122.13 40993.87 98.23 1.59 321.71 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.29 1.20 29.12 vdb 95.47 11246.13 0.00 0.00 0.80 117.80 772.47 312089.33 59396.47 98.72 1.22 404.02 0.00 0.00 0.00 0.00 0.00 0.00 81.40 0.30 1.04 36.43 vdb 416.87 27695.73 0.00 0.00 1.34 66.44 852.40 275337.87 52466.33 98.40 4.25 323.01 0.00 0.00 0.00 0.00 0.00 0.00 124.27 0.73 4.27 45.28 vdb 0.00 0.00 0.00 0.00 0.00 0.00 690.93 269222.40 49188.07 98.61 1.23 389.65 0.00 0.00 0.00 0.00 0.00 0.00 74.60 0.40 0.88 31.07 vdb 0.07 1.07 0.00 0.00 0.00 16.00 838.40 295674.40 55691.40 98.52 1.61 352.67 0.00 0.00 0.00 0.00 0.00 0.00 101.07 0.23 1.37 33.68 vdb 0.00 0.00 0.00 0.00 0.00 0.00 674.73 290930.93 55346.40 98.80 1.25 431.18 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.24 0.87 32.80 vdb 2.87 142.13 0.00 0.00 0.23 49.58 809.27 286785.60 53820.53 98.52 1.32 354.38 0.00 0.00 0.00 0.00 0.00 0.00 102.00 0.22 1.09 32.75 vdb 16.20 1880.80 0.00 0.00 3.30 116.10 859.53 323764.00 65429.80 98.70 1.58 376.67 0.00 0.00 0.00 0.00 0.00 0.00 112.60 0.46 1.46 40.00 vdb 838.87 106450.93 0.00 0.00 0.58 126.90 661.93 218886.13 37578.07 98.27 2.28 330.68 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.55 2.04 55.57 vdb 0.07 0.27 0.00 0.00 1.00 4.00 901.60 378569.33 77062.40 98.84 1.72 419.89 0.00 0.00 0.00 0.00 0.00 0.00 105.13 0.27 1.58 41.31 vdb 0.00 0.00 0.00 0.00 0.00 0.00 764.00 352281.87 70652.27 98.93 1.20 461.10 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.23 0.94 38.53 vdb 50.07 5750.93 0.00 0.00 0.70 114.87 878.27 418755.47 89242.87 99.03 1.46 476.80 0.00 0.00 0.00 0.00 0.00 0.00 98.27 0.12 1.33 48.43 vdb 0.00 0.00 0.00 0.00 0.00 0.00 649.40 219117.07 37200.33 98.28 1.07 337.41 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.21 0.71 26.32 vdb 0.00 0.00 0.00 0.00 0.00 0.00 894.73 342192.00 71081.87 98.76 1.14 382.45 0.00 0.00 0.00 0.00 0.00 0.00 119.07 0.17 1.04 39.63 vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.00 232256.00 41335.87 98.46 0.92 359.53 0.00 0.00 0.00 0.00 0.00 0.00 77.20 0.29 0.61 28.64 vdb 0.00 0.00 0.00 0.00 0.00 0.00 1005.27 364122.93 70386.33 98.59 2.16 362.22 0.00 0.00 0.00 0.00 0.00 0.00 130.13 0.35 2.22 40.59 vdb 14.33 1710.67 0.00 0.00 0.28 119.35 911.33 361817.60 72186.87 98.75 1.67 397.02 0.00 0.00 0.00 0.00 0.00 0.00 104.67 0.36 1.57 40.45 vdb 308.27 38892.80 0.00 0.00 0.71 126.17 598.20 194449.33 32753.53 98.21 1.84 325.06 0.00 0.00 0.00 0.00 0.00 0.00 76.80 0.39 1.35 40.77 vdb 482.67 60978.40 0.00 0.00 1.11 126.34 816.80 331697.87 67881.73 98.81 3.37 406.09 0.00 0.00 0.00 0.00 0.00 0.00 92.93 0.70 3.36 58.40 vdb 1388.87 176792.80 0.00 0.00 0.52 127.29 631.53 204010.13 37077.07 98.33 2.42 323.04 0.00 0.00 0.00 0.00 0.00 0.00 77.73 0.74 2.31 78.91 vdb 0.40 1.60 0.00 0.00 0.50 4.00 798.67 428524.53 94237.07 99.16 1.89 536.55 0.00 0.00 0.00 0.00 0.00 0.00 71.80 0.21 1.52 48.61 ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [January 5, 2026, 10:48am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/185 "2026-01-05T10:48:04Z")
> [@Lakshya\_Gupta](#): > > I created 2 virtual machines using the same number of cores (20) and memory (53G), but with different disk sizes, 320Gb and 640Gb, and established a single node Elastic Search cluster on top of it. Thereafter I created an index (similar on both the Clusters) with a single primary shard and no routing (because it’s 1 shard anyways) and then ran NFRs for 15minutes on both of them (the NFR was creating random documents and inserting them). I looked at the iostat and sar outputs, tbh I don't think it moves us forward. There's to my eye not much difference between the 2 iostat outputs. The w/s ramps up, but r/s stays close to zero for a while, then r/s also increases a lot to a sort of mini sine-wave like behaviour, likely the merges kicking in. I made some gnuplots of the data, just the r/s, w/s, f/s, and %util(right hand y-axis). Here's the images: ![gnuplot-320GB](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9e9f0153c2b54f9218c28161b9587312d98ca2c.jpeg) ![gnuplot-640GB](https://us1.discourse-cdn.com/elastic/original/3X/7/9/793522155a6a70ac131a9f8ee4b2753b13c98aa3.jpeg) Look at those 2 graphs and tell me one system is really "better" than the other? ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 11:12am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/186 "2026-01-05T11:12:11Z")
> [@RainTown](#): > > Look at those 2 graphs and tell me one system is really "better" than the other? You are right, I don’t see any improvements with disk size (and thereby more IOPS) either. Based on the outcome of these 2 graphs, it feels like the bottleneck is not IOPS. What do you think? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [January 5, 2026, 11:28am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/187 "2026-01-05T11:28:46Z")
> [@Lakshya\_Gupta](#): > > Based on the outcome of these 2 graphs, it feels like the bottleneck is not IOPS. What do you think? I agree. In fact I agreed on this already before Christmas, in part by seeing the fio results, and we pivoted back to the original architecture decision, which gives you little chance to fully utilize your hardware. Suggestions were to "special case" the most skewed IDs, or simply remove the customer routing altogether. > [@RainTown](#): > > So, we could try and spend time getting a fio profile that matches real-world elasticsearch profile closer and closer. I don't think that is now worth the effort. You are neither bandwidth nor IOPS bound - the storage is not the core problem, and faster/better storage would not "solve" the problem, at best it could just squeeze a little bit more out of it. > ... > I don't work for elastic, so this is just my view, but you, er, self-limited the scalability in your design choices. That is only demonstrated with the reality of the skew in data, that you likely were unable to predict, so don't be too hard on yourselves > [@Christian\_Dahlqvist](#): > > As @RainTown pointed out you have limited how efficiently Elasticsearch can work through your design choices, so I think a change in the design may be required if nothing else works. I wish I had a different answer right now, or a really great suggestion of something else to try, but I don't. ---
**Author:** ![Lakshya\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lakshya_gupta/32/146116_2.png) [@Lakshya\_Gupta](https://discuss.elastic.co/u/Lakshya_Gupta)\ **Post date:** [January 5, 2026, 11:36am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/188 "2026-01-05T11:36:29Z")
> [@RainTown](#): > > Suggestions were to "special case" the most skewed IDs, or simply remove the customer routing altogether. Implementing both of these might be a bit tricky, but if this is the only option, then might as well need to carry this out 😕 . ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [January 5, 2026, 11:50am UTC](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954/189 "2026-01-05T11:50:51Z")
> [@Lakshya\_Gupta](#): > > Implementing both of these might be a bit tricky, but if this is the only option, then might as well need to carry this out 😕 . Which is why I also asked gently if it is worth the hassle, operational effort, disruption, ... I'm obviously not bound by your SLAs, nor do I know how often you are breaching them, nor the penalty, but ... not all problems are worth solving. As even if you "solve" this one, you can easily inadvertently create a regression elsewhere in what is probably quite a complex overall solution. Every perturbation on a working system is attached to some risk. You are likely wasting (under utilizing might be nicer) the resources in your existing cluster, and that has a business cost too. [Previous page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=8) [Next page](https://discuss.elastic.co/t/high-cpu-usage-on-a-few-data-nodes-hotspotting-of-data/383954.md?page=10)