RainTown
December 21, 2025, 1:00pm
170
Of course, but "updating" same document _id to be exactly the same (well, different _version) is generally pointless. It's also easy-ish to remove, e.g. use the doc (or hash thereof) as a key. eg in redis
Hash the document (e.g. SHA-256, ...)
Obviously the width of time window is easy to tune and show be low-cost.
If the doc is not always the same, rather just rapidly changing, it's a bit harder to define what "de-dup" would really mean, it's more sampling than de-deduplication.
@Lakshya-Gupta showed his model, but I didn't quite follow, sorry again.
Er, this is a bit like the "high CPU load" moments. If there are none in the real data, dont feel obliged to generate them. The pseudo code above I suggested is trivial to adapt to keep counts of how many dups you are removing - you really should be tracking this anyways.
RainTown
December 21, 2025, 1:11pm
171
I was also thinking along these lines. Special cases can be handled differently. if you have some small number of IDs that are very clearly "special", completely different access pattern, then don't just chuck them in same bucket as the zillions of other IDs.
You have 5 people yelling in a library, and we are discussing the building's acoustic design ?
1 Like
RainTown:
I was also thinking along these lines. Special cases can be handled differently. if you have some small number of IDs that are very clearly "special", completely different access pattern, then don't just chuck them in same bucket as the zillions of other IDs.
Indeed. The issue driving the selection of a number iof routing shards was if I recall correctly the number of open context. By separating the high volume IDs into a separate index it may be possible to use tighter routing, maybe only 1 shard per ID, for the current index while you could have a significantly higher number for the high volume IDs. Depending on the query patterns, this may overall reduce the number of open contexts while at the same time spread the load more evenly.
Doing this would give more room for fine-tuning.
We are interested in architecture, right?
RainTown
December 21, 2025, 4:46pm
173
In a different sphere, it's well known you cannot architecture away anti-social behaviour, all you can do is try constrain it. A few "bad" people can ruin an area.
1 Like
I’ve got sick
Yeah, figured out a strategy to test this…will share the results once tested (although you can consider that the deduplication logic should be working as we are expecting it to).
We though of this, but the issue is we have at least 10 different consumers firing read queries. If we create 2 indices, then managing it can become a bit of a challenge and moreover if another cold id becomes a hot id tomorrow, then we will end up with the same issue.
Christian_Dahlqvist:
Elasticsearch is a search engine and is optimised for retrieval of reasonably small result sets based on relevance. It is not optimised for highly concurrent retrieval of large bulk result sets. It is also, as you have noted, not optimised for high update rates.
True
RainTown:
Of course, but "updating" same document _id to be exactly the same (well, different _version) is generally pointless. It's also easy-ish to remove, e.g. use the doc (or hash thereof) as a key. eg in redis
Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go.
Lakshya_Gupta:
Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go.
I know it should
Could you perhaps provide some additional details on exactly how you intend to test this?
Lakshya_Gupta:
We though of this, but the issue is we have at least 10 different consumers firing read queries. If we create 2 indices, then managing it can become a bit of a challenge and moreover if another cold id becomes a hot id tomorrow, then we will end up with the same issue.
It would require a process to move specific routing IDs to the specialised index but this is something you may not need to use frequently as it likely will take time for new IDs to start causing severe problems. Estimating the probability and frequency of this requires knowledge of the data and use case so I will leave this to you.
With many clients I would likely look to create a custom query proxy used by all clients that can rewrite and redirect queries where required.
RainTown
December 26, 2025, 1:10pm
178
I stretch my metaphor, but if you have five anti-social neighbors, and find a solution for that problem, move them to a out-of-the-way location, and a sixth emerges, just use the same solution ?
But, there's something not quite adding up here:
Me too. You never did share the iostat output when under highest load, but it could be that there's another "load generator" we've not covered yet.
At start of thread you wrote:
What are the doc counts for those 5 "special" values for this field ? How does that compare to say the next 100 field values in terms of count? What's the mapping type of that field? Without divulging stuff you dont want to divulge, what is that field?
btw 1,315,608 = ( 2 * 2 * 2 * 3 * 7 * 41 * 191) was a very specific number, is there some meaning in it, or was just the count the time you checked it.
Lakshya_Gupta:
Yeah, we have taken care of this as well, if 30 different types of updates come for a document in a 30 second window, then all of those are batched together (till 30 seconds from the first update time) and upserted in a single go.
This description is interesting. How are these updates batched together? Are they sent as a number of different individual updates in a single bulk request (this is my default interpretation of batched together) or are they actually combined into a single logical update request for that document?
Hey, I’m back with some new data points.
I created 2 virtual machines using the same number of cores (20) and memory (53G), but with different disk sizes, 320Gb and 640Gb, and established a single node Elastic Search cluster on top of it. Thereafter I created an index (similar on both the Clusters) with a single primary shard and no routing (because it’s 1 shard anyways) and then ran NFRs for 15minutes on both of them (the NFR was creating random documents and inserting them).
I ran 2 commands in parallel to record the output
nohup sar -u ALL -P ALL 1 > sar_cpu.log 2>&1 &
nohup iostat -ztcxd 15 > iostat_output.log 2>&1 &
Attaching the detailed metrics for both the machines.
320 GB - Iostat output - Unique Download Link | WeTransfer
320 GB - sar output - (Not able to publish due to Firewell, will post some content in next message itself)
640 GB - Iostate output - Unique Download Link | WeTransfer
640 GB - sar output - (Not able to publish due to Firewell, will post some content in next message itself)
320 Gb SAR output for some duration
14:00:50 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:00:51 all 84.80 0.00 7.02 0.75 1.05 0.00 3.46 0.00 0.00 2.91
14:00:51 0 86.27 0.00 6.86 1.96 0.98 0.00 2.94 0.00 0.00 0.98
14:00:51 1 79.38 0.00 5.15 1.03 4.12 0.00 3.09 0.00 0.00 7.22
14:00:51 2 86.87 0.00 6.06 1.01 0.00 0.00 4.04 0.00 0.00 2.02
14:00:51 3 86.27 0.00 4.90 0.00 0.98 0.00 4.90 0.00 0.00 2.94
14:00:51 4 86.00 0.00 6.00 0.00 1.00 0.00 4.00 0.00 0.00 3.00
14:00:51 5 88.12 0.00 4.95 0.99 0.99 0.00 1.98 0.00 0.00 2.97
14:00:51 6 89.11 0.00 6.93 0.99 0.00 0.00 0.99 0.00 0.00 1.98
14:00:51 7 73.00 0.00 18.00 1.00 1.00 0.00 4.00 0.00 0.00 3.00
14:00:51 8 80.61 0.00 7.14 1.02 2.04 0.00 6.12 0.00 0.00 3.06
14:00:51 9 94.00 0.00 3.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00
14:00:51 10 90.82 0.00 6.12 0.00 0.00 0.00 1.02 0.00 0.00 2.04
14:00:51 11 87.13 0.00 7.92 0.00 0.00 0.00 2.97 0.00 0.00 1.98
14:00:51 12 75.00 0.00 12.00 1.00 3.00 0.00 5.00 0.00 0.00 4.00
14:00:51 13 73.74 0.00 11.11 1.01 1.01 0.00 6.06 0.00 0.00 7.07
14:00:51 14 91.09 0.00 4.95 0.99 0.99 0.00 1.98 0.00 0.00 0.00
14:00:51 15 79.00 0.00 8.00 1.00 2.00 0.00 7.00 0.00 0.00 3.00
14:00:51 16 81.82 0.00 7.07 2.02 1.01 0.00 4.04 0.00 0.00 4.04
14:00:51 17 95.88 0.00 2.06 0.00 0.00 0.00 1.03 0.00 0.00 1.03
14:00:51 18 86.60 0.00 7.22 0.00 1.03 0.00 2.06 0.00 0.00 3.09
14:00:51 19 85.15 0.00 4.95 0.99 0.99 0.00 2.97 0.00 0.00 4.95
14:00:51 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:00:52 all 76.02 0.00 15.33 0.05 0.66 0.00 3.95 0.00 0.00 4.00
14:00:52 0 90.82 0.00 4.08 0.00 0.00 0.00 2.04 0.00 0.00 3.06
14:00:52 1 83.00 0.00 7.00 0.00 1.00 0.00 5.00 0.00 0.00 4.00
14:00:52 2 61.00 0.00 30.00 0.00 0.00 0.00 4.00 0.00 0.00 5.00
14:00:52 3 82.83 0.00 8.08 0.00 0.00 0.00 3.03 0.00 0.00 6.06
14:00:52 4 85.86 0.00 11.11 0.00 0.00 0.00 1.01 0.00 0.00 2.02
14:00:52 5 81.82 0.00 13.13 0.00 0.00 0.00 1.01 0.00 0.00 4.04
14:00:52 6 49.00 0.00 48.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00
14:00:52 7 72.45 0.00 21.43 0.00 0.00 0.00 2.04 0.00 0.00 4.08
14:00:52 8 86.73 0.00 5.10 0.00 0.00 0.00 4.08 0.00 0.00 4.08
14:00:52 9 80.61 0.00 7.14 0.00 0.00 0.00 6.12 0.00 0.00 6.12
14:00:52 10 42.00 0.00 41.00 0.00 7.00 0.00 7.00 0.00 0.00 3.00
14:00:52 11 85.71 0.00 8.16 0.00 0.00 0.00 4.08 0.00 0.00 2.04
14:00:52 12 91.92 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 1.01
14:00:52 13 81.63 0.00 8.16 0.00 1.02 0.00 3.06 0.00 0.00 6.12
14:00:52 14 65.66 0.00 22.22 0.00 1.01 0.00 6.06 0.00 0.00 5.05
14:00:52 15 86.87 0.00 7.07 0.00 0.00 0.00 3.03 0.00 0.00 3.03
14:00:52 16 66.67 0.00 16.16 0.00 1.01 0.00 8.08 0.00 0.00 8.08
14:00:52 17 56.57 0.00 25.25 1.01 1.01 0.00 10.10 0.00 0.00 6.06
14:00:52 18 86.87 0.00 6.06 0.00 1.01 0.00 3.03 0.00 0.00 3.03
14:00:52 19 83.67 0.00 11.22 0.00 0.00 0.00 3.06 0.00 0.00 2.04
14:00:52 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:00:53 all 76.14 0.00 14.19 0.10 0.90 0.00 3.76 0.00 0.00 4.91
14:00:53 0 59.41 0.00 27.72 0.00 0.99 0.00 4.95 0.00 0.00 6.93
14:00:53 1 70.71 0.00 10.10 0.00 2.02 0.00 6.06 0.00 0.00 11.11
14:00:53 2 78.00 0.00 8.00 0.00 0.00 0.00 10.00 0.00 0.00 4.00
14:00:53 3 78.12 0.00 8.33 0.00 1.04 0.00 6.25 0.00 0.00 6.25
14:00:53 4 79.59 0.00 7.14 0.00 1.02 0.00 5.10 0.00 0.00 7.14
14:00:53 5 82.65 0.00 7.14 0.00 0.00 0.00 7.14 0.00 0.00 3.06
14:00:53 6 64.00 0.00 23.00 0.00 1.00 0.00 5.00 0.00 0.00 7.00
14:00:53 7 72.73 0.00 21.21 0.00 0.00 0.00 1.01 0.00 0.00 5.05
14:00:53 8 76.24 0.00 12.87 0.00 0.99 0.00 4.95 0.00 0.00 4.95
14:00:53 9 90.00 0.00 4.00 0.00 1.00 0.00 1.00 0.00 0.00 4.00
14:00:53 10 50.50 0.00 33.66 0.99 6.93 0.00 2.97 0.00 0.00 4.95
14:00:53 11 80.00 0.00 8.00 0.00 1.00 0.00 3.00 0.00 0.00 8.00
14:00:53 12 86.14 0.00 4.95 0.00 0.99 0.00 1.98 0.00 0.00 5.94
14:00:53 13 90.10 0.00 3.96 0.99 0.00 0.00 2.97 0.00 0.00 1.98
14:00:53 14 83.67 0.00 7.14 0.00 0.00 0.00 5.10 0.00 0.00 4.08
14:00:53 15 92.16 0.00 3.92 0.00 0.00 0.00 0.00 0.00 0.00 3.92
14:00:53 16 60.61 0.00 37.37 0.00 0.00 0.00 1.01 0.00 0.00 1.01
14:00:53 17 63.73 0.00 32.35 0.00 0.00 0.00 2.94 0.00 0.00 0.98
14:00:53 18 90.00 0.00 2.00 0.00 1.00 0.00 3.00 0.00 0.00 4.00
14:00:53 19 74.75 0.00 20.20 0.00 0.00 0.00 1.01 0.00 0.00 4.04
14:00:53 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:00:54 all 75.14 0.00 15.34 0.15 0.80 0.00 3.61 0.00 0.00 4.96
14:00:54 0 74.00 0.00 19.00 0.00 1.00 0.00 3.00 0.00 0.00 3.00
14:00:54 1 68.32 0.00 21.78 0.00 0.00 0.00 2.97 0.00 0.00 6.93
14:00:54 2 86.00 0.00 8.00 0.00 0.00 0.00 2.00 0.00 0.00 4.00
14:00:54 3 85.15 0.00 4.95 0.00 0.99 0.00 3.96 0.00 0.00 4.95
14:00:54 4 89.11 0.00 5.94 0.00 0.99 0.00 1.98 0.00 0.00 1.98
14:00:54 5 87.76 0.00 4.08 0.00 0.00 0.00 3.06 0.00 0.00 5.10
14:00:54 6 76.47 0.00 15.69 0.00 0.98 0.00 3.92 0.00 0.00 2.94
14:00:54 7 42.00 0.00 53.00 1.00 2.00 0.00 1.00 0.00 0.00 1.00
14:00:54 8 76.77 0.00 10.10 0.00 2.02 0.00 5.05 0.00 0.00 6.06
14:00:54 9 68.69 0.00 12.12 1.01 1.01 0.00 7.07 0.00 0.00 10.10
14:00:54 10 89.80 0.00 6.12 0.00 1.02 0.00 0.00 0.00 0.00 3.06
14:00:54 11 79.59 0.00 8.16 1.02 1.02 0.00 3.06 0.00 0.00 7.14
14:00:54 12 67.33 0.00 18.81 0.00 0.99 0.00 6.93 0.00 0.00 5.94
14:00:54 13 78.35 0.00 12.37 0.00 1.03 0.00 2.06 0.00 0.00 6.19
14:00:54 14 78.57 0.00 7.14 0.00 1.02 0.00 7.14 0.00 0.00 6.12
14:00:54 15 62.63 0.00 29.29 0.00 0.00 0.00 4.04 0.00 0.00 4.04
14:00:54 16 81.00 0.00 12.00 0.00 1.00 0.00 2.00 0.00 0.00 4.00
14:00:54 17 90.10 0.00 4.95 0.00 0.00 0.00 1.98 0.00 0.00 2.97
14:00:54 18 75.73 0.00 10.68 0.00 0.97 0.00 5.83 0.00 0.00 6.80
14:00:54 19 45.45 0.00 42.42 0.00 0.00 0.00 5.05 0.00 0.00 7.07
14:00:54 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:00:55 all 67.80 0.00 15.82 0.05 1.02 0.00 3.97 0.00 0.00 11.34
14:00:55 0 65.66 0.00 18.18 1.01 0.00 0.00 4.04 0.00 0.00 11.11
14:00:55 1 65.62 0.00 8.33 0.00 4.17 0.00 5.21 0.00 0.00 16.67
14:00:55 2 47.47 0.00 41.41 0.00 1.01 0.00 4.04 0.00 0.00 6.06
14:00:55 3 68.32 0.00 15.84 0.00 0.00 0.00 4.95 0.00 0.00 10.89
14:00:55 4 68.00 0.00 10.00 0.00 1.00 0.00 6.00 0.00 0.00 15.00
14:00:55 5 89.00 0.00 6.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00
14:00:55 6 78.79 0.00 8.08 0.00 0.00 0.00 3.03 0.00 0.00 10.10
14:00:55 7 72.45 0.00 7.14 0.00 1.02 0.00 2.04 0.00 0.00 17.35
14:00:55 8 80.61 0.00 3.06 0.00 0.00 0.00 5.10 0.00 0.00 11.22
14:00:55 9 82.18 0.00 7.92 0.00 0.99 0.00 3.96 0.00 0.00 4.95
14:00:55 10 74.23 0.00 7.22 0.00 0.00 0.00 3.09 0.00 0.00 15.46
14:00:55 11 71.88 0.00 9.38 0.00 0.00 0.00 2.08 0.00 0.00 16.67
14:00:55 12 70.41 0.00 14.29 0.00 2.04 0.00 4.08 0.00 0.00 9.18
14:00:55 13 79.80 0.00 6.06 0.00 0.00 0.00 1.01 0.00 0.00 13.13
14:00:55 14 75.00 0.00 9.38 0.00 0.00 0.00 6.25 0.00 0.00 9.38
14:00:55 15 67.01 0.00 10.31 0.00 0.00 0.00 6.19 0.00 0.00 16.49
14:00:55 16 65.31 0.00 20.41 0.00 1.02 0.00 3.06 0.00 0.00 10.20
14:00:55 17 40.00 0.00 36.84 0.00 3.16 0.00 5.26 0.00 0.00 14.74
14:00:55 18 57.58 0.00 35.35 0.00 1.01 0.00 3.03 0.00 0.00 3.03
14:00:55 19 36.00 0.00 41.00 0.00 5.00 0.00 4.00 0.00 0.00 14.00
640 Gb SAR output for some duration
14:22:59 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:00 all 87.17 0.00 6.12 0.05 0.35 0.00 2.71 0.00 0.00 3.61
14:23:00 0 84.85 0.00 6.06 0.00 0.00 0.00 3.03 0.00 0.00 6.06
14:23:00 1 87.13 0.00 3.96 0.00 0.00 0.00 2.97 0.00 0.00 5.94
14:23:00 2 65.69 0.00 27.45 0.00 0.00 0.00 4.90 0.00 0.00 1.96
14:23:00 3 93.00 0.00 3.00 0.00 1.00 0.00 1.00 0.00 0.00 2.00
14:23:00 4 93.94 0.00 2.02 0.00 0.00 0.00 3.03 0.00 0.00 1.01
14:23:00 5 85.15 0.00 6.93 0.00 0.99 0.00 1.98 0.00 0.00 4.95
14:23:00 6 86.00 0.00 4.00 0.00 0.00 0.00 5.00 0.00 0.00 5.00
14:23:00 7 85.00 0.00 7.00 0.00 1.00 0.00 4.00 0.00 0.00 3.00
14:23:00 8 74.00 0.00 15.00 0.00 0.00 0.00 3.00 0.00 0.00 8.00
14:23:00 9 77.08 0.00 8.33 0.00 2.08 0.00 3.12 0.00 0.00 9.38
14:23:00 10 88.78 0.00 6.12 0.00 0.00 0.00 3.06 0.00 0.00 2.04
14:23:00 11 91.00 0.00 4.00 0.00 1.00 0.00 3.00 0.00 0.00 1.00
14:23:00 12 88.89 0.00 6.06 0.00 0.00 0.00 2.02 0.00 0.00 3.03
14:23:00 13 95.00 0.00 2.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00
14:23:00 14 84.00 0.00 5.00 0.00 0.00 0.00 4.00 0.00 0.00 7.00
14:23:00 15 88.24 0.00 3.92 0.98 0.98 0.00 1.96 0.00 0.00 3.92
14:23:00 16 97.98 0.00 1.01 0.00 0.00 0.00 1.01 0.00 0.00 0.00
14:23:00 17 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
14:23:00 18 91.00 0.00 4.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00
14:23:00 19 87.88 0.00 5.05 0.00 0.00 0.00 4.04 0.00 0.00 3.03
14:23:00 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:01 all 84.80 0.00 8.20 0.40 0.55 0.00 2.32 0.00 0.00 3.72
14:23:01 0 85.71 0.00 6.12 2.04 0.00 0.00 4.08 0.00 0.00 2.04
14:23:01 1 87.76 0.00 5.10 0.00 1.02 0.00 2.04 0.00 0.00 4.08
14:23:01 2 71.72 0.00 25.25 0.00 1.01 0.00 1.01 0.00 0.00 1.01
14:23:01 3 88.00 0.00 6.00 2.00 1.00 0.00 2.00 0.00 0.00 1.00
14:23:01 4 87.38 0.00 6.80 0.00 0.00 0.00 1.94 0.00 0.00 3.88
14:23:01 5 87.00 0.00 4.00 0.00 1.00 0.00 2.00 0.00 0.00 6.00
14:23:01 6 82.83 0.00 8.08 1.01 1.01 0.00 2.02 0.00 0.00 5.05
14:23:01 7 71.13 0.00 9.28 2.06 2.06 0.00 3.09 0.00 0.00 12.37
14:23:01 8 80.81 0.00 9.09 1.01 1.01 0.00 2.02 0.00 0.00 6.06
14:23:01 9 81.00 0.00 9.00 0.00 1.00 0.00 3.00 0.00 0.00 6.00
14:23:01 10 87.76 0.00 8.16 0.00 0.00 0.00 2.04 0.00 0.00 2.04
14:23:01 11 90.91 0.00 5.05 0.00 0.00 0.00 1.01 0.00 0.00 3.03
14:23:01 12 86.14 0.00 8.91 0.00 0.00 0.00 0.99 0.00 0.00 3.96
14:23:01 13 87.88 0.00 8.08 0.00 0.00 0.00 2.02 0.00 0.00 2.02
14:23:01 14 77.78 0.00 9.09 0.00 0.00 0.00 7.07 0.00 0.00 6.06
14:23:01 15 80.81 0.00 7.07 0.00 1.01 0.00 6.06 0.00 0.00 5.05
14:23:01 16 90.10 0.00 6.93 0.00 0.00 0.00 0.99 0.00 0.00 1.98
14:23:01 17 89.80 0.00 9.18 0.00 0.00 0.00 1.02 0.00 0.00 0.00
14:23:01 18 91.92 0.00 6.06 0.00 0.00 0.00 1.01 0.00 0.00 1.01
14:23:01 19 89.11 0.00 6.93 0.00 0.99 0.00 0.99 0.00 0.00 1.98
14:23:01 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:02 all 83.22 0.00 8.29 0.00 0.66 0.00 2.68 0.00 0.00 5.16
14:23:02 0 95.00 0.00 4.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00
14:23:02 1 78.00 0.00 8.00 0.00 2.00 0.00 4.00 0.00 0.00 8.00
14:23:02 2 82.83 0.00 14.14 0.00 1.01 0.00 1.01 0.00 0.00 1.01
14:23:02 3 65.00 0.00 17.00 0.00 4.00 0.00 2.00 0.00 0.00 12.00
14:23:02 4 91.92 0.00 6.06 0.00 0.00 0.00 2.02 0.00 0.00 0.00
14:23:02 5 92.93 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 0.00
14:23:02 6 76.34 0.00 5.38 0.00 0.00 0.00 6.45 0.00 0.00 11.83
14:23:02 7 66.33 0.00 8.16 0.00 3.06 0.00 9.18 0.00 0.00 13.27
14:23:02 8 72.16 0.00 12.37 0.00 1.03 0.00 5.15 0.00 0.00 9.28
14:23:02 9 82.29 0.00 9.38 0.00 0.00 0.00 2.08 0.00 0.00 6.25
14:23:02 10 84.00 0.00 11.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00
14:23:02 11 86.87 0.00 4.04 0.00 0.00 0.00 2.02 0.00 0.00 7.07
14:23:02 12 82.00 0.00 11.00 0.00 1.00 0.00 1.00 0.00 0.00 5.00
14:23:02 13 88.78 0.00 4.08 0.00 0.00 0.00 1.02 0.00 0.00 6.12
14:23:02 14 79.41 0.00 18.63 0.00 0.98 0.00 0.98 0.00 0.00 0.00
14:23:02 15 85.86 0.00 5.05 0.00 0.00 0.00 3.03 0.00 0.00 6.06
14:23:02 16 84.85 0.00 11.11 0.00 0.00 0.00 2.02 0.00 0.00 2.02
14:23:02 17 89.00 0.00 4.00 0.00 0.00 0.00 2.00 0.00 0.00 5.00
14:23:02 18 96.04 0.00 1.98 0.00 0.00 0.00 1.98 0.00 0.00 0.00
14:23:02 19 83.84 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 9.09
14:23:02 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:03 all 89.22 0.20 4.53 0.05 0.35 0.00 2.92 0.00 0.00 2.72
14:23:03 0 87.13 0.99 3.96 0.00 0.99 0.00 3.96 0.00 0.00 2.97
14:23:03 1 92.86 0.00 5.10 0.00 0.00 0.00 1.02 0.00 0.00 1.02
14:23:03 2 96.00 1.00 2.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00
14:23:03 3 90.91 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 2.02
14:23:03 4 98.99 0.00 1.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
14:23:03 5 93.07 0.00 2.97 0.00 0.00 0.00 1.98 0.00 0.00 1.98
14:23:03 6 84.85 0.00 7.07 0.00 1.01 0.00 2.02 0.00 0.00 5.05
14:23:03 7 82.35 0.98 5.88 0.00 0.00 0.00 4.90 0.00 0.00 5.88
14:23:03 8 77.55 1.02 11.22 0.00 1.02 0.00 4.08 0.00 0.00 5.10
14:23:03 9 74.23 0.00 5.15 0.00 1.03 0.00 10.31 0.00 0.00 9.28
14:23:03 10 93.94 0.00 3.03 0.00 1.01 0.00 1.01 0.00 0.00 1.01
14:23:03 11 90.00 0.00 5.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00
14:23:03 12 92.86 0.00 4.08 0.00 0.00 0.00 3.06 0.00 0.00 0.00
14:23:03 13 84.00 0.00 6.00 1.00 0.00 0.00 6.00 0.00 0.00 3.00
14:23:03 14 88.78 0.00 3.06 0.00 0.00 0.00 5.10 0.00 0.00 3.06
14:23:03 15 92.08 0.00 4.95 0.00 0.00 0.00 1.98 0.00 0.00 0.99
14:23:03 16 88.89 0.00 5.05 0.00 1.01 0.00 2.02 0.00 0.00 3.03
14:23:03 17 97.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
14:23:03 18 90.00 0.00 4.00 0.00 0.00 0.00 4.00 0.00 0.00 2.00
14:23:03 19 88.66 0.00 3.09 0.00 1.03 0.00 2.06 0.00 0.00 5.15
14:23:03 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:04 all 88.13 0.25 6.39 0.00 0.25 0.00 3.12 0.00 0.00 1.86
14:23:04 0 88.00 0.00 7.00 0.00 0.00 0.00 2.00 0.00 0.00 3.00
14:23:04 1 94.00 0.00 3.00 0.00 0.00 0.00 2.00 0.00 0.00 1.00
14:23:04 2 88.89 0.00 6.06 0.00 0.00 0.00 4.04 0.00 0.00 1.01
14:23:04 3 79.59 1.02 7.14 0.00 0.00 0.00 7.14 0.00 0.00 5.10
14:23:04 4 81.00 0.00 15.00 0.00 0.00 0.00 4.00 0.00 0.00 0.00
14:23:04 5 90.00 0.00 7.00 0.00 1.00 0.00 1.00 0.00 0.00 1.00
14:23:04 6 91.00 1.00 5.00 0.00 0.00 0.00 2.00 0.00 0.00 1.00
14:23:04 7 80.41 0.00 11.34 0.00 0.00 0.00 5.15 0.00 0.00 3.09
14:23:04 8 84.85 0.00 7.07 0.00 0.00 0.00 6.06 0.00 0.00 2.02
14:23:04 9 87.00 1.00 7.00 0.00 0.00 0.00 3.00 0.00 0.00 2.00
14:23:04 10 94.95 0.00 3.03 0.00 0.00 0.00 2.02 0.00 0.00 0.00
14:23:04 11 90.91 1.01 3.03 0.00 0.00 0.00 4.04 0.00 0.00 1.01
14:23:04 12 91.92 0.00 5.05 0.00 0.00 0.00 2.02 0.00 0.00 1.01
14:23:04 13 90.10 0.00 5.94 0.00 0.99 0.00 2.97 0.00 0.00 0.00
14:23:04 14 91.92 0.00 5.05 0.00 0.00 0.00 1.01 0.00 0.00 2.02
14:23:04 15 87.76 0.00 6.12 0.00 0.00 0.00 2.04 0.00 0.00 4.08
14:23:04 16 86.14 0.00 7.92 0.00 0.00 0.00 2.97 0.00 0.00 2.97
14:23:04 17 97.03 0.00 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00
14:23:04 18 84.69 0.00 6.12 0.00 0.00 0.00 5.10 0.00 0.00 4.08
14:23:04 19 82.18 0.99 7.92 0.00 1.98 0.00 3.96 0.00 0.00 2.97
14:23:04 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle
14:23:05 all 83.08 0.15 8.43 0.05 0.51 0.00 3.18 0.00 0.00 4.60
14:23:05 0 86.87 0.00 4.04 0.00 0.00 0.00 3.03 0.00 0.00 6.06
14:23:05 1 92.78 0.00 2.06 0.00 0.00 0.00 1.03 0.00 0.00 4.12
14:23:05 2 78.79 0.00 7.07 1.01 1.01 0.00 3.03 0.00 0.00 9.09
14:23:05 3 64.58 0.00 13.54 0.00 2.08 0.00 7.29 0.00 0.00 12.50
14:23:05 4 67.00 0.00 25.00 0.00 1.00 0.00 3.00 0.00 0.00 4.00
14:23:05 5 82.00 1.00 7.00 0.00 4.00 0.00 1.00 0.00 0.00 5.00
14:23:05 6 87.63 0.00 6.19 0.00 0.00 0.00 3.09 0.00 0.00 3.09
14:23:05 7 88.00 0.00 5.00 0.00 0.00 0.00 4.00 0.00 0.00 3.00
14:23:05 8 82.83 0.00 5.05 0.00 0.00 0.00 6.06 0.00 0.00 6.06
14:23:05 9 78.79 0.00 8.08 0.00 1.01 0.00 5.05 0.00 0.00 7.07
14:23:05 10 82.83 1.01 6.06 0.00 0.00 0.00 4.04 0.00 0.00 6.06
14:23:05 11 80.81 0.00 8.08 0.00 1.01 0.00 4.04 0.00 0.00 6.06
14:23:05 12 90.10 0.00 2.97 0.00 0.00 0.00 3.96 0.00 0.00 2.97
14:23:05 13 90.91 1.01 3.03 0.00 0.00 0.00 3.03 0.00 0.00 2.02
14:23:05 14 87.13 0.00 7.92 0.00 0.00 0.00 1.98 0.00 0.00 2.97
14:23:05 15 78.79 0.00 18.18 0.00 0.00 0.00 1.01 0.00 0.00 2.02
14:23:05 16 84.85 0.00 9.09 0.00 0.00 0.00 3.03 0.00 0.00 3.03
14:23:05 17 72.00 0.00 22.00 0.00 0.00 0.00 3.00 0.00 0.00 3.00
14:23:05 18 91.84 0.00 5.10 0.00 0.00 0.00 2.04 0.00 0.00 1.02
14:23:05 19 92.93 0.00 3.03 0.00 0.00 0.00 1.01 0.00 0.00 3.03
RainTown:
btw 1,315,608 = ( 2 * 2 * 2 * 3 * 7 * 41 * 191) was a very specific number, is there some meaning in it, or was just the count the time you checked it.
Nope, not any special number, just the count.
So let’s say there are 10 different updates for a single document in a 30 second window, then all of these updates are combined together in our application and upserted as a single document, meaning that all updates for a single document are combined into a single logical request for that document.
1 Like
RainTown
January 5, 2026, 10:02am
184
OK to save others wetransfer, here's the sdb disk data, taken at 15 second intervals in both cases.
## iostat_320GB_2026-01-05_1409.log
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
vdb 0.03 3.56 0.00 0.27 0.67 105.53 5.73 393.90 52.54 90.17 1.21 68.75 0.00 406.25 0.00 0.00 1.82 230509.24 1.45 0.65 0.01 0.34
vdb 0.00 0.00 0.00 0.00 0.00 0.00 10.73 198.93 38.33 78.12 0.77 18.53 0.00 0.00 0.00 0.00 0.00 0.00 3.80 0.74 0.01 0.85
vdb 0.00 0.00 0.00 0.00 0.00 0.00 360.93 115226.93 14029.33 97.49 1.38 319.25 0.00 0.00 0.00 0.00 0.00 0.00 37.07 0.31 0.51 12.29
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1219.40 371660.80 63065.00 98.10 0.97 304.79 0.00 0.00 0.00 0.00 0.00 0.00 151.07 0.14 1.21 36.91
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1155.60 414278.40 74973.07 98.48 1.09 358.50 0.00 0.00 0.00 0.00 0.00 0.00 133.60 0.23 1.29 39.65
vdb 0.00 0.00 0.00 0.00 0.00 0.00 979.27 351356.53 62913.47 98.47 0.81 358.80 0.00 0.00 0.00 0.00 0.00 0.00 110.60 0.19 0.82 35.81
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1048.27 347034.40 62437.87 98.35 1.06 331.06 0.00 0.00 0.00 0.00 0.00 0.00 138.53 0.16 1.13 37.47
vdb 0.00 0.00 0.00 0.00 0.00 0.00 797.93 306260.53 55359.47 98.58 0.88 383.82 0.00 0.00 0.00 0.00 0.00 0.00 93.33 0.13 0.71 31.84
vdb 0.00 0.00 0.00 0.00 0.00 0.00 782.61 298732.84 54074.22 98.57 1.14 381.71 0.00 0.00 0.00 0.00 0.00 0.00 77.95 0.21 0.90 31.87
vdb 0.00 0.00 0.00 0.00 0.00 0.00 847.63 313932.76 60229.89 98.61 0.85 370.36 0.00 0.00 0.00 0.00 0.00 0.00 111.54 0.12 0.73 35.49
vdb 0.00 0.00 0.00 0.00 0.00 0.00 781.15 240481.55 41002.20 98.13 0.89 307.86 0.00 0.00 0.00 0.00 0.00 0.00 81.41 0.16 0.70 28.49
vdb 0.00 0.00 0.00 0.00 0.00 0.00 823.00 263872.80 48211.13 98.32 0.88 320.62 0.00 0.00 0.00 0.00 0.00 0.00 110.67 0.27 0.75 35.04
vdb 0.00 0.00 0.00 0.00 0.00 0.00 702.87 248756.80 45463.00 98.48 1.02 353.92 0.00 0.00 0.00 0.00 0.00 0.00 74.87 0.17 0.73 31.36
vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.80 254354.40 46731.47 98.63 1.17 393.25 0.00 0.00 0.00 0.00 0.00 0.00 72.47 0.16 0.77 31.31
vdb 0.00 0.00 0.00 0.00 0.00 0.00 640.13 177248.00 29948.80 97.91 0.76 276.89 0.00 0.00 0.00 0.00 0.00 0.00 95.00 0.11 0.49 24.48
vdb 0.27 8.80 1.93 87.88 0.50 33.00 796.33 364409.07 77641.20 98.98 1.28 457.61 0.00 0.00 0.00 0.00 0.00 0.00 96.60 0.28 1.04 41.07
vdb 0.00 0.00 0.00 0.00 0.00 0.00 729.27 328409.33 67852.13 98.94 1.13 450.33 0.00 0.00 0.00 0.00 0.00 0.00 72.27 0.13 0.84 37.92
vdb 0.00 0.00 0.00 0.00 0.00 0.00 590.40 242202.93 46730.13 98.75 0.84 410.24 0.00 0.00 0.00 0.00 0.00 0.00 71.73 0.12 0.51 28.77
vdb 0.00 0.00 0.00 0.00 0.00 0.00 604.33 235453.07 44514.33 98.66 0.94 389.61 0.00 0.00 0.00 0.00 0.00 0.00 73.27 0.21 0.58 27.89
vdb 0.00 0.00 0.00 0.00 0.00 0.00 573.67 187122.40 33198.87 98.30 0.81 326.19 0.00 0.00 0.00 0.00 0.00 0.00 82.60 0.14 0.48 24.08
vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.20 220578.67 41140.60 98.45 0.90 341.35 0.00 0.00 0.00 0.00 0.00 0.00 105.13 0.19 0.60 29.65
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1038.93 395617.60 75117.13 98.64 1.02 380.79 0.00 0.00 0.00 0.00 0.00 0.00 123.80 0.13 1.08 38.16
vdb 0.60 2.40 0.00 0.00 0.33 4.00 1073.80 376683.20 70257.67 98.49 1.30 350.79 0.00 0.00 0.00 0.00 0.00 0.00 144.47 0.16 1.42 38.21
vdb 0.13 0.53 0.00 0.00 0.50 4.00 559.93 264561.60 56500.87 99.02 1.25 472.49 0.00 0.00 0.00 0.00 0.00 0.00 88.67 0.14 0.71 21.55
vdb 382.93 38029.07 0.00 0.00 0.28 99.31 515.00 215640.27 37577.93 98.65 1.33 418.72 0.00 0.00 0.00 0.00 0.00 0.00 77.60 0.22 0.81 24.83
vdb 30.87 2789.07 0.00 0.00 0.41 90.36 1242.13 621722.93 132224.80 99.07 1.29 500.53 0.00 0.00 0.00 0.00 0.00 0.00 120.53 0.24 1.64 56.05
vdb 35.33 3567.73 0.00 0.00 0.28 100.97 1220.80 464354.13 93318.13 98.71 0.82 380.37 0.00 0.00 0.00 0.00 0.00 0.00 176.33 0.12 1.04 45.17
vdb 7.27 382.93 0.00 0.00 0.46 52.70 1010.80 330330.67 54465.47 98.18 0.78 326.80 0.00 0.00 0.00 0.00 0.00 0.00 133.27 0.13 0.81 34.32
vdb 8.00 404.27 0.00 0.00 0.44 50.53 1129.47 432742.13 85700.67 98.70 1.36 383.14 0.00 0.00 0.00 0.00 0.00 0.00 153.73 0.18 1.57 42.61
vdb 78.40 8293.33 0.00 0.00 0.62 105.78 1141.40 430768.27 83246.93 98.65 1.36 377.40 0.00 0.00 0.00 0.00 0.00 0.00 172.60 0.30 1.65 45.33
vdb 363.13 43784.27 0.40 0.11 0.70 120.57 979.27 338804.80 61585.33 98.43 1.68 345.98 0.00 0.00 0.00 0.00 0.00 0.00 129.13 0.30 1.94 44.99
vdb 20.00 2042.93 0.00 0.00 0.31 102.15 1011.60 410913.60 83870.13 98.81 1.05 406.20 0.00 0.00 0.00 0.00 0.00 0.00 146.27 0.24 1.11 42.91
vdb 39.27 4570.40 0.00 0.00 0.31 116.39 880.80 290303.20 50768.80 98.29 0.83 329.59 0.00 0.00 0.00 0.00 0.00 0.00 124.67 0.14 0.76 31.65
vdb 27.07 2176.00 0.00 0.00 0.79 80.39 881.53 385717.87 76757.87 98.86 1.03 437.55 0.00 0.00 0.00 0.00 0.00 0.00 103.53 0.20 0.95 39.25
vdb 0.53 2.13 0.00 0.00 0.12 4.00 960.80 346204.53 65026.07 98.54 1.11 360.33 0.00 0.00 0.00 0.00 0.00 0.00 98.47 0.17 1.08 36.75
vdb 4.00 16.00 0.00 0.00 0.25 4.00 848.93 321556.27 62153.60 98.65 1.78 378.78 0.00 0.00 0.00 0.00 0.00 0.00 100.53 0.54 1.57 39.55
vdb 528.40 38596.53 0.00 0.00 1.00 73.04 891.87 354328.00 70124.87 98.74 2.31 397.29 0.00 0.00 0.00 0.00 0.00 0.00 101.93 0.34 2.62 51.04
vdb 352.53 18881.87 0.00 0.00 0.67 53.56 1011.67 551422.67 119157.00 99.16 1.52 545.06 0.00 0.00 0.00 0.00 0.00 0.00 85.67 0.34 1.81 58.85
vdb 18.47 1226.40 0.00 0.00 0.23 66.41 799.13 306472.27 57286.53 98.62 0.82 383.51 0.00 0.00 0.00 0.00 0.00 0.00 98.27 0.12 0.67 33.55
vdb 2.47 10.40 0.13 5.13 0.30 4.22 898.60 340812.00 67165.73 98.68 0.86 379.27 0.00 0.00 0.00 0.00 0.00 0.00 98.20 0.10 0.78 35.33
vdb 1.07 12.27 2.00 65.22 0.31 11.50 785.93 273817.33 49750.87 98.44 0.90 348.40 0.00 0.00 0.00 0.00 0.00 0.00 97.60 0.25 0.73 32.08
vdb 3.07 12.27 0.00 0.00 0.37 4.00 814.33 341875.20 67947.60 98.82 1.14 419.82 0.00 0.00 0.00 0.00 0.00 0.00 98.20 0.24 0.95 39.57
vdb 1.33 36.53 0.00 0.00 0.30 27.40 751.80 241671.47 41905.73 98.24 0.77 321.46 0.00 0.00 0.00 0.00 0.00 0.00 97.20 0.15 0.59 29.36
vdb 2.13 10.40 0.47 17.95 0.69 4.88 869.20 354629.60 71582.40 98.80 2.02 408.00 0.00 0.00 0.00 0.00 0.00 0.00 97.87 0.24 1.78 38.85
vdb 613.40 52583.20 0.00 0.00 0.77 85.72 719.60 277639.47 52604.00 98.65 2.22 385.82 0.00 0.00 0.00 0.00 0.00 0.00 96.60 0.80 2.15 53.89
vdb 7.60 648.00 0.00 0.00 1.04 85.26 758.93 297681.07 58154.47 98.71 1.59 392.24 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.27 1.24 36.56
vdb 29.00 3484.80 0.00 0.00 0.63 120.17 680.33 274476.53 50714.73 98.68 1.00 403.44 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.25 0.72 32.05
vdb 2.13 156.53 0.00 0.00 0.25 73.38 667.53 225117.87 41565.67 98.42 1.01 337.24 0.00 0.00 0.00 0.00 0.00 0.00 78.00 0.14 0.68 28.59
vdb 0.07 0.27 0.00 0.00 1.00 4.00 905.47 383426.40 81464.80 98.90 1.21 423.46 0.00 0.00 0.00 0.00 0.00 0.00 113.07 0.24 1.12 42.19
vdb 0.20 0.80 0.00 0.00 0.33 4.00 607.33 186962.93 30445.13 98.04 0.85 307.84 0.00 0.00 0.00 0.00 0.00 0.00 71.87 0.11 0.53 23.39
vdb 17.40 1910.67 0.00 0.00 0.25 109.81 604.53 247407.47 46532.53 98.72 0.88 409.25 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.19 0.55 28.77
vdb 1077.20 107264.00 0.00 0.00 0.75 99.58 843.60 296176.27 61161.33 98.64 3.12 351.09 0.00 0.00 0.00 0.00 0.00 0.00 117.33 0.92 3.55 76.51
vdb 401.07 25366.67 0.00 0.00 0.98 63.25 777.40 411032.27 87238.33 99.12 2.67 528.73 0.00 0.00 0.00 0.00 0.00 0.00 66.07 0.62 2.51 59.95
vdb 43.80 4234.40 2.27 4.92 0.72 96.68 853.53 492707.73 110831.60 99.24 1.72 577.26 0.00 0.00 0.00 0.00 0.00 0.00 64.27 0.37 1.53 53.49
vdb 67.27 7241.87 0.93 1.37 0.68 107.66 692.67 279851.73 55327.80 98.76 1.45 404.02 0.00 0.00 0.00 0.00 0.00 0.00 71.13 0.21 1.07 37.97
vdb 225.53 28188.53 0.00 0.00 0.87 124.99 573.13 228129.60 43783.13 98.71 1.86 398.04 0.00 0.00 0.00 0.00 0.00 0.00 69.67 0.65 1.30 39.47
vdb 28.07 3540.27 0.00 0.00 0.42 126.14 620.73 197980.53 35245.00 98.27 0.82 318.95 0.00 0.00 0.00 0.00 0.00 0.00 79.20 0.15 0.53 26.29
vdb 116.93 14670.93 0.00 0.00 0.33 125.46 604.07 244633.07 47383.80 98.74 0.95 404.98 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.11 0.62 33.65
vdb 25.73 2795.47 0.00 0.00 0.47 108.63 674.87 215106.13 42645.33 98.44 1.23 318.74 0.00 0.00 0.00 0.00 0.00 0.00 95.20 0.14 0.85 29.17
vdb 1.47 70.40 0.00 0.00 0.32 48.00 862.13 352412.53 71098.80 98.80 1.32 408.77 0.00 0.00 0.00 0.00 0.00 0.00 109.33 0.17 1.15 38.03
vdb 309.40 38766.67 0.00 0.00 0.90 125.30 833.67 287650.67 51645.13 98.41 1.99 345.04 0.00 0.00 0.00 0.00 0.00 0.00 110.80 0.51 2.00 41.25
vdb 118.20 14795.47 0.00 0.00 1.70 125.17 602.20 248080.53 50837.47 98.83 2.74 411.96 0.00 0.00 0.00 0.00 0.00 0.00 70.93 0.63 1.89 36.27
vdb 191.27 24146.40 0.00 0.00 1.16 126.24 562.67 186333.07 33426.87 98.34 2.23 331.16 0.00 0.00 0.00 0.00 0.00 0.00 72.87 0.51 1.52 33.33
vdb 2.07 88.80 0.00 0.00 0.29 42.97 562.47 191215.73 33472.73 98.35 0.96 339.96 0.00 0.00 0.00 0.00 0.00 0.00 71.60 0.21 0.56 25.17
vdb 0.20 0.80 0.00 0.00 0.33 4.00 644.93 277751.47 57686.60 98.89 1.51 430.67 0.00 0.00 0.00 0.00 0.00 0.00 69.20 0.39 1.00 34.83
vdb 31.47 3969.07 0.00 0.00 0.45 126.14 557.60 210196.00 41199.67 98.66 1.04 376.97 0.00 0.00 0.00 0.00 0.00 0.00 73.20 0.15 0.61 28.40
vdb 0.00 0.00 0.00 0.00 0.00 0.00 530.93 181512.53 33837.40 98.46 1.50 341.87 0.00 0.00 0.00 0.00 0.00 0.00 69.00 0.31 0.82 26.05
vdb 36.67 4111.20 0.00 0.00 0.52 112.12 560.80 208841.07 41924.87 98.68 1.14 372.40 0.00 0.00 0.00 0.00 0.00 0.00 78.93 0.16 0.67 30.35
vdb 1.20 22.93 0.00 0.00 0.28 19.11 709.13 249012.27 49458.73 98.59 0.97 351.15 0.00 0.00 0.00 0.00 0.00 0.00 97.53 0.19 0.71 32.19
## iostat_640GB_2026-01-05_1432.log
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
vdb 0.01 1.61 0.00 0.02 0.52 118.16 6.43 327.39 33.05 83.72 0.94 50.95 0.00 830.31 0.00 0.00 1.13 175039.68 1.49 0.49 0.01 0.29
vdb 0.00 0.00 0.00 0.00 0.00 0.00 12.67 291.73 60.67 82.73 0.57 23.03 0.00 0.00 0.00 0.00 0.00 0.00 4.47 0.48 0.01 0.85
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1230.80 371603.47 56267.60 97.86 2.11 301.92 0.00 0.00 0.00 0.00 0.00 0.00 134.53 0.47 2.66 34.93
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1417.47 483401.33 86083.73 98.38 0.96 341.03 0.00 0.00 0.00 0.00 0.00 0.00 170.67 0.13 1.38 40.67
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1159.73 422145.60 75298.27 98.48 1.58 364.00 0.00 0.00 0.00 0.00 0.00 0.00 149.27 0.15 1.85 37.89
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1134.07 407770.93 74040.67 98.49 1.73 359.57 0.00 0.00 0.00 0.00 0.00 0.00 129.40 0.27 2.00 37.65
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1068.60 398468.00 72778.53 98.55 1.71 372.89 0.00 0.00 0.00 0.00 0.00 0.00 121.07 0.45 1.88 39.39
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1042.33 390344.00 73901.60 98.61 1.20 374.49 0.00 0.00 0.00 0.00 0.00 0.00 122.20 0.17 1.27 38.37
vdb 0.00 0.00 0.00 0.00 0.00 0.00 788.07 289938.67 50836.80 98.47 1.27 367.91 0.00 0.00 0.00 0.00 0.00 0.00 101.20 0.30 1.03 32.21
vdb 0.00 0.00 0.00 0.00 0.00 0.00 995.47 353162.67 64973.67 98.49 1.28 354.77 0.00 0.00 0.00 0.00 0.00 0.00 125.93 0.13 1.29 37.20
vdb 0.00 0.00 0.00 0.00 0.00 0.00 916.07 317304.53 57496.20 98.43 1.13 346.38 0.00 0.00 0.00 0.00 0.00 0.00 125.47 0.11 1.05 33.87
vdb 0.00 0.00 0.00 0.00 0.00 0.00 926.47 310887.20 56513.67 98.39 2.38 335.56 0.00 0.00 0.00 0.00 0.00 0.00 129.87 0.22 2.23 35.31
vdb 0.00 0.00 0.00 0.00 0.00 0.00 866.67 360320.00 72203.13 98.81 1.35 415.75 0.00 0.00 0.00 0.00 0.00 0.00 104.67 0.13 1.18 39.07
vdb 0.00 0.00 0.00 0.00 0.00 0.00 807.27 326996.53 61410.60 98.70 0.97 405.07 0.00 0.00 0.00 0.00 0.00 0.00 91.20 0.14 0.80 34.72
vdb 5.60 516.80 0.00 0.00 0.85 92.29 946.13 427712.27 89597.53 98.96 0.99 452.06 0.00 0.00 0.00 0.00 0.00 0.00 127.40 0.19 0.96 42.69
vdb 65.40 7930.93 1.27 1.90 1.17 121.27 713.33 258136.27 47356.07 98.52 1.25 361.87 0.00 0.00 0.00 0.00 0.00 0.00 83.00 0.34 0.99 35.52
vdb 2.87 190.67 0.00 0.00 0.28 66.51 759.80 286863.20 56985.40 98.68 1.28 377.55 0.00 0.00 0.00 0.00 0.00 0.00 107.00 0.38 1.01 36.27
vdb 11.67 1036.27 0.00 0.00 0.31 88.82 611.07 205282.93 34224.73 98.25 2.00 335.94 0.00 0.00 0.00 0.00 0.00 0.00 80.47 0.20 1.24 26.61
vdb 3.00 56.80 0.27 8.16 0.56 18.93 719.87 259650.67 50056.80 98.58 1.07 360.69 0.00 0.00 0.00 0.00 0.00 0.00 102.73 0.32 0.80 32.11
vdb 48.07 5618.67 0.00 0.00 1.67 116.89 699.87 259702.67 47204.40 98.54 2.05 371.07 0.00 0.00 0.00 0.00 0.00 0.00 80.80 0.44 1.55 33.63
vdb 308.20 38897.87 0.00 0.00 0.98 126.21 1171.67 424724.00 83513.13 98.62 2.20 362.50 0.00 0.00 0.00 0.00 0.00 0.00 163.73 0.66 2.98 55.15
vdb 0.07 0.27 0.00 0.00 0.00 4.00 1222.20 472392.53 90645.80 98.67 1.57 386.51 0.00 0.00 0.00 0.00 0.00 0.00 156.53 0.26 1.96 42.11
vdb 891.47 109743.73 0.00 0.00 0.58 123.10 892.00 531333.87 118240.27 99.25 3.24 595.67 0.00 0.00 0.00 0.00 0.00 0.00 76.00 0.84 3.47 55.39
vdb 45.00 4273.60 0.00 0.00 0.34 94.97 240.27 140334.13 30744.60 99.22 1.88 584.08 0.00 0.00 0.00 0.00 0.00 0.00 47.33 0.07 0.47 12.00
vdb 45.67 5248.27 0.00 0.00 0.27 114.93 740.80 277701.33 46834.40 98.44 1.58 374.87 0.00 0.00 0.00 0.00 0.00 0.00 95.60 0.20 1.20 24.96
vdb 307.33 38948.27 0.00 0.00 0.76 126.73 1079.40 422967.20 75879.60 98.60 1.63 391.85 0.00 0.00 0.00 0.00 0.00 0.00 136.13 0.32 2.03 43.73
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1291.67 500264.80 92085.20 98.62 1.06 387.30 0.00 0.00 0.00 0.00 0.00 0.00 144.53 0.19 1.40 40.45
vdb 39.20 4896.80 0.00 0.00 0.35 124.92 1245.60 462864.80 86248.93 98.58 0.83 371.60 0.00 0.00 0.00 0.00 0.00 0.00 163.20 0.12 1.06 40.35
vdb 36.87 4097.60 0.00 0.00 0.40 111.15 1338.27 541657.33 107145.40 98.77 1.57 404.75 0.00 0.00 0.00 0.00 0.00 0.00 162.67 0.12 2.13 47.89
vdb 349.27 44054.93 0.07 0.02 0.48 126.14 1268.93 509679.47 98026.80 98.72 1.80 401.66 0.00 0.00 0.00 0.00 0.00 0.00 156.93 0.42 2.52 54.56
vdb 155.33 19547.73 0.00 0.00 1.15 125.84 1435.13 644687.47 133423.40 98.94 2.42 449.22 0.00 0.00 0.00 0.00 0.00 0.00 144.33 0.48 3.72 58.27
vdb 12.13 1458.67 0.00 0.00 0.45 120.22 1254.47 545881.07 106459.27 98.84 1.32 435.15 0.00 0.00 0.00 0.00 0.00 0.00 128.73 0.16 1.68 44.35
vdb 0.07 0.27 0.00 0.00 0.00 4.00 1131.87 446450.67 86985.73 98.72 1.54 394.44 0.00 0.00 0.00 0.00 0.00 0.00 132.07 0.48 1.81 43.47
vdb 175.20 16444.53 0.00 0.00 0.89 93.86 1110.13 413014.67 80836.60 98.65 2.06 372.04 0.00 0.00 0.00 0.00 0.00 0.00 145.07 0.34 2.50 45.25
vdb 90.07 10244.80 0.00 0.00 0.65 113.75 1108.47 424598.13 84175.73 98.70 3.24 383.05 0.00 0.00 0.00 0.00 0.00 0.00 131.67 0.30 3.69 45.65
vdb 39.40 4760.27 0.00 0.00 0.37 120.82 931.53 343535.73 61653.60 98.51 1.12 368.79 0.00 0.00 0.00 0.00 0.00 0.00 108.53 0.18 1.07 35.31
vdb 153.87 18186.40 1.80 1.16 0.87 118.20 1066.73 417408.00 85193.27 98.76 2.35 391.30 0.00 0.00 0.00 0.00 0.00 0.00 123.33 0.27 2.68 48.35
vdb 191.20 23293.07 0.00 0.00 0.81 121.83 986.33 368774.93 69115.67 98.59 2.38 373.88 0.00 0.00 0.00 0.00 0.00 0.00 118.53 0.26 2.53 41.84
vdb 3.47 17.87 0.00 0.00 1.12 5.15 971.40 399100.80 78724.93 98.78 1.52 410.85 0.00 0.00 0.00 0.00 0.00 0.00 102.47 0.19 1.50 40.43
vdb 0.00 0.00 0.00 0.00 0.00 0.00 823.27 284509.60 49577.00 98.37 1.11 345.59 0.00 0.00 0.00 0.00 0.00 0.00 105.67 0.21 0.93 31.65
vdb 0.27 1.07 0.00 0.00 0.25 4.00 1038.07 384037.60 77545.20 98.68 1.45 369.95 0.00 0.00 0.00 0.00 0.00 0.00 121.80 0.26 1.54 41.41
vdb 1.07 4.80 0.00 0.00 0.62 4.50 842.33 309266.67 56807.33 98.54 1.31 367.15 0.00 0.00 0.00 0.00 0.00 0.00 106.27 0.18 1.12 33.76
vdb 674.80 77366.93 0.13 0.02 0.97 114.65 966.33 375238.40 73028.60 98.69 2.82 388.31 0.00 0.00 0.00 0.00 0.00 0.00 95.93 0.74 3.46 64.13
vdb 0.00 0.00 0.00 0.00 0.00 0.00 910.80 432904.53 89790.40 99.00 1.75 475.30 0.00 0.00 0.00 0.00 0.00 0.00 93.53 0.21 1.61 45.52
vdb 0.13 0.53 0.00 0.00 0.00 4.00 934.47 443489.33 92719.67 99.00 1.14 474.59 0.00 0.00 0.00 0.00 0.00 0.00 93.87 0.19 1.08 44.53
vdb 66.20 4892.27 0.00 0.00 0.63 73.90 838.07 290399.73 51988.00 98.41 1.52 346.51 0.00 0.00 0.00 0.00 0.00 0.00 103.60 0.18 1.33 33.89
vdb 0.20 0.80 0.00 0.00 0.33 4.00 849.20 357271.73 69929.93 98.80 1.53 420.72 0.00 0.00 0.00 0.00 0.00 0.00 96.40 0.27 1.32 36.83
vdb 0.47 1.87 0.00 0.00 1.57 4.00 807.67 280924.27 50872.53 98.44 1.24 347.82 0.00 0.00 0.00 0.00 0.00 0.00 100.27 0.18 1.02 30.85
vdb 0.87 19.73 0.00 0.00 0.69 22.77 786.93 359258.13 72853.20 98.93 1.83 456.53 0.00 0.00 0.00 0.00 0.00 0.00 71.20 0.36 1.47 41.79
vdb 0.27 1.07 0.00 0.00 0.25 4.00 737.07 237122.13 40993.87 98.23 1.59 321.71 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.29 1.20 29.12
vdb 95.47 11246.13 0.00 0.00 0.80 117.80 772.47 312089.33 59396.47 98.72 1.22 404.02 0.00 0.00 0.00 0.00 0.00 0.00 81.40 0.30 1.04 36.43
vdb 416.87 27695.73 0.00 0.00 1.34 66.44 852.40 275337.87 52466.33 98.40 4.25 323.01 0.00 0.00 0.00 0.00 0.00 0.00 124.27 0.73 4.27 45.28
vdb 0.00 0.00 0.00 0.00 0.00 0.00 690.93 269222.40 49188.07 98.61 1.23 389.65 0.00 0.00 0.00 0.00 0.00 0.00 74.60 0.40 0.88 31.07
vdb 0.07 1.07 0.00 0.00 0.00 16.00 838.40 295674.40 55691.40 98.52 1.61 352.67 0.00 0.00 0.00 0.00 0.00 0.00 101.07 0.23 1.37 33.68
vdb 0.00 0.00 0.00 0.00 0.00 0.00 674.73 290930.93 55346.40 98.80 1.25 431.18 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.24 0.87 32.80
vdb 2.87 142.13 0.00 0.00 0.23 49.58 809.27 286785.60 53820.53 98.52 1.32 354.38 0.00 0.00 0.00 0.00 0.00 0.00 102.00 0.22 1.09 32.75
vdb 16.20 1880.80 0.00 0.00 3.30 116.10 859.53 323764.00 65429.80 98.70 1.58 376.67 0.00 0.00 0.00 0.00 0.00 0.00 112.60 0.46 1.46 40.00
vdb 838.87 106450.93 0.00 0.00 0.58 126.90 661.93 218886.13 37578.07 98.27 2.28 330.68 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.55 2.04 55.57
vdb 0.07 0.27 0.00 0.00 1.00 4.00 901.60 378569.33 77062.40 98.84 1.72 419.89 0.00 0.00 0.00 0.00 0.00 0.00 105.13 0.27 1.58 41.31
vdb 0.00 0.00 0.00 0.00 0.00 0.00 764.00 352281.87 70652.27 98.93 1.20 461.10 0.00 0.00 0.00 0.00 0.00 0.00 79.00 0.23 0.94 38.53
vdb 50.07 5750.93 0.00 0.00 0.70 114.87 878.27 418755.47 89242.87 99.03 1.46 476.80 0.00 0.00 0.00 0.00 0.00 0.00 98.27 0.12 1.33 48.43
vdb 0.00 0.00 0.00 0.00 0.00 0.00 649.40 219117.07 37200.33 98.28 1.07 337.41 0.00 0.00 0.00 0.00 0.00 0.00 75.67 0.21 0.71 26.32
vdb 0.00 0.00 0.00 0.00 0.00 0.00 894.73 342192.00 71081.87 98.76 1.14 382.45 0.00 0.00 0.00 0.00 0.00 0.00 119.07 0.17 1.04 39.63
vdb 0.00 0.00 0.00 0.00 0.00 0.00 646.00 232256.00 41335.87 98.46 0.92 359.53 0.00 0.00 0.00 0.00 0.00 0.00 77.20 0.29 0.61 28.64
vdb 0.00 0.00 0.00 0.00 0.00 0.00 1005.27 364122.93 70386.33 98.59 2.16 362.22 0.00 0.00 0.00 0.00 0.00 0.00 130.13 0.35 2.22 40.59
vdb 14.33 1710.67 0.00 0.00 0.28 119.35 911.33 361817.60 72186.87 98.75 1.67 397.02 0.00 0.00 0.00 0.00 0.00 0.00 104.67 0.36 1.57 40.45
vdb 308.27 38892.80 0.00 0.00 0.71 126.17 598.20 194449.33 32753.53 98.21 1.84 325.06 0.00 0.00 0.00 0.00 0.00 0.00 76.80 0.39 1.35 40.77
vdb 482.67 60978.40 0.00 0.00 1.11 126.34 816.80 331697.87 67881.73 98.81 3.37 406.09 0.00 0.00 0.00 0.00 0.00 0.00 92.93 0.70 3.36 58.40
vdb 1388.87 176792.80 0.00 0.00 0.52 127.29 631.53 204010.13 37077.07 98.33 2.42 323.04 0.00 0.00 0.00 0.00 0.00 0.00 77.73 0.74 2.31 78.91
vdb 0.40 1.60 0.00 0.00 0.50 4.00 798.67 428524.53 94237.07 99.16 1.89 536.55 0.00 0.00 0.00 0.00 0.00 0.00 71.80 0.21 1.52 48.61
RainTown
January 5, 2026, 10:48am
185
Lakshya_Gupta:
I created 2 virtual machines using the same number of cores (20) and memory (53G), but with different disk sizes, 320Gb and 640Gb, and established a single node Elastic Search cluster on top of it. Thereafter I created an index (similar on both the Clusters) with a single primary shard and no routing (because it’s 1 shard anyways) and then ran NFRs for 15minutes on both of them (the NFR was creating random documents and inserting them).
I looked at the iostat and sar outputs, tbh I don't think it moves us forward. There's to my eye not much difference between the 2 iostat outputs. The w/s ramps up, but r/s stays close to zero for a while, then r/s also increases a lot to a sort of mini sine-wave like behaviour, likely the merges kicking in. I made some gnuplots of the data, just the r/s, w/s, f/s, and %util(right hand y-axis). Here's the images:
Look at those 2 graphs and tell me one system is really "better" than the other?
You are right, I don’t see any improvements with disk size (and thereby more IOPS) either.
Based on the outcome of these 2 graphs, it feels like the bottleneck is not IOPS. What do you think?
RainTown
January 5, 2026, 11:28am
187
I agree. In fact I agreed on this already before Christmas, in part by seeing the fio results, and we pivoted back to the original architecture decision, which gives you little chance to fully utilize your hardware. Suggestions were to "special case" the most skewed IDs, or simply remove the customer routing altogether.
RainTown:
So, we could try and spend time getting a fio profile that matches real-world elasticsearch profile closer and closer. I don't think that is now worth the effort. You are neither bandwidth nor IOPS bound - the storage is not the core problem, and faster/better storage would not "solve" the problem, at best it could just squeeze a little bit more out of it.
I wish I had a different answer right now, or a really great suggestion of something else to try, but I don't.
Implementing both of these might be a bit tricky, but if this is the only option, then might as well need to carry this out
RainTown
January 5, 2026, 11:50am
189
Which is why I also asked gently if it is worth the hassle, operational effort, disruption, ... I'm obviously not bound by your SLAs, nor do I know how often you are breaching them, nor the penalty, but ... not all problems are worth solving. As even if you "solve" this one, you can easily inadvertently create a regression elsewhere in what is probably quite a complex overall solution. Every perturbation on a working system is attached to some risk.
You are likely wasting (under utilizing might be nicer) the resources in your existing cluster, and that has a business cost too.
will come back in a day.
