If you have 15 hot shards and you know which ones they are you might benefit from monitoring their distribution and reroute when required through a script in order to ensure they are spread out. It is not elegant, but might reduce overload on individual nodes.
This only means if you happened to have 3 hot shards on same node, your IO might be more efficient, and you'll see more raw IOPS on that node as the node can use different threads.
But main issue will still boil down to the skewed routing keys, which means only 15 threads (across whole cluster) are doing most of the IO. It's this limited concurrency that creates a bottleneck. The iotune settings probably just bound that bottleneck. I hope, if you are able to remove/increase those limits, you will see some, maybe substantial, improvement. The deduping helps, as lots of updates is sort of worst case scenario here, so its important to validate (as per @Christian_Dahlqvist ) thats doing what you think its doing.
Yeah, Iāve not been able to generate the load and verify that using the commands specified by Christian, however, there are other metrics using which I can 99% say that the de-dup layer is working fine (Iāve compared the number of writes which used to happen before the de-dup layer and after itā¦there is a 90% reduction in the number of writes, and the number of high cpu instances have also dropped from 8-10 times daily to at max 1-2 occurrence a day).
However, most likely, by tomorrow I should have the load test ready along with the iostat results.
If the current deduplication solution reduced the write/update load by 90% that is indeed a great reduction and a sign that it is working, at least to some extent. It does not necessarily prove that it is working as you specified though. If there was a mistake in the logic or implementation and a correct implementation could have reduced the writes by 95% (instead of 90%), this would have halved the number of writes you are experiencing now, so it is in my opinion worth verifying this.