i see all cluster working correctly until this force merge appears this write load shift to one hot node. it impacts the cluster when ever i see the server logs it is there when cpu load on any hot node is high.
From Elastic (the company)? Anyways, better not to comment, who knows what she/he was told, maybe your usage/load is not as anticipated.
If I were you, I'd prioritize resolving that ASAP. In part because this sort of thread has limits, people like me ask questions, you do your best to answer them, that leads to more questions .... 4-5 days can easily pass without too much progress. With a support ticket it can be a lot more interactive.
8192pb? A bug in the tool maybe 
- not important.
... which is why a limited warm tier might help. Next time, look at the number of shards (and segments) of the index that it is being force-merged. It was only 4 days ago you wrote:
and I think you also increased to 6 sometime later?, so those changes might not have helped in the force-merge sense yet, depending on the specifics of the ILM policy.
Also looking at the docs
force_merge_index is true by default and force_merge_on_clone is also true by default. So your index is likely cloned then force-merged. So if the index had just 1 primary shard and 1 replica shard, then the force-merge, as executed on the cloned primary shard, can be executed entirely on one hot node?
You also missed answering below - what are you re-routing and how? Might not be important, but ... who knows.
Yes we i have done that as well 6 primary in tier one level data streams like apm with 3 to 4.5 tb primary data per day. and kubernetes logs in this tier 1 which highest ingest of 300 GB per day to 6 primary and remaining were low in ingestion so to 3 primary shards.
by this i mean i am using reroute processor in ingest pipelines to reroute not needed apm error to delete after 2 days.
changes i have made they are helping me to reduce shards count. from 2000 to 1200 t0 1600 shards.
Just few days more then i will surely open the ticket. there some blocker to go with them right now.
yes, probably in this 9.3.4 version, because UI > Indices i can't see the indices i have to upgrade the cluster as well.
In passing, have only 5/6 of your hot nodes having the ingest role is a asymetrry that I don't like (elastic-hot6 is the exception)
This pretty much has to be a rolling upgrade, so I'd be cautious about starting that until you better understand your current issues.
shard count is not the metric you should be optimizing, cluster stability and availability are. Since you opened the thread, are nodes continuing to crash? If so, at what frequency? If not, do you have other operational issues directly impacting availability, and if so what are they? A heavily loaded and/or unevenly loaded cluster is not ideal. Neither is "wasting" some $ of ERUs. But "service availability" is more crucial, at least in the places I worked.
Possibly someone from Elastic who is reading this might be able to assist you in getting a support ticket opened. My hunch is your difficulties may be quickly understood, and even addressable, with some engineer from Elastic looking at your overall solution interactively.
Good luck!
Hello @kkumar123
Did you make progress with this via Elastic Support, or other means? Even if its resolved now, it is always useful for those that stumble on the thread in the future to know how a problem report like yours ended ...
And if not resovled, you can still ask more questions 
I am currently waiting for our license renewal to complete before opening an official support ticket. In the meantime, I have taken a few remediation steps. I disabled high-I/O force-merges, and earlier today, I successfully converted an ingest node into a hot data node by attaching a new 8 TB disk (7 TB usable volume). This added capacity should help stabilise our immediate write queues, though I am still tuning the cluster for long-term consistency.
Currently, we face brief APM ingestion drops about three times a day. This happens during ILM rollovers when write and snapshot threads spike—especially when replica shards are temporarily unassigned. Retention demands are low, so this footprint is unexpected.
Additionally, I am troubleshooting a storage issue on our frozen/searchable snapshot tier. I allocated a 3,500 GB cache on the frozen tier disk, but df -h shows it remains at 75% utilization. Even though the corresponding indices have been deleted and I have triggered snapshot cache clear commands, the disk space is not being released. I am still investigating why this cache is not clearing out.
clearing the old snapshot and indices stuck at ilm phases due to change ilm of removing force merge
This is expected on the frozen tier. The frozen cache is allocated up-front regardless of how much data it holds.
Can you explain why? Waiting to renew shouldn't be necessary, support tickets from your current license will keep running with the renewed one.
Thanks for updating.
You did this in the ILM policy via setting force_merge_index to false, for some specific indices/data-streams? Or some other way?
I don't follow here, sorry. What's stuck? What does the /{index}/_ilm/explain API return for an index in this "suck" state ?
I hope you can get the support ticket stuff worked out soon.
We had cron job to change ilm policy with forcemerge and then remvoe it after few hours in UTC at low traffic. so i disabled that cron to run.
I don't have that support account as well once they renew they will give me those details for support account access currently i don't have