# Urgent Help Needed Elastic Shutting down ingestion & All the write load is being transferred to one of the hot nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [July 9, 2026, 8:07pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083 "2026-07-09T20:07:07Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 13, 2026, 8:42pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/21 "2026-07-13T20:42:27Z")
i see all cluster working correctly until this force merge appears this write load shift to one hot node. it impacts the cluster when ever i see the server logs it is there when cpu load on any hot node is high. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 13, 2026, 11:58pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/22 "2026-07-13T23:58:44Z")
> [@kkumar123](#): > > The design was purposed by solution architect. From Elastic (the company)? Anyways, better not to comment, who knows what she/he was told, maybe your usage/load is not as anticipated. > [@kkumar123](#): > > I would have open that ticket but right now i don't access to support portal, will surely do that. If I were you, I'd prioritize resolving that ASAP. In part because this sort of thread has limits, people like me ask questions, you do your best to answer them, that leads to more questions .... 4-5 days can easily pass without too much progress. With a support ticket it _can_ be a lot more interactive. > [@kkumar123](#): > > `elastic-ml1 31.3gb 8192pb 8191.9pb 19.2tb lt` 8192pb? A bug in the tool maybe 😉 - not important. > [@kkumar123](#): > > i see all cluster working correctly until this force merge appears this write load shift to one hot node. ... which is why a limited warm tier _might_ help. Next time, look at the number of shards (and segments) of the index that it is being force-merged. It was only 4 days ago you wrote: > [@kkumar123](#): > > i have increased the number of primaries to 3 for few data-streams which were having highest ingestion. and I think you also increased to 6 sometime later?, so those changes might not have helped in the force-merge sense yet, depending on the specifics of the ILM policy. Also looking at the [docs](https://www.elastic.co/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot) `force_merge_index` is true by default and `force_merge_on_clone` is also true by default. So your index is likely cloned then force-merged. So if the index had just 1 primary shard and 1 replica shard, then the force-merge, as executed on the cloned primary shard, can be executed entirely on one hot node? You also missed answering below - what are you re-routing and how? Might not be important, but ... who knows. > [@RainTown](#): > > > [@kkumar123](#): > > > > i am using reroute to route some of errors > > Can you be more precise as to what you mean here ? ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 14, 2026, 4:02pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/23 "2026-07-14T16:02:48Z")
> [@RainTown](#): > > and I think you also increased to 6 sometime later?, so those changes might not have helped in the force-merge sense yet, depending on the specifics of the ILM policy. Yes we i have done that as well 6 primary in tier one level data streams like apm with 3 to 4.5 tb primary data per day. and kubernetes logs in this tier 1 which highest ingest of 300 GB per day to 6 primary and remaining were low in ingestion so to 3 primary shards. > [@RainTown](#): > > Can you be more precise as to what you mean here ? by this i mean i am using reroute processor in ingest pipelines to reroute not needed apm error to delete after 2 days. changes i have made they are helping me to reduce shards count. from 2000 to 1200 t0 1600 shards. Just few days more then i will surely open the ticket. there some blocker to go with them right now. > [@RainTown](#): > > 8192pb? A bug in the tool maybe - not important. yes, probably in this 9.3.4 version, because UI \> Indices i can't see the indices i have to upgrade the cluster as well. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 14, 2026, 5:16pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/24 "2026-07-14T17:16:36Z")
In passing, have only 5/6 of your hot nodes having the ingest role is a asymetrry that I don't like (elastic-hot6 is the exception) > [@kkumar123](#): > > I have to upgrade the cluster as well. This pretty much has to be a rolling upgrade, so I'd be cautious about starting that until you better understand your current issues. > [@kkumar123](#): > > changes i have made they are helping me to reduce shards count. from 2000 to 1200 t0 1600 shards. shard count is not the metric you should be optimizing, cluster stability and availability are. Since you opened the thread, are nodes continuing to crash? If so, at what frequency? If not, do you have other operational issues directly impacting availability, and if so what are they? A heavily loaded and/or unevenly loaded cluster is not ideal. Neither is "wasting" some $ of ERUs. But "service availability" is more crucial, at least in the places I worked. Possibly someone from Elastic who is reading this might be able to assist you in getting a support ticket opened. My hunch is your difficulties _may_ be quickly understood, and even addressable, with some engineer from Elastic looking at your overall solution interactively. Good luck! ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 20, 2026, 12:11pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/25 "2026-07-20T12:11:03Z")
Hello @kkumar123 > [@kkumar123](#): > > Just few days more then i will surely open the ticket Did you make progress with this via Elastic Support, or other means? Even if its resolved now, it is always useful for those that stumble on the thread in the future to know how a problem report like yours ended ... And if not resovled, you can still ask more questions 🙂 ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 20, 2026, 4:20pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/26 "2026-07-20T16:20:14Z")
I am currently waiting for our license renewal to complete before opening an official support ticket. In the meantime, I have taken a few remediation steps. I disabled high-I/O force-merges, and earlier today, I successfully converted an ingest node into a hot data node by attaching a new 8 TB disk (7 TB usable volume). This added capacity should help stabilise our immediate write queues, though I am still tuning the cluster for long-term consistency. Currently, we face brief APM ingestion drops about three times a day. This happens during ILM rollovers when write and snapshot threads spike—especially when replica shards are temporarily unassigned. Retention demands are low, so this footprint is unexpected. Additionally, I am troubleshooting a storage issue on our frozen/searchable snapshot tier. I allocated a 3,500 GB cache on the frozen tier disk, but `df -h` shows it remains at 75% utilization. Even though the corresponding indices have been deleted and I have triggered snapshot cache clear commands, the disk space is not being released. I am still investigating why this cache is not clearing out. clearing the old snapshot and indices stuck at ilm phases due to change ilm of removing force merge ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 21, 2026, 7:21am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/27 "2026-07-21T07:21:59Z")
> [@kkumar123](#): > > a 3,500 GB cache on the frozen tier disk, but `df -h` shows it remains at 75% utilization. Even though the corresponding indices have been deleted and I have triggered snapshot cache clear commands, the disk space is not being released. This is expected on the frozen tier. The frozen cache is allocated up-front regardless of how much data it holds. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 21, 2026, 7:23am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/28 "2026-07-21T07:23:19Z")
> [@kkumar123](#): > > I am currently waiting for our license renewal to complete before opening an official support ticket. Can you explain why? Waiting to renew shouldn't be necessary, support tickets from your current license will keep running with the renewed one. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [July 21, 2026, 11:06am UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/29 "2026-07-21T11:06:16Z")
Thanks for updating. > [@kkumar123](#): > > I disabled high-I/O force-merges You did this in the ILM policy via setting force\_merge\_index to false, for some specific indices/data-streams? Or some other way? > [@kkumar123](#): > > clearing the old snapshot and indices stuck at ilm phases due to change ilm of removing force merge I don't follow here, sorry. What's stuck? What does the `/{index}/_ilm/explain` API return for an index in this "suck" state ? I hope you can get the support ticket stuff worked out soon. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 21, 2026, 2:42pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/30 "2026-07-21T14:42:14Z")
> [@RainTown](#): > > You did this in the ILM policy via setting force\_merge\_index to false, for some specific indices/data-streams? Or some other way? We had cron job to change ilm policy with forcemerge and then remvoe it after few hours in UTC at low traffic. so i disabled that cron to run. ---
**Author:** ![kkumar123](https://avatars.discourse-cdn.com/v4/letter/k/e9c0ed/32.png) [@kkumar123](https://discuss.elastic.co/u/kkumar123)\ **Post date:** [July 21, 2026, 2:48pm UTC](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083/31 "2026-07-21T14:48:33Z")
I don't have that support account as well once they renew they will give me those details for support account access currently i don't have [Previous page](https://discuss.elastic.co/t/urgent-help-needed-elastic-shutting-down-ingestion-all-the-write-load-is-being-transferred-to-one-of-the-hot-nodes/388083.md?page=1)