# Elastic cluster is getting down after 2 - 3 hours **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [December 11, 2025, 7:03am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971 "2025-12-11T07:03:07Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2025, 12:37pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/22 "2025-12-11T12:37:15Z")
> [@sathish12](#): > > ```auto > "store" : { > "size_in_bytes" : 425100974480, > > ``` > [@sathish12](#): > > ```auto > "shards" : { > "total" : 3024, > > ``` If I calculate correctly that is an average shard size of around 130MB which is very small when it comes to Elasticsearch. If your data has a long retention period I would recommend that you switch to monthly indices instead of daily, which should bring the average size up a bit. If your retention period is too short to warrant monthly indices I suspect you have a lot of index patterns, and in that case I would probably advise you to look into combining indices. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 11, 2025, 12:42pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/23 "2025-12-11T12:42:10Z")
> [@Christian\_Dahlqvist](#): > > If I calculate correctly that is an average shard size of around 130MB which is very small when it comes to Elasticsearch. If your data has a long retention period I would recommend that you switch to monthly indices instead of daily, which should bring the average size up a bit. > > If your retention period is too short to warrant monthly indices I suspect you have a lot of index patterns, and in that case I would probably advise you to look into combining indices. Sure @Christian_Dahlqvist I will update it to the monthly indices and try ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2025, 12:47pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/24 "2025-12-11T12:47:26Z")
I recall running some tests a long time back where I tried to put as much data on a single node as possible. If I remember correctly there was a higher proportion of memory overhead for small shards compared to larger ones. If this is still the case (a lot has however changed in Elasticsearch since I ran that test…) having fewer larger shards may decrease heap pressure and improve stability. If you have indices that are no longer written to you can also try to merge these into monthly indices through reindexing. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 11, 2025, 12:56pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/25 "2025-12-11T12:56:52Z")
I agree with @Christian_Dahlqvist on moving to monthly indices , but I I still do not see how the 6-node cluster across 3 physical nodes benefits you, over a 3 node cluster? What's the logic here ? Are there 2 separate IO paths that the 2 different instances can utilise? Elara / Kore / Europa were "dm" , i.e. so data nodes and master eligibkle Amalthea / Kale / Callisto were "d", i.e. just data nodes So I hope you have Elara / Kore / Europa on different physical nodes, otherwise your cluster cannot survive a node failure. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 11, 2025, 1:07pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/26 "2025-12-11T13:07:54Z")
> [@Christian\_Dahlqvist](#): > > I recall running some tests a long time back where I tried to put as much data on a single node as possible. If I remember correctly there was a higher proportion of memory overhead for small shards compared to larger ones. If this is still the case (a lot has however changed in Elasticsearch since I ran that test…) having fewer larger shards may decrease heap pressure and improve stability. > > If you have indices that are no longer written to you can also try to merge these into monthly indices through reindexing. How can I merge existing indices into monthly indices? And As far as I know the index might store around 10GB if the application under test? How to avoid the memory pressure if I use monthly indices? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 11, 2025, 1:12pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/27 "2025-12-11T13:12:52Z")
> [@sathish12](#): > > How can I merge existing indices into monthly indices? You can use the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/docs-reindex.html). > [@sathish12](#): > > And As far as I know the index might store around 10GB if the application under test? How to avoid the memory pressure if I use monthly indices? You are still querying the same data set and have memory pressure from querying all the daily indices. I suspect a single 10GB shard may require less memory overhead than 30 correponding daily shards of 333MB each. Note that 10GB is not large when it comes to Elasticsearch as the recommended shard size for log and metrics use cases is generally between 30Gb and 50GB. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 11, 2025, 1:15pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/28 "2025-12-11T13:15:35Z")
> [@RainTown](#): > > I agree with @Christian_Dahlqvist on moving to monthly indices , but I I still do not see how the 6-node cluster across 3 physical nodes benefits you, over a 3 node cluster? What's the logic here ? Are there 2 separate IO paths that the 2 different instances can utilise? > > Elara / Kore / Europa were "dm" , i.e. so data nodes and master eligibkle > > Amalthea / Kale / Callisto were "d", i.e. just data nodes > > So I hope you have Elara / Kore / Europa on different physical nodes, otherwise your cluster cannot survive a node failure. @RainTown We will drill down to 1 node per machine if required to eliminate the issue. But we are looking for a smooth way so that we may not loose the data by doing so? Do you have any solution for this.? I would like to convert it to 3 nodes cluster by making 1 node per each machine. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 11, 2025, 1:17pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/29 "2025-12-11T13:17:04Z")
> [@Christian\_Dahlqvist](#): > > You are still querying the same data set and have memory pressure from querying all the daily indices. I suspect a single 10GB shard may require less memory overhead than 30 correponding daily shards of 333MB each. Note that 10GB is not large when it comes to Elasticsearch as the recommended shard size for log and metrics use cases is generally between 30Gb and 50GB. OK @Christian_Dahlqvist ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 11, 2025, 1:33pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/30 "2025-12-11T13:33:36Z")
It's actually quite easy, just eject all the data from a node, wait til the shards all relocate elsewhere, then shut it down. You will have a green, 5-node cluster. It might take a long time, but it's not a complex procedure. Rinse and repeat 2 more times. But you need be a bit careful that primary and replica shards dont end up on the same (physical) host. This may be a problem already, unless you have configured otherwise. Personally, from what you have shared, I'd prioritize the monthly indices first. And that's less easy tbh. Because of my very first response: > [@RainTown](#): > > why was it trying to auto-create an index, with 2025.11.26 in the index name, on 2025-12-10? If all the blabla-2025-11-\* indices were all effectively frozen, you can just reindex them into say blabla-2025-11, just wait for that too complete, and eventually just delete all the blabla-2025-11-\* indices. But if during that reindex you are still getting new documents into those existing indices, in sort of random date order, then that's a bit harder. And it seems you do have that issue! ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 11, 2025, 1:54pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/31 "2025-12-11T13:54:34Z")
> [@RainTown](#): > > If all the blabla-2025-11-\* indices were all effectively frozen, you can just reindex them into say blabla-2025-11, just wait for that too complete, and eventually just delete all the blabla-2025-11-\* indices. But if during that reindex you are still getting new documents into those existing indices, in sort of random date order, then that's a bit harder. And it seems you do have that issue! So reIndexing is suggestible? or I should follow any other process? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 11, 2025, 2:31pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/32 "2025-12-11T14:31:57Z")
Good question. Yes, I would still do it. You can block writes to the blabla-2025-11-\* indices too. But you know your data flows, why are logs, if they are logs, from 26 Nov being ingested on 10th Dec? doesn't that indicate an issue somewhere else? EDIT: if you start 6 months ago, do July 2025 first, then I hope this is less of an issue then. But without downtime it's going to be hard to do piecemeal if data and indices are just appearing is semi random order. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 11, 2025, 5:30pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/33 "2025-12-11T17:30:19Z")
All this discussion about sizing & sharding & memory usage and so on is good but it’s not really answering the OP’s question. Why is this node going down? > [@sathish12](#): > > > `[2025-12-10T15:04:23,537][INFO][o.e.n.Node] [Kale] stopping ...` > > `[2025-12-10T15:04:23,540][INFO][o.e.x.w.WatcherService] [Kale] stopping watch service, reason [shutdown initiated]` > > `[2025-12-10T15:04:23,541][INFO][o.e.x.w.WatcherLifeCycleService] [Kale] watcher has stopped and shutdown` > > `[2025-12-10T15:04:23,595][INFO][o.e.c.c.Coordinator] [Kale] master node [{Europa}{asrawfdsvdzvdc}{asrawfdsvdzvdc}{Europa}{x.x.x.x.}{x.x.x.x.:9300}{dm}{8.13.4}{7000099-8503000}] disconnected, restarting discovery` > > `[2025-12-10T15:04:23,608][INFO][o.e.h.AbstractHttpServerTransport] [Kale] channel [Netty4HttpChannel{localAddress=/x.x.x.x.:9201, remoteAddress=/x.x.x.y:64422}] already closed` > > `[2025-12-10T15:04:30,351][INFO][o.e.n.Node] [Kale] stopped` > > `[2025-12-10T15:04:30,351][INFO][o.e.n.Node] [Kale] closing ...` > > `[2025-12-10T15:04:30,444][INFO][o.e.n.Node] [Kale] closed` This sequence of messages indicates a normal process exit: the node is shutting down gracefully because it’s being asked to shut down by some external influence sending it a signal (usually a `SIGTERM` but possibly something else, depending on how exactly the process is being managed). It doesn’t matter how much you change any of the aspects of this cluster in the ways discussed above, it will always exit if asked to do so as we are seeing here. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 11, 2025, 5:54pm UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/34 "2025-12-11T17:54:00Z")
> [@DavidTurner](#): > > It doesn’t matter how much you change any of the aspects of this cluster in the ways discussed above, it will always exit if asked to do so as we are seeing here. You are absolutely right. Maybe if setup differently, meant in a generic way, it wont be gracefully shut down every 2-3 hours. But indeed, cart (what to do) got a bit before horse (what is actually wrong in your specific case). The question was asked but rather skipped. > [@RainTown](#): > > Please check all nodes logs for ERROR and WARN before the crashes. Also check the system logs for any limits you maybe have reached, OOMs, or similar. @sathish12 Your system reported as CentOS 8. You said you used the .tar.gz file, so how do you start/stop the 2x elasticsearch instances per host? Did you integrate with systemd? You need to look at the system logs and try to find out _why_ Elasticsearch was being shutdown, as per David. It's maybe a memory issue, but that's just a guess and it's for sure better to actually know, as it could be something entirely unrelated to anything on the thread. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 3:02am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/35 "2025-12-12T03:02:38Z")
> [@DavidTurner](#): > > This sequence of messages indicates a normal process exit: the node is shutting down gracefully because it’s being asked to shut down by some external influence sending it a signal (usually a `SIGTERM` but possibly something else, depending on how exactly the process is being managed). > > It doesn’t matter how much you change any of the aspects of this cluster in the ways discussed above, it will always exit if asked to do so as we are seeing here. @DavidTurner Could you let me know how to check this? ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 3:03am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/36 "2025-12-12T03:03:45Z")
> [@RainTown](#): > > > [@RainTown](#): > > > > Please check all nodes logs for ERROR and WARN before the crashes. Also check the system logs for any limits you maybe have reached, OOMs, or similar. I did not find any ERROR or WARN logs before the Native controller issue ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 3:04am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/37 "2025-12-12T03:04:31Z")
> [@RainTown](#): > > Your system reported as CentOS 8. You said you used the .tar.gz file, so how do you start/stop the 2x elasticsearch instances per host? Did you integrate with systemd? You need to look at the system logs and try to find out _why_ Elasticsearch was being shutdown, as per David. It's maybe a memory issue, but that's just a guess and it's for sure better to actually know, as it could be something entirely unrelated to anything on the thread. @RainTown I am running the elastic in background using '&' for each node ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 3:07am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/38 "2025-12-12T03:07:25Z")
@RainTown @DavidTurner @Christian_Dahlqvist I am not sure. But yesterday I have changed to 7GB heap for each node and I have observed that the cluster was staying few hours longer till 6 hours and then went down ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 12, 2025, 6:41am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/39 "2025-12-12T06:41:33Z")
Could you check this? ```auto grep -i kill /var/log/messages* host kernel: Out of Memory: Killed process 2592 (xxx). ``` May be you have an OOM Killer which killed the process? ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 6:51am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/40 "2025-12-12T06:51:25Z")
> [@dadoonet](#): > > Could you check this? > > ```auto > grep -i kill /var/log/messages* > host kernel: Out of Memory: Killed process 2592 (xxx). > > ``` > > May be you have an OOM Killer which killed the process? I am seeing the logs of around 20 days back regarding OOM. It has NOV21 not the recent ones ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 8:40am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/41 "2025-12-12T08:40:11Z")
> [@sathish12](#): > > I am seeing the logs of around 20 days back regarding OOM. It has NOV21 not the recent ones OK, but even that is a bit suggestive? Can you post the exact message matched. Also note some of the file in /var/log/ might be compressed. Also, can you share the elasticsearch.yml files for both instances. Are the elastic processes running as same (local) user? If running as "elasticsearch: user, then please share output of `ps -uelasticsearch --cols 9999 -opid,ppid,stime,etime,rss,command` replace elasticsearch by whichever user or users you are using. > [@sathish12](#): > > @RainTown > > I am running the elastic in background using '&' for each node What about stdout and stderr for these background processes, where do they go? can you share the full command you use to start the instances please. > [@sathish12](#): > > I am not sure. But yesterday I have changed to 7GB heap for each node and I have observed that the cluster was staying few hours longer till 6 hours and then went down If you start with less heap, then there is more "general" memory available for all processes, so that might mean it would take longer to reach point where it crashes than if started with a larger heap. But there 101 other factors too. It is a bit suggestive of course. When you write "then went down" can you be more specific. Does one instance stop? Is there a cascade and all the nodes stop? Something else? [Previous page](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971.md?page=1) [Next page](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971.md?page=3)