# Elastic cluster is getting down after 2 - 3 hours **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [December 11, 2025, 7:03am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971 "2025-12-11T07:03:07Z")\ **Posts on this page:** 16\ **Page:** 3
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 12, 2025, 8:55am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/42 "2025-12-12T08:55:23Z")
The sequence of logs shared in the OP are not consistent with the shutdown being caused by the OOM killer. The OOM killer sends a `SIGKILL` which causes immediate process exit, not a graceful shutdown. You don’t get any log messages in that case, the process just dies. If you were running it from the command line then sometimes you get the one-word message `Killed` on the last line, and an exit code of 137. The logs in the OP are also not consistent with a Java OOM exception. That tends to be rather visible in the logs and again doesn’t go through the full `stopping` and `closing` sequence seen here. It is unclear whether _all_ of the shutdowns in question involve those log messages or whether @sathish12 just happened to pick an outlier. But all the ones that say `stopping` and `closing` are graceful shutdowns. Note that if you run Elasticsearch from the console using `&` and then close the console then I expect the process will shut down like this. That’d be my first suspicion. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 10:05am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/43 "2025-12-12T10:05:49Z")
> [@RainTown](#): > > OK, but even that is a bit suggestive? Can you post the exact message matched. Also note some of the file in /var/log/ might be compressed. ```auto /var/log/messages-20251123:Nov 21 11:29:56 x.x.x.x kernel: GC Thread#0 invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0 /var/log/messages-20251123:Nov 21 11:29:57 x.x.x.x kernel: oom_kill_process.cold.32+0xb/0x10 /var/log/messages-20251123:Nov 21 11:29:59 x.x.x.x kernel: [2127] 42 2127 57597 0 208896 251 0 gsd-rfkill /var/log/messages-20251123:Nov 21 11:29:59 x.x.x.x kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1002.slice/session-40.scope,task=java,pid=119441,uid=1002 /var/log/messages-20251123:Nov 21 11:29:59 x.x.x.x kernel: Out of memory: Killed process 119441 (java) total-vm:20097204kB, anon-rss:9454160kB, file-rss:0kB, shmem-rss:0kB, UID:1002 pgtables:33412kB oom_score_adj:0 ``` ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 10:09am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/44 "2025-12-12T10:09:48Z")
> [@RainTown](#): > > Also, can you share the elasticsearch.yml files for both instances. Are the elastic processes running as same (local) user? If running as "elasticsearch: user, then please share output of > > `ps -uelasticsearch --cols 9999 -opid,ppid,stime,etime,rss,command` > > replace elasticsearch by whichever user or users you are using. ```auto 497567 497290 14:23 40:57 26412 /home/demouser/Europa/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=Europa/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/home/demouser/Europa -Des.path.conf=/home/demouser/Europa/config -Des.distribution.type=tar -cp /home/demouser/Europa/lib/*:/home/demouser/Europa/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher 497632 497567 14:23 40:54 9410404 /home/demouser/Europa/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=org.elasticsearch.preallocate --enable-native-access=org.elasticsearch.nativeaccess -XX:ReplayDataFile=logs/replay_pid%p.log -Des.distribution.type=tar -Xms7g -Xmx7g -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Xms7g -Xmx7g -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=3758096384 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /home/demouser/Europa/lib --add-modules=jdk.net --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch 497655 497632 14:23 40:52 132 /home/demouser/Europa/modules/x-pack-ml/platform/linux-x86_64/bin/controller 497663 497290 14:23 40:52 77124 /home/demouser/Kale/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=Kale/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/home/demouser/Kale -Des.path.conf=/home/demouser/Kale/config -Des.distribution.type=tar -cp /home/demouser/Kale/lib/*:/home/demouser/Kale/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher 497727 497663 14:23 40:48 10267476 /home/demouser/Kale/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=org.elasticsearch.preallocate --enable-native-access=org.elasticsearch.nativeaccess -XX:ReplayDataFile=logs/replay_pid%p.log -Des.distribution.type=tar -Xms7g -Xmx7g -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -Xms7g -Xmx7g -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=3758096384 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 --module-path /home/demouser/Kale/lib --add-modules=jdk.net --add-modules=ALL-MODULE-PATH -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch 497751 497727 14:23 40:46 852 /home/demouser/Kale/modules/x-pack-ml/platform/linux-x86_64/bin/controller ``` ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 10:21am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/45 "2025-12-12T10:21:08Z")
> [@RainTown](#): > > Also, can you share the elasticsearch.yml files for both instances. Are the elastic processes running as same (local) user? If running as "elasticsearch: user, then please share output of ```auto # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: demoCLuster # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: Europa # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # By default Elasticsearch is only accessible on localhost. Set a different # address here to expose this node on the network: # network.host: x.x.x.x # # By default Elasticsearch listens for HTTP traffic on the first free port it # finds starting at 9200. Set a specific HTTP port here: # http.port: 9200 transport.port: 9300 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Various ----------------------------------- # # Allow wildcard deletion of indices: # #action.destructive_requires_name: false #----------------------- BEGIN SECURITY AUTO CONFIGURATION ----------------------- # # The following settings, TLS certificates, and keys have been automatically # generated to configure Elasticsearch security features on 24-06-2024 14:28:16 # # -------------------------------------------------------------------------------- # Enable security features xpack.security.enabled: true xpack.security.enrollment.enabled: true # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 # Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 # Discover existing nodes in the cluster #discovery.seed_hosts: ["a.b.c.d:9300"] #cluster.initial_master_nodes: ["a.b.c.d:9300","x.x.x.y:9300","x.x.x.z:9300"] #cluster.initial_master_nodes: ["Europa","Elara","Kore"] discovery.seed_hosts: ["x.x.x.x:9300", "x.x.x.x:9301", "x.x.x.y:9300", "x.x.x.y:9301","x.x.x.z:9300","x.x.x.z:9301"] node.roles: [master, data] # Allow HTTP API connections from anywhere # Connections are encrypted and require user authentication http.host: 0.0.0.0 # Allow other nodes to join the cluster from anywhere # Connections are encrypted and mutually authenticated transport.host: 0.0.0.0 #----------------------- END SECURITY AUTO CONFIGURATION ------------------------- ``` ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 10:22am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/46 "2025-12-12T10:22:05Z")
> [@RainTown](#): > > What about stdout and stderr for these background processes, where do they go? can you share the full command you use to start the instances please. I am using MRemoteNG to connect to the VMs. So I cannot monitor it till the end. As soon as I disconnect the network, the mremoteng session wears off ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 12, 2025, 10:22am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/47 "2025-12-12T10:22:50Z")
> [@DavidTurner](#): > > The sequence of logs shared in the OP are not consistent with the shutdown being caused by the OOM killer. The OOM killer sends a `SIGKILL` which causes immediate process exit, not a graceful shutdown. You don’t get any log messages in that case, the process just dies. If you were running it from the command line then sometimes you get the one-word message `Killed` on the last line, and an exit code of 137. > > The logs in the OP are also not consistent with a Java OOM exception. That tends to be rather visible in the logs and again doesn’t go through the full `stopping` and `closing` sequence seen here. > > It is unclear whether _all_ of the shutdowns in question involve those log messages or whether @sathish12 just happened to pick an outlier. But all the ones that say `stopping` and `closing` are graceful shutdowns. > > Note that if you run Elasticsearch from the console using `&` and then close the console then I expect the process will shut down like this. That’d be my first suspicion. Hi @DavidTurner Do you need any other information, Let me know ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 12, 2025, 11:03am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/48 "2025-12-12T11:03:10Z")
I think @DavidTurner has pinpointed it already. I started elasticsearch, over ssh connection, in background. ```auto % ps -ukevin | fgrep java 1651997 pts/1 00:00:02 java 1652057 pts/1 00:00:30 java ``` I then exit the shell. First of all I see ```auto % exit zsh: you have running jobs. ``` Then I exit anyways and see: ```auto zsh: warning: 1 jobs SIGHUPed [2025-12-12T11:50:12,734][INFO][o.e.x.m.p.NativeController] [u2024] Native controller process has stopped - no new native processes can be started [2025-12-12T11:50:12,737][INFO][o.e.n.Node] [u2024] stopping ... [2025-12-12T11:50:12,738][INFO][o.e.c.f.AbstractFileWatchingService] [u2024] shutting down watcher thread [2025-12-12T11:50:12,739][INFO][o.e.c.f.AbstractFileWatchingService] [u2024] watcher service stopped [2025-12-12T11:50:12,741][INFO][o.e.x.w.WatcherService] [u2024] stopping watch service, reason [shutdown initiated] [2025-12-12T11:50:12,742][INFO][o.e.x.w.WatcherLifeCycleService] [u2024] watcher has stopped and shutdown ``` Notice the `warning: 1 jobs SIGHUPed` message, the shell sent a signal to the node to shut itself down. You can avoid this by using `disown` shell command after backgrounding it. But it would be better to integrate into systemd, which you would have "for free" had you had used the rpm install instead of .tar.gz. Anyways, try this: ```auto % bin/elasticsearch >> ./stdout.txt 2>> stderr.txt & [1] 1652489 % disown %1 % exit ``` (the "1" is the number in square brackets, technically the job number). The JVM process(es), elasticsearch, will continue to run even if/when shell exits. You can also use nohup. Or start within screen sessions. Or a few other ways. If this is the cause of recent crashes, its not clear why > [@sathish12](#): > > The cluster was working file for months without any issue if you were starting the exact same way months ago? That notwithstanding, the Nov 21 crash was a OOMkiller crash. ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 15, 2025, 7:12am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/49 "2025-12-15T07:12:46Z")
Hi @RainTown @Christian_Dahlqvist @DavidTurner I am running the currently the existing elastic as daemon with -d -p in the command line. To avoid SIGHUP related to session. Lets see how it goes. Keep you posted Thanks, Sathish ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 16, 2025, 5:22am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/50 "2025-12-16T05:22:50Z")
> [@sathish12](#): > > Hi @RainTown @Christian_Dahlqvist @DavidTurner > > I am running the currently the existing elastic as daemon with -d -p in the command line. To avoid SIGHUP related to session. Lets see how it goes. > > Keep you posted > > Thanks, > Sathish Hi @RainTown @Christian_Dahlqvist @DavidTurner The issue is resolved as the cluster is stable for around 2nd day now Thanks & Regards ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 16, 2025, 5:23am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/51 "2025-12-16T05:23:49Z")
> [@sathish12](#): > > Hi @RainTown @Christian_Dahlqvist @DavidTurner > > The issue is resolved as the cluster is stable for around 2nd day now > > Thanks & Regards The only question I have right now is how was it able to run around 6 months without even getting stopped? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 16, 2025, 7:33am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/52 "2025-12-16T07:33:24Z")
> [@sathish12](#): > > The issue is resolved as the cluster is stable for around 2nd day now Are you going to let us into the secret? What did you change? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 16, 2025, 8:08am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/53 "2025-12-16T08:08:27Z")
> [@sathish12](#): > > I am running the currently the existing elastic as daemon with -d -p in the command line would seem to have resolved the exiting-shell (via MRemoteNG ) not HUP-ing the elasticsearch processes, as you identified! One might say that we are now waiting for the OOM 🙂 ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 17, 2025, 7:15am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/54 "2025-12-17T07:15:14Z")
> [@DavidTurner](#): > > Are you going to let us into the secret? What did you change? @DavidTurner I am running the process with -d -p option to make it run as daemon. So now the process is detached from the session and running as daemon ---
**Author:** ![sathish12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sathish12/32/143619_2.png) [@sathish12](https://discuss.elastic.co/u/sathish12)\ **Post date:** [December 17, 2025, 7:16am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/55 "2025-12-17T07:16:33Z")
Just curious that I have been running this process using '&' as suffix for more than a year. So why this issue came now and I had to run it as daemon ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 17, 2025, 8:00am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/56 "2025-12-17T08:00:54Z")
No idea, sorry. I’d guess something has changed in your environment. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 17, 2025, 10:50am UTC](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971/57 "2025-12-17T10:50:55Z")
couple of possibilities. Your shell changed (either in behaviour or config). e.g. some update changed version or defaults in /etc/.... that you pickup automatically and unknowingly. Same for MRemoteNG, or some systemd/logind/sshd config, or ... Something in between changed, e.g. firewall that started killing idle connections more aggressively. Best is to try to integrate into the "init system", which nowadays is most commonly systemd on Linux. You can maybe accept @DavidTurner 's answer as the solution here, or wait for the next OOM (might be long wait). [Previous page](https://discuss.elastic.co/t/elastic-cluster-is-getting-down-after-2-3-hours/383971.md?page=2)