Hi all,
Sometime Elasticsearch service stops unexpectedly in the weekend and there are no details in logs to identify the exact issue. Please suggest any solution if you already come across this issue in past.
log details.
[2023-12-16T01:30:00,715][INFO ][o.e.x.s.SnapshotRetentionTask] [node-1] starting SLM retention snapshot cleanup task
[2023-12-16T01:30:00,715][INFO ][o.e.x.s.SnapshotRetentionTask] [node-1] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2023-12-16T05:20:00,462][INFO ][o.e.b.Elasticsearch ] [node-1] running graceful exit on windows
[2023-12-16T05:20:00,508][INFO ][o.e.n.Node ] [node-1] stopping ...
[2023-12-16T05:20:00,508][INFO ][o.e.c.f.AbstractFileWatchingService] [node-1] shutting down watcher thread
[2023-12-16T05:20:00,524][INFO ][o.e.c.f.AbstractFileWatchingService] [node-1] watcher service stopped
[2023-12-16T05:20:00,571][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/800] [Main.cc@176] ML controller exiting
[2023-12-16T05:20:00,571][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
[2023-12-16T05:20:00,587][INFO ][o.e.x.w.WatcherService ] [node-1] stopping watch service, reason [shutdown initiated]
[2023-12-16T05:20:00,602][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-timeslices-90d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [2/-1]
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-occurrences-7d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [2/-1]
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-occurrences-monthly-aligned] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [2/-1]
[2023-12-16T05:20:02,480][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,479][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-timeslices-7d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-occurrences-weekly-aligned] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,476][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-occurrences-30d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,479][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-occurrences-90d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [3/-1]
[2023-12-16T05:20:02,479][INFO ][o.e.x.t.t.TransformFailureHandler] [node-1] [slo-summary-timeslices-30d-rolling] Transform encountered an exception: [org.elasticsearch.node.NodeClosedException: node closed {node-1}{fN5d2aecR4Wz38XzClTslQ}{w0piyMCPRIujTdov6CGS5w}{node-1}{10.50.162.4}{10.50.162.4:9300}{cdfhilmrstw}{8.11.1}{7000099-8500003}{ml.allocated_processors_double=2.0, ml.allocated_processors=2, ml.machine_memory=17126223872, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=8564768768}]; Will automatically retry [2/-1]
[2023-12-16T05:20:02,526][WARN ][o.e.x.t.t.ClientTransformIndexer] [node-1] [slo-summary-timeslices-30d-rolling] updating stats of transform failed.
java.lang.RuntimeException: Failed to persist transform statistics for transform [slo-summary-timeslices-30d-rolling]
at org.elasticsearch.xpack.transform.persistence.IndexBasedTransformConfigManager.lambda$putOrUpdateTransformStoredDoc$21(IndexBasedTransformConfigManager.java:726) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.11.1.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.11.1.jar:?]
at