Observed that elasticsearch service keep restarting time to time. Node has 16GB memory and allocated 8GB for the heap.
Elasticsearch version - 7.13
Below is the log. This is one of the nodes from 5 node cluster.
[2023-01-18T15:34:48,799][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] stopping ...
[2023-01-18T15:34:48,819][INFO ][o.e.x.w.WatcherService ] [AZDTEELSPRDAPP01] stopping watch service, reason [shutdown initiated]
[2023-01-18T15:34:48,820][INFO ][o.e.x.w.WatcherLifeCycleService] [AZDTEELSPRDAPP01] watcher has stopped and shutdown
[2023-01-18T15:34:48,827][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [AZDTEELSPRDAPP01] [controller/3007] [Main.cc@169] ML controller exiting
[2023-01-18T15:34:48,830][INFO ][o.e.x.m.p.NativeController] [AZDTEELSPRDAPP01] Native controller process has stopped - no new native processes can be started
[2023-01-18T15:34:49,150][INFO ][o.e.c.c.Coordinator ] [AZDTEELSPRDAPP01] master node [{AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}{ml.machine_memory=16825581568, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8589934592, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [AZDTEELSPRDAPP05][10.8.26.19:9300][disconnected] disconnected
[2023-01-18T15:34:58,094][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] stopped
[2023-01-18T15:34:58,094][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] closing ...
[2023-01-18T15:34:58,123][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] closed
[2023-01-18T15:35:02,533][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] version[7.13.3], pid[26274], build[default/rpm/5d21bea28db1e89ecc1f66311ebdec9dc3aa7d64/2021-07-02T12:06:10.804015202Z], OS[Linux/3.10.0-1160.31.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2023-01-18T15:35:02,534][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-01-18T15:35:02,535][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms8g, -Xmx8g, -Dlog4j2.formatMsgNoLookups=true, -XX:+UseG1GC, -Djava.io.tmpdir=/data1/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2023-01-18T15:35:05,488][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [aggs-matrix-stats]
[2023-01-18T15:35:05,488][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [analysis-common]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [constant-keyword]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [frozen-indices]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [ingest-common]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [ingest-geoip]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [ingest-user-agent]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [kibana]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [lang-expression]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [lang-mustache]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [lang-painless]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [mapper-extras]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [mapper-version]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [parent-join]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [percolator]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [rank-eval]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [reindex]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [repositories-metering-api]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [repository-encrypted]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [repository-url]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [runtime-fields-common]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [search-business-rules]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [searchable-snapshots]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [snapshot-repo-test-kit]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [spatial]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [systemd]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [transform]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [transport-netty4]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [unsigned-long]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [vectors]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [wildcard]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-aggregate-metric]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-analytics]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-async]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-async-search]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-autoscaling]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-ccr]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-core]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-data-streams]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-deprecation]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-enrich]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-eql]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-fleet]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-graph]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-identity-provider]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-ilm]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-logstash]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-ml]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-monitoring]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-ql]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-rollup]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-security]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-shutdown]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-sql]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-stack]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-text-structure]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-voting-only-node]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] loaded module [x-pack-watcher]
[2023-01-18T15:35:05,501][INFO ][o.e.p.PluginsService ] [AZDTEELSPRDAPP01] no plugins loaded
[2023-01-18T15:35:05,545][INFO ][o.e.e.NodeEnvironment ] [AZDTEELSPRDAPP01] using [1] data paths, mounts [[/data1 (/dev/sdb1)]], net usable_space [433.1gb], net total_space [1023.8gb], types [xfs]
[2023-01-18T15:35:05,545][INFO ][o.e.e.NodeEnvironment ] [AZDTEELSPRDAPP01] heap size [8gb], compressed ordinary object pointers [true]
[2023-01-18T15:35:07,764][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] node name [AZDTEELSPRDAPP01], node ID [zxP3dDsaTpixfMPWq2cDcA], cluster name [dte-elasticsearch-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-01-18T15:35:12,736][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [AZDTEELSPRDAPP01] [controller/26488] [Main.cc@117] controller (64 bit): Version 7.13.3 (Build d74de47cfda9c3) Copyright (c) 2021 Elasticsearch BV
[2023-01-18T15:35:13,128][INFO ][o.e.x.s.a.s.FileRolesStore] [AZDTEELSPRDAPP01] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-01-18T15:35:13,935][INFO ][o.e.i.g.LocalDatabases ] [AZDTEELSPRDAPP01] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2023-01-18T15:35:13,935][INFO ][o.e.i.g.DatabaseRegistry ] [AZDTEELSPRDAPP01] initialized database registry, using geoip-databases directory [/data1/elasticsearch/tmp/geoip-databases/zxP3dDsaTpixfMPWq2cDcA]
[2023-01-18T15:35:14,565][INFO ][o.e.t.NettyAllocator ] [AZDTEELSPRDAPP01] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-01-18T15:35:14,633][INFO ][o.e.d.DiscoveryModule ] [AZDTEELSPRDAPP01] using discovery type [zen] and seed hosts providers [settings]
[2023-01-18T15:35:15,098][INFO ][o.e.g.DanglingIndicesState] [AZDTEELSPRDAPP01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-01-18T15:35:15,518][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] initialized
[2023-01-18T15:35:15,519][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] starting ...
[2023-01-18T15:35:15,588][INFO ][o.e.x.s.c.f.PersistentCache] [AZDTEELSPRDAPP01] persistent cache index loaded
[2023-01-18T15:35:15,748][INFO ][o.e.t.TransportService ] [AZDTEELSPRDAPP01] publish_address {10.8.26.4:9300}, bound_addresses {10.8.26.4:9300}
[2023-01-18T15:35:18,673][INFO ][o.e.b.BootstrapChecks ] [AZDTEELSPRDAPP01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-18T15:35:18,676][INFO ][o.e.c.c.Coordinator ] [AZDTEELSPRDAPP01] cluster UUID [2FxEMIyPSoagAyX5lnSqwA]
[2023-01-18T15:35:20,802][INFO ][o.e.c.s.ClusterApplierService] [AZDTEELSPRDAPP01] master node changed {previous [], current [{AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}]}, added {{AZDTEELSPRDAPP02}{SXXq1aqwRJurELS8kXPGBA}{Lwl1dRXrQwuFlHoR7mpf-A}{10.8.26.5}{10.8.26.5:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP03}{vtpTm3N7RkSNLWT15ozMAA}{1RivTho2RU6fR2e-PfAixg}{10.8.26.6}{10.8.26.6:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP04}{UH9qGNqrSZG5Ef4MDlyG1g}{yPyw6TkkQJ26qrnlLmqVWg}{10.8.26.8}{10.8.26.8:9300}{cdfhilmrstw}}, term: 43, version: 88679, reason: ApplyCommitRequest{term=43, version=88679, sourceNode={AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}{ml.machine_memory=16825581568, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8589934592, transform.node=true}}
[2023-01-18T15:35:22,074][INFO ][o.e.x.s.a.TokenService ] [AZDTEELSPRDAPP01] refresh keys
[2023-01-18T15:35:22,334][INFO ][o.e.x.s.a.TokenService ] [AZDTEELSPRDAPP01] refreshed keys
[2023-01-18T15:35:22,540][INFO ][o.e.l.LicenseService ] [AZDTEELSPRDAPP01] license [78f4d2ba-5ca8-4a95-b188-122be6438524] mode [basic] - valid
[2023-01-18T15:35:22,542][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [AZDTEELSPRDAPP01] Active license is now [BASIC]; Security is enabled
[2023-01-18T15:35:22,556][INFO ][o.e.h.AbstractHttpServerTransport] [AZDTEELSPRDAPP01] publish_address {10.8.26.4:9200}, bound_addresses {10.8.26.4:9200}
[2023-01-18T15:35:22,557][INFO ][o.e.n.Node ] [AZDTEELSPRDAPP01] started
Below are Linux server logs to the issue timestamp /var/log/messages
Jan 15 15:34:17 AZDTEELSPRDAPP01 trace-agent: 2023-01-15 15:34:17 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 15 15:34:17 AZDTEELSPRDAPP01 trace-agent: 2023-01-15 15:34:17 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 16 15:34:07 AZDTEELSPRDAPP01 trace-agent: 2023-01-16 15:34:07 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 16 15:34:07 AZDTEELSPRDAPP01 trace-agent: 2023-01-16 15:34:07 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 17 15:34:21 AZDTEELSPRDAPP01 trace-agent: 2023-01-17 15:34:21 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 17 15:34:21 AZDTEELSPRDAPP01 trace-agent: 2023-01-17 15:34:21 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:11 AZDTEELSPRDAPP01 trace-agent: 2023-01-18 15:34:11 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:11 AZDTEELSPRDAPP01 trace-agent: 2023-01-18 15:34:11 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:30 AZDTEELSPRDAPP01 polkitd[835]: Registered Authentication Agent for unix-process:26086:9829187 (system bus name :1.5382 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jan 18 15:34:30 AZDTEELSPRDAPP01 systemd: Stopping Kibana...
Jan 18 15:34:30 AZDTEELSPRDAPP01 systemd: Stopping Kibana...
Jan 18 15:34:39 AZDTEELSPRDAPP01 systemd: Stopped Kibana.
Jan 18 15:34:39 AZDTEELSPRDAPP01 systemd: Stopped Kibana.
Jan 18 15:34:39 AZDTEELSPRDAPP01 polkitd[835]: Unregistered Authentication Agent for unix-process:26086:9829187 (system bus name :1.5382, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jan 18 15:34:48 AZDTEELSPRDAPP01 polkitd[835]: Registered Authentication Agent for unix-process:26173:9830981 (system bus name :1.5384 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jan 18 15:34:48 AZDTEELSPRDAPP01 systemd: Stopping Elasticsearch...
Jan 18 15:34:48 AZDTEELSPRDAPP01 systemd: Stopping Elasticsearch...
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Stopped Elasticsearch.
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Stopped Elasticsearch.
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Starting Elasticsearch...
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Starting Elasticsearch...