Elasticsearch service keep restarting time to time

Observed that elasticsearch service keep restarting time to time. Node has 16GB memory and allocated 8GB for the heap.

Elasticsearch version - 7.13
Below is the log. This is one of the nodes from 5 node cluster.

[2023-01-18T15:34:48,799][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] stopping ...
[2023-01-18T15:34:48,819][INFO ][o.e.x.w.WatcherService   ] [AZDTEELSPRDAPP01] stopping watch service, reason [shutdown initiated]
[2023-01-18T15:34:48,820][INFO ][o.e.x.w.WatcherLifeCycleService] [AZDTEELSPRDAPP01] watcher has stopped and shutdown
[2023-01-18T15:34:48,827][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [AZDTEELSPRDAPP01] [controller/3007] [Main.cc@169] ML controller exiting
[2023-01-18T15:34:48,830][INFO ][o.e.x.m.p.NativeController] [AZDTEELSPRDAPP01] Native controller process has stopped - no new native processes can be started
[2023-01-18T15:34:49,150][INFO ][o.e.c.c.Coordinator      ] [AZDTEELSPRDAPP01] master node [{AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}{ml.machine_memory=16825581568, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8589934592, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [AZDTEELSPRDAPP05][10.8.26.19:9300][disconnected] disconnected
[2023-01-18T15:34:58,094][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] stopped
[2023-01-18T15:34:58,094][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] closing ...
[2023-01-18T15:34:58,123][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] closed
[2023-01-18T15:35:02,533][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] version[7.13.3], pid[26274], build[default/rpm/5d21bea28db1e89ecc1f66311ebdec9dc3aa7d64/2021-07-02T12:06:10.804015202Z], OS[Linux/3.10.0-1160.31.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2023-01-18T15:35:02,534][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-01-18T15:35:02,535][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms8g, -Xmx8g, -Dlog4j2.formatMsgNoLookups=true, -XX:+UseG1GC, -Djava.io.tmpdir=/data1/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2023-01-18T15:35:05,488][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [aggs-matrix-stats]
[2023-01-18T15:35:05,488][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [analysis-common]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [constant-keyword]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [frozen-indices]
[2023-01-18T15:35:05,489][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [ingest-common]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [ingest-geoip]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [ingest-user-agent]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [kibana]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [lang-expression]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [lang-mustache]
[2023-01-18T15:35:05,490][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [lang-painless]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [mapper-extras]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [mapper-version]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [parent-join]
[2023-01-18T15:35:05,491][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [percolator]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [rank-eval]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [reindex]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [repositories-metering-api]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [repository-encrypted]
[2023-01-18T15:35:05,492][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [repository-url]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [runtime-fields-common]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [search-business-rules]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [searchable-snapshots]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [snapshot-repo-test-kit]
[2023-01-18T15:35:05,493][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [spatial]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [systemd]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [transform]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [transport-netty4]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [unsigned-long]
[2023-01-18T15:35:05,494][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [vectors]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [wildcard]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-aggregate-metric]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-analytics]
[2023-01-18T15:35:05,495][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-async]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-async-search]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-autoscaling]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-ccr]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-core]
[2023-01-18T15:35:05,496][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-data-streams]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-deprecation]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-enrich]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-eql]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-fleet]
[2023-01-18T15:35:05,497][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-graph]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-identity-provider]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-ilm]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-logstash]
[2023-01-18T15:35:05,498][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-ml]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-monitoring]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-ql]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-rollup]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-security]
[2023-01-18T15:35:05,499][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-shutdown]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-sql]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-stack]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-text-structure]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-voting-only-node]
[2023-01-18T15:35:05,500][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] loaded module [x-pack-watcher]
[2023-01-18T15:35:05,501][INFO ][o.e.p.PluginsService     ] [AZDTEELSPRDAPP01] no plugins loaded
[2023-01-18T15:35:05,545][INFO ][o.e.e.NodeEnvironment    ] [AZDTEELSPRDAPP01] using [1] data paths, mounts [[/data1 (/dev/sdb1)]], net usable_space [433.1gb], net total_space [1023.8gb], types [xfs]
[2023-01-18T15:35:05,545][INFO ][o.e.e.NodeEnvironment    ] [AZDTEELSPRDAPP01] heap size [8gb], compressed ordinary object pointers [true]
[2023-01-18T15:35:07,764][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] node name [AZDTEELSPRDAPP01], node ID [zxP3dDsaTpixfMPWq2cDcA], cluster name [dte-elasticsearch-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-01-18T15:35:12,736][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [AZDTEELSPRDAPP01] [controller/26488] [Main.cc@117] controller (64 bit): Version 7.13.3 (Build d74de47cfda9c3) Copyright (c) 2021 Elasticsearch BV
[2023-01-18T15:35:13,128][INFO ][o.e.x.s.a.s.FileRolesStore] [AZDTEELSPRDAPP01] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2023-01-18T15:35:13,935][INFO ][o.e.i.g.LocalDatabases   ] [AZDTEELSPRDAPP01] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2023-01-18T15:35:13,935][INFO ][o.e.i.g.DatabaseRegistry ] [AZDTEELSPRDAPP01] initialized database registry, using geoip-databases directory [/data1/elasticsearch/tmp/geoip-databases/zxP3dDsaTpixfMPWq2cDcA]
[2023-01-18T15:35:14,565][INFO ][o.e.t.NettyAllocator     ] [AZDTEELSPRDAPP01] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-01-18T15:35:14,633][INFO ][o.e.d.DiscoveryModule    ] [AZDTEELSPRDAPP01] using discovery type [zen] and seed hosts providers [settings]
[2023-01-18T15:35:15,098][INFO ][o.e.g.DanglingIndicesState] [AZDTEELSPRDAPP01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-01-18T15:35:15,518][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] initialized
[2023-01-18T15:35:15,519][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] starting ...
[2023-01-18T15:35:15,588][INFO ][o.e.x.s.c.f.PersistentCache] [AZDTEELSPRDAPP01] persistent cache index loaded
[2023-01-18T15:35:15,748][INFO ][o.e.t.TransportService   ] [AZDTEELSPRDAPP01] publish_address {10.8.26.4:9300}, bound_addresses {10.8.26.4:9300}
[2023-01-18T15:35:18,673][INFO ][o.e.b.BootstrapChecks    ] [AZDTEELSPRDAPP01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-18T15:35:18,676][INFO ][o.e.c.c.Coordinator      ] [AZDTEELSPRDAPP01] cluster UUID [2FxEMIyPSoagAyX5lnSqwA]
[2023-01-18T15:35:20,802][INFO ][o.e.c.s.ClusterApplierService] [AZDTEELSPRDAPP01] master node changed {previous [], current [{AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}]}, added {{AZDTEELSPRDAPP02}{SXXq1aqwRJurELS8kXPGBA}{Lwl1dRXrQwuFlHoR7mpf-A}{10.8.26.5}{10.8.26.5:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP03}{vtpTm3N7RkSNLWT15ozMAA}{1RivTho2RU6fR2e-PfAixg}{10.8.26.6}{10.8.26.6:9300}{cdfhilmrstw}, {AZDTEELSPRDAPP04}{UH9qGNqrSZG5Ef4MDlyG1g}{yPyw6TkkQJ26qrnlLmqVWg}{10.8.26.8}{10.8.26.8:9300}{cdfhilmrstw}}, term: 43, version: 88679, reason: ApplyCommitRequest{term=43, version=88679, sourceNode={AZDTEELSPRDAPP05}{AbEUim-nT7Kd1BMAN3h_BA}{aQYBuuyxSJWWq1syBz3mqA}{10.8.26.19}{10.8.26.19:9300}{cdfhilmrstw}{ml.machine_memory=16825581568, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8589934592, transform.node=true}}
[2023-01-18T15:35:22,074][INFO ][o.e.x.s.a.TokenService   ] [AZDTEELSPRDAPP01] refresh keys
[2023-01-18T15:35:22,334][INFO ][o.e.x.s.a.TokenService   ] [AZDTEELSPRDAPP01] refreshed keys
[2023-01-18T15:35:22,540][INFO ][o.e.l.LicenseService     ] [AZDTEELSPRDAPP01] license [78f4d2ba-5ca8-4a95-b188-122be6438524] mode [basic] - valid
[2023-01-18T15:35:22,542][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [AZDTEELSPRDAPP01] Active license is now [BASIC]; Security is enabled
[2023-01-18T15:35:22,556][INFO ][o.e.h.AbstractHttpServerTransport] [AZDTEELSPRDAPP01] publish_address {10.8.26.4:9200}, bound_addresses {10.8.26.4:9200}
[2023-01-18T15:35:22,557][INFO ][o.e.n.Node               ] [AZDTEELSPRDAPP01] started

Below are Linux server logs to the issue timestamp /var/log/messages

Jan 15 15:34:17 AZDTEELSPRDAPP01 trace-agent: 2023-01-15 15:34:17 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 15 15:34:17 AZDTEELSPRDAPP01 trace-agent: 2023-01-15 15:34:17 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 16 15:34:07 AZDTEELSPRDAPP01 trace-agent: 2023-01-16 15:34:07 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 16 15:34:07 AZDTEELSPRDAPP01 trace-agent: 2023-01-16 15:34:07 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 17 15:34:21 AZDTEELSPRDAPP01 trace-agent: 2023-01-17 15:34:21 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 17 15:34:21 AZDTEELSPRDAPP01 trace-agent: 2023-01-17 15:34:21 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:11 AZDTEELSPRDAPP01 trace-agent: 2023-01-18 15:34:11 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:11 AZDTEELSPRDAPP01 trace-agent: 2023-01-18 15:34:11 +08 | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Jan 18 15:34:30 AZDTEELSPRDAPP01 polkitd[835]: Registered Authentication Agent for unix-process:26086:9829187 (system bus name :1.5382 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jan 18 15:34:30 AZDTEELSPRDAPP01 systemd: Stopping Kibana...
Jan 18 15:34:30 AZDTEELSPRDAPP01 systemd: Stopping Kibana...
Jan 18 15:34:39 AZDTEELSPRDAPP01 systemd: Stopped Kibana.
Jan 18 15:34:39 AZDTEELSPRDAPP01 systemd: Stopped Kibana.
Jan 18 15:34:39 AZDTEELSPRDAPP01 polkitd[835]: Unregistered Authentication Agent for unix-process:26086:9829187 (system bus name :1.5382, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jan 18 15:34:48 AZDTEELSPRDAPP01 polkitd[835]: Registered Authentication Agent for unix-process:26173:9830981 (system bus name :1.5384 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jan 18 15:34:48 AZDTEELSPRDAPP01 systemd: Stopping Elasticsearch...
Jan 18 15:34:48 AZDTEELSPRDAPP01 systemd: Stopping Elasticsearch...
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Stopped Elasticsearch.
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Stopped Elasticsearch.
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Starting Elasticsearch...
Jan 18 15:34:58 AZDTEELSPRDAPP01 systemd: Starting Elasticsearch...

The Elasticsearch log shows that it start and doesn't stop, so it's not clear why systemd things it's stopping and starting it.

Please note though that 7.13 will soon be EOL and you should be looking to upgrade.