Elasticsearh Error : Pods not coming back up

Elastic Stack Elasticsearch
1

Hi
We have Elasticsearch running on openshift.

Image: docker.elastic.co/elasticsearch/elasticsearch:6.8.15

All of a sudden on the the pods went down and are not restarting. Trying to figure out what the cause is.
Log files are below . Any help to figure out the issue would be appreciated.

TIA

[2022-01-03T19:04:54,174][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-0] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: 
   {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{K4JeVu6PRVyxWYVEoSBkoQ}{10.130.3.17}{10.130.3.17:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, local
   {elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, master

[2022-01-03T19:04:59,213][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][4361] overhead, spent [2.8s] collecting in the last [3.6s]
[2022-01-03T19:04:59,214][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{K4JeVu6PRVyxWYVEoSBkoQ}{10.130.3.17}{10.130.3.17:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=2369}]], but needed [2]), pinging again
[2022-01-03T19:04:59,219][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152357] timed out after [5050ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:04:59,219][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152360] timed out after [4050ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:04:59,928][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152361] timed out after [3649ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:01,638][WARN ][o.e.t.TransportService   ] [elasticsearch-master-0] Received response for a request that has timed out, sent [7476ms] ago, timed out [2426ms] ago, action [internal:discovery/zen/unicast], node [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [152357]
[2022-01-03T19:05:02,215][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-0] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{K4JeVu6PRVyxWYVEoSBkoQ}{10.130.3.17}{10.130.3.17:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=2369}]], but needed [2]), pinging again
[2022-01-03T19:05:02,965][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152364] timed out after [3627ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:03,965][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152366] timed out after [3626ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:04,965][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [152369] timed out after [3627ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:05,045][WARN ][o.e.t.TransportService   ] [elasticsearch-master-0] Received response for a request that has timed out, sent [9878ms] ago, timed out [5828ms] ago, action [internal:discovery/zen/unicast], node [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [152360]
[2022-01-03T19:05:06,967][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [153110] timed out after [3801ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:07,969][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-master-0] failed to send ping to [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-1][10.131.2.43:9300][internal:discovery/zen/unicast] request_id [153111] timed out after [3802ms]
	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1028) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:08,631][WARN ][o.e.t.TransportService   ] [elasticsearch-master-0] Received response for a request that has timed out, sent [5402ms] ago, timed out [1601ms] ago, action [internal:discovery/zen/unicast], node [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [153110]
[2022-01-03T19:05:08,631][WARN ][o.e.t.TransportService   ] [elasticsearch-master-0] Received response for a request that has timed out, sent [8228ms] ago, timed out [4602ms] ago, action [internal:discovery/zen/unicast], node [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], id [152366]
[2022-01-03T19:05:27,567][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:28,665][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:36,969][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:40,112][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][4399] overhead, spent [2.8s] collecting in the last [3.8s]
[2022-01-03T19:05:46,668][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:55,253][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:191) ~[elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:479) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:357) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:513) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:05:56,903][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:262) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:564) [elasticsearch-6.8.15.jar:6.8.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:708) [elasticsearch-6.8.15.jar:6.8.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-03T19:06:05,218][INFO ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-0] failed to send join request to master [{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{unPMtAz-QEWBDCCKh3CgXQ}{10.131.2.43}{10.131.2.43:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [ElasticsearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.]; ]
[2022-01-03T19:06:06,819][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at ..
..
2

Hi
Just checking to see if anyone has any thoughts.

3

Hello,

Do you have enough free space in all of your nodes?

Also, share your configurations files so people can try to understand how your cluster works, and logs from the other node.

4

Yes there is free space .

Node1:

[2022-01-07T19:11:18,047][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (172.18.35.129:vol_d40f39186a226377a87ad816a9cb94a2)]], net usable_space [16.4gb], net total_space [29.9gb], types [fuse.glusterfs]
[2022-01-07T19:11:18,049][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] heap size [1gb], compressed ordinary object pointers [true]
[2022-01-07T19:11:49,215][INFO ][o.e.n.Node               ] [elasticsearch-master-1] node name [elasticsearch-master-1], node ID [lAEQgI8UQuia5tPmB-WOCQ]
[2022-01-07T19:11:49,217][INFO ][o.e.n.Node               ] [elasticsearch-master-1] version[6.8.15], pid[1], build[default/docker/c9a8c60/2021-03-18T06:33:32.588487Z], OS[Linux/3.10.0-1160.49.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-01-07T19:11:49,218][INFO ][o.e.n.Node               ] [elasticsearch-master-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-1325547667723646708, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -Dlog4j2.formatMsgNoLookups=true, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [aggs-matrix-stats]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [analysis-common]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-common]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-geoip]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-user-agent]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-expression]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-mustache]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-painless]
[2022-01-07T19:11:53,961][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [mapper-extras]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [parent-join]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [percolator]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [rank-eval]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [reindex]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [repository-url]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [transport-netty4]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [tribe]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-ccr]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-core]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-deprecation]
[2022-01-07T19:11:53,962][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-graph]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-ilm]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-logstash]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-ml]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-monitoring]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-rollup]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-security]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-sql]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-upgrade]
[2022-01-07T19:11:53,963][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [x-pack-watcher]
[2022-01-07T19:11:53,964][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] no plugins loaded
[2022-01-07T19:12:02,870][INFO ][o.e.x.s.a.s.FileRolesStore] [elasticsearch-master-1] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2022-01-07T19:12:04,188][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elasticsearch-master-1] [controller/80] [Main.cc@114] controller (64 bit): Version 6.8.15 (Build 4cd69e3d9d9390) Copyright (c) 2021 Elasticsearch BV
[2022-01-07T19:12:10,550][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-1] using discovery type [zen] and host providers [settings]
[2022-01-07T19:12:12,268][INFO ][o.e.n.Node               ] [elasticsearch-master-1] initialized
[2022-01-07T19:12:12,269][INFO ][o.e.n.Node               ] [elasticsearch-master-1] starting ...
[2022-01-07T19:12:12,557][INFO ][o.e.t.TransportService   ] [elasticsearch-master-1] publish_address {10.131.3.171:9300}, bound_addresses {0.0.0.0:9300}
[2022-01-07T19:12:14,161][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-01-07T19:12:17,331][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] detected_master {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{elasticsearch-master-2}{esaH2xs6TxysGV1U8ygwYQ}{kLp8HNLGTGGRwowTLbm3Kw}{10.130.3.235}{10.130.3.235:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},{elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [2]])
[2022-01-07T19:12:17,439][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch-master-1] publish_address {10.131.3.171:9200}, bound_addresses {0.0.0.0:9200}
[2022-01-07T19:12:17,439][INFO ][o.e.n.Node               ] [elasticsearch-master-1] started
[2022-01-07T19:12:34,522][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [elasticsearch-master-1] Failed to clear cache for realms [[]]
[2022-01-07T19:12:34,600][INFO ][o.e.l.LicenseService     ] [elasticsearch-master-1] license [ed7e96b2-abec-4937-be06-39175d2e4822] mode [basic] - valid
[2022-01-07T19:12:44,821][INFO ][o.e.x.s.a.TokenService   ] [elasticsearch-master-1] refresh keys
[2022-01-07T19:12:45,084][INFO ][o.e.x.s.a.TokenService   ] [elasticsearch-master-1] refreshed keys
[2022-01-07T19:12:46,058][WARN ][o.e.d.i.m.MapperService  ] [elasticsearch-master-1] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2022-01-07T19:12:46,146][WARN ][o.e.d.i.m.AllFieldMapper ] [elasticsearch-master-1] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
[2022-01-07T19:12:46,160][WARN ][o.e.d.i.m.AllFieldMapper ] [elasticsearch-master-1] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
5

Node 2:

common]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-geoip]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-user-agent]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-expression]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-mustache]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-painless]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [mapper-extras]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [parent-join]
[2022-01-07T19:11:50,794][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [percolator]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [rank-eval]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [reindex]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [repository-url]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [transport-netty4]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [tribe]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-ccr]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-core]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-deprecation]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-graph]
[2022-01-07T19:11:50,795][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-ilm]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-logstash]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-ml]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-monitoring]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-rollup]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-security]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-sql]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-upgrade]
[2022-01-07T19:11:50,796][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [x-pack-watcher]
[2022-01-07T19:11:50,797][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] no plugins loaded
[2022-01-07T19:11:59,870][INFO ][o.e.x.s.a.s.FileRolesStore] [elasticsearch-master-2] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2022-01-07T19:12:01,261][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elasticsearch-master-2] [controller/80] [Main.cc@114] controller (64 bit): Version 6.8.15 (Build 4cd69e3d9d9390) Copyright (c) 2021 Elasticsearch BV
[2022-01-07T19:12:07,593][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-2] using discovery type [zen] and host providers [settings]
[2022-01-07T19:12:09,373][INFO ][o.e.n.Node               ] [elasticsearch-master-2] initialized
[2022-01-07T19:12:09,373][INFO ][o.e.n.Node               ] [elasticsearch-master-2] starting ...
[2022-01-07T19:12:09,597][INFO ][o.e.t.TransportService   ] [elasticsearch-master-2] publish_address {10.130.3.235:9300}, bound_addresses {0.0.0.0:9300}
[2022-01-07T19:12:11,406][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-01-07T19:12:15,856][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] detected_master {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [1]])
[2022-01-07T19:12:15,883][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch-master-2] publish_address {10.130.3.235:9200}, bound_addresses {0.0.0.0:9200}
[2022-01-07T19:12:15,883][INFO ][o.e.n.Node               ] [elasticsearch-master-2] started
[2022-01-07T19:12:17,362][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-master-1}{lAEQgI8UQuia5tPmB-WOCQ}{g1lwx0wMRGqyRD_4CN9FwQ}{10.131.3.171}{10.131.3.171:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [2]])
[2022-01-07T19:12:17,375][INFO ][o.e.x.m.e.l.LocalExporter] [elasticsearch-master-2] waiting for elected master node [{elasticsearch-master-0}{dvu34af4SEmyF6dIvNwaPw}{hmX8aT_4T7KoSvDrWe2esQ}{10.129.2.126}{10.129.2.126:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2022-01-07T19:12:35,704][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [elasticsearch-master-2] Failed to clear cache for realms [[]]
[2022-01-07T19:12:35,771][INFO ][o.e.l.LicenseService     ] [elasticsearch-master-2] license [ed7e96b2-abec-4937-be06-39175d2e4822] mode [basic] - valid
[2022-01-07T19:12:44,812][INFO ][o.e.x.s.a.TokenService   ] [elasticsearch-master-2] refresh keys
[2022-01-07T19:12:45,068][INFO ][o.e.x.s.a.TokenService   ] [elasticsearch-master-2] refreshed keys
[2022-01-07T19:12:45,969][WARN ][o.e.d.i.m.MapperService  ] [elasticsearch-master-2] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2022-01-07T19:12:46,067][WARN ][o.e.d.i.m.AllFieldMapper ] [elasticsearch-master-2] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
[2022-01-07T19:12:46,080][WARN ][o.e.d.i.m.AllFieldMapper ] [elasticsearch-master-2] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a
6

Config:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  namespace: adsd-cumulus-prod-db
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch-master
  serviceName: elasticsearch-master-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch-master
        chart: elasticsearch
        release: elasticsearch
      name: elasticsearch-master
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - elasticsearch-master
              topologyKey: kubernetes.io/hostname
      containers:
        - env:
            - name: node.name
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: discovery.zen.minimum_master_nodes
              value: '2'
            - name: discovery.zen.ping.unicast.hosts
              value: elasticsearch-master-headless
            - name: cluster.name
              value: elasticsearch
            - name: network.host
              value: 0.0.0.0
            - name: ES_JAVA_OPTS
              value: '-Xmx1g -Xms1g -Dlog4j2.formatMsgNoLookups=true'
            - name: node.data
              value: 'true'
            - name: node.ingest
              value: 'true'
            - name: node.master
              value: 'true'
          image: 'docker.elastic.co/elasticsearch/elasticsearch:6.8.15'
          imagePullPolicy: IfNotPresent
          name: elasticsearch
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
          readinessProbe:
            exec:
              command:
                - sh
                - '-c'
                - >
                  #!/usr/bin/env bash -e

                  # If the node is starting up wait for the cluster to be ready
                  (request params: "wait_for_status=green&timeout=1s" )

                  # Once it has started only check that the node itself is
                  responding

                  START_FILE=/tmp/.es_start_file


                  # Disable nss cache to avoid filling dentry cache when calling
                  curl

                  # This is required with Elasticsearch Docker using nss < 3.52

                  export NSS_SDB_USE_CACHE=no


                  http () {
                    local path="${1}"
                    local args="${2}"
                    set -- -XGET -s

                    if [ "$args" != "" ]; then
                      set -- "$@" $args
                    fi

                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    fi

                    curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"
                  }


                  if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    HTTP_CODE=$(http "/" "-w %{http_code}")
                    RC=$?
                    if [[ ${RC} -ne 0 ]]; then
                      echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}"
                      exit ${RC}
                    fi
                    # ready if HTTP code 200, 503 is tolerable if ES version is 6.x
                    if [[ ${HTTP_CODE} == "200" ]]; then
                      exit 0
                    elif [[ ${HTTP_CODE} == "503" && "6" == "6" ]]; then
                      exit 0
                    else
                      echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
                      exit 1
                    fi

                  else
                    echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
                      touch ${START_FILE}
                      exit 0
                    else
                      echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                      exit 1
                    fi
                  fi
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 3
            timeoutSeconds: 5
          resources:
            limits:
              cpu: '1'
              memory: 2Gi
            requests:
              cpu: '1'
              memory: 2Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/data
              name: elasticsearch-master
            - mountPath: /db-backup
              name: db-backup
            - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              name: esconfig
              subPath: elasticsearch.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 120
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: elasticsearch-master
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 30Gi
7

If you do a reboot, Elasticsearch comes back up After 30 minutes or so it crashes.

Ran this command and get

sh-4.2$ curl -XGET localhost:9200/_cluster/allocation/explain?pretty
{
  "index" : "jaeger-service-2021-10-10",
  "shard" : 2,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2022-01-07T19:12:22.800Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "throttled",
  "allocate_explanation" : "allocation temporarily throttled",
  "node_allocation_decisions" : [
    {
      "node_id" : "dvu34af4SEmyF6dIvNwaPw",
      "node_name" : "elasticsearch-master-0",
      "transport_address" : "10.129.2.126:9300",
      "node_attributes" : {
        "ml.machine_memory" : "2147483648",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      },
      "node_decision" : "throttled",
      "deciders" : [
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    },
    {
      "node_id" : "esaH2xs6TxysGV1U8ygwYQ",
      "node_name" : "elasticsearch-master-2",
      "transport_address" : "10.130.3.235:9300",
      "node_attributes" : {
        "ml.machine_memory" : "2147483648",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      },
      "node_decision" : "throttled",
      "store" : {
        "matching_sync_id" : true
      },
      "deciders" : [
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    },
    {
      "node_id" : "lAEQgI8UQuia5tPmB-WOCQ",
      "node_name" : "elasticsearch-master-1",
      "transport_address" : "10.131.3.171:9300",
      "node_attributes" : {
        "ml.machine_memory" : "2147483648",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "matching_sync_id" : true
      },
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[jaeger-service-2021-10-10][2], node[lAEQgI8UQuia5tPmB-WOCQ], [P], s[STARTED], a[id=7A95E9cyShWmdEEI_u3kiA]]"
        },
        {
          "decider" : "throttling",
          "decision" : "THROTTLE",
          "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
        }
      ]
    }
  ]
}
sh-4.2$
8 
[2022-01-07T20:08:20,695][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2922] overhead, spent [3.1s] collecting in the last [3.1s]
[2022-01-07T20:08:24,023][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2923] overhead, spent [3.2s] collecting in the last [3.3s]
[2022-01-07T20:08:27,495][WARN ][o.e.c.InternalClusterInfoService] [elasticsearch-master-0] Failed to update shard information for ClusterInfoUpdateJob within 15s timeout
[2022-01-07T20:08:27,496][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2924] overhead, spent [3.3s] collecting in the last [3.4s]
[2022-01-07T20:08:31,345][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2925] overhead, spent [3.8s] collecting in the last [3.8s]
[2022-01-07T20:08:35,131][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2926] overhead, spent [3.6s] collecting in the last [3.7s]
[2022-01-07T20:08:38,705][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2927] overhead, spent [3.4s] collecting in the last [3.5s]
[2022-01-07T20:08:42,199][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2928] overhead, spent [3.4s] collecting in the last [3.4s]
[2022-01-07T20:08:45,534][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2929] overhead, spent [3.2s] collecting in the last [3.3s]
[2022-01-07T20:08:48,800][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2930] overhead, spent [3.2s] collecting in the last [3.2s]
[2022-01-07T20:08:51,919][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2931] overhead, spent [3s] collecting in the last [3.1s]
[2022-01-07T20:08:55,416][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2932] overhead, spent [3.4s] collecting in the last [3.4s]
[2022-01-07T20:09:01,430][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2933] overhead, spent [5.9s] collecting in the last [6s]
[2022-01-07T20:09:04,509][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2934] overhead, spent [3s] collecting in the last [3s]
[2022-01-07T20:09:29,123][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2935] overhead, spent [24.5s] collecting in the last [24.6s]
[2022-01-07T20:09:32,704][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2936] overhead, spent [3.5s] collecting in the last [3.5s]
[2022-01-07T20:09:40,043][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2937] overhead, spent [7.3s] collecting in the last [3.7s]
[2022-01-07T20:09:46,912][WARN ][o.e.c.InternalClusterInfoService] [elasticsearch-master-0] Failed to update node information for ClusterInfoUpdateJob within 15s timeout
[2022-01-07T20:10:03,746][WARN ][o.e.c.InternalClusterInfoService] [elasticsearch-master-0] Failed to update shard information for ClusterInfoUpdateJob within 15s timeout
[2022-01-07T20:10:44,113][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2938] overhead, spent [1m] collecting in the last [1.1m]
[2022-01-07T20:12:39,783][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2939] overhead, spent [1.9m] collecting in the last [1.8m]
[2022-01-07T20:14:43,563][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2940] overhead, spent [2m] collecting in the last [2m]
[2022-01-07T20:15:07,298][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][2941] overhead, spent [19.7s] collecting in the last [19.4s]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to data/java_pid1.hprof ...
Unable to create data/java_pid1.hprof: File exists
[2022-01-07T20:34:01,021][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] fatal error in thread [elasticsearch[elasticsearch-master-0][management][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-07T20:34:45,791][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] fatal error in thread [elasticsearch[elasticsearch-master-0][generic][T#90]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-07T20:36:06,723][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] fatal error in thread [elasticsearch[elasticsearch-master-0][generic][T#126]], exiting
java.lang.OutOfMemoryError: Java heap space
[2022-01-07T20:36:09,801][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] fatal error in thread [elasticsearch[elasticsearch-master-0][generic][T#111]], exiting
java.lang.OutOfMemoryError: Java heap space
9

It seems that you are getting Out of Memory errors on some of your nodes:

[2022-01-07T20:34:01,021][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-0] fatal error in thread [elasticsearch[elasticsearch-master-0][management][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space

Can you try to increase the HEAP? I would say that 1 GB is too small for your use case.

Change the value of Xmx and Xms.

            - name: ES_JAVA_OPTS
              value: '-Xmx1g -Xms1g -Dlog4j2.formatMsgNoLookups=true'
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.