Hello,
I'm running Elasticsearch 5.6 and when my Cluster became yellow I have these logs
In master:
[2020-05-15T14:50:57,908][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [node-elasticsearch_master_02] failed to execute on node [HlG_sniCQLO4KdpkV_EgjA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [node-elasticsearch_data_03][172.26.44.163:9300][cluster:monitor/nodes/stats[n]] request_id [42924423] timed out after [15001ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
[2020-05-15T14:51:49,057][INFO ][o.e.c.r.a.AllocationService] [node-elasticsearch_master_02] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
[2020-05-15T14:51:49,060][INFO ][o.e.c.s.ClusterService ] [node-elasticsearch_master_02] removed {{node-elasticsearch_data_03}{HlG_sniCQLO4KdpkV_EgjA}{BhBWwYheS8GrJmt0yvmAEQ}{172.26.44.163}{172.26.44.163:9300}{rack=rack-prod},}, reason: zen-disco-node-failed({node-elasticsearch_data_03}{HlG_sniCQLO4KdpkV_EgjA}{BhBWwYheS8GrJmt0yvmAEQ}{172.26.44.163}{172.26.44.163:9300}{rack=rack-prod}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)
[2020-05-15T14:51:51,434][WARN ][o.e.t.TransportService ] [node-elasticsearch_master_02] Received response for a request that has timed out, sent [68527ms] ago, timed out [53526ms] ago, action [
In data
[2020-05-15T14:49:37,709][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184550] overhead, spent [1.2m] collecting in the last [1.2m]
[2020-05-15T14:49:38,711][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184551] overhead, spent [718ms] collecting in the last [1s]
[2020-05-15T14:50:05,341][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184577] overhead, spent [1.3s] collecting in the last [1.5s]
[2020-05-15T14:50:14,464][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184586] overhead, spent [698ms] collecting in the last [1.1s]
[2020-05-15T14:51:51,485][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][old][184590][2267] duration [1.5m], collections [1]/[1.5m], total [1.5m]/[1.1h], memory [21.8gb]->[21.2gb]/[21.9gb], all_pools {[young] [498.9mb]->[72.5mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [21.2gb]->[21.1gb]/[21.3gb]}
[2020-05-15T14:51:51,485][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184590] overhead, spent [1.5m] collecting in the last [1.5m]
[2020-05-15T14:51:52,485][WARN ][o.e.m.j.JvmGcMonitorService] [node-elasticsearch_data_03] [gc][184591] overhead, spent [651ms] collecting in the last [1s]
[2020-05-15T14:51:52,601][WARN ][o.e.t.n.Netty4Transport ] [node-elasticsearch_data_03] write and flush on the network layer failed (channel: [id: 0x6924cae3, L:/172.26.44.163:9300 ! R:/172.26.44.169:49866])
java.nio.channels.ClosedChannelException: null
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
Any idea?
Many thanks
