Nodes constantly turn off and on again

RusseL · December 7, 2020, 7:22am

Hi guys, I have a 14 nodes of cluster (11 data, 3 master).I am using Elasticsearch and kibana version 7.4.2. When everything is going well, suddenly the nodes started to shut down and the kibana status turns red or yellow. When I restart the services on the shut down nodes, the status turns green again. But I experience this at least twice a day. My system team said that there is no hardware-related situation. Below I am sharing the log log of the cluster.log and the closed node. I hope you can help.

github.com

Resul-Hasturk/Elasticsearch/blob/main/clusterlog.txt

[2020-12-07T09:20:21,267][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:20:26,442][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-3] collector [index-stats] timed out when collecting data
[2020-12-07T09:20:51,281][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:21:22,241][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:21:53,060][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:22:24,672][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:22:47,189][WARN ][o.e.t.TransportService   ] [master-3] Received response for a request that has timed out, sent [12807ms] ago, timed out [2802ms] ago, action [internal:coordination/fault_detection/follower_check], node [{data-1}{_BUmAFaYTGi-6aaYnPlqvw}{R8bdK4JlRVK7CpqNOk4Qeg}{192.168.3.68}{192.168.3.68:9300}{dil}{ml.machine_memory=270073581568, ml.max_open_jobs=20, xpack.installed=true, disk_type=ssd}], id [331367427]
[2020-12-07T09:23:05,635][WARN ][o.e.t.TransportService   ] [master-3] Received response for a request that has timed out, sent [17609ms] ago, timed out [7604ms] ago, action [internal:coordination/fault_detection/follower_check], node [{data-1}{_BUmAFaYTGi-6aaYnPlqvw}{R8bdK4JlRVK7CpqNOk4Qeg}{192.168.3.68}{192.168.3.68:9300}{dil}{ml.machine_memory=270073581568, ml.max_open_jobs=20, xpack.installed=true, disk_type=ssd}], id [331367651]
[2020-12-07T09:23:09,673][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [master-3] failed to execute on node [_BUmAFaYTGi-6aaYnPlqvw]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [data-1][192.168.3.68:9300][cluster:monitor/nodes/stats[n]] request_id [331367733] timed out after [15008ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1022) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.2.jar:7.4.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-12-07T09:23:24,674][WARN ][o.e.c.InternalClusterInfoService] [master-3] Failed to update shard information for ClusterInfoUpdateJob within 15s timeout
[2020-12-07T09:23:24,675][WARN ][o.e.c.r.a.DiskThresholdMonitor] [master-3] flood stage disk watermark [20gb] exceeded on [hcPCtDaERsWlwXUdl5vmiw][data-11][/var/lib/elasticsearch/nodes/0] free: 2.3gb[48.3%], all indices on this node will be marked read-only
[2020-12-07T09:23:39,784][WARN ][o.e.t.TransportService   ] [master-3] Received response for a request that has timed out, sent [22012ms] ago, timed out [12006ms] ago, action [internal:coordination/fault_detection/follower_check], node [{data-1}{_BUmAFaYTGi-6aaYnPlqvw}{R8bdK4JlRVK7CpqNOk4Qeg}{192.168.3.68}{192.168.3.68:9300}{dil}{ml.machine_memory=270073581568, ml.max_open_jobs=20, xpack.installed=true, disk_type=ssd}], id [331368118]
[2020-12-07T09:23:39,784][WARN ][o.e.t.TransportService   ] [master-3] Received response for a request that has timed out, sent [11006ms] ago, timed out [1001ms] ago, action [internal:coordination/fault_detection/follower_check], node [{data-1}{_BUmAFaYTGi-6aaYnPlqvw}{R8bdK4JlRVK7CpqNOk4Qeg}{192.168.3.68}{192.168.3.68:9300}{dil}{ml.machine_memory=270073581568, ml.max_open_jobs=20, xpack.installed=true, disk_type=ssd}], id [331368274]
[2020-12-07T09:24:09,677][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [master-3] failed to execute on node [_BUmAFaYTGi-6aaYnPlqvw]

This file has been truncated. show original

github.com

Resul-Hasturk/Elasticsearch/blob/main/nodelog.txt

[2020-12-06T21:30:31,700][WARN ][o.e.c.s.ClusterApplierService] [data-1] cluster state applier task [ApplyCommitRequest{term=13, version=721312, sourceNode={master-3}{wHJJAON-TS-9uMYtbisMig}{WYIoYMs3QhS3sVGCi0_Ukw}
{192.168.3.59}{192.168.3.59:9300}{m}{xpack.installed=true, disk_type=ssd}}] took [59s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=13, version=721312,
 sourceNode={master-3}{wHJJAON-TS-9uMYtbisMig}{WYIoYMs3QhS3sVGCi0_Ukw}{192.168.3.59}{192.168.3.59:9300}{m}{xpack.installed=true, disk_type=ssd}}]] took [0ms], [connecting to new nodes] took [0ms], [applying settings] took [0ms],
 [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@3ae7fb08]] took [59119ms], [running applier [org.elasticsearch.script.ScriptService@6d46732f]] took [0ms],
 [running applier [org.elasticsearch.xpack.ilm.IndexLifecycleService@69db2d8d]] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@21617839]] took [0ms],
 [running applier [org.elasticsearch.snapshots.RestoreService@68bd683]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@588a0276]] took [0ms],
 [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@2c988be8]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$3560/0x00000008015bf840@1498e774]] took [0ms],
 [running applier [org.elasticsearch.tasks.TaskManager@6f28cc35]] took [0ms], [running applier [org.elasticsearch.gateway.GatewayMetaState@5fefa68d]] took [6ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@45f02322]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.security.support.SecurityIndexManager@fe24583]] took [0ms], [notifying listener [org.elasticsearch.xpack.security.support.SecurityIndexManager@67bb5a6f]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.security.authc.TokenService$$Lambda$1978/0x000000080120cc40@75eb3f3d]] took [0ms], [notifying listener [org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$$Lambda$2066/0x000000080124b440@3cd55de6]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.watcher.support.WatcherIndexTemplateRegistry@1001031]] took [0ms], [notifying listener [org.elasticsearch.xpack.watcher.WatcherLifeCycleService@2dbec5fb]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.watcher.WatcherIndexingListener@6de524]] took [0ms], [notifying listener [org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager@1501bade]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.ml.datafeed.DatafeedManager$TaskRunner@1b3b7b6b]] took [0ms], [notifying listener [org.elasticsearch.xpack.ml.MlInitializationService@2c64f03b]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.ml.MlAssignmentNotifier@7c08a8fb]] took [0ms], [notifying listener [org.elasticsearch.xpack.ilm.IndexLifecycleService@69db2d8d]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.core.slm.history.SnapshotLifecycleTemplateRegistry@c08eb1d]] took [0ms], [notifying listener [org.elasticsearch.xpack.slm.SnapshotLifecycleService@4ca829d4]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.ccr.action.ShardFollowTaskCleaner@4746ed52]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@7eb0da31]] took [0ms],
 [notifying listener [org.elasticsearch.node.ResponseCollectorService@35ab96f9]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@3840ef11]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.ml.action.TransportOpenJobAction$OpenJobPersistentTasksExecutor$$Lambda$3113/0x00000008014c4840@481ca936]] took [0ms],
 [notifying listener [org.elasticsearch.xpack.ml.action.TransportStartDataFrameAnalyticsAction$TaskExecutor$$Lambda$3117/0x00000008014c5840@30058356]] took [0ms], 
[notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@59f0d7f1]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@667783b9]] took [0ms],

This file has been truncated. show original

Christian_Dahlqvist · December 7, 2020, 7:32am

It looks like you are running low on disk space causing indices to be set to read-only. What is the full output of the cluster stats API?

RusseL · December 7, 2020, 7:42am

when i run

GET /cluster_stats

github.com

Resul-Hasturk/Elasticsearch/blob/main/_cluster stats

{
  "_nodes" : {
    "total" : 14,
    "successful" : 14,
    "failed" : 0
  },
  "cluster_name" : "escls01",
  "cluster_uuid" : "a14P_D4BQmaCeJalR-zU6w",
  "timestamp" : 1607326429417,
  "status" : "yellow",
  "indices" : {
    "count" : 102,
    "shards" : {
      "total" : 942,
      "primaries" : 472,
      "replication" : 0.9957627118644068,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 78,

This file has been truncated. show original

Christian_Dahlqvist · December 7, 2020, 7:54am

How is your cluster configured? How are you monitoring this cluster? Is this a hot/warm cluster?

What type of node is data-11, which seems to be running low on disk space?

RusseL · December 7, 2020, 8:20am

I took data-11 out of the cluster with the following code. It has 0 shard in it.

PUT cluster/settings 
{
"transient":{
"cluster.routing.allocation.exclude._ip":"192.168.3.72"    
}
}

And there is a continuous flow of data to the cluster.

I watch the cluster on 'Kibana 7.4.2'.

I don't fully understand this question. Is there a file you want me to share?

How is your cluster configured?

Maybe it will help

github.com

Resul-Hasturk/Elasticsearch/blob/main/cluster_info

-------------/etc/security/limits.conf-------------
elasticsearch   -       nofile          65535
elasticsearch   hard    nproc           16384

-------------/etc/sysctl.conf-------------
vm.max_map_count=262144

-------------elasticsearch.service-------------
[Service]
LimitMEMLOCK=infinity

-------------/etc/elasticsearch/jvm.options-------------
-Xms30g
-Xmx30g
## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
10-:-XX:-UseConcMarkSweepGC
10-:-XX:-UseCMSInitiatingOccupancyOnly
10-:-XX:+UseG1GC

This file has been truncated. show original

Christian_Dahlqvist · December 7, 2020, 8:27am

With respect to the question about how your cluster is configured I was wondering if all data nodes have the same specification or whether you are using a hot/warm architecture.

Can you share the logs from the nodes that shut down?

RusseL · December 9, 2020, 11:24am

Thank you for taking care of my problem. I removed the two nodes I added from the cluster. It took some time to recover. Everything is fine for now.

system · January 6, 2021, 11:24am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.