Nodes constantly turn off and on again

Hi guys, I have a 14 nodes of cluster (11 data, 3 master).I am using Elasticsearch and kibana version 7.4.2. When everything is going well, suddenly the nodes started to shut down and the kibana status turns red or yellow. When I restart the services on the shut down nodes, the status turns green again. But I experience this at least twice a day. My system team said that there is no hardware-related situation. Below I am sharing the log log of the cluster.log and the closed node. I hope you can help.

It looks like you are running low on disk space causing indices to be set to read-only. What is the full output of the cluster stats API?

when i run

GET /cluster_stats

How is your cluster configured? How are you monitoring this cluster? Is this a hot/warm cluster?

What type of node is data-11, which seems to be running low on disk space?

I took data-11 out of the cluster with the following code. It has 0 shard in it.

PUT cluster/settings 

And there is a continuous flow of data to the cluster.

I watch the cluster on 'Kibana 7.4.2'.

I don't fully understand this question. Is there a file you want me to share?

How is your cluster configured?

Maybe it will help

With respect to the question about how your cluster is configured I was wondering if all data nodes have the same specification or whether you are using a hot/warm architecture.

Can you share the logs from the nodes that shut down?

Thank you for taking care of my problem. I removed the two nodes I added from the cluster. It took some time to recover. Everything is fine for now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.