# Nodes constantly turn off and on again

**URL:** <https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808>\
**Category:** Elasticsearch\
**Created:** [December 7, 2020, 7:22am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808 "2020-12-07T07:22:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![RusseL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russel/32/77328_2.png) [@RusseL](https://discuss.elastic.co/u/RusseL)\
**Post date:** [December 7, 2020, 7:22am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/1 "2020-12-07T07:22:44Z")

</div>

Hi guys, I have a 14 nodes of cluster (11 data, 3 master).I am using Elasticsearch and kibana version 7.4.2. When everything is going well, suddenly the nodes started to shut down and the kibana status turns red or yellow. When I restart the services on the shut down nodes, the status turns green again. But I experience this at least twice a day. My system team said that there is no hardware-related situation. Below I am sharing the log log of the cluster.log and the closed node. I hope you can help.

> <https://github.com/Resul-Hasturk/Elasticsearch/blob/main/clusterlog.txt>

> <https://github.com/Resul-Hasturk/Elasticsearch/blob/main/nodelog.txt>

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 7, 2020, 7:32am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/2 "2020-12-07T07:32:49Z")

</div>

It looks like you are running low on disk space causing indices to be set to read-only. What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![RusseL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russel/32/77328_2.png) [@RusseL](https://discuss.elastic.co/u/RusseL)\
**Post date:** [December 7, 2020, 7:42am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/3 "2020-12-07T07:42:46Z")

</div>

when i run

```
GET /cluster_stats

```

> <https://github.com/Resul-Hasturk/Elasticsearch/blob/main/_cluster%20stats>

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 7, 2020, 7:54am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/4 "2020-12-07T07:54:47Z")

</div>

How is your cluster configured? How are you monitoring this cluster? Is this a hot/warm cluster?

What type of node is `data-11`, which seems to be running low on disk space?

---

<div class="post-metadata">

**Author:** ![RusseL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russel/32/77328_2.png) [@RusseL](https://discuss.elastic.co/u/RusseL)\
**Post date:** [December 7, 2020, 8:20am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/5 "2020-12-07T08:20:15Z")

</div>

I took data-11 out of the cluster with the following code. It has 0 shard in it.

```
PUT cluster/settings 
{
"transient":{
"cluster.routing.allocation.exclude._ip":"192.168.3.72"    
}
}

```

And there is a continuous flow of data to the cluster.

I watch the cluster on 'Kibana 7.4.2'.

I don't fully understand this question. Is there a file you want me to share?

```
How is your cluster configured?

```

Maybe it will help

> <https://github.com/Resul-Hasturk/Elasticsearch/blob/main/cluster_info>

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 7, 2020, 8:27am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/6 "2020-12-07T08:27:42Z")

</div>

With respect to the question about how your cluster is configured I was wondering if all data nodes have the same specification or whether you are using a hot/warm architecture.

Can you share the logs from the nodes that shut down?

---

<div class="post-metadata">

**Author:** ![RusseL](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russel/32/77328_2.png) [@RusseL](https://discuss.elastic.co/u/RusseL)\
**Post date:** [December 9, 2020, 11:24am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/7 "2020-12-09T11:24:04Z")

</div>

Thank you for taking care of my problem. I removed the two nodes I added from the cluster. It took some time to recover. Everything is fine for now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2021, 11:24am UTC](https://discuss.elastic.co/t/nodes-constantly-turn-off-and-on-again/257808/8 "2021-01-06T11:24:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.