# One Master Node removes and adds cluster nodes on regular base
**URL:** https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700
**Category:** Elasticsearch
**Created:** [November 14, 2018, 3:02pm UTC](https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700 "2018-11-14T15:02:07Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Thomas\_Tomski](https://discuss.elastic.co/u/Thomas_Tomski)
#### Post date: [November 14, 2018, 3:02pm UTC](https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700/1 "2018-11-14T15:02:07Z")
Hello,
I have an ELS Cluster running with 8 Data nodes and 3 Master only Nodes.
One of these Master nodes remove and adds cluster members on regular base.
Here is an extract from the log file:
Here, the Node gets removed:
`[2018-11-14T15:34:07,372][INFO][o.e.c.s.ClusterApplierService] [logstash01.mgm.xxx] removed {{elasticsearch03.mgm.xxx}{nldBGzbgQ62AV6k29wG2dQ}{zc8HBwN6SDqksV0A1Xx8qA}{10.201.x.xx}{10.201.x.xx:9300}{ml.machine_memory=8352514048, rack=live, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {logstash01.mgm.xxx}{90Vy_3eISGONm0lkcmdmig}{rI7B6L0uQNyPNTbvzMIWew}{10.83.x.xx}{10.83.x.xx:9300}{ml.machine_memory=4124659712, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [123681]])`
Here, the previously removed Node gets added to the cluster again:
`[2018-11-14T15:34:11,191][INFO][o.e.c.s.ClusterApplierService] [logstash01.mgm.xxx] added {{elasticsearch03.mgm.xxx}{nldBGzbgQ62AV6k29wG2dQ}{zc8HBwN6SDqksV0A1Xx8qA}{10.201.x.xx}{10.201.x.xx:9300}{ml.machine_memory=8352514048, rack=live, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {logstash01.mgm.xxx}{90Vy_3eISGONm0lkcmdmig}{rI7B6L0uQNyPNTbvzMIWew}{10.83.xx.xx}{10.83.xx.xx:9300}{ml.machine_memory=4124659712, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [123683]])`
What can cause this behavior ?
Best regards, Thomas
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 15, 2018, 8:11am UTC](https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700/2 "2018-11-15T08:11:30Z")
Do you have a very large cluster state, e.g. lots of indices and shards? Are you nodes under heavy load and/or suffering from long or frequent GC?
---
### Author:  [@Thomas\_Tomski](https://discuss.elastic.co/u/Thomas_Tomski)
#### Post date: [November 26, 2018, 9:39am UTC](https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700/3 "2018-11-26T09:39:52Z")
Hello,
The problem was the same as here:
> [@Node disconnects every hour](https://discuss.elastic.co/t/node-disconnects-every-hour/37162/16):
>
> ES 2.1.0 When you say it removed and re-adds, does it mean it drops the node and re-adds the node, making many shards go unassigned and have to be assigned again? This is exactly what is happening. I see cluster goes into yellow every hour and many shards become unassigned.
It was solved by modifying the sysctl parameters:
```
# Reduce keep alive values to increase frequency of keep alive packets sent across network in order to prevent ElasticSearch nodes losing connection between each other
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 20
```
Cheers!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 24, 2018, 9:39am UTC](https://discuss.elastic.co/t/one-master-node-removes-and-adds-cluster-nodes-on-regular-base/156700/4 "2018-12-24T09:39:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.