# Adding new node to cluster is not so easy -- new node is busted **URL:** \ **Category:** Elasticsearch\ **Created:** [January 21, 2016, 8:54pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819 "2016-01-21T20:54:40Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jm650](https://avatars.discourse-cdn.com/v4/letter/j/5f9b8f/32.png) [@jm650](https://discuss.elastic.co/u/jm650)\ **Post date:** [January 21, 2016, 8:54pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/1 "2016-01-21T20:54:40Z")
Life was beautiful on a single node but I index 300gb of logs per day and after ten or eleven days I added a new storage node. The new node doesnt work. I had high hopes for the cluster to automagically provision it but I guess it needs help or I need to change my expectations. I am using ES 2.1.1 on java 1.7 The nodes are on the same same ip subnet. The original node seems to see the new node but the new node complains about missing indexes and has other errors including messages about a node leaving. I can ping and connect to port 9200 & 9300 from each server to the other. There are no firewalls in between. I have blindly included some logs and data from the cluster in hopes they make sense to someone with a clue. Does anyone know what is going wrong here? Cluster status information: [http://pastebin.com/VWz9jNU7](http://pastebin.com/VWz9jNU7) Logs from new node: [http://pastebin.com/wvS8SBZ1](http://pastebin.com/wvS8SBZ1) thanks, j. ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 22, 2016, 1:41am UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/2 "2016-01-22T01:41:28Z")
> [@jm650](#): > > Does anyone know what is going wrong here? Did you configure unicast and a unicast host list on either of the nodes? ---
**Author:** ![jm650](https://avatars.discourse-cdn.com/v4/letter/j/5f9b8f/32.png) [@jm650](https://discuss.elastic.co/u/jm650)\ **Post date:** [January 22, 2016, 3:04am UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/3 "2016-01-22T03:04:21Z")
I have made some progress on this. I have added a dedicated master node, plus the master on the existing data node for a total of two. I changed the minimum master count to 2. I no longer see the second data node disconnecting. However, the new node seems unable to create indexes. When midnight zulu time rolled around the cluster tried to create a new logstash index on the new data node. The index went red and the cluster stopped. deleting the new index moved the cluster from red to yellow for about two seconds until the cluster tried to create a new index on the new data node and it went back to red. I turned down elasticsearch on the new node, deleted the index from the cluster, and life moved on with the new index now on the old node. how can i determine why the new node is having problems? es005 = new node es004 = original node es005 log can be found here: [http://broken.net/pbcluster002.txt](http://broken.net/pbcluster002.txt) here are the configs: [root@es005 elasticsearch]# egrep -v ^# elasticsearch.yml [cluster.name](http://cluster.name): pbcluster002 [node.name](http://node.name): es005 bootstrap.mlockall: false network.host: 192.168.134.29 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.134.27"] discovery.zen.ping.multicast.enabled: false discovery.zen.minimum\_master\_nodes: 2 node.master: false node.data: true * * * [root@es004 elasticsearch]# egrep -v ^# elasticsearch.yml [cluster.name](http://cluster.name): pbcluster002 [node.name](http://node.name): es004 bootstrap.mlockall: false network.host: 192.168.134.27 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.134.27", "192.168.134.29"] discovery.zen.minimum\_master\_nodes: 2 threadpool.bulk.queue\_size: 1000 index: number\_of\_shards: 1 node.master: true node.data: true ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 23, 2016, 3:30pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/4 "2016-01-23T15:30:01Z")
Am I correct in reading that you have exactly two nodes, one of them is set to be master-eligible, the other is not, but you have `discovery.zen.minimum_master_nodes` set to `2`? That will not work, you have to have at least as many master-eligible nodes as you have this setting equal to. Preferably this setting is equal to at least a quorum of your nodes. Note that with two nodes you're in a tricky position because `discovery.zen.minimum_master_nodes` being set to `1` risks split brain and set to `2` drops the possibility of high availability. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 23, 2016, 3:58pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/5 "2016-01-23T15:58:28Z")
Can you provide the output from the [cluster nodes info API](https://www.elastic.co/guide/en/elasticsearch/reference/2.1/cluster-nodes-info.html)? ---
**Author:** ![jm650](https://avatars.discourse-cdn.com/v4/letter/j/5f9b8f/32.png) [@jm650](https://discuss.elastic.co/u/jm650)\ **Post date:** [January 23, 2016, 5:20pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/6 "2016-01-23T17:20:06Z")
I am over the first hurdle. The problem was I installed marvel on the first node which required the license plugin. There is some bug where new nodes wont join if they also do not also have the license plugin. Apparently the license plugin changes the cluster metadata and custom cluster metadata has to be synchronized for nodes to join. I installed license & marvel on the new nodes allowed them to join (marvel is now broken, but it was going to break at some point any way). The very next thing that happened is primary shards starting moving over to the new data node and after twenty minutes the two hot shards from todays logstash index went red. I ran a lucene heathcheck against the shards and they came back clean. In frustration, I excluded the new node (curl -XPUT es005:9200/\_cluster/settings -d '{ "transient": {"cluster.routing.allocation.exclude.\_id": "C1hpqpRzQ4KR6MtlIMGz3w"}}') and deleted the index creating a time gap in the log data which I guess I will have to replay. So, now the new node has joined the cluster and I am gun shy about putting data on it. What are appropriate levels of logging to turn to determine what is causing the indexes to go red? thanks, j. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:22pm UTC](https://discuss.elastic.co/t/adding-new-node-to-cluster-is-not-so-easy-new-node-is-busted/39819/7 "2017-07-05T23:22:10Z")