# Failed to add a new node to a single node cluster on the fly

**URL:** <https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031>\
**Category:** Elasticsearch\
**Created:** [December 1, 2015, 11:15am UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031 "2015-12-01T11:15:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Enniu\_51](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/enniu_51/32/5310_2.png) [@Enniu\_51](https://discuss.elastic.co/u/Enniu_51)\
**Post date:** [December 1, 2015, 11:15am UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031/1 "2015-12-01T11:15:13Z")

</div>

We have an one node cluster for a long time, and the old node is `crawler_service_001(172.16.11.17)`. We want to add a new node to that cluster(don't restart `crawler_service_001`), and named it `crawler_service_002(172.16.11.16)`.

But when I trying to setup `crawler_service_002`, the logs gives:

```
[2015-12-01 19:04:59,750][INFO][discovery.zen] [crawler_service_002] failed to send join request to master [{crawler_service_001}{rCqrCf10SGuAVqW_YYiDng}{127.0.0.1}{127.0.0.1:9300}], reason [RemoteTransportException[[crawler_service_002][127.0.0.1:9300][internal:discovery/zen/join]]; nested: IllegalStateException[Node [{crawler_service_002}{1DoKt04tQAGFiaUIAzkdOA}{127.0.0.1}{127.0.0.1:9300}{master=false}] not master for join request]; ]
[2015-12-01 19:05:02,764][INFO][discovery.zen] [crawler_service_002] failed to send join request to master [{crawler_service_001}{rCqrCf10SGuAVqW_YYiDng}{127.0.0.1}{127.0.0.1:9300}], reason [RemoteTransportException[[crawler_service_002][127.0.0.1:9300][internal:discovery/zen/join]]; nested: IllegalStateException[Node [{crawler_service_002}{1DoKt04tQAGFiaUIAzkdOA}{127.0.0.1}{127.0.0.1:9300}{master=false}] not master for join request]; ]
[2015-12-01 19:05:05,773][INFO][discovery.zen] [crawler_service_002] failed to send join request to master [{crawler_service_001}{rCqrCf10SGuAVqW_YYiDng}{127.0.0.1}{127.0.0.1:9300}], reason [RemoteTransportException[[crawler_service_002][127.0.0.1:9300][internal:discovery/zen/join]]; nested: IllegalStateException[Node [{crawler_service_002}{1DoKt04tQAGFiaUIAzkdOA}{127.0.0.1}{127.0.0.1:9300}{master=false}] not master for join request]; ]
[2015-12-01 19:05:08,785][INFO][discovery.zen] [crawler_service_002] failed to send join request to master [{crawler_service_001}{rCqrCf10SGuAVqW_YYiDng}{127.0.0.1}{127.0.0.1:9300}], reason [RemoteTransportException[[crawler_service_002][127.0.0.1:9300][internal:discovery/zen/join]]; nested: IllegalStateException[Node [{crawler_service_002}{1DoKt04tQAGFiaUIAzkdOA}{127.0.0.1}{127.0.0.1:9300}{master=false}] not master for join request]; ]

```

`elasticsearch.yml` of `crawler_service_002`(new node):

```
cluster.name: elasticsearch_dc_001
node.name: crawler_service_002

network.bind_host: 0.0.0.0
transport.tcp.port: 9300
transport.publish_port: 9300
node.master: false

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["172.16.11.16", "172.16.11.17"]

```

I tried change `node.master: false` to `true`, but still result the same error logs.

`elasticsearch.yml` of `crawler_service_001` (old node):

```
cluster.name: elasticsearch_dc_001
node.name: crawler_service_001

network.bind_host: 0.0.0.0
transport.tcp.port: 9300
transport.publish_port: 9300

```

Is there any mistake in my configuration file? Or some other reason?

---

<div class="post-metadata">

**Author:** ![Enniu\_51](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/enniu_51/32/5310_2.png) [@Enniu\_51](https://discuss.elastic.co/u/Enniu_51)\
**Post date:** [December 2, 2015, 2:20am UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031/2 "2015-12-02T02:20:10Z")

</div>

After a wild google and checked everything carefully, finally I noticed the log: `{crawler_service_001}{rCqrCf10SGuAVqW_YYiDng}{127.0.0.1}{127.0.0.1:9300}`

That's really strange, in `cralwer_service_002`'s opinion, the address of `crawler_service_001` is `127.0.0.1`!

I guess, `cralwer_service_002` used the ip address in unicast hosts found `cralwer_service_001`, but `cralwer_service_001` didn't know its publish address itself and tell `cralwer_service_002`, "Hi new guys, I'm `cralwer_service_001` from `127.0.0.1`!".

So I add `network.host: <their_ip>` to both node configuration, and restarted the whole cluster, it worked.

---

<div class="post-metadata">

**Author:** ![slentzen](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@slentzen](https://discuss.elastic.co/u/slentzen)\
**Post date:** [April 17, 2016, 11:59am UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031/3 "2016-04-17T11:59:53Z")

</div>

Thank you, I was having the exact same problem and this resolved it. I thought I had configured everything properly with network.bind\_host...

network.bind\_host: "0.0.0.0"

..but the logs indicated that discovery was failing and was using 127.0.0.1 for discovery. I added the following to each of my nodes and they discovered each other right away.

network.host: "0.0.0.0"

Note: This is on EC2 and I'd already set up the security group that enables the nodes to query EC2 for other nodes.

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [April 23, 2017, 10:58am UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031/4 "2017-04-23T10:58:41Z")

</div>

my log look better but I still have a question in mine I get

{node\_name}{rCqrCf10SGuAVqW\_YYiDng}{ **another\_String** }{good\_ip}{same\_as\_good\_ip:9300}

Is the other "another\_String" a problem?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:00pm UTC](https://discuss.elastic.co/t/failed-to-add-a-new-node-to-a-single-node-cluster-on-the-fly/36031/5 "2017-07-05T22:00:55Z")

</div>


