Zen discovery not working

Tamizh · June 13, 2017, 7:44am

When I try to add a node to my single node cluster. These are my node configurations

Node - 1
cluster.name: <Cluster Name>
node.name: <Node Name>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<new-node-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

Node - 2
cluster.name: <Cluster Name>
node.name: <Node Name - 2>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<Older-node-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

Both get started but not getting connected.

[2017-06-13T06:33:14,232][INFO ][o.e.t.TransportService   ] [Q-ZXfxh] publish_address 
{127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-06-13T06:33:17,398][INFO ][o.e.c.s.ClusterService   ] [Q-ZXfxh] new_master {Q-ZXfxh}{Q-
ZXfxhxQaOLkvT5SojlmQ}{mNwfOVNRQaiKCrE-s_5m0g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-06-13T06:33:17,455][INFO ][o.e.h.HttpServer         ] [Q-ZXfxh] publish_address 
{127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-06-13T06:33:17,456][INFO ][o.e.n.Node               ] [Q-ZXfxh] started
[2017-06-13T06:33:17,702][INFO ][o.e.g.GatewayService     ] [Q-ZXfxh] recovered [1] indices into cluster_state
[2017-06-13T06:33:18,059][INFO ][o.e.c.r.a.AllocationService] [Q-ZXfxh] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

Christian_Dahlqvist · June 13, 2017, 7:47am

Looks like your node binds to localhost, which is not reachable from outside. Change network.host to 0.0.0.0 or the external IP of the host.

Tamizh · June 13, 2017, 8:01am

I changed my configuration as follows

Node - 1
cluster.name: <Cluster Name>
node.name: <Node Name>
network.host: <node2-ip>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<new-node-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

Node - 2
cluster.name: <Cluster Name>
node.name: <Node Name - 2>
network.host: <node2-ip>
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300", "<Older-node-ip>:9300"]
discovery.zen.minimum_master_nodes: 1

But still, i got the following error.

org.elasticsearch.transport.BindTransportException: Failed to bind to [9300-9400]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:626) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:591) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:182) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:182) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.node.Node.start(Node.java:525) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:211) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:288) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) [elasticsearch-5.0.0.jar:5.0.0]
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:554) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1258) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:511) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:496) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:980) ~[?:?]
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:250) ~[?:?]
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:363) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[?:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
 [2017-06-13T07:52:36,144][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9300-9400]]; nested: BindException[Cannot assign requested address];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-5.0.0.jar:5.0.0]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9300-9400]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:626) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:591) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:182) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:182) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.node.Node.start(Node.java:525) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:211) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:288) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.0.jar:5.0.0]
... 6 more

Christian_Dahlqvist · June 13, 2017, 8:05am

network.host on node 1 should be <node1-ip>, which should be the same as <Older-node-ip>. If both nodes are master eligible, minimum_master_nodes should be set to 2 in order to prevent split-brain scenarios.

Tamizh · June 13, 2017, 8:24am

Thanks for the solution. it worked. Still, i have some question that I want to ask.

why I can't access my elasticsearch from my localhost. After change host to my localhost IP address. What is the difference between the empty network.host to node-ip network.host.

root@ubuntu:~# curl -XGET "localhost:9200/_cluster/health"
curl: (7) Failed to connect to localhost port 9200: Connection refused

The following the startup log of elasticsearch

[2017-06-13T08:15:07,508][INFO ][o.e.c.s.ClusterService   ] [Q-ZXfxh] detected_master {MhnGR-D}{MhnGR-DSRyi3kfcR8w9KvQ}{8444RarNRSW1rWTsiEi8Mg}{139.162.34.250}{139.162.34.250:9300}, added {{MhnGR-D}{MhnGR-DSRyi3kfcR8w9KvQ}{8444RarNRSW1rWTsiEi8Mg}{139.162.34.250}{139.162.34.250:9300},}, reason: zen-disco-receive(from master [master {MhnGR-D}{MhnGR-DSRyi3kfcR8w9KvQ}{8444RarNRSW1rWTsiEi8Mg}{139.162.34.250}{139.162.34.250:9300} committed version [1]])
[2017-06-13T08:15:07,544][INFO ][o.e.h.HttpServer         ] [Q-ZXfxh] publish_address {172.104.50.221:9200}, bound_addresses {172.104.50.221:9200}
[2017-06-13T08:15:07,544][INFO ][o.e.n.Node               ] [Q-ZXfxh] started
[2017-06-13T08:15:07,821][WARN ][o.e.g.DanglingIndicesState] [Q-ZXfxh] [[.kibana/ph_lW1xWT2CClFUPw-525w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2017-06-13T08:15:07,866][WARN ][o.e.g.DanglingIndicesState] [Q-ZXfxh] [[.kibana/ph_lW1xWT2CClFUPw-525w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2017-06-13T08:15:08,044][WARN ][o.e.g.DanglingIndicesState] [Q-ZXfxh] [[.kibana/ph_lW1xWT2CClFUPw-525w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2017-06-13T08:15:08,297][WARN ][o.e.g.DanglingIndicesState] [Q-ZXfxh] [[.kibana/ph_lW1xWT2CClFUPw-525w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata
[2017-06-13T08:15:08,638][WARN ][o.e.g.DanglingIndicesState] [Q-ZXfxh] [[.kibana/ph_lW1xWT2CClFUPw-525w]] can not be imported as a dangling index, as index with same name already exists in cluster metadata

From this log, i have one more concern. What about the dangling index and what it will cause.

Christian_Dahlqvist · June 13, 2017, 8:40am

Set network.host to 0.0.0.0 to bind to all interfaces.

Tamizh · June 13, 2017, 8:48am

Really thanks for your help. I was struggling with this last two days.

But I don't want to put network.host to 0.0.0.0. If I give like that, anybody can access my IP. So I will add Nginx in my node and port forward to access my elasticsearch. Is this the right way ??

Christian_Dahlqvist · June 13, 2017, 9:00am

Nginx will not work for the transport protocol, which Elasticsearch uses to communicate between nodes. You may however be able to limit access to port 9300 from the outside using e.g .iptables.

Tamizh · June 13, 2017, 9:20am

currently, i secure my elasticsearch and kibana API with basic authentication using Nginx. Because of I am adding a new node, I can't compromise with security.

So is there any way to make both possible.

Christian_Dahlqvist · June 13, 2017, 9:26am

Securing port 9300 through IP tables so that it is only accessible from the other node does not need to change the rest of your setup as you can configure HTTP to bind to a different host, e.g. localhost.

Tamizh · June 13, 2017, 11:07am

iptables -I INPUT -p tcp ! -s <ip address> --dport 9300 -j DROP

After I gave the command iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       tcp  -- !li1447-250.members.linode.com  anywhere             tcp dpt:9300

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I gave network.publish_host: <node-ip> and comment the network.host so that it can only connect from localhost and I can use Nginx over it. But the network.publish_host not working as I expect. so what I am doing wrong here ??

system · July 11, 2017, 11:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.