# Help please with ES and EC2 cluster formation
**URL:** https://discuss.elastic.co/t/help-please-with-es-and-ec2-cluster-formation/17659
**Category:** Elasticsearch
**Created:** [May 21, 2014, 10:32pm UTC](https://discuss.elastic.co/t/help-please-with-es-and-ec2-cluster-formation/17659 "2014-05-21T22:32:56Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)
#### Post date: [May 21, 2014, 10:32pm UTC](https://discuss.elastic.co/t/help-please-with-es-and-ec2-cluster-formation/17659/1 "2014-05-21T22:32:56Z")
Hello.
Hoping someone can help with my cluster formation problem. I have 2 EC2
instances, both with EIPs assigned. All outbound ports over 1024 are open
(for now anyway). I've installed the ES 1.1.1 RPM for CentOS successfully.
I have multicast disabled, and have both servers listed in the
elasticsearch.yml file for unicast discovery. My elasticsearch.yml file
looks like this:
cluster.name: elasticsearch-dev
node.name: elasticsearch-ip-10-0-0-45
index.number\_of\_shards: 5
index.number\_of\_replicas: 0
path.data: /usr/local/ebs01/elasticsearch,/usr/local/ebs02/elasticsearch
gateway.recover\_after\_nodes: 2
discovery.zen.minimum\_master\_nodes: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:
["ip-10-0-0-45.us-west-2.compute.internal:9200",
"ip-10-0-0-41.us-west-2.compute.internal:9200"]
I start a node on server 1, it comes up as a master (discovery set to
TRACE):
[2014-05-21 22:04:24,511][INFO][node]
[elasticsearch-ip-10-0-0-45] version[1.1.1], pid[30874],
build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-21 22:04:24,511][INFO][node]
[elasticsearch-ip-10-0-0-45] initializing ...
[2014-05-21 22:04:24,514][INFO][plugins]
[elasticsearch-ip-10-0-0-45] loaded [], sites []
[2014-05-21 22:04:25,745][DEBUG][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] using initial hosts
[ip-10-0-0-45.us-west-2.compute.internal:9200,
ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent\_connects [10]
[2014-05-21 22:04:25,747][DEBUG][discovery.zen]
[elasticsearch-ip-10-0-0-45] using ping.timeout [3s],
master\_election.filter\_client [true], master\_election.filter\_data [false]
[2014-05-21 22:04:25,747][DEBUG][discovery.zen.elect]
[elasticsearch-ip-10-0-0-45] using minimum\_master\_nodes [1]
[2014-05-21 22:04:25,748][DEBUG][discovery.zen.fd]
[elasticsearch-ip-10-0-0-45] [master] uses ping\_interval [1s], ping\_timeout
[30s], ping\_retries [3]
[2014-05-21 22:04:25,757][DEBUG][discovery.zen.fd]
[elasticsearch-ip-10-0-0-45] [node] uses ping\_interval [1s], ping\_timeout
[30s], ping\_retries [3]
[2014-05-21 22:04:26,688][INFO][node]
[elasticsearch-ip-10-0-0-45] initialized
[2014-05-21 22:04:26,688][INFO][node]
[elasticsearch-ip-10-0-0-45] starting ...
[2014-05-21 22:04:26,801][INFO][transport]
[elasticsearch-ip-10-0-0-45] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]},
publish\_address {inet[/10.0.0.45:9300]}
[2014-05-21 22:04:26,811][TRACE][discovery]
[elasticsearch-ip-10-0-0-45] waiting for 30s for the initial state to be
set by the discovery
[2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:26,828][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:26,833][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:28,314][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:28,315][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:28,322][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:28,324][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,821][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:29,822][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:29,823][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] disconnecting from
[#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] disconnecting from
[#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:29,825][TRACE][discovery.zen]
[elasticsearch-ip-10-0-0-45] full ping responses: {none}
[2014-05-21 22:04:29,825][DEBUG][discovery.zen]
[elasticsearch-ip-10-0-0-45] filtered ping responses: (filter\_client[true],
filter\_data[false]) {none}
[2014-05-21 22:04:29,829][INFO][cluster.service]
[elasticsearch-ip-10-0-0-45] new\_master
[elasticsearch-ip-10-0-0-45][41HM3OKaTKae7E3IgYYgDA][ip-10-0-0-45.us-west-2.compute.internal][inet[/10.0.0.45:9300]],
reason: zen-disco-join (elected\_as\_master)
[2014-05-21 22:04:29,842][TRACE][discovery]
[elasticsearch-ip-10-0-0-45] initial state set from discovery
[2014-05-21 22:04:29,842][INFO][discovery]
[elasticsearch-ip-10-0-0-45] elasticsearch-dev/41HM3OKaTKae7E3IgYYgDA
[2014-05-21 22:04:29,850][INFO][http]
[elasticsearch-ip-10-0-0-45] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish\_address {inet[/10.0.0.45:9200]}
[2014-05-21 22:04:29,851][INFO][node]
[elasticsearch-ip-10-0-0-45] started
I start a node on server 2, server 1 gets the following exception 3 times,
and server 2 comes up as a master as well:
server1:
[2014-05-21 22:20:39,313][WARN][http.netty]
[elasticsearch-ip-10-0-0-45] Caught exception while handling client http
traffic, closing connection [id: 0xf151775e, /10.0.0.41:49606 =\> /
10.0.0.45:9200]
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:97)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:189)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:101)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
server2:
[2014-05-21 22:20:36,938][INFO][node]
[elasticsearch-ip-10-0-0-41] version[1.1.1], pid[15666],
build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-21 22:20:36,938][INFO][node]
[elasticsearch-ip-10-0-0-41] initializing ...
[2014-05-21 22:20:36,941][INFO][plugins]
[elasticsearch-ip-10-0-0-41] loaded [], sites []
[2014-05-21 22:20:38,168][DEBUG][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] using initial hosts
[ip-10-0-0-45.us-west-2.compute.internal:9200,
ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent\_connects [10]
[2014-05-21 22:20:38,170][DEBUG][discovery.zen]
[elasticsearch-ip-10-0-0-41] using ping.timeout [3s],
master\_election.filter\_client [true], master\_election.filter\_data [false]
[2014-05-21 22:20:38,170][DEBUG][discovery.zen.elect]
[elasticsearch-ip-10-0-0-41] using minimum\_master\_nodes [1]
[2014-05-21 22:20:38,171][DEBUG][discovery.zen.fd]
[elasticsearch-ip-10-0-0-41] [master] uses ping\_interval [1s], ping\_timeout
[30s], ping\_retries [3]
[2014-05-21 22:20:38,181][DEBUG][discovery.zen.fd]
[elasticsearch-ip-10-0-0-41] [node] uses ping\_interval [1s], ping\_timeout
[30s], ping\_retries [3]
[2014-05-21 22:20:39,144][INFO][node]
[elasticsearch-ip-10-0-0-41] initialized
[2014-05-21 22:20:39,144][INFO][node]
[elasticsearch-ip-10-0-0-41] starting ...
[2014-05-21 22:20:39,258][INFO][transport]
[elasticsearch-ip-10-0-0-41] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]},
publish\_address {inet[/10.0.0.41:9300]}
[2014-05-21 22:20:39,268][TRACE][discovery]
[elasticsearch-ip-10-0-0-41] waiting for 30s for the initial state to be
set by the discovery
[2014-05-21 22:20:39,270][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,271][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:39,287][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:39,288][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,290][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,316][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:40,776][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,275][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect\_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] disconnecting from
[#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] disconnecting from
[#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:42,278][TRACE][discovery.zen]
[elasticsearch-ip-10-0-0-41] full ping responses: {none}
[2014-05-21 22:20:42,279][DEBUG][discovery.zen]
[elasticsearch-ip-10-0-0-41] filtered ping responses: (filter\_client[true],
filter\_data[false]) {none}
[2014-05-21 22:20:42,283][INFO][cluster.service]
[elasticsearch-ip-10-0-0-41] new\_master
[elasticsearch-ip-10-0-0-41][9mhq1\_x8Ts-snk1M7SaXRA][ip-10-0-0-41.us-west-2.compute.internal][inet[/10.0.0.41:9300]],
reason: zen-disco-join (elected\_as\_master)
[2014-05-21 22:20:42,291][TRACE][discovery]
[elasticsearch-ip-10-0-0-41] initial state set from discovery
[2014-05-21 22:20:42,292][INFO][discovery]
[elasticsearch-ip-10-0-0-41] elasticsearch-dev/9mhq1\_x8Ts-snk1M7SaXRA
[2014-05-21 22:20:42,300][INFO][http]
[elasticsearch-ip-10-0-0-41] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish\_address {inet[/10.0.0.41:9200]}
[2014-05-21 22:20:42,302][INFO][node]
[elasticsearch-ip-10-0-0-41] started
Hoping someone can shed some light on what's gone wrong ☹
I am stumped, and have been Googling and reading all day :S
Any help would me much appreciated!
Thanks in advance for your time.
Chris
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DpjRSpM8RzTQGnghrJFOb68pWjctxDOUF3xxcC-FW-p38g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DpjRSpM8RzTQGnghrJFOb68pWjctxDOUF3xxcC-FW-p38g%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)
#### Post date: [May 22, 2014, 3:13pm UTC](https://discuss.elastic.co/t/help-please-with-es-and-ec2-cluster-formation/17659/2 "2014-05-22T15:13:00Z")
Solved my own issue. That's how it works. As soon as you post the
question, you get that lightbulb moment...
Anyway, just in case someone else screws up like I did, here is the
solution for me: Configure unicast properly.
This:
discovery.zen.ping.unicast.hosts:
["ip-10-0-0-45.us-west-2.compute.internal:9200",
"ip-10-0-0-41.us-west-2.compute.internal:9200"]
Should be this:
discovery.zen.ping.unicast.hosts: ["ip-10-0-0-45.us-west-2.compute.internal",
"ip-10-0-0-41.us-west-2.compute.internal"]
Or maybe this would work too:
discovery.zen.ping.unicast.hosts:
["ip-10-0-0-45.us-west-2.compute.internal:9300",
"ip-10-0-0-41.us-west-2.compute.internal:9300"]
In either case, port 9200 is not correct. TRACE logs show 9300 on a
working system, but it seems to default to 9300 so no port is actually
necessary.
Chris
On Wed, May 21, 2014 at 5:32 PM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)wrote:
> Hello.
>
> Hoping someone can help with my cluster formation problem. I have 2 EC2
> instances, both with EIPs assigned. All outbound ports over 1024 are open
> (for now anyway). I've installed the ES 1.1.1 RPM for CentOS successfully.
>
> I have multicast disabled, and have both servers listed in the
> elasticsearch.yml file for unicast discovery. My elasticsearch.yml file
> looks like this:
>
> cluster.name: elasticsearch-dev
> node.name: elasticsearch-ip-10-0-0-45
> index.number\_of\_shards: 5
> index.number\_of\_replicas: 0
> path.data: /usr/local/ebs01/elasticsearch,/usr/local/ebs02/elasticsearch
> gateway.recover\_after\_nodes: 2
> discovery.zen.minimum\_master\_nodes: 1
> discovery.zen.ping.multicast.enabled: false
> discovery.zen.ping.unicast.hosts:
> ["ip-10-0-0-45.us-west-2.compute.internal:9200",
> "ip-10-0-0-41.us-west-2.compute.internal:9200"]
>
> I start a node on server 1, it comes up as a master (discovery set to
> TRACE):
>
> [2014-05-21 22:04:24,511][INFO][node]
> [elasticsearch-ip-10-0-0-45] version[1.1.1], pid[30874],
> build[f1585f0/2014-04-16T14:27:12Z]
> [2014-05-21 22:04:24,511][INFO][node]
> [elasticsearch-ip-10-0-0-45] initializing ...
> [2014-05-21 22:04:24,514][INFO][plugins]
> [elasticsearch-ip-10-0-0-45] loaded , sites
> [2014-05-21 22:04:25,745][DEBUG][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] using initial hosts
> [ip-10-0-0-45.us-west-2.compute.internal:9200,
> ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent\_connects [10]
> [2014-05-21 22:04:25,747][DEBUG][discovery.zen]
> [elasticsearch-ip-10-0-0-45] using ping.timeout [3s],
> master\_election.filter\_client [true], master\_election.filter\_data [false]
> [2014-05-21 22:04:25,747][DEBUG][discovery.zen.elect]
> [elasticsearch-ip-10-0-0-45] using minimum\_master\_nodes [1]
> [2014-05-21 22:04:25,748][DEBUG][discovery.zen.fd]
> [elasticsearch-ip-10-0-0-45] [master] uses ping\_interval [1s], ping\_timeout
> [30s], ping\_retries [3]
> [2014-05-21 22:04:25,757][DEBUG][discovery.zen.fd]
> [elasticsearch-ip-10-0-0-45] [node] uses ping\_interval [1s], ping\_timeout
> [30s], ping\_retries [3]
> [2014-05-21 22:04:26,688][INFO][node]
> [elasticsearch-ip-10-0-0-45] initialized
> [2014-05-21 22:04:26,688][INFO][node]
> [elasticsearch-ip-10-0-0-45] starting ...
> [2014-05-21 22:04:26,801][INFO][transport]
> [elasticsearch-ip-10-0-0-45] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]},
> publish\_address {inet[/10.0.0.45:9300]}
> [2014-05-21 22:04:26,811][TRACE][discovery]
> [elasticsearch-ip-10-0-0-45] waiting for 30s for the initial state to be
> set by the discovery
> [2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:04:26,828][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:26,833][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:28,314][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:04:28,315][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:04:28,322][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:28,324][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:29,821][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:04:29,822][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:04:29,823][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] disconnecting from
> [#zen\_unicast\_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-45] [1] disconnecting from
> [#zen\_unicast\_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:04:29,825][TRACE][discovery.zen]
> [elasticsearch-ip-10-0-0-45] full ping responses: {none}
> [2014-05-21 22:04:29,825][DEBUG][discovery.zen]
> [elasticsearch-ip-10-0-0-45] filtered ping responses: (filter\_client[true],
> filter\_data[false]) {none}
> [2014-05-21 22:04:29,829][INFO][cluster.service]
> [elasticsearch-ip-10-0-0-45] new\_master
> [elasticsearch-ip-10-0-0-45][41HM3OKaTKae7E3IgYYgDA][ip-10-0-0-45.us-west-2.compute.internal][inet[/10.0.0.45:9300]],
> reason: zen-disco-join (elected\_as\_master)
> [2014-05-21 22:04:29,842][TRACE][discovery]
> [elasticsearch-ip-10-0-0-45] initial state set from discovery
> [2014-05-21 22:04:29,842][INFO][discovery]
> [elasticsearch-ip-10-0-0-45] elasticsearch-dev/41HM3OKaTKae7E3IgYYgDA
> [2014-05-21 22:04:29,850][INFO][http]
> [elasticsearch-ip-10-0-0-45] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]},
> publish\_address {inet[/10.0.0.45:9200]}
> [2014-05-21 22:04:29,851][INFO][node]
> [elasticsearch-ip-10-0-0-45] started
>
> I start a node on server 2, server 1 gets the following exception 3 times,
> and server 2 comes up as a master as well:
>
> server1:
> [2014-05-21 22:20:39,313][WARN][http.netty]
> [elasticsearch-ip-10-0-0-45] Caught exception while handling client http
> traffic, closing connection [id: 0xf151775e, /10.0.0.41:49606 =\> /
> 10.0.0.45:9200]
> java.lang.IllegalArgumentException: empty text
> at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:97)
> at
> org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
> at
> org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
> at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:189)
> at
> org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:101)
> at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
> at
> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> server2:
> [2014-05-21 22:20:36,938][INFO][node]
> [elasticsearch-ip-10-0-0-41] version[1.1.1], pid[15666],
> build[f1585f0/2014-04-16T14:27:12Z]
> [2014-05-21 22:20:36,938][INFO][node]
> [elasticsearch-ip-10-0-0-41] initializing ...
> [2014-05-21 22:20:36,941][INFO][plugins]
> [elasticsearch-ip-10-0-0-41] loaded , sites
> [2014-05-21 22:20:38,168][DEBUG][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] using initial hosts
> [ip-10-0-0-45.us-west-2.compute.internal:9200,
> ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent\_connects [10]
> [2014-05-21 22:20:38,170][DEBUG][discovery.zen]
> [elasticsearch-ip-10-0-0-41] using ping.timeout [3s],
> master\_election.filter\_client [true], master\_election.filter\_data [false]
> [2014-05-21 22:20:38,170][DEBUG][discovery.zen.elect]
> [elasticsearch-ip-10-0-0-41] using minimum\_master\_nodes [1]
> [2014-05-21 22:20:38,171][DEBUG][discovery.zen.fd]
> [elasticsearch-ip-10-0-0-41] [master] uses ping\_interval [1s], ping\_timeout
> [30s], ping\_retries [3]
> [2014-05-21 22:20:38,181][DEBUG][discovery.zen.fd]
> [elasticsearch-ip-10-0-0-41] [node] uses ping\_interval [1s], ping\_timeout
> [30s], ping\_retries [3]
> [2014-05-21 22:20:39,144][INFO][node]
> [elasticsearch-ip-10-0-0-41] initialized
> [2014-05-21 22:20:39,144][INFO][node]
> [elasticsearch-ip-10-0-0-41] starting ...
> [2014-05-21 22:20:39,258][INFO][transport]
> [elasticsearch-ip-10-0-0-41] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]},
> publish\_address {inet[/10.0.0.41:9300]}
> [2014-05-21 22:20:39,268][TRACE][discovery]
> [elasticsearch-ip-10-0-0-41] waiting for 30s for the initial state to be
> set by the discovery
> [2014-05-21 22:20:39,270][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:39,271][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:20:39,287][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:20:39,288][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connected to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:39,290][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] sending to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:39,316][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.NodeDisconnectedException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
> disconnected
> [2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connected to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] sending to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:20:40,776][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.NodeDisconnectedException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
> disconnected
> [2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connecting (light) to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] connected to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] sending to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:42,275][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] failed to connect to
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> org.elasticsearch.transport.ConnectTransportException:
> [inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
> connect\_timeout[30s]
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
> at
> org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
> at
> org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
> at
> org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
> [2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] failed to connect to
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> org.elasticsearch.transport.NodeDisconnectedException:
> [inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
> disconnected
> [2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] disconnecting from
> [#zen\_unicast\_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
> 10.0.0.45:9200]]
> [2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
> [elasticsearch-ip-10-0-0-41] [1] disconnecting from
> [#zen\_unicast\_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
> 10.0.0.41:9200]]
> [2014-05-21 22:20:42,278][TRACE][discovery.zen]
> [elasticsearch-ip-10-0-0-41] full ping responses: {none}
> [2014-05-21 22:20:42,279][DEBUG][discovery.zen]
> [elasticsearch-ip-10-0-0-41] filtered ping responses: (filter\_client[true],
> filter\_data[false]) {none}
> [2014-05-21 22:20:42,283][INFO][cluster.service]
> [elasticsearch-ip-10-0-0-41] new\_master
> [elasticsearch-ip-10-0-0-41][9mhq1\_x8Ts-snk1M7SaXRA][ip-10-0-0-41.us-west-2.compute.internal][inet[/10.0.0.41:9300]],
> reason: zen-disco-join (elected\_as\_master)
> [2014-05-21 22:20:42,291][TRACE][discovery]
> [elasticsearch-ip-10-0-0-41] initial state set from discovery
> [2014-05-21 22:20:42,292][INFO][discovery]
> [elasticsearch-ip-10-0-0-41] elasticsearch-dev/9mhq1\_x8Ts-snk1M7SaXRA
> [2014-05-21 22:20:42,300][INFO][http]
> [elasticsearch-ip-10-0-0-41] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]},
> publish\_address {inet[/10.0.0.41:9200]}
> [2014-05-21 22:20:42,302][INFO][node]
> [elasticsearch-ip-10-0-0-41] started
>
> Hoping someone can shed some light on what's gone wrong ☹
> I am stumped, and have been Googling and reading all day :S
> Any help would me much appreciated!
>
> Thanks in advance for your time.
> Chris
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DpjhTX64VoMGKHMfebckJCUC3bHXPoMiAS8xHgL477R18w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DpjhTX64VoMGKHMfebckJCUC3bHXPoMiAS8xHgL477R18w%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:27am UTC](https://discuss.elastic.co/t/help-please-with-es-and-ec2-cluster-formation/17659/3 "2017-07-06T01:27:36Z")