Hello.
Hoping someone can help with my cluster formation problem. I have 2 EC2
instances, both with EIPs assigned. All outbound ports over 1024 are open
(for now anyway). I've installed the ES 1.1.1 RPM for CentOS successfully.
I have multicast disabled, and have both servers listed in the
elasticsearch.yml file for unicast discovery. My elasticsearch.yml file
looks like this:
cluster.name: elasticsearch-dev
node.name: elasticsearch-ip-10-0-0-45
index.number_of_shards: 5
index.number_of_replicas: 0
path.data: /usr/local/ebs01/elasticsearch,/usr/local/ebs02/elasticsearch
gateway.recover_after_nodes: 2
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:
["ip-10-0-0-45.us-west-2.compute.internal:9200",
"ip-10-0-0-41.us-west-2.compute.internal:9200"]
I start a node on server 1, it comes up as a master (discovery set to
TRACE):
[2014-05-21 22:04:24,511][INFO ][node ]
[elasticsearch-ip-10-0-0-45] version[1.1.1], pid[30874],
build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-21 22:04:24,511][INFO ][node ]
[elasticsearch-ip-10-0-0-45] initializing ...
[2014-05-21 22:04:24,514][INFO ][plugins ]
[elasticsearch-ip-10-0-0-45] loaded [], sites []
[2014-05-21 22:04:25,745][DEBUG][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] using initial hosts
[ip-10-0-0-45.us-west-2.compute.internal:9200,
ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent_connects [10]
[2014-05-21 22:04:25,747][DEBUG][discovery.zen ]
[elasticsearch-ip-10-0-0-45] using ping.timeout [3s],
master_election.filter_client [true], master_election.filter_data [false]
[2014-05-21 22:04:25,747][DEBUG][discovery.zen.elect ]
[elasticsearch-ip-10-0-0-45] using minimum_master_nodes [1]
[2014-05-21 22:04:25,748][DEBUG][discovery.zen.fd ]
[elasticsearch-ip-10-0-0-45] [master] uses ping_interval [1s], ping_timeout
[30s], ping_retries [3]
[2014-05-21 22:04:25,757][DEBUG][discovery.zen.fd ]
[elasticsearch-ip-10-0-0-45] [node ] uses ping_interval [1s], ping_timeout
[30s], ping_retries [3]
[2014-05-21 22:04:26,688][INFO ][node ]
[elasticsearch-ip-10-0-0-45] initialized
[2014-05-21 22:04:26,688][INFO ][node ]
[elasticsearch-ip-10-0-0-45] starting ...
[2014-05-21 22:04:26,801][INFO ][transport ]
[elasticsearch-ip-10-0-0-45] bound_address {inet[/0:0:0:0:0:0:0:0:9300]},
publish_address {inet[/10.0.0.45:9300]}
[2014-05-21 22:04:26,811][TRACE][discovery ]
[elasticsearch-ip-10-0-0-45] waiting for 30s for the initial state to be
set by the discovery
[2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:26,813][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:26,828][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:26,833][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:28,314][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:28,315][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:28,322][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:28,324][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,821][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:29,822][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:29,823][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] disconnecting from
[#zen_unicast_1#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:04:29,824][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-45] [1] disconnecting from
[#zen_unicast_2#][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:04:29,825][TRACE][discovery.zen ]
[elasticsearch-ip-10-0-0-45] full ping responses: {none}
[2014-05-21 22:04:29,825][DEBUG][discovery.zen ]
[elasticsearch-ip-10-0-0-45] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2014-05-21 22:04:29,829][INFO ][cluster.service ]
[elasticsearch-ip-10-0-0-45] new_master
[elasticsearch-ip-10-0-0-45][41HM3OKaTKae7E3IgYYgDA][ip-10-0-0-45.us-west-2.compute.internal][inet[/10.0.0.45:9300]],
reason: zen-disco-join (elected_as_master)
[2014-05-21 22:04:29,842][TRACE][discovery ]
[elasticsearch-ip-10-0-0-45] initial state set from discovery
[2014-05-21 22:04:29,842][INFO ][discovery ]
[elasticsearch-ip-10-0-0-45] elasticsearch-dev/41HM3OKaTKae7E3IgYYgDA
[2014-05-21 22:04:29,850][INFO ][http ]
[elasticsearch-ip-10-0-0-45] bound_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish_address {inet[/10.0.0.45:9200]}
[2014-05-21 22:04:29,851][INFO ][node ]
[elasticsearch-ip-10-0-0-45] started
I start a node on server 2, server 1 gets the following exception 3 times,
and server 2 comes up as a master as well:
server1:
[2014-05-21 22:20:39,313][WARN ][http.netty ]
[elasticsearch-ip-10-0-0-45] Caught exception while handling client http
traffic, closing connection [id: 0xf151775e, /10.0.0.41:49606 => /
10.0.0.45:9200]
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:97)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:189)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:101)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
server2:
[2014-05-21 22:20:36,938][INFO ][node ]
[elasticsearch-ip-10-0-0-41] version[1.1.1], pid[15666],
build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-21 22:20:36,938][INFO ][node ]
[elasticsearch-ip-10-0-0-41] initializing ...
[2014-05-21 22:20:36,941][INFO ][plugins ]
[elasticsearch-ip-10-0-0-41] loaded [], sites []
[2014-05-21 22:20:38,168][DEBUG][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] using initial hosts
[ip-10-0-0-45.us-west-2.compute.internal:9200,
ip-10-0-0-41.us-west-2.compute.internal:9200], with concurrent_connects [10]
[2014-05-21 22:20:38,170][DEBUG][discovery.zen ]
[elasticsearch-ip-10-0-0-41] using ping.timeout [3s],
master_election.filter_client [true], master_election.filter_data [false]
[2014-05-21 22:20:38,170][DEBUG][discovery.zen.elect ]
[elasticsearch-ip-10-0-0-41] using minimum_master_nodes [1]
[2014-05-21 22:20:38,171][DEBUG][discovery.zen.fd ]
[elasticsearch-ip-10-0-0-41] [master] uses ping_interval [1s], ping_timeout
[30s], ping_retries [3]
[2014-05-21 22:20:38,181][DEBUG][discovery.zen.fd ]
[elasticsearch-ip-10-0-0-41] [node ] uses ping_interval [1s], ping_timeout
[30s], ping_retries [3]
[2014-05-21 22:20:39,144][INFO ][node ]
[elasticsearch-ip-10-0-0-41] initialized
[2014-05-21 22:20:39,144][INFO ][node ]
[elasticsearch-ip-10-0-0-41] starting ...
[2014-05-21 22:20:39,258][INFO ][transport ]
[elasticsearch-ip-10-0-0-41] bound_address {inet[/0:0:0:0:0:0:0:0:9300]},
publish_address {inet[/10.0.0.41:9300]}
[2014-05-21 22:20:39,268][TRACE][discovery ]
[elasticsearch-ip-10-0-0-41] waiting for 30s for the initial state to be
set by the discovery
[2014-05-21 22:20:39,270][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,271][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:39,287][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:39,288][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,290][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:39,316][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:40,771][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:40,773][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:40,776][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:42,273][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connecting (light) to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] connected to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,274][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] sending to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,275][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] failed to connect to
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
org.elasticsearch.transport.ConnectTransportException:
[][inet[ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused:
ip-10-0-0-41.us-west-2.compute.internal/10.0.0.41:9200
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] failed to connect to
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
org.elasticsearch.transport.NodeDisconnectedException:
[][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9200]][discovery/zen/unicast]
disconnected
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] disconnecting from
[#zen_unicast_1#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/
10.0.0.45:9200]]
[2014-05-21 22:20:42,278][TRACE][discovery.zen.ping.unicast]
[elasticsearch-ip-10-0-0-41] [1] disconnecting from
[#zen_unicast_2#][ip-10-0-0-41.us-west-2.compute.internal][inet[ip-10-0-0-41.us-west-2.compute.internal/
10.0.0.41:9200]]
[2014-05-21 22:20:42,278][TRACE][discovery.zen ]
[elasticsearch-ip-10-0-0-41] full ping responses: {none}
[2014-05-21 22:20:42,279][DEBUG][discovery.zen ]
[elasticsearch-ip-10-0-0-41] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2014-05-21 22:20:42,283][INFO ][cluster.service ]
[elasticsearch-ip-10-0-0-41] new_master
[elasticsearch-ip-10-0-0-41][9mhq1_x8Ts-snk1M7SaXRA][ip-10-0-0-41.us-west-2.compute.internal][inet[/10.0.0.41:9300]],
reason: zen-disco-join (elected_as_master)
[2014-05-21 22:20:42,291][TRACE][discovery ]
[elasticsearch-ip-10-0-0-41] initial state set from discovery
[2014-05-21 22:20:42,292][INFO ][discovery ]
[elasticsearch-ip-10-0-0-41] elasticsearch-dev/9mhq1_x8Ts-snk1M7SaXRA
[2014-05-21 22:20:42,300][INFO ][http ]
[elasticsearch-ip-10-0-0-41] bound_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish_address {inet[/10.0.0.41:9200]}
[2014-05-21 22:20:42,302][INFO ][node ]
[elasticsearch-ip-10-0-0-41] started
Hoping someone can shed some light on what's gone wrong 
I am stumped, and have been Googling and reading all day :S
Any help would me much appreciated!
Thanks in advance for your time.
Chris
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3DpjRSpM8RzTQGnghrJFOb68pWjctxDOUF3xxcC-FW-p38g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
