Problem with Zen discovery


(IronMan2014) #1

I can't quite figure out the problem with my 2 node EC2 cluster, here is
log, Anyone?

[2014-04-16 16:50:21,351][INFO ][node ] [node 1] version
[1.1.0], pid[21635], build[2181e11/2014-03-25T15:59:51Z]

[2014-04-16 16:50:21,351][INFO ][node ] [node 1]initializing
...

[2014-04-16 16:50:21,499][INFO ][plugins ] [node 1] loaded
[mapper-attachments, cloud-aws], sites []

[2014-04-16 16:50:24,309][DEBUG][discovery.zen.ping.unicast] [node 1] usinginitial hosts
[23.23.190.242, 54.83.207.170], with concurrent_connects [10]

[2014-04-16 16:50:24,333][DEBUG][discovery.zen ] [node 1] usingping
.timeout [3s], master_election.filter_client [true], master_election.filter_data
[false]

[2014-04-16 16:50:24,334][DEBUG][discovery.zen.elect ] [node 1] usingminimum_master_nodes
[2]

[2014-04-16 16:50:24,335][DEBUG][discovery.zen.fd ] [node 1] [master
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2014-04-16 16:50:24,351][DEBUG][discovery.zen.fd ] [node 1] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2014-04-16 16:50:25,973][INFO ][node ] [node 1]initialized

[2014-04-16 16:50:25,974][INFO ][node ] [node 1]starting
...

[2014-04-16 16:50:26,091][INFO ][transport ] [node 1]bound_address
{inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.138.39.99:9300]}

[2014-04-16 16:50:26,098][TRACE][discovery ] [node 1]
waiting for 30s for the initial state to be set by the discovery

[2014-04-16 16:50:26,168][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:26,170][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:27,669][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:27,671][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:30,674][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:30,675][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:30,675][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:33,678][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:33,679][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:35,179][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:35,180][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:35,180][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:35,181][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:36,681][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:36,682][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:39,683][TRACE][discovery.zen.ping.unicast] [node 1] [3]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen.ping.unicast] [node 1] [3]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:39,684][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:39,684][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:41,186][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:41,187][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:42,687][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:42,688][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:44,188][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:44,189][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:44,189][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:44,190][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:45,691][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:45,691][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:47,192][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:47,192][TRACE][discovery.zen.ping.unicast] [node 1] [5]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:48,692][TRACE][discovery.zen.ping.unicast] [node 1] [5]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:48,693][TRACE][discovery.zen.ping.unicast] [node 1] [5]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:48,693][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:48,693][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:48,693][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:48,694][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:48,694][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:50,195][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:50,196][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:51,696][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:51,697][TRACE][discovery.zen.ping.unicast] [node 1] [6]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen.ping.unicast] [node 1] [6]
disconnecting from
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen.ping.unicast] [node 1] [6]
disconnecting from
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:53,198][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:53,198][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:53,198][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:53,199][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:54,700][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:54,700][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,098][WARN ][discovery ] [node 1] waited
for 30s and no initial state was set by the discovery

[2014-04-16 16:50:56,098][INFO ][discovery ] [node 1]
rexCluster/3-8t7WtESc-9mWMcHomqUw

[2014-04-16 16:50:56,105][INFO ][http ] [node 1]bound_address
{inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.138.39.99:9200]}

[2014-04-16 16:50:56,136][INFO ][node ] [node 1] started

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,331][TRACE][discovery.zen.ping.unicast] [node 1] [1]
failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException:
[][inet[/54.83.207.170:9300]] connect_timeout[30s]

    at 

org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)

    at 

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)

    at 

org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)

    at 

org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)

    at 

org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)

    at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:
connection timed out: /54.83.207.170:9300

    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

processConnectTimeout(NioClientBoss.java:137)

    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

process(NioClientBoss.java:83)

    at org.elasticsearch.common.netty.channel.socket.nio.

AbstractNioSelector.run(AbstractNioSelector.java:318)

    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

run(NioClientBoss.java:42)

    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(

ThreadRenamingRunnable.java:108)

    at org.elasticsearch.common.netty.util.internal.

DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

    ... 3 more

[2014-04-16 16:50:56,364][TRACE][discovery.zen.ping.unicast] [node 1] [1]failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.
207.170:9300]] connect_timeout[30s]

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:683)

    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

NettyTransport.java:643)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToNodeLight(NettyTransport.java:610)

    at org.elasticsearch.transport.TransportService.connectToNodeLight(

TransportService.java:133)

    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

(UnicastZenPing.java:279)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Caused by: java.nio.channels.ClosedByInterruptException

    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

AbstractInterruptibleChannel.java:202)

    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

sendDownstream(DefaultChannelPipeline.java:574)

    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

java:634)

    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

AbstractChannel.java:207)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:229)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:182)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:680)

    ... 7 more

[2014-04-16 16:50:56,363][TRACE][discovery.zen.ping.unicast] [node 1] [6]failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.
207.170:9300]] connect_timeout[30s]

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:683)

    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

NettyTransport.java:643)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToNodeLight(NettyTransport.java:610)

    at org.elasticsearch.transport.TransportService.connectToNodeLight(

TransportService.java:133)

    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

(UnicastZenPing.java:279)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Caused by: java.nio.channels.ClosedByInterruptException

    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

AbstractInterruptibleChannel.java:202)

    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

sendDownstream(DefaultChannelPipeline.java:574)

    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

java:634)

    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

AbstractChannel.java:207)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:229)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:182)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:680)

    ... 7 more

[2014-04-16 16:50:56,362][TRACE][discovery.zen.ping.unicast] [node 1] [6]failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.
207.170:9300]] connect_timeout[30s]

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:683)

    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

NettyTransport.java:643)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToNodeLight(NettyTransport.java:610)

    at org.elasticsearch.transport.TransportService.connectToNodeLight(

TransportService.java:133)

    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

(UnicastZenPing.java:279)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Caused by: java.nio.channels.ClosedByInterruptException

    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

AbstractInterruptibleChannel.java:202)

    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

    at org.elasticsearch.common.netty.channel.socket.nio.

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

sendDownstream(DefaultChannelPipeline.java:574)

    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

java:634)

    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

AbstractChannel.java:207)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:229)

    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

ClientBootstrap.java:182)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:680)

    ... 7 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d0fc703-66c2-4961-85ee-d434a9ada70e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(IronMan2014) #2

Some continuation to the log:

[2014-04-16 16:50:56,136][INFO ][node ] [node 1] started

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]connecting
(light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]
connecting (light) to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,331][TRACE][discovery.zen.ping.unicast] [node 1] [1]
failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException:
[][inet[/54.83.207.170:9300]] connect_timeout[30s]

   at 

org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)

   at 

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)

   at 

org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)

   at 

org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)

   at 

org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145)

   at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615)

   at java.lang.Thread.run(Thread.java:744)

Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:connection timed
out: /54.83.207.170:9300

   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

processConnectTimeout(NioClientBoss.java:137)

   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

process(NioClientBoss.java:83)

   at org.elasticsearch.common.netty.channel.socket.nio.

AbstractNioSelector.run(AbstractNioSelector.java:318)

   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

run(NioClientBoss.java:42)

   at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(

ThreadRenamingRunnable.java:108)

   at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1

.run(DeadLockProofWorker.java:42)

   ... 3 more

[2014-04-16 16:50:56,364][TRACE][discovery.zen.ping.unicast] [node 1] [1]failed to connect to
[#zen_unicast_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.
207.170:9300]] connect_timeout[30s]

    at org.elasticsearch.transport.netty.NettyTransport.

connectToChannelsLight(NettyTransport.java:683)

    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

NettyTransport.java:643)

    at org.elasticsearch.transport.netty.NettyTransport.

connectToNodeLight(NettyTransport.java:610)

    at org.elasticsearch.transport.TransportService.connectToNodeLight(

TransportService.java:133)

    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

(UnicastZenPing.java:279)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

Caused by: java.nio.channels.ClosedByInterruptException

On Wednesday, April 16, 2014 1:11:04 PM UTC-4, IronMan2014 wrote:

I can't quite figure out the problem with my 2 node EC2 cluster, here is
log, Anyone?

[2014-04-16 16:50:21,351][INFO ][node ] [node 1]version
[1.1.0], pid[21635], build[2181e11/2014-03-25T15:59:51Z]

[2014-04-16 16:50:21,351][INFO ][node ] [node 1]initializing
...

[2014-04-16 16:50:21,499][INFO ][plugins ] [node 1]loaded
[mapper-attachments, cloud-aws], sites []

[2014-04-16 16:50:24,309][DEBUG][discovery.zen.ping.unicast] [node 1]
using initial hosts [23.23.190.242, 54.83.207.170], withconcurrent_connects
[10]

[2014-04-16 16:50:24,333][DEBUG][discovery.zen ] [node 1] usingping
.timeout [3s], master_election.filter_client [true], master_election.filter_data
[false]

[2014-04-16 16:50:24,334][DEBUG][discovery.zen.elect ] [node 1] usingminimum_master_nodes
[2]

[2014-04-16 16:50:24,335][DEBUG][discovery.zen.fd ] [node 1] [
master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2014-04-16 16:50:24,351][DEBUG][discovery.zen.fd ] [node 1] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

[2014-04-16 16:50:25,973][INFO ][node ] [node 1]initialized

[2014-04-16 16:50:25,974][INFO ][node ] [node 1]starting
...

[2014-04-16 16:50:26,091][INFO ][transport ] [node 1]bound_address
{inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.138.39.99:9300]}

[2014-04-16 16:50:26,098][TRACE][discovery ] [node 1]
waiting for 30s for the initial state to be set by the discovery

[2014-04-16 16:50:26,168][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:26,170][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:27,669][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:27,671][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]
disconnecting from [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]
disconnecting from [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:30,674][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:30,675][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:30,675][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:33,678][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:33,679][TRACE][discovery.zen.ping.unicast] [node 1] [2]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]
disconnecting from [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]
disconnecting from [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:35,179][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:35,180][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:35,180][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:35,181][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:36,681][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:36,682][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:39,683][TRACE][discovery.zen.ping.unicast] [node 1] [3]
disconnecting from [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen.ping.unicast] [node 1] [3]
disconnecting from [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen ] [node 1] full
ping responses: {none}

[2014-04-16 16:50:39,684][DEBUG][discovery.zen ] [node 1]
filtered ping responses: (filter_client[true], filter_data[false]) {none}

[2014-04-16 16:50:39,684][TRACE][discovery.zen ] [node 1] no
masterNode returned

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:41,186][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:41,187][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:42,687][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]

[2014-04-16 16:50:42,688][TRACE][discovery.zen.ping.unicast] [node 1] [4]
connecting (light) to [#zen_unicast_2#][ip-10-138-39-99][inet[/
54.83.207.170:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]
disconnecting from [#zen_unicast_1#][ip-10-138-39-99][inet[/
23.23.190.242:9300]]
<
...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/29b9b97a-5b0d-497c-9f93-09c22a94cbbd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3