# Problem with Zen discovery

**URL:** <https://discuss.elastic.co/t/problem-with-zen-discovery/17052>\
**Category:** Elasticsearch\
**Created:** [April 16, 2014, 5:11pm UTC](https://discuss.elastic.co/t/problem-with-zen-discovery/17052 "2014-04-16T17:11:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![IronMan2014](https://avatars.discourse-cdn.com/v4/letter/i/94ad74/32.png) [@IronMan2014](https://discuss.elastic.co/u/IronMan2014)\
**Post date:** [April 16, 2014, 5:11pm UTC](https://discuss.elastic.co/t/problem-with-zen-discovery/17052/1 "2014-04-16T17:11:04Z")

</div>

I can't quite figure out the problem with my 2 node EC2 cluster, here is  
log, Anyone?

[2014-04-16 16:50:21,351][INFO][node] [node 1] version  
[1.1.0], pid[21635], build[2181e11/2014-03-25T15:59:51Z]

[2014-04-16 16:50:21,351][INFO][node] [node 1]initializing  
...

[2014-04-16 16:50:21,499][INFO][plugins] [node 1] loaded  
[mapper-attachments, cloud-aws], sites []

[2014-04-16 16:50:24,309][DEBUG][discovery.zen.ping.unicast] [node 1] usinginitial hosts  
[23.23.190.242, 54.83.207.170], with concurrent\_connects [10]

[2014-04-16 16:50:24,333][DEBUG][discovery.zen] [node 1] usingping  
.timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data  
[false]

[2014-04-16 16:50:24,334][DEBUG][discovery.zen.elect] [node 1] usingminimum\_master\_nodes  
[2]

[2014-04-16 16:50:24,335][DEBUG][discovery.zen.fd] [node 1] [master  
] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]

[2014-04-16 16:50:24,351][DEBUG][discovery.zen.fd] [node 1] [node  
] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]

[2014-04-16 16:50:25,973][INFO][node] [node 1]initialized

[2014-04-16 16:50:25,974][INFO][node] [node 1]starting  
...

[2014-04-16 16:50:26,091][INFO][transport] [node 1]bound\_address  
{inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/10.138.39.99:9300]}

[2014-04-16 16:50:26,098][TRACE][discovery] [node 1]  
waiting for 30s for the initial state to be set by the discovery

[2014-04-16 16:50:26,168][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:26,170][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:27,669][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:27,671][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:30,674][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:30,675][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:30,675][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:33,678][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:33,679][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:35,179][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:35,179][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:35,180][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:35,180][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:35,181][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:36,681][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:36,682][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:39,683][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:39,684][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:39,684][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:39,684][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:41,186][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:41,187][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:42,687][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:42,688][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:44,188][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:44,188][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:44,189][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:44,189][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:44,190][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:45,691][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:45,691][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:47,192][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:47,192][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:48,692][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:48,693][TRACE][discovery.zen.ping.unicast] [node 1] [5]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:48,693][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:48,693][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:48,693][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:48,694][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:48,694][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:50,195][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:50,196][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:51,696][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:51,697][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
disconnecting from  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen.ping.unicast] [node 1] [6]  
disconnecting from  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:53,197][TRACE][discovery.zen] [node 1] full  
ping responses: {none}

[2014-04-16 16:50:53,198][DEBUG][discovery.zen] [node 1]  
filtered ping responses: (filter\_client[true], filter\_data[false]) {none}

[2014-04-16 16:50:53,198][TRACE][discovery.zen] [node 1] no  
masterNode returned

[2014-04-16 16:50:53,198][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:53,199][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:54,700][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:54,700][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,098][WARN][discovery] [node 1] waited  
for 30s and no initial state was set by the discovery

[2014-04-16 16:50:56,098][INFO][discovery] [node 1]  
rexCluster/3-8t7WtESc-9mWMcHomqUw

[2014-04-16 16:50:56,105][INFO][http] [node 1]bound\_address  
{inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/10.138.39.99:9200]}

[2014-04-16 16:50:56,136][INFO][node] [node 1] started

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,331][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException:  
[][inet[/54.83.207.170:9300]] connect\_timeout[30s]

```
    at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)

```
    at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)

```
    at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)

```
    at 

```

org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)

```
    at 

```

org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)

```
    at 

```

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

```
    at 

```

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

```
    at java.lang.Thread.run(Thread.java:744)

```

Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:  
connection timed out: /54.83.207.170:9300

```
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

processConnectTimeout(NioClientBoss.java:137)

```
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

process(NioClientBoss.java:83)

```
    at org.elasticsearch.common.netty.channel.socket.nio.

```

AbstractNioSelector.run(AbstractNioSelector.java:318)

```
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

run(NioClientBoss.java:42)

```
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(

```

ThreadRenamingRunnable.java:108)

```
    at org.elasticsearch.common.netty.util.internal.

```

DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

```
    ... 3 more

```

[2014-04-16 16:50:56,364][TRACE][discovery.zen.ping.unicast] [node 1] [1]failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.  
207.170:9300]] connect\_timeout[30s]

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:683)

```
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

```

NettyTransport.java:643)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToNodeLight(NettyTransport.java:610)

```
    at org.elasticsearch.transport.TransportService.connectToNodeLight(

```

TransportService.java:133)

```
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

```

(UnicastZenPing.java:279)

```
    at java.util.concurrent.ThreadPoolExecutor.runWorker(

```

ThreadPoolExecutor.java:1145)

```
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

```

ThreadPoolExecutor.java:615)

```
    at java.lang.Thread.run(Thread.java:744)

```

Caused by: java.nio.channels.ClosedByInterruptException

```
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

```

AbstractInterruptibleChannel.java:202)

```
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

```
    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

```
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

```

sendDownstream(DefaultChannelPipeline.java:574)

```
    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

```

java:634)

```
    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

```

AbstractChannel.java:207)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:229)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:182)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:680)

```
    ... 7 more

```

[2014-04-16 16:50:56,363][TRACE][discovery.zen.ping.unicast] [node 1] [6]failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.  
207.170:9300]] connect\_timeout[30s]

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:683)

```
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

```

NettyTransport.java:643)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToNodeLight(NettyTransport.java:610)

```
    at org.elasticsearch.transport.TransportService.connectToNodeLight(

```

TransportService.java:133)

```
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

```

(UnicastZenPing.java:279)

```
    at java.util.concurrent.ThreadPoolExecutor.runWorker(

```

ThreadPoolExecutor.java:1145)

```
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

```

ThreadPoolExecutor.java:615)

```
    at java.lang.Thread.run(Thread.java:744)

```

Caused by: java.nio.channels.ClosedByInterruptException

```
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

```

AbstractInterruptibleChannel.java:202)

```
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

```
    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

```
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

```

sendDownstream(DefaultChannelPipeline.java:574)

```
    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

```

java:634)

```
    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

```

AbstractChannel.java:207)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:229)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:182)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:680)

```
    ... 7 more

```

[2014-04-16 16:50:56,362][TRACE][discovery.zen.ping.unicast] [node 1] [6]failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [][inet[/54.83.  
207.170:9300]] connect\_timeout[30s]

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:683)

```
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

```

NettyTransport.java:643)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToNodeLight(NettyTransport.java:610)

```
    at org.elasticsearch.transport.TransportService.connectToNodeLight(

```

TransportService.java:133)

```
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

```

(UnicastZenPing.java:279)

```
    at java.util.concurrent.ThreadPoolExecutor.runWorker(

```

ThreadPoolExecutor.java:1145)

```
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

```

ThreadPoolExecutor.java:615)

```
    at java.lang.Thread.run(Thread.java:744)

```

Caused by: java.nio.channels.ClosedByInterruptException

```
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(

```

AbstractInterruptibleChannel.java:202)

```
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)

```
    at org.elasticsearch.common.netty.channel.socket.nio.

```

NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)

```
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.

```

sendDownstream(DefaultChannelPipeline.java:574)

```
    at org.elasticsearch.common.netty.channel.Channels.connect(Channels.

```

java:634)

```
    at org.elasticsearch.common.netty.channel.AbstractChannel.connect(

```

AbstractChannel.java:207)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:229)

```
    at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(

```

ClientBootstrap.java:182)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:680)

```
    ... 7 more

```

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7d0fc703-66c2-4961-85ee-d434a9ada70e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d0fc703-66c2-4961-85ee-d434a9ada70e%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![IronMan2014](https://avatars.discourse-cdn.com/v4/letter/i/94ad74/32.png) [@IronMan2014](https://discuss.elastic.co/u/IronMan2014)\
**Post date:** [April 16, 2014, 5:15pm UTC](https://discuss.elastic.co/t/problem-with-zen-discovery/17052/2 "2014-04-16T17:15:22Z")

</div>

Some continuation to the log:

[2014-04-16 16:50:56,136][INFO][node] [node 1] started

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]connecting  
(light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/23.23.190.242:9300]]

[2014-04-16 16:50:56,201][TRACE][discovery.zen.ping.unicast] [node 1] [7]  
connecting (light) to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

[2014-04-16 16:50:56,331][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException:  
[inet[/54.83.207.170:9300]] connect\_timeout[30s]

```
   at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:683)

```
   at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:643)

```
   at 

```

org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:610)

```
   at 

```

org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:133)

```
   at 

```

org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:279)

```
    at java.util.concurrent.ThreadPoolExecutor.runWorker(

```

ThreadPoolExecutor.java:1145)

```
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(

```

ThreadPoolExecutor.java:615)

```
   at java.lang.Thread.run(Thread.java:744)

```

Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:connection timed  
out: /54.83.207.170:9300

```
   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

processConnectTimeout(NioClientBoss.java:137)

```
   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

process(NioClientBoss.java:83)

```
   at org.elasticsearch.common.netty.channel.socket.nio.

```

AbstractNioSelector.run(AbstractNioSelector.java:318)

```
   at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.

```

run(NioClientBoss.java:42)

```
   at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(

```

ThreadRenamingRunnable.java:108)

```
   at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1

```

.run(DeadLockProofWorker.java:42)

```
   ... 3 more

```

[2014-04-16 16:50:56,364][TRACE][discovery.zen.ping.unicast] [node 1] [1]failed to connect to  
[#zen\_unicast\_2#][ip-10-138-39-99][inet[/54.83.207.170:9300]]

org.elasticsearch.transport.ConnectTransportException: [inet[/54.83.  
207.170:9300]] connect\_timeout[30s]

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToChannelsLight(NettyTransport.java:683)

```
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(

```

NettyTransport.java:643)

```
    at org.elasticsearch.transport.netty.NettyTransport.

```

connectToNodeLight(NettyTransport.java:610)

```
    at org.elasticsearch.transport.TransportService.connectToNodeLight(

```

TransportService.java:133)

```
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run

```

(UnicastZenPing.java:279)

```
    at java.util.concurrent.ThreadPoolExecutor.runWorker(

```

ThreadPoolExecutor.java:1145)

```
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(

```

ThreadPoolExecutor.java:615)

```
    at java.lang.Thread.run(Thread.java:744)

```

Caused by: java.nio.channels.ClosedByInterruptException

On Wednesday, April 16, 2014 1:11:04 PM UTC-4, IronMan2014 wrote:

> I can't quite figure out the problem with my 2 node EC2 cluster, here is  
> log, Anyone?
> 
> [2014-04-16 16:50:21,351][INFO][node] [node 1]version  
> [1.1.0], pid[21635], build[2181e11/2014-03-25T15:59:51Z]
> 
> [2014-04-16 16:50:21,351][INFO][node] [node 1]initializing  
> ...
> 
> [2014-04-16 16:50:21,499][INFO][plugins] [node 1]loaded  
> [mapper-attachments, cloud-aws], sites
> 
> [2014-04-16 16:50:24,309][DEBUG][discovery.zen.ping.unicast] [node 1]  
> using initial hosts [23.23.190.242, 54.83.207.170], withconcurrent\_connects  
> [10]
> 
> [2014-04-16 16:50:24,333][DEBUG][discovery.zen] [node 1] usingping  
> .timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data  
> [false]
> 
> [2014-04-16 16:50:24,334][DEBUG][discovery.zen.elect] [node 1] usingminimum\_master\_nodes  
> [2]
> 
> [2014-04-16 16:50:24,335][DEBUG][discovery.zen.fd] [node 1] [  
> master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]
> 
> [2014-04-16 16:50:24,351][DEBUG][discovery.zen.fd] [node 1] [node  
> ] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]
> 
> [2014-04-16 16:50:25,973][INFO][node] [node 1]initialized
> 
> [2014-04-16 16:50:25,974][INFO][node] [node 1]starting  
> ...
> 
> [2014-04-16 16:50:26,091][INFO][transport] [node 1]bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/10.138.39.99:9300]}
> 
> [2014-04-16 16:50:26,098][TRACE][discovery] [node 1]  
> waiting for 30s for the initial state to be set by the discovery
> 
> [2014-04-16 16:50:26,168][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:26,170][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:27,669][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:27,671][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:29,172][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> disconnecting from [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:30,673][TRACE][discovery.zen.ping.unicast] [node 1] [1]  
> disconnecting from [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:30,674][TRACE][discovery.zen] [node 1] full  
> ping responses: {none}
> 
> [2014-04-16 16:50:30,675][DEBUG][discovery.zen] [node 1]  
> filtered ping responses: (filter\_client[true], filter\_data[false]) {none}
> 
> [2014-04-16 16:50:30,675][TRACE][discovery.zen] [node 1] no  
> masterNode returned
> 
> [2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:30,676][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:32,177][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:33,678][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:33,679][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> disconnecting from [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:35,179][TRACE][discovery.zen.ping.unicast] [node 1] [2]  
> disconnecting from [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:35,179][TRACE][discovery.zen] [node 1] full  
> ping responses: {none}
> 
> [2014-04-16 16:50:35,179][DEBUG][discovery.zen] [node 1]  
> filtered ping responses: (filter\_client[true], filter\_data[false]) {none}
> 
> [2014-04-16 16:50:35,180][TRACE][discovery.zen] [node 1] no  
> masterNode returned
> 
> [2014-04-16 16:50:35,180][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:35,181][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:36,681][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:36,682][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:38,183][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:39,683][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> disconnecting from [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:39,684][TRACE][discovery.zen.ping.unicast] [node 1] [3]  
> disconnecting from [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:39,684][TRACE][discovery.zen] [node 1] full  
> ping responses: {none}
> 
> [2014-04-16 16:50:39,684][DEBUG][discovery.zen] [node 1]  
> filtered ping responses: (filter\_client[true], filter\_data[false]) {none}
> 
> [2014-04-16 16:50:39,684][TRACE][discovery.zen] [node 1] no  
> masterNode returned
> 
> [2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:39,685][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:41,186][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:41,187][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:42,687][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]
> 
> [2014-04-16 16:50:42,688][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> connecting (light) to [#zen\_unicast\_2#][ip-10-138-39-99][inet[/  
> 54.83.207.170:9300]]
> 
> [2014-04-16 16:50:44,188][TRACE][discovery.zen.ping.unicast] [node 1] [4]  
> disconnecting from [#zen\_unicast\_1#][ip-10-138-39-99][inet[/  
> 23.23.190.242:9300]]  
> \<  
> ...

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29b9b97a-5b0d-497c-9f93-09c22a94cbbd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29b9b97a-5b0d-497c-9f93-09c22a94cbbd%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:35am UTC](https://discuss.elastic.co/t/problem-with-zen-discovery/17052/3 "2017-07-06T01:35:28Z")

</div>