I am trying to form an ES clsuter in AWS private subnets
When I add second node, its getting added to the cluster, but only some shards are getting allocated
I see these errors in logs
`[2016-05-06 13:43:16,921][TRACE][discovery.zen.ping.unicast] [Llyron] [7] failed to connect to {#zen_unicast_8#}{::1}{[::1]:9302}
ConnectTransportException[[][[::1]:9302] connect_timeout[30s]]; nested: ConnectException[Connection refused: /0:0:0:0:0:0:0:1:9302];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:880)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:852)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:250)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /0:0:0:0:0:0:0:1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-05-06 13:43:16,921][TRACE][discovery.zen.ping.unicast] [Llyron] [7] failed to connect to {#zen_unicast_7#}{::1}{[::1]:9301}
ConnectTransportException[[][[::1]:9301] connect_timeout[30s]]; nested: ConnectException[Connection refused: /0:0:0:0:0:0:0:1:9301];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:880)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:852)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:250)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /0:0:0:0:0:0:0:1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
ockProofWorker.java:42)
... 3 more
network.host: "10.0.136.205"
action.destructive_requires_name: false
script.inline: on
script.indexed: on
node.master: true
node.data: true
index.number_of_shards: "5"
index.number_of_replicas: 1
path.conf: /etc/elasticsearch
path.work: /tmp/elasticsearch
path.data: /data/elasticsearch
path.logs: /data/logs/elasticsearch
bootstrap.mlockall: true
discovery.zen.minimum_master_nodes: 1
# EC2 discovery allows to use AWS EC2 API in order to perform discovery.
cloud:
aws:
protocol: https
region: ap-southeast-1
discovery.type: ec2
any ideas what might be going on here?