Zen discovery plugin - Connection refused: /127.0.0.1:9302

Elastic Stack Elasticsearch
1

Hello everyone,
Just started using Elasticsearch and trying out Zen discovery plugin for the first time. I'm currently using version 5.0.0-alpha4.

My data node is unable to join the cluster, I'm getting Connection refused: /0:0:0:0:0:0:0:1:9301 error. Here is what I have in my elasticsearch.yml file:

cluster:
   name: Elastic-POC
node:
   name: ${HOSTNAME}-data
   master: false
   data: true
cloud:
    aws:
        access_key: xxxxxx
        secret_key: xxxxxx
        region: us-west-2
        ec2:
           protocol: http
           access_key: xxxxxx
           secret_key: xxxxxx
discovery:
    type: ec2
    zen.minimum_master_nodes: 1
    ec2.any_group: true
    ec2.groups: sg-xxxxxx
network:
    host: _ec2:privateIp_

What I'm missing? Any helps or suggestions would be highly appreciated.

Thanks,
chok

2

I have enabled "TRACE" for discover plugin and here is the log:

[2016-07-12 00:30:39,377][INFO ][env                      ] [ip-172-29-1-44-data] heap size [15.8gb], compressed ordinary object pointers [true]
    [2016-07-12 00:30:40,563][DEBUG][discovery.zen.elect      ] [ip-172-29-1-44-data] using minimum_master_nodes [1]
    [2016-07-12 00:30:40,913][DEBUG][discovery.ec2            ] [ip-172-29-1-44-data] using host_type [PRIVATE_IP], tags [{}], groups [[sg-xxxxxx]] with any_group [true], availability_zones [[]]
    [2016-07-12 00:30:40,914][DEBUG][discovery.zen.ping.unicast] [ip-172-29-1-44-data] using initial hosts [127.0.0.1, [::1]], with concurrent_connects [10]
    [2016-07-12 00:30:40,922][DEBUG][discovery.zen            ] [ip-172-29-1-44-data] using ping_timeout [3s], join.timeout [1m], master_election.ignore_non_master [false]
    [2016-07-12 00:30:40,925][DEBUG][discovery.zen.fd         ] [ip-172-29-1-44-data] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
    [2016-07-12 00:30:40,938][DEBUG][discovery.zen.fd         ] [ip-172-29-1-44-data] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
    [2016-07-12 00:30:41,250][DEBUG][discovery.ec2            ] [ip-172-29-1-44-data] using host_type [PRIVATE_IP], tags [{}], groups [[sg-xxxxxx]] with any_group [true], availability_zones [[]]
    [2016-07-12 00:30:41,250][DEBUG][discovery.ec2            ] [ip-172-29-1-44-data] using host_type [PRIVATE_IP], tags [{}], groups [[sg-xxxxxx]] with any_group [true], availability_zones [[]]
    [2016-07-12 00:30:41,252][INFO ][node                     ] [ip-172-29-1-44-data] initialized
    [2016-07-12 00:30:41,252][INFO ][node                     ] [ip-172-29-1-44-data] starting ...
    [2016-07-12 00:30:41,546][INFO ][transport                ] [ip-172-29-1-44-data] publish_address {172.29.1.44:9300}, bound_addresses {172.29.1.44:9300}
    [2016-07-12 00:30:41,561][TRACE][discovery.zen            ] [ip-172-29-1-44-data] starting an election context, will accumulate joins
    [2016-07-12 00:30:41,562][TRACE][discovery.zen            ] [ip-172-29-1-44-data] starting to ping
    [2016-07-12 00:30:42,477][TRACE][discovery.ec2            ] [ip-172-29-1-44-data] building dynamic unicast discovery nodes...
    [2016-07-12 00:30:42,477][DEBUG][discovery.ec2            ] [ip-172-29-1-44-data] using dynamic discovery nodes []
    [2016-07-12 00:30:42,480][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}
    [2016-07-12 00:30:42,480][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_2#}{127.0.0.1}{127.0.0.1:9301}
    [2016-07-12 00:30:42,482][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_4#}{127.0.0.1}{127.0.0.1:9303}
    [2016-07-12 00:30:42,482][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_5#}{127.0.0.1}{127.0.0.1:9304}
    [2016-07-12 00:30:42,482][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_3#}{127.0.0.1}{127.0.0.1:9302}
    [2016-07-12 00:30:42,483][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_6#}{::1}{[::1]:9300}
    [2016-07-12 00:30:42,485][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_7#}{::1}{[::1]:9301}
    [2016-07-12 00:30:42,485][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_8#}{::1}{[::1]:9302}
    [2016-07-12 00:30:42,487][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_9#}{::1}{[::1]:9303}
    [2016-07-12 00:30:42,487][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] connecting (light) to {#zen_unicast_10#}{::1}{[::1]:9304}
    [2016-07-12 00:30:42,508][TRACE][discovery.zen.ping.unicast] [ip-172-29-1-44-data] [1] failed to connect to {#zen_unicast_3#}{127.0.0.1}{127.0.0.1:9302}
    ConnectTransportException[[][127.0.0.1:9302] connect_timeout[30s]]; nested: ConnectException[Connection refused: /127.0.0.1:9302];
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:1008)
3

A quick sanity check first - did you install the ec2 discovery plugin?

4

Thanks for your response. I was about to figure out. Those instances were unable to connect to AWS or outside world, as a result, they didn't the actual host of ES, so unable to connect. Once I configured NAT gateway in that private subnet, all of them were able to discover rest of the ES hosts.