Elasticsearch nodes join cluster using discovery

[sharon.c]

I installed 2 Elasticsearch 5.2 nodes on EC2, inside the same VPC. I set these two nodes with the security group to allow inbund port 9200, 9300 and ICMP within the same VPC. I installed ec2 discovery plugin for nodes to discover the cluster

I need these two Elasticsearch nodes to join the same cluster, the following is my settings.
The followings are two elasticsearch.yml settings

discovery:
      type: ec2
cluster.name: mycluster
node.name: datanode1
node.master: true
node.data: true
node.ingest: false
path.data: /data
network.host: 10.XXX.XXX.XXX
network.publish_host: 10.XXX.XXX.XXX
network.bind_host: 10.XXX.XXX.XXX


discovery:
      type: ec2
cluster.name: mycluster
node.name: datanode2
node.master: true
node.data: true
node.ingest: false
path.data: /data
network.host: 10.XXX.XXX.XXX
network.publish_host: 10.XXX.XXX.XXX
network.bind_host: 10.XXX.XXX.XXX

The two nodes can do curl and ping to each other successfully, but they cannot discover each other.

I used to use Elasticsearch 2.3, we use zen discovery, and it works. But when I use the same settings in Elasticsearch 5.2, elasticsearch starts for few seconds and stops
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1","host2"]

How can I make these two Elasticsearch 5.2 nodes to join in one cluster?

[ywelsch (Yannick Welsch)]

Do you want to use Zen or EC2 discovery? Your config is a mix of the two. A properly configured EC2 discovery makes it so you don't have to provide an explicit list of hosts (see here: https://www.elastic.co/guide/en/elasticsearch/plugins/5.2/discovery-ec2.html )
The discovery.zen.ping.unicast.hosts setting is not used for EC2 discovery but for the standard zen discovery mechanism (which you can get by setting the discovery type back to "zen") .

[sharon.c]

I want to use zen, because the ec2 instances are in the same vpc, I think it would be ok to use zen, right?

discovery:
      type: zen
cluster.name: mycluster
node.name: datanode1
node.master: true
node.data: true
node.ingest: false
path.data: /data
network.host: 10.XXX.XXX.XXX
network.publish_host: 10.XXX.XXX.XXX
network.bind_host: 10.XXX.XXX.XXX
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.XXX.XXX.XXX","10.XXX.XXX.XXX"]

is this correct configuration for zen?
The nodes still cannot find each other to join a cluster

[ywelsch (Yannick Welsch)]

Can you establish a manual connection between the instances by running Elasticsearch on one instance and using telnet 10.XXX.XXX.XXX 9300 on the other instance to check the connection? (Port 9300 is the default port used for inter-node communication). Have you properly set-up the security groups? See "Step 6: Configure Security Group" at https://www.elastic.co/blog/running-elasticsearch-on-aws