[Solved] Discovery_ec2 with ES 6.3.2 not finding other nodes

Elastic Stack Elasticsearch
1

I'm doing my first 6.x install and trying to get the new discovery_ec2 plugin to work and it isn't finding any of my other nodes:

I have 1 master node and 2 data only nodes. The master node comes up and votes itself master. The two data nodes during master discovery don't appear to be able to find the master node:

[2018-08-10T23:43:31,480][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [es_ip-10-174-0-164] no known master node, scheduling a retry
[2018-08-10T23:43:38,617][WARN ][o.e.d.z.ZenDiscovery     ] [es_ip-10-174-0-164] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-08-10T23:44:01,481][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [es_ip-10-174-0-164] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])

My understanding is that as of 6.x the discovery_ec2 plugin uses the IAM profile credentials of the EC2 to do the discovery. I have verified that my IAM role the EC2 is running under has permissions to do the discovery.

I look for other members of the cluster by searching EC2's by tags:

# discovery
discovery.zen.hosts_provider: ec2
discovery.ec2.host_type: private_ip
discovery.zen.ping_timeout: 120s
discovery.ec2.groups: sg-064eba472ffef646d
discovery.ec2.tag.Owner: devops
discovery.ec2.tag.Mode: user
discovery.ec2.tag.Carrier: XX
discovery.ec2.tag.Group: user

Do I need to set up anything special on the EC2? Does the user account need an ~/.aws folder with valid creditials/secret key? Or should it be able to determine all of that from the IAM profile?

Also, we're running these EC2 behind a web proxy and have also set the discovery proxy:

discovery.ec2.protocol: https
discovery.ec2.proxy.host: foo.proxy.int
discovery.ec2.proxy.port: 1234

Any suggestions would be really appreciated... trying to migrate off of a really old 5.1 install ...

Thanks!

2

In which region are running your nodes?

3

us-west-2

4

Looks like you have to supply an endpoint and the availability zones in order for it to work properly... would be nice to update the 6.x documentation to make this obvious...

Solution found here:

5

Yeah. There is a pending PR to fix the doc but this has not been merged yet.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.