I'm doing my first 6.x install and trying to get the new discovery_ec2 plugin to work and it isn't finding any of my other nodes:
I have 1 master node and 2 data only nodes. The master node comes up and votes itself master. The two data nodes during master discovery don't appear to be able to find the master node:
[2018-08-10T23:43:31,480][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [es_ip-10-174-0-164] no known master node, scheduling a retry [2018-08-10T23:43:38,617][WARN ][o.e.d.z.ZenDiscovery ] [es_ip-10-174-0-164] not enough master nodes discovered during pinging (found [], but needed [-1]), pinging again [2018-08-10T23:44:01,481][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [es_ip-10-174-0-164] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
My understanding is that as of 6.x the discovery_ec2 plugin uses the IAM profile credentials of the EC2 to do the discovery. I have verified that my IAM role the EC2 is running under has permissions to do the discovery.
I look for other members of the cluster by searching EC2's by tags:
# discovery discovery.zen.hosts_provider: ec2 discovery.ec2.host_type: private_ip discovery.zen.ping_timeout: 120s discovery.ec2.groups: sg-064eba472ffef646d discovery.ec2.tag.Owner: devops discovery.ec2.tag.Mode: user discovery.ec2.tag.Carrier: XX discovery.ec2.tag.Group: user
Do I need to set up anything special on the EC2? Does the user account need an ~/.aws folder with valid creditials/secret key? Or should it be able to determine all of that from the IAM profile?
Also, we're running these EC2 behind a web proxy and have also set the discovery proxy:
discovery.ec2.protocol: https discovery.ec2.proxy.host: foo.proxy.int discovery.ec2.proxy.port: 1234
Any suggestions would be really appreciated... trying to migrate off of a really old 5.1 install ...