Hello,
I am trying to get a cluster up and running on AWS. I have the ECS cluster created, containers assigned, but whenever the Elasticsearch tasks try to begin, it fails on the bootstrapping section with the following error:
[2019-08-01T22:44:38,730][WARN ][o.e.c.c.ClusterFormationFailureHelper] [80224704d1af] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{a45dbc021a35}{z4gdvtVIR6mZlk9_YmS5iA}{hHgsihl_SsW2yCeTUq6zvA}{10.19.27.206}{10.19.27.206:9300}{aws_availability_zone=us-west-2a, ml.machine_memory=30064771072, ml.max_open_jobs=20, xpack.installed=true}, {f2c7fc02cc7d}{7EoblbjKQtyh9UsRC8VKzg}{sdUY893nS0u7y9PGBYDg5g}{10.19.151.235}{10.19.151.235:9300}{aws_availability_zone=us-west-2c, ml.machine_memory=30064771072, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [127.0.0.1:9300, 10.19.29.62:9300, 10.19.27.206:9300, 10.19.151.235:9300, 10.19.70.82:9300, 10.19.25.225:9300, 10.19.108.106:9300, 10.19.4.78:9300, 10.19.136.101:9300, 10.19.85.235:9300] from hosts providers and [{80224704d1af}{9teQjoEMTYeQZ6wAVngDJQ}{QWnPveM8RmidzPfghN-JCA}{10.19.85.235}{10.19.85.235:9300}{aws_availability_zone=us-west-2b, ml.machine_memory=30064771072, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
From this document here, it looks like when starting a cluster, Elasticsearch needs to know the full list of master-eligible nodes. However, if I'm using ECS Auto Scaling Group to create the instances for the nodes, I won't know the node hosts ahead of time.
How should I go about addressing this issue? Previously, on v. 5.6.1, we used the discovery.ec2 plugin to find all the nodes that belong in a cluster. Why isn't this continuing to work as expected?