# Elasticsearch 7.1.1 not forming a cluster using EC2 auto discovery plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2019, 1:46pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199 "2019-06-04T13:46:34Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\ **Post date:** [June 4, 2019, 1:46pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/1 "2019-06-04T13:46:34Z")
I have a 3 node cluster which I have setup in AWS using the AWS EC2 discovery plugin, each of the nodes can communicate with each other and autodiscovery finds the other nodes, but they dont form a cluster. I am using a basic licence and have not yet turned on SSL, not sure if this is now a requirement? There are no errors in the logs, but each node just forms its own cluster and not one single cluster. **elasticsearch.yml** discovery.seed\_providers: ec2 discovery.ec2.tag.discovery: my-cluster-discovery-tag cloud.node.auto\_attributes: true cluster.routing.allocation.awareness.attributes: aws\_availability\_zone cluster.name: my-cluster-name network.host: [_local_, _site_] logger.org.elasticsearch.discovery: TRACE Logs [2019-06-04T13:25:17,830][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] not active [2019-06-04T13:25:17,883][INFO][o.e.c.s.MasterService] [ip-10-241-0-24.ec2.internal] elected-as-master ([1] nodes joined)[{ip-10-241-0-24.ec2.internal}{xI8fYZUmQ8WAZfIMjgp9sQ}{4QwPqtsFTr6H8WyImS25gg}{10.241.0.24}{10.241.0.24:9300}{aws\_availability\_zone=us-east-1a, ml.machine\_memory=8362668032, xpack.installed=true, ml.max\_open\_jobs=20} elect leader, _BECOME\_MASTER\_TASK_, _FINISH\_ELECTION_], term: 18, version: 64, reason: master node changed {previous , current [{ip-10-241-0-24.ec2.internal}{xI8fYZUmQ8WAZfIMjgp9sQ}{4QwPqtsFTr6H8WyImS25gg}{10.241.0.24}{10.241.0.24:9300}{aws\_availability\_zone=us-east-1a, ml.machine\_memory=8362668032, xpack.installed=true, ml.max\_open\_jobs=20}]} [2019-06-04T13:25:18,012][INFO][o.e.c.s.ClusterApplierService] [ip-10-241-0-24.ec2.internal] master node changed {previous , current [{ip-10-241-0-24.ec2.internal}{xI8fYZUmQ8WAZfIMjgp9sQ}{4QwPqtsFTr6H8WyImS25gg}{10.241.0.24}{10.241.0.24:9300}{aws\_availability\_zone=us-east-1a, ml.machine\_memory=8362668032, xpack.installed=true, ml.max\_open\_jobs=20}]}, term: 18, version: 64, reason: Publication{term=18, version=64} [2019-06-04T13:25:18,200][INFO][o.e.h.AbstractHttpServerTransport] [ip-10-241-0-24.ec2.internal] publish\_address {10.241.0.24:9200}, bound\_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.241.0.24:9200} [2019-06-04T13:25:18,201][INFO][o.e.n.Node] [ip-10-241-0-24.ec2.internal] started [2019-06-04T13:25:18,546][WARN][o.e.x.s.a.s.m.NativeRoleMappingStore] [ip-10-241-0-24.ec2.internal] Failed to clear cache for realms [] [2019-06-04T13:25:18,637][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] not active [2019-06-04T13:25:18,762][INFO][o.e.l.LicenseService] [ip-10-241-0-24.ec2.internal] license [4cbbefce-620e-48e7-aada-64b0f04d8f51] mode [basic] - valid [2019-06-04T13:25:18,786][INFO][o.e.g.GatewayService] [ip-10-241-0-24.ec2.internal] recovered [0] indices into cluster\_state [2019-06-04T13:25:19,183][TRACE][o.e.d.e.AwsEc2SeedHostsProvider] [ip-10-241-0-24.ec2.internal] finding seed nodes... [2019-06-04T13:25:19,184][TRACE][o.e.d.e.AwsEc2SeedHostsProvider] [ip-10-241-0-24.ec2.internal] adding i-003c7e2a6e0e097a5, address 10.241.0.105, transport\_address 10.241.0.105:9300 [2019-06-04T13:25:19,184][TRACE][o.e.d.e.AwsEc2SeedHostsProvider] [ip-10-241-0-24.ec2.internal] adding i-0d7c3be1d22faa9b7, address 10.241.0.144, transport\_address 10.241.0.144:9300 [2019-06-04T13:25:19,184][TRACE][o.e.d.e.AwsEc2SeedHostsProvider] [ip-10-241-0-24.ec2.internal] adding i-0c99fd54453835c02, address 10.241.0.24, transport\_address 10.241.0.24:9300 [2019-06-04T13:25:19,185][DEBUG][o.e.d.e.AwsEc2SeedHostsProvider] [ip-10-241-0-24.ec2.internal] using dynamic transport addresses [10.241.0.105:9300, 10.241.0.144:9300, 10.241.0.24:9300] **[2019-06-04T13:25:19,185][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] probing resolved transport addresses [127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, 10.241.0.105:9300, 10.241.0.144:9300, 10.241.0.24:9300]** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(127.0.0.1:9301) not running** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(127.0.0.1:9302) not running** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(127.0.0.1:9303) not running** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(127.0.0.1:9304) not running** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe([::1]:9301) not running** **[2019-06-04T13:25:19,186][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe([::1]:9302) not running** **[2019-06-04T13:25:19,187][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe([::1]:9303) not running** **[2019-06-04T13:25:19,187][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe([::1]:9304) not running** **[2019-06-04T13:25:19,187][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(10.241.0.105:9300) not running** **[2019-06-04T13:25:19,187][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(10.241.0.144:9300) not running** **[2019-06-04T13:25:19,187][TRACE][o.e.d.PeerFinder] [ip-10-241-0-24.ec2.internal] startProbe(10.241.0.24:9300) not running** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 4, 2019, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/2 "2019-06-04T14:42:23Z")
Hi @Pat_Humphreys, > [@Pat\_Humphreys](#): > > There are no errors in the logs, but each node just forms its own cluster and not one single cluster. I suspect that the first time you started these nodes they did not have `discovery.seed_providers: ec2` and therefore they [auto-bootstrapped into separate clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#_auto_bootstrapping_in_development_mode). The note at the bottom of that manual page goes into more detail about what this means and what you should do to fix it. ---
**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\ **Post date:** [June 4, 2019, 4:05pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/3 "2019-06-04T16:05:55Z")
Thanks @DavidTurner that was what was preventing me from getting the errors. But now I am getting a master\_not\_discovered\_exception. Which is a bit odd as I can see the coms to all 3 hosts Also do I need to set discovery.seed\_hosts and cluster.initial\_master\_nodes? As I have discovery.seed\_providers: ec2 Startup logs [https://pastebin.com/CYSnxebs](https://pastebin.com/CYSnxebs) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 4, 2019, 5:08pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/4 "2019-06-04T17:08:20Z")
> [@Pat\_Humphreys](#): > > But now I am getting a master\_not\_discovered\_exception. The logs you quoted only last a short time and look normal enough. Can you share more? I'm particularly looking for a message containing the string `ClusterFormationFailureHelper`. > [@Pat\_Humphreys](#): > > Also do I need to set discovery.seed\_hosts and cluster.initial\_master\_nodes? As I have discovery.seed\_providers: ec2 You do not need `discovery.seed_hosts` since this is taken care of by `discovery.seed_providers: ec2`. You do, however, need `cluster.initial_master_nodes` to get the cluster going in the first place. ---
**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\ **Post date:** [June 5, 2019, 9:55am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/5 "2019-06-05T09:55:54Z")
I have v fixed it thanks a lot for your help. So my problem was not specifying the value cluster.initial\_master\_nodes Its not great now having to hardcode this value as I have many clusters (with all nodes master eligible ) which I spin up dynamically on AWS with an AMI with all the config and a userdata script for cluster specific settings. So im going to end up writing a script to query the AWS describe API to get the ips for that cluster, but the EC2 discovery plugin already has this available to it. Seems like a missing feature of EC2 autodiscovery ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 5, 2019, 12:43pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/6 "2019-06-05T12:43:32Z")
> [@Pat\_Humphreys](#): > > So im going to end up writing a script to query the AWS describe API to get the ips for that cluster, but the EC2 discovery plugin already has this available to it. Unfortunately the `DescribeInstances` API doesn't give strong enough consistency guarantees for setting `cluster.initial_master_nodes`. There's a risk you will get inconsistent results on the different nodes, and this could result in forming multiple clusters without realising. It's quite a shame, because if it was consistent enough then we'd certainly be using it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2019, 12:43pm UTC](https://discuss.elastic.co/t/elasticsearch-7-1-1-not-forming-a-cluster-using-ec2-auto-discovery-plugin/184199/7 "2019-07-03T12:43:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.