# Struggling with EC2 Discovery **URL:** \ **Category:** Elasticsearch\ **Created:** [March 5, 2013, 5:54pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021 "2013-03-05T17:54:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 5:54pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/1 "2013-03-05T17:54:35Z")
I'm trying to follow this tutorial (though without S3; just using EC2): [http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html](http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html) Everything seems to go well until I start the 2nd server (where it should discover the 1st server via EC2). Here's what ElasticSearch is saying. It looks exactly the same as the 1st server -- that is, it does not seem to be discovering the other server. I've checked that ports 22, 9200, and 9300 are open on my security group (using RightScale, FWIW). [root@ip-10-197-21-96 elasticsearch-0.20.5]# bin/elasticsearch -Des.max-open-files=true -Xmx3g -Xms3g -f [2013-03-05 09:34:48,201][INFO][bootstrap] max\_open\_files [63981] [2013-03-05 09:34:49,859][INFO][node] [Moonhunter] {0.20.5}[23946]: initializing ... [2013-03-05 09:34:49,930][INFO][plugins] [Moonhunter] loaded [cloud-aws], sites [] [2013-03-05 09:34:51,601][DEBUG][discovery.zen.ping.multicast] [Moonhunter] using group [224.2.2.4], with port [54328], ttl [3], and address [null] [2013-03-05 09:34:51,606][DEBUG][discovery.zen.ping.unicast] [Moonhunter] using initial hosts [], with concurrent\_connects [10] [2013-03-05 09:34:51,607][DEBUG][discovery.ec2] [Moonhunter] using ping.timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data [false] [2013-03-05 09:34:51,608][DEBUG][discovery.zen.elect] [Moonhunter] using minimum\_master\_nodes [-1] [2013-03-05 09:34:51,609][DEBUG][discovery.zen.fd] [Moonhunter] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-03-05 09:34:51,614][DEBUG][discovery.zen.fd] [Moonhunter] [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-03-05 09:34:51,819][DEBUG][discovery.ec2] [Moonhunter] using host\_type [PRIVATE\_IP], tags [{}], groups [[]] with any\_group [true], availability\_zones [[]] [2013-03-05 09:34:52,668][DEBUG][gateway.local] [Moonhunter] using initial\_shards [quorum], list\_timeout [30s] [2013-03-05 09:34:52,826][DEBUG][gateway.local.state.meta] [Moonhunter] using gateway.local.auto\_import\_dangled [YES], with gateway.local.dangling\_timeout [2h] [2013-03-05 09:34:52,855][DEBUG][gateway.local.state.meta] [Moonhunter] took 28ms to load state [2013-03-05 09:34:52,856][DEBUG][gateway.local.state.shards] [Moonhunter] took 0s to load started shards state [2013-03-05 09:34:52,860][INFO][node] [Moonhunter] {0.20.5}[23946]: initialized [2013-03-05 09:34:52,861][INFO][node] [Moonhunter] {0.20.5}[23946]: starting ... [2013-03-05 09:34:52,998][INFO][transport] [Moonhunter] bound\_address {inet[/0.0.0.0:9300]}, publish\_address {inet[/10.197.21.96:9300]} [2013-03-05 09:34:53,004][TRACE][discovery] [Moonhunter] waiting for 30s for the initial state to be set by the discovery [2013-03-05 09:34:53,845][TRACE][discovery.ec2] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:53,846][DEBUG][discovery.ec2] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:55,514][TRACE][discovery.ec2] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:55,514][DEBUG][discovery.ec2] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:57,166][TRACE][discovery.ec2] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:57,166][DEBUG][discovery.ec2] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:57,167][TRACE][discovery.ec2] [Moonhunter] full ping responses: {none} [2013-03-05 09:34:57,168][DEBUG][discovery.ec2] [Moonhunter] filtered ping responses: (filter\_client[true], filter\_data[false]) {none} [2013-03-05 09:34:57,172][INFO][cluster.service] [Moonhunter] new\_master [Moonhunter][1xNSvy2ISMq43o3GNiH9Lg][inet[/10.197.21.96:9300]], reason: zen-disco-join (elected\_as\_master) [2013-03-05 09:34:57,207][TRACE][discovery] [Moonhunter] initial state set from discovery [2013-03-05 09:34:57,208][INFO][discovery] [Moonhunter] streamifiedstaging/1xNSvy2ISMq43o3GNiH9Lg [2013-03-05 09:34:57,224][INFO][http] [Moonhunter] bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/10.197.21.96:9200]} [2013-03-05 09:34:57,224][INFO][node] [Moonhunter] {0.20.5}[23946]: started [2013-03-05 09:34:57,295][INFO][gateway] [Moonhunter] recovered [0] indices into cluster\_state On a related note -- I'm also curious what happens with an EC2 cluster as servers come up and go down. I have an auto scaling array, so let's say Servers A and B are running... then traffic to our site increases, so servers C and D are launched. Then traffic decreases, and the auto-scaling array terminates servers A and B. Assuming that everything worked right, there is no data loss, correct? That is -- Servers C and D joined the cluster and retrieved their indexes from servers A and B, thus even though A and B are now gone, C and D contain all the necessary data? ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [March 5, 2013, 6:42pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/2 "2013-03-05T18:42:51Z")
Hi Zane, can you confirm that you have your elasticsearch.yml configured to use security groups for auto-discovery, like so? discovery: type: ec2 groups: my-security-group If so, is "my-security-group" used by any other servers? because as far as I know, ES will try to discover on every node using that security group, and if it's not running elasticsearch, then it will sometimes hang on discovery. I noticed this when I tried to use an existing security group for my elasticsearch nodes, and they started trying to cluster up with machines that were in the same security group, but not running elasticsearch. hope this helps! --paul On Tuesday, March 5, 2013 12:54:35 PM UTC-5, Zane Claes wrote: > I'm trying to follow this tutorial (though without S3; just using EC2): > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html) > > Everything seems to go well until I start the 2nd server (where it should > discover the 1st server via EC2). Here's what Elasticsearch is saying. It > looks exactly the same as the 1st server -- that is, it does not seem to > be > discovering the other server. I've checked that ports 22, 9200, and 9300 > are > open on my security group (using RightScale, FWIW). > > On a related note -- I'm also curious what happens with an EC2 cluster as > servers come up and go down. I have an auto scaling array, so let's say > Servers A and B are running... then traffic to our site increases, so > servers C and D are launched. Then traffic decreases, and the auto-scaling > array terminates servers A and B. Assuming that everything worked right, > there is no data loss, correct? That is -- Servers C and D joined the > cluster and retrieved their indexes from servers A and B, thus even though > A > and B are now gone, C and D contain all the necessary data? > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [March 5, 2013, 6:44pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/3 "2013-03-05T18:44:21Z")
to answer the second part of your question, there is no data loss as part of autoscaling. the shards will just be re-allocated. you can autoscale to your heart's content. --paul -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 7:23pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/4 "2013-03-05T19:23:46Z")
Hmm... no, I guess I'm not using discovery.groups. I didn't see any reference to including that in the tutorial. I'm also confused what value I'd use here... I'm using RightScale, so, I'm not sure if I put in the name of the RightScale security group...? Here's what my bash script does... echo 'Modifying ElasticSearch config...' echo "cluster.name: $ELASTICSEARCH\_CLUSTER" \>\> config/elasticsearch.yml echo "cloud.aws.access\_key: $ACCESS\_KEY" \>\> config/elasticsearch.yml echo "cloud.aws.secret\_key: $ACCESS\_SECRET" \>\> config/elasticsearch.yml echo "discovery.type: ec2" \>\> config/elasticsearch.yml echo "bootstrap.mlockall: true" \>\> config/elasticsearch.yml ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [March 5, 2013, 7:32pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/5 "2013-03-05T19:32:32Z")
ok, so it looks like you're using the section under "the simplest configuration". if you have any ec2 instances that aren't running elasticsearch, then this configuration won't work in my experience. scroll down to the "Filtering the discovery" section in the tutorial[http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html](http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html)to see the section I'm referring to. the "simplest configuration" assumes that all your ec2 instances are running elasticsearch, which probably isn't the case. what is happening is that your elasticsearch instances that are running, are trying to join with ec2 instances that aren't listening on 9200, and they are hanging. In order for this not to happen, you need some way of filtering out machines not running elasticsearch, hence the "filtering the discovery" section of the tutorial. the good news is that it sounds like this is probably the problem, half the battle! --paul On Tuesday, March 5, 2013 2:23:46 PM UTC-5, Zane Claes wrote: > Hmm... no, I guess I'm not using discovery.groups. I didn't see any > reference > to including that in the tutorial. I'm also confused what value I'd use > here... I'm using RightScale, so, I'm not sure if I put in the name of the > RightScale security group...? Here's what my bash script does... > > echo 'Modifying Elasticsearch config...' > echo "cluster.name: $ELASTICSEARCH\_CLUSTER" \>\> config/elasticsearch.yml > echo "cloud.aws.access\_key: $ACCESS\_KEY" \>\> config/elasticsearch.yml > echo "cloud.aws.secret\_key: $ACCESS\_SECRET" \>\> config/elasticsearch.yml > echo "discovery.type: ec2" \>\> config/elasticsearch.yml > echo "bootstrap.mlockall: true" \>\> config/elasticsearch.yml > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031067.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031067.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 9:19pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/6 "2013-03-05T21:19:55Z")
Okay; I set "discovery.groups: streamifiedstaging" but it did not have any change on the discovery. The new logs are below; the group seems to be used, but it does not help the discovery. I'm confused a little about exactly what this parameter is, though -- I've just supplied the same config name to both instances, on the assumption that the discovery.groups is an internal string used by ElasticSearch -- in other words, it does not "reference" the name of an EC2 security bucket (for example.) [2013-03-05 13:15:59,904][INFO][bootstrap] max\_open\_files [63981] [2013-03-05 13:16:03,074][INFO][node] [Sin] {0.20.5}[18148]: initializing ... [2013-03-05 13:16:03,090][INFO][plugins] [Sin] loaded [cloud-aws], sites [] [2013-03-05 13:16:04,995][DEBUG][discovery.zen.ping.multicast] [Sin] using group [224.2.2.4], with port [54328], ttl [3], and address [null] [2013-03-05 13:16:05,001][DEBUG][discovery.zen.ping.unicast] [Sin] using initial hosts [], with concurrent\_connects [10] [2013-03-05 13:16:05,003][DEBUG][discovery.ec2] [Sin] using ping.timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data [false] [2013-03-05 13:16:05,004][DEBUG][discovery.zen.elect] [Sin] using minimum\_master\_nodes [-1] [2013-03-05 13:16:05,006][DEBUG][discovery.zen.fd] [Sin] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-03-05 13:16:05,011][DEBUG][discovery.zen.fd] [Sin] [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-03-05 13:16:05,241][DEBUG][discovery.ec2] [Sin] using host\_type [PRIVATE\_IP], tags [{}], groups [[]] with any\_group [true], availability\_zones [[]] [2013-03-05 13:16:06,237][DEBUG][gateway.local] [Sin] using initial\_shards [quorum], list\_timeout [30s] [2013-03-05 13:16:06,515][DEBUG][gateway.local.state.meta] [Sin] using gateway.local.auto\_import\_dangled [YES], with gateway.local.dangling\_timeout [2h] [2013-03-05 13:16:06,564][DEBUG][gateway.local.state.meta] [Sin] took 48ms to load state [2013-03-05 13:16:06,565][DEBUG][gateway.local.state.shards] [Sin] took 0s to load started shards state [2013-03-05 13:16:06,570][INFO][node] [Sin] {0.20.5}[18148]: initialized [2013-03-05 13:16:06,571][INFO][node] [Sin] {0.20.5}[18148]: starting ... [2013-03-05 13:16:06,708][INFO][transport] [Sin] bound\_address {inet[/0.0.0.0:9300]}, publish\_address {inet[/10.197.54.29:9300]} [2013-03-05 13:16:06,720][TRACE][discovery] [Sin] waiting for 30s for the initial state to be set by the discovery [2013-03-05 13:16:07,655][TRACE][discovery.ec2] [Sin] building dynamic unicast discovery nodes... [2013-03-05 13:16:07,655][DEBUG][discovery.ec2] [Sin] using dynamic discovery nodes [] [2013-03-05 13:16:09,298][TRACE][discovery.ec2] [Sin] building dynamic unicast discovery nodes... [2013-03-05 13:16:09,298][DEBUG][discovery.ec2] [Sin] using dynamic discovery nodes [] [2013-03-05 13:16:10,940][TRACE][discovery.ec2] [Sin] building dynamic unicast discovery nodes... [2013-03-05 13:16:10,940][DEBUG][discovery.ec2] [Sin] using dynamic discovery nodes [] [2013-03-05 13:16:10,941][TRACE][discovery.ec2] [Sin] full ping responses: {none} [2013-03-05 13:16:10,941][DEBUG][discovery.ec2] [Sin] filtered ping responses: (filter\_client[true], filter\_data[false]) {none} [2013-03-05 13:16:10,946][INFO][cluster.service] [Sin] new\_master [Sin][VTTZwAPcQHSGNiAHVC616A][inet[/10.197.54.29:9300]], reason: zen-disco-join (elected\_as\_master) [2013-03-05 13:16:10,990][TRACE][discovery] [Sin] initial state set from discovery [2013-03-05 13:16:10,990][INFO][discovery] [Sin] streamifiedstaging/VTTZwAPcQHSGNiAHVC616A [2013-03-05 13:16:11,038][INFO][http] [Sin] bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/10.197.54.29:9200]} [2013-03-05 13:16:11,039][INFO][node] [Sin] {0.20.5}[18148]: started [2013-03-05 13:16:11,117][INFO][gateway] [Sin] recovered [0] indices into cluster\_state ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [March 5, 2013, 9:33pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/7 "2013-03-05T21:33:17Z")
This parameter isn't internal to Elasticsearch, it says: 'Any running ec2 instance that has the security group "streamifiedstaging" is an Elasticsearch node, and should join the cluster'. Your ec2 instances need to have the security group "streamifiedstaging", or else things won't work correctly. --paul On Tuesday, March 5, 2013 4:19:55 PM UTC-5, Zane Claes wrote: > Okay; I set "discovery.groups: streamifiedstaging" but it did not have any > change on the discovery. The new logs are below; the group seems to be > used, > but it does not help the discovery. > > I'm confused a little about exactly what this parameter is, though -- I've > just supplied the same config name to both instances, on the assumption > that > the discovery.groups is an internal string used by Elasticsearch -- in > other > words, it does not "reference" the name of an EC2 security bucket (for > example.) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031074.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031074.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 9:59pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/8 "2013-03-05T21:59:50Z")
Ugh; apparently I must be really thick, because I just can't get this to work. 1. I've tried both the RightScale security group "name" and ID (eg "sg-6d437528") 2. I've noticed a difference in the syntax you pasted above and that in the tutorial. Namely, the tutorial has an extra "hanging indedent" before the "groups:" Here's my security group: \ And my elasticsearch.yml: cluster.name: streamifiedstaging cloud.aws.access\_key: [REDACTED] cloud.aws.secret\_key: [REDACTED] discovery: type: ec2 groups: sg-6d437528 bootstrap.mlockall: true ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 5, 2013, 10:13pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/9 "2013-03-05T22:13:40Z")
We actually use tag filtering with great success on AWS\> we have multiple clusters in the same security group. For each ES conifg, you have to define something like this: discovery.ec2.tag.mytag: mycluster Then for each AWS node you want in the cluster, you need to define the tag as one of the name value pairs. So you would set the key as "mytag" and the value as "mycluster". When the nodes start up, they look for other nodes that have the same tag and value. It works really slick. - Craig On Tue, Mar 5, 2013 at 2:59 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > Ugh; apparently I must be really thick, because I just can't get this to > work. > > 1. I've tried both the RightScale security group "name" and ID (eg > "sg-6d437528") > 2. I've noticed a difference in the syntax you pasted above and that in the > tutorial. Namely, the tutorial has an extra "hanging indedent" before the > "groups:" > > Here's my security group: > \< > [http://elasticsearch-users.115913.n3.nabble.com/file/n4031077/Screen\_Shot\_2013-03-05\_at\_1.58.37\_PM.png](http://elasticsearch-users.115913.n3.nabble.com/file/n4031077/Screen_Shot_2013-03-05_at_1.58.37_PM.png) > > > > > And my elasticsearch.yml: > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031077.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031077.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- … CRAIG BROWN Search Architect youwho, Inc. [www.youwho.com](http://www.youwho.com) M: 801.680.7300 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 10:29pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/10 "2013-03-05T22:29:54Z")
Craig, To be clear -- are you using the tag WITH the group, or instead of the group field? I still get the same result when using the tag... in fact, the output has always been the same... It almost sounds like the servers cannot even communicate... in other words, it seems the problem is not the filtration, but the multicast/discovery itself... ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 5, 2013, 10:40pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/11 "2013-03-05T22:40:25Z")
Instead of the group. The servers all need to be in the same security group though. Check your security group settings I think you actually need to open the port ranges 9200-9400, not just 9200 and 9300. I need to double check our settings, but try that. - Craig On Tue, Mar 5, 2013 at 3:29 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > Craig, > > To be clear -- are you using the tag WITH the group, or instead of the > group > field? > I still get the same result when using the tag... in fact, the output has > always been the same... > It almost sounds like the servers cannot even communicate... > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031079.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031079.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- … CRAIG BROWN Search Architect youwho, Inc. [www.youwho.com](http://www.youwho.com) M: 801.680.7300 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 10:48pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/12 "2013-03-05T22:48:06Z")
I tried the whole range (9200-9400) and no luck- though I'm assuming we're talking TCP 😉 You can actually see my entire security bucket settings in the above screenshot... ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 10:49pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/13 "2013-03-05T22:49:51Z")
I just noticed one of the first lines in my above logs seems to imply that the multicast is using port 54328, not 9200/9300: [2013-03-05 14:43:44,305][DEBUG][discovery.zen.ping.multicast] [Lila Cheney] using group [224.2.2.4], with port [54328], ttl [3], and address [null] Thoughts? ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 5, 2013, 11:09pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/14 "2013-03-05T23:09:34Z")
Multicast should be disabled. It doesn't work on AWS. You can also enable trace logging for ES. That may help to track down the issue. - Craig On Tue, Mar 5, 2013 at 3:49 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > I just noticed one of the first lines in my above logs seems to imply that > the multicast is using port 54328, not 9200/9300: > > [2013-03-05 14:43:44,305][DEBUG][discovery.zen.ping.multicast] [Lila > Cheney] > using group [224.2.2.4], with port [54328], ttl [3], and address [null] > > Thoughts? > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031088.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031088.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- … CRAIG BROWN Search Architect youwho, Inc. [www.youwho.com](http://www.youwho.com) M: 801.680.7300 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 11:17pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/15 "2013-03-05T23:17:52Z")
The logs I've pasted above have **discovery: TRACE** enabled in logging.yml. Or are you talking about something else? Because I don't see much discovery information in them. ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 11:22pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/16 "2013-03-05T23:22:44Z")
I just confirmed that I can telnet between the 2 servers, eg: [zane@ip-10-197-54-29 ~]$ telnet 10.197.54.152 9200 Trying 10.197.54.152... Connected to 10.197.54.152. So connectivity/firewalling is not a problem... ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 5, 2013, 11:44pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/17 "2013-03-05T23:44:30Z")
How about telneting to 9300? It looks like trace logging is set up. Are the nodes just all electing themselves as master? You might want to try unicast discovery first to make sure that you can get all of your nodes working together. Or maybe try unicast with just 2 nodes first. I'm not sure now many nodes you're working with. - Craig On Tue, Mar 5, 2013 at 4:22 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > I just confirmed that I can telnet between the 2 servers, eg: > [zane@ip-10-197-54-29 ~]$ telnet 10.197.54.152 9200 > Trying 10.197.54.152... > Connected to 10.197.54.152. > > So connectivity/firewalling is not a problem... > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031093.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031093.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- … CRAIG BROWN Search Architect youwho, Inc. [www.youwho.com](http://www.youwho.com) M: 801.680.7300 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 5, 2013, 11:49pm UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/18 "2013-03-05T23:49:16Z")
Telnetting to 9300 works exactly the same. I'm running exactly 2 servers at the moment. Yes; the nodes are each setting themselves as master. The below paste seems to be the relevant part of the logs. I'm unclear exactly what's happening here, but at a guess, it sounds like there are no "dynamic discover nodes" (and there should be?) [2013-03-05 15:05:28,855][TRACE][discovery] [Patriot II] waiting for 30s for the initial state to be set by the discovery [2013-03-05 15:05:29,694][TRACE][discovery.ec2] [Patriot II] building dynamic unicast discovery nodes... [2013-03-05 15:05:29,695][DEBUG][discovery.ec2] [Patriot II] using dynamic discovery nodes [] [2013-03-05 15:05:31,348][TRACE][discovery.ec2] [Patriot II] building dynamic unicast discovery nodes... [2013-03-05 15:05:31,348][DEBUG][discovery.ec2] [Patriot II] using dynamic discovery nodes [] [2013-03-05 15:05:32,994][TRACE][discovery.ec2] [Patriot II] building dynamic unicast discovery nodes... [2013-03-05 15:05:32,995][DEBUG][discovery.ec2] [Patriot II] using dynamic discovery nodes [] [2013-03-05 15:05:32,996][TRACE][discovery.ec2] [Patriot II] full ping responses: {none} [2013-03-05 15:05:32,996][DEBUG][discovery.ec2] [Patriot II] filtered ping responses: (filter\_client[true], filter\_data[false]) {none} [2013-03-05 15:05:33,003][INFO][cluster.service] [Patriot II] new\_master [Patriot II][MGFdecz9TNeWOs8lDnWUvA][inet[/10.197.54.152:9300]], reason: zen-disco-join (elected\_as\_master) ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 6, 2013, 12:03am UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/19 "2013-03-06T00:03:38Z")
It looks like aws discovery is not finding a list of nodes to try to connect to, so each node thinks it's alone. I'd try unicast discovery first and make sure that works, then go back and try aws discovery again. - Craig On Tue, Mar 5, 2013 at 4:49 PM, inZania [zaneclaes@gmail.com](mailto:zaneclaes@gmail.com) wrote: > Telnetting to 9300 works exactly the same. I'm running exactly 2 servers at > the moment. > Yes; the nodes are each setting themselves as master. The below paste seems > to be the relevant part of the logs. I'm unclear exactly what's happening > here, but at a guess, it sounds like there are no "dynamic discover nodes" > (and there should be?) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031096.html](http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031096.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- … CRAIG BROWN Search Architect youwho, Inc. [www.youwho.com](http://www.youwho.com) M: 801.680.7300 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![inZania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inzania/32/2305_2.png) [@inZania](https://discuss.elastic.co/u/inZania)\ **Post date:** [March 6, 2013, 12:05am UTC](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021/20 "2013-03-06T00:05:55Z")
Okay, I got unicasting to work! Hard-coding the IP of the 1st server into the unicast setting of the 2nd allowed it to start up and discover the first. Subsequently, I get a good health check (below). Now where do I go from here to get EC2 working? [zane@ip-10-197-54-29 ~]$ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' { "cluster\_name" : "streamifiedstaging", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 }[zane@ip-10-197-54-29 ~]$ [Next page](https://discuss.elastic.co/t/struggling-with-ec2-discovery/11021.md?page=2)