Struggling with EC2 Discovery

inZania · March 5, 2013, 5:54pm

I'm trying to follow this tutorial (though without S3; just using EC2): http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html

Everything seems to go well until I start the 2nd server (where it should discover the 1st server via EC2). Here's what ElasticSearch is saying. It looks exactly the same as the 1st server -- that is, it does not seem to be discovering the other server. I've checked that ports 22, 9200, and 9300 are open on my security group (using RightScale, FWIW).

[root@ip-10-197-21-96 elasticsearch-0.20.5]# bin/elasticsearch -Des.max-open-files=true -Xmx3g -Xms3g -f [2013-03-05 09:34:48,201][INFO ][bootstrap ] max_open_files [63981] [2013-03-05 09:34:49,859][INFO ][node ] [Moonhunter] {0.20.5}[23946]: initializing ... [2013-03-05 09:34:49,930][INFO ][plugins ] [Moonhunter] loaded [cloud-aws], sites [] [2013-03-05 09:34:51,601][DEBUG][discovery.zen.ping.multicast] [Moonhunter] using group [224.2.2.4], with port [54328], ttl [3], and address [null] [2013-03-05 09:34:51,606][DEBUG][discovery.zen.ping.unicast] [Moonhunter] using initial hosts [], with concurrent_connects [10] [2013-03-05 09:34:51,607][DEBUG][discovery.ec2 ] [Moonhunter] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false] [2013-03-05 09:34:51,608][DEBUG][discovery.zen.elect ] [Moonhunter] using minimum_master_nodes [-1] [2013-03-05 09:34:51,609][DEBUG][discovery.zen.fd ] [Moonhunter] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3] [2013-03-05 09:34:51,614][DEBUG][discovery.zen.fd ] [Moonhunter] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3] [2013-03-05 09:34:51,819][DEBUG][discovery.ec2 ] [Moonhunter] using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true], availability_zones [[]] [2013-03-05 09:34:52,668][DEBUG][gateway.local ] [Moonhunter] using initial_shards [quorum], list_timeout [30s] [2013-03-05 09:34:52,826][DEBUG][gateway.local.state.meta ] [Moonhunter] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h] [2013-03-05 09:34:52,855][DEBUG][gateway.local.state.meta ] [Moonhunter] took 28ms to load state [2013-03-05 09:34:52,856][DEBUG][gateway.local.state.shards] [Moonhunter] took 0s to load started shards state [2013-03-05 09:34:52,860][INFO ][node ] [Moonhunter] {0.20.5}[23946]: initialized [2013-03-05 09:34:52,861][INFO ][node ] [Moonhunter] {0.20.5}[23946]: starting ... [2013-03-05 09:34:52,998][INFO ][transport ] [Moonhunter] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.197.21.96:9300]} [2013-03-05 09:34:53,004][TRACE][discovery ] [Moonhunter] waiting for 30s for the initial state to be set by the discovery [2013-03-05 09:34:53,845][TRACE][discovery.ec2 ] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:53,846][DEBUG][discovery.ec2 ] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:55,514][TRACE][discovery.ec2 ] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:55,514][DEBUG][discovery.ec2 ] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:57,166][TRACE][discovery.ec2 ] [Moonhunter] building dynamic unicast discovery nodes... [2013-03-05 09:34:57,166][DEBUG][discovery.ec2 ] [Moonhunter] using dynamic discovery nodes [] [2013-03-05 09:34:57,167][TRACE][discovery.ec2 ] [Moonhunter] full ping responses: {none} [2013-03-05 09:34:57,168][DEBUG][discovery.ec2 ] [Moonhunter] filtered ping responses: (filter_client[true], filter_data[false]) {none} [2013-03-05 09:34:57,172][INFO ][cluster.service ] [Moonhunter] new_master [Moonhunter][1xNSvy2ISMq43o3GNiH9Lg][inet[/10.197.21.96:9300]], reason: zen-disco-join (elected_as_master) [2013-03-05 09:34:57,207][TRACE][discovery ] [Moonhunter] initial state set from discovery [2013-03-05 09:34:57,208][INFO ][discovery ] [Moonhunter] streamifiedstaging/1xNSvy2ISMq43o3GNiH9Lg [2013-03-05 09:34:57,224][INFO ][http ] [Moonhunter] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/10.197.21.96:9200]} [2013-03-05 09:34:57,224][INFO ][node ] [Moonhunter] {0.20.5}[23946]: started [2013-03-05 09:34:57,295][INFO ][gateway ] [Moonhunter] recovered [0] indices into cluster_state

On a related note -- I'm also curious what happens with an EC2 cluster as servers come up and go down. I have an auto scaling array, so let's say Servers A and B are running... then traffic to our site increases, so servers C and D are launched. Then traffic decreases, and the auto-scaling array terminates servers A and B. Assuming that everything worked right, there is no data loss, correct? That is -- Servers C and D joined the cluster and retrieved their indexes from servers A and B, thus even though A and B are now gone, C and D contain all the necessary data?

Paul_Sanwald_2 · March 5, 2013, 6:42pm

Hi Zane,
can you confirm that you have your elasticsearch.yml configured to use
security groups for auto-discovery, like so?

discovery:
type: ec2
groups: my-security-group

If so, is "my-security-group" used by any other servers? because as far as
I know, ES will try to discover on every node using that security group,
and if it's not running elasticsearch, then it will sometimes hang on
discovery. I noticed this when I tried to use an existing security group
for my elasticsearch nodes, and they started trying to cluster up with
machines that were in the same security group, but not running
elasticsearch.

hope this helps!

--paul

On Tuesday, March 5, 2013 12:54:35 PM UTC-5, Zane Claes wrote:

I'm trying to follow this tutorial (though without S3; just using EC2):
Elasticsearch Platform — Find real-time answers at scale | Elastic

Everything seems to go well until I start the 2nd server (where it should
discover the 1st server via EC2). Here's what Elasticsearch is saying. It
looks exactly the same as the 1st server -- that is, it does not seem to
be
discovering the other server. I've checked that ports 22, 9200, and 9300
are
open on my security group (using RightScale, FWIW).

On a related note -- I'm also curious what happens with an EC2 cluster as
servers come up and go down. I have an auto scaling array, so let's say
Servers A and B are running... then traffic to our site increases, so
servers C and D are launched. Then traffic decreases, and the auto-scaling
array terminates servers A and B. Assuming that everything worked right,
there is no data loss, correct? That is -- Servers C and D joined the
cluster and retrieved their indexes from servers A and B, thus even though
A
and B are now gone, C and D contain all the necessary data?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Paul_Sanwald_2 · March 5, 2013, 6:44pm

to answer the second part of your question, there is no data loss as part
of autoscaling. the shards will just be re-allocated. you can autoscale to
your heart's content.

--paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 7:23pm

Hmm... no, I guess I'm not using discovery.groups. I didn't see any reference to including that in the tutorial. I'm also confused what value I'd use here... I'm using RightScale, so, I'm not sure if I put in the name of the RightScale security group...? Here's what my bash script does...

echo 'Modifying ElasticSearch config...'
echo "cluster.name: $ELASTICSEARCH_CLUSTER" >> config/elasticsearch.yml
echo "cloud.aws.access_key: $ACCESS_KEY" >> config/elasticsearch.yml
echo "cloud.aws.secret_key: $ACCESS_SECRET" >> config/elasticsearch.yml
echo "discovery.type: ec2" >> config/elasticsearch.yml
echo "bootstrap.mlockall: true" >> config/elasticsearch.yml

Paul_Sanwald_2 · March 5, 2013, 7:32pm

ok, so it looks like you're using the section under "the simplest
configuration". if you have any ec2 instances that aren't running
elasticsearch, then this configuration won't work in my experience. scroll
down to the "Filtering the discovery" section in the tutorialhttp://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.htmlto see the section I'm referring to.

the "simplest configuration" assumes that all your ec2 instances are
running elasticsearch, which probably isn't the case. what is happening is
that your elasticsearch instances that are running, are trying to join with
ec2 instances that aren't listening on 9200, and they are hanging. In order
for this not to happen, you need some way of filtering out machines not
running elasticsearch, hence the "filtering the discovery" section of the
tutorial.

the good news is that it sounds like this is probably the problem, half the
battle!

--paul

On Tuesday, March 5, 2013 2:23:46 PM UTC-5, Zane Claes wrote:

Hmm... no, I guess I'm not using discovery.groups. I didn't see any
reference
to including that in the tutorial. I'm also confused what value I'd use
here... I'm using RightScale, so, I'm not sure if I put in the name of the
RightScale security group...? Here's what my bash script does...

echo 'Modifying Elasticsearch config...'
echo "cluster.name: $ELASTICSEARCH_CLUSTER" >> config/elasticsearch.yml
echo "cloud.aws.access_key: $ACCESS_KEY" >> config/elasticsearch.yml
echo "cloud.aws.secret_key: $ACCESS_SECRET" >> config/elasticsearch.yml
echo "discovery.type: ec2" >> config/elasticsearch.yml
echo "bootstrap.mlockall: true" >> config/elasticsearch.yml

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031067.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 9:19pm

Okay; I set "discovery.groups: streamifiedstaging" but it did not have any change on the discovery. The new logs are below; the group seems to be used, but it does not help the discovery.

I'm confused a little about exactly what this parameter is, though -- I've just supplied the same config name to both instances, on the assumption that the discovery.groups is an internal string used by ElasticSearch -- in other words, it does not "reference" the name of an EC2 security bucket (for example.)

[2013-03-05 13:15:59,904][INFO ][bootstrap [2013-03-05 13:16:03,074][INFO ][node [2013-03-05 13:16:03,090][INFO ][plugins [2013-03-05 13:16:04,995][DEBUG][discovery.zen [2013-03-05 13:16:05,001][DEBUG][discovery.zen [2013-03-05 13:16:05,003][DEBUG][discovery.ec2 [2013-03-05 13:16:05,004][DEBUG][discovery.zen.elect [2013-03-05 13:16:05,006][DEBUG][discovery.zen.fd [2013-03-05 13:16:05,011][DEBUG][discovery.zen.fd [2013-03-05 13:16:05,241][DEBUG][discovery.ec2 [2013-03-05 13:16:06,237][DEBUG][gateway.local [2013-03-05 13:16:06,515][DEBUG][gateway.local.state.meta [2013-03-05 13:16:06,564][DEBUG][gateway.local.state.meta [2013-03-05 13:16:06,565][DEBUG][gateway.local [2013-03-05 13:16:06,570][INFO ][node [2013-03-05 13:16:06,571][INFO ][node [2013-03-05 13:16:06,708][INFO ][transport [2013-03-05 13:16:06,720][TRACE][discovery [2013-03-05 13:16:07,655][TRACE][discovery.ec2 [2013-03-05 13:16:07,655][DEBUG][discovery.ec2 [2013-03-05 13:16:09,298][TRACE][discovery.ec2 [2013-03-05 13:16:09,298][DEBUG][discovery.ec2 [2013-03-05 13:16:10,940][TRACE][discovery.ec2 [2013-03-05 13:16:10,940][DEBUG][discovery.ec2 [2013-03-05 13:16:10,941][TRACE][discovery.ec2 [2013-03-05 13:16:10,941][DEBUG][discovery.ec2 [2013-03-05 13:16:10,946][INFO ][cluster.service [2013-03-05 13:16:10,990][TRACE][discovery [2013-03-05 13:16:10,990][INFO ][discovery [2013-03-05 13:16:11,038][INFO ][http [2013-03-05 13:16:11,039][INFO ][node [2013-03-05 13:16:11,117][INFO ][gateway ] max_open_files [63981]
] [Sin] {0.20.5}[18148]: initializing ...
] [Sin] loaded [cloud-aws], sites []
.ping.multicast] [Sin] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
.ping.unicast] [Sin] using initial hosts [], with concurrent_connects [10]
] [Sin] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
] [Sin] using minimum_master_nodes [-1]
] [Sin] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
] [Sin] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
] [Sin] using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true], availability_zones [[]]
] [Sin] using initial_shards [quorum], list_timeout [30s]
] [Sin] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
] [Sin] took 48ms to load state
.state.shards] [Sin] took 0s to load started shards state
] [Sin] {0.20.5}[18148]: initialized
] [Sin] {0.20.5}[18148]: starting ...
] [Sin] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.197.54.29:9300]}
] [Sin] waiting for 30s for the initial state to be set by the discovery
] [Sin] building dynamic unicast discovery nodes...
] [Sin] using dynamic discovery nodes []
] [Sin] building dynamic unicast discovery nodes...
] [Sin] using dynamic discovery nodes []
] [Sin] building dynamic unicast discovery nodes...
] [Sin] using dynamic discovery nodes []
] [Sin] full ping responses: {none}
] [Sin] filtered ping responses: (filter_client[true], filter_data[false]) {none}
] [Sin] new_master [Sin][VTTZwAPcQHSGNiAHVC616A][inet[/10.197.54.29:9300]], reason: zen-disco-join (elected_as_master)
] [Sin] initial state set from discovery
] [Sin] streamifiedstaging/VTTZwAPcQHSGNiAHVC616A
] [Sin] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/10.197.54.29:9200]}
] [Sin] {0.20.5}[18148]: started
] [Sin] recovered [0] indices into cluster_state

Paul_Sanwald_2 · March 5, 2013, 9:33pm

This parameter isn't internal to Elasticsearch, it says:
'Any running ec2 instance that has the security group "streamifiedstaging"
is an Elasticsearch node, and should join the cluster'.

Your ec2 instances need to have the security group "streamifiedstaging", or
else things won't work correctly.

--paul

On Tuesday, March 5, 2013 4:19:55 PM UTC-5, Zane Claes wrote:

Okay; I set "discovery.groups: streamifiedstaging" but it did not have any
change on the discovery. The new logs are below; the group seems to be
used,
but it does not help the discovery.

I'm confused a little about exactly what this parameter is, though -- I've
just supplied the same config name to both instances, on the assumption
that
the discovery.groups is an internal string used by Elasticsearch -- in
other
words, it does not "reference" the name of an EC2 security bucket (for
example.)

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031074.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 9:59pm

Ugh; apparently I must be really thick, because I just can't get this to work.

I've tried both the RightScale security group "name" and ID (eg "sg-6d437528")
I've noticed a difference in the syntax you pasted above and that in the tutorial. Namely, the tutorial has an extra "hanging indedent" before the "groups:"

Here's my security group:
<nabble_img src="Screen_Shot_2013-03-05_at_1.58.37_PM.png" border="0"/>

And my elasticsearch.yml:
cluster.name: streamifiedstaging
cloud.aws.access_key: [REDACTED]
cloud.aws.secret_key: [REDACTED]
discovery:
type: ec2
groups: sg-6d437528
bootstrap.mlockall: true

Craig_Brown · March 5, 2013, 10:13pm

We actually use tag filtering with great success on AWS> we have multiple
clusters in the same security group.

For each ES conifg, you have to define something like this:
discovery.ec2.tag.mytag: mycluster

Then for each AWS node you want in the cluster, you need to define the tag
as one of the name value pairs.
So you would set the key as "mytag" and the value as "mycluster".

When the nodes start up, they look for other nodes that have the same tag
and value. It works really slick.

Craig

On Tue, Mar 5, 2013 at 2:59 PM, inZania zaneclaes@gmail.com wrote:

Ugh; apparently I must be really thick, because I just can't get this to
work.

I've tried both the RightScale security group "name" and ID (eg
"sg-6d437528")

I've noticed a difference in the syntax you pasted above and that in the
tutorial. Namely, the tutorial has an extra "hanging indedent" before the
"groups:"

Here's my security group:
<
http://elasticsearch-users.115913.n3.nabble.com/file/n4031077/Screen_Shot_2013-03-05_at_1.58.37_PM.png

And my elasticsearch.yml:

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031077.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
…
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 10:29pm

Craig,

To be clear -- are you using the tag WITH the group, or instead of the group field?
I still get the same result when using the tag... in fact, the output has always been the same...
It almost sounds like the servers cannot even communicate... in other words, it seems the problem is not the filtration, but the multicast/discovery itself...

Craig_Brown · March 5, 2013, 10:40pm

Instead of the group. The servers all need to be in the same security group
though.

Check your security group settings I think you actually need to open the
port ranges 9200-9400, not just 9200 and 9300. I need to double check our
settings, but try that.

Craig

On Tue, Mar 5, 2013 at 3:29 PM, inZania zaneclaes@gmail.com wrote:

Craig,

To be clear -- are you using the tag WITH the group, or instead of the
group
field?
I still get the same result when using the tag... in fact, the output has
always been the same...
It almost sounds like the servers cannot even communicate...

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031079.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
…
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 10:48pm

I tried the whole range (9200-9400) and no luck- though I'm assuming we're talking TCP
You can actually see my entire security bucket settings in the above screenshot...

inZania · March 5, 2013, 10:49pm

I just noticed one of the first lines in my above logs seems to imply that the multicast is using port 54328, not 9200/9300:

[2013-03-05 14:43:44,305][DEBUG][discovery.zen.ping.multicast] [Lila Cheney] using group [224.2.2.4], with port [54328], ttl [3], and address [null]

Thoughts?

Craig_Brown · March 5, 2013, 11:09pm

Multicast should be disabled. It doesn't work on AWS. You can also enable
trace logging for ES. That may help to track down the issue.

Craig

On Tue, Mar 5, 2013 at 3:49 PM, inZania zaneclaes@gmail.com wrote:

I just noticed one of the first lines in my above logs seems to imply that
the multicast is using port 54328, not 9200/9300:

[2013-03-05 14:43:44,305][DEBUG][discovery.zen.ping.multicast] [Lila
Cheney]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]

Thoughts?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031088.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
…
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 11:17pm

The logs I've pasted above have discovery: TRACE enabled in logging.yml. Or are you talking about something else? Because I don't see much discovery information in them.

inZania · March 5, 2013, 11:22pm

I just confirmed that I can telnet between the 2 servers, eg:
[zane@ip-10-197-54-29 ~]$ telnet 10.197.54.152 9200
Trying 10.197.54.152...
Connected to 10.197.54.152.

So connectivity/firewalling is not a problem...

Craig_Brown · March 5, 2013, 11:44pm

How about telneting to 9300?
It looks like trace logging is set up.
Are the nodes just all electing themselves as master?
You might want to try unicast discovery first to make sure that you can get
all of your nodes working together. Or maybe try unicast with just 2 nodes
first. I'm not sure now many nodes you're working with.

Craig

On Tue, Mar 5, 2013 at 4:22 PM, inZania zaneclaes@gmail.com wrote:

I just confirmed that I can telnet between the 2 servers, eg:
[zane@ip-10-197-54-29 ~]$ telnet 10.197.54.152 9200
Trying 10.197.54.152...
Connected to 10.197.54.152.

So connectivity/firewalling is not a problem...

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031093.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
…
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 5, 2013, 11:49pm

Telnetting to 9300 works exactly the same. I'm running exactly 2 servers at the moment.
Yes; the nodes are each setting themselves as master. The below paste seems to be the relevant part of the logs. I'm unclear exactly what's happening here, but at a guess, it sounds like there are no "dynamic discover nodes" (and there should be?)

[2013-03-05 15:05:28,855][TRACE][discovery ] [Patriot II] waiting for 30s for the initial state to be set by the discovery
[2013-03-05 15:05:29,694][TRACE][discovery.ec2 ] [Patriot II] building dynamic unicast discovery nodes...
[2013-03-05 15:05:29,695][DEBUG][discovery.ec2 ] [Patriot II] using dynamic discovery nodes []
[2013-03-05 15:05:31,348][TRACE][discovery.ec2 ] [Patriot II] building dynamic unicast discovery nodes...
[2013-03-05 15:05:31,348][DEBUG][discovery.ec2 ] [Patriot II] using dynamic discovery nodes []
[2013-03-05 15:05:32,994][TRACE][discovery.ec2 ] [Patriot II] building dynamic unicast discovery nodes...
[2013-03-05 15:05:32,995][DEBUG][discovery.ec2 ] [Patriot II] using dynamic discovery nodes []
[2013-03-05 15:05:32,996][TRACE][discovery.ec2 ] [Patriot II] full ping responses: {none}
[2013-03-05 15:05:32,996][DEBUG][discovery.ec2 ] [Patriot II] filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-03-05 15:05:33,003][INFO ][cluster.service ] [Patriot II] new_master [Patriot II][MGFdecz9TNeWOs8lDnWUvA][inet[/10.197.54.152:9300]], reason: zen-disco-join (elected_as_master)

Craig_Brown · March 6, 2013, 12:03am

It looks like aws discovery is not finding a list of nodes to try to
connect to, so each node thinks it's alone.

I'd try unicast discovery first and make sure that works, then go back and
try aws discovery again.

Craig

On Tue, Mar 5, 2013 at 4:49 PM, inZania zaneclaes@gmail.com wrote:

Telnetting to 9300 works exactly the same. I'm running exactly 2 servers at
the moment.
Yes; the nodes are each setting themselves as master. The below paste seems
to be the relevant part of the logs. I'm unclear exactly what's happening
here, but at a guess, it sounds like there are no "dynamic discover nodes"
(and there should be?)

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Struggling-with-EC2-Discovery-tp4031058p4031096.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
…
CRAIG BROWN
Search Architect
youwho, Inc.

www.youwho.com

M: 801.680.7300

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

inZania · March 6, 2013, 12:05am

Okay, I got unicasting to work!

Hard-coding the IP of the 1st server into the unicast setting of the 2nd allowed it to start up and discover the first. Subsequently, I get a good health check (below).

Now where do I go from here to get EC2 working?

[zane@ip-10-197-54-29 ~]$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "streamifiedstaging",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}[zane@ip-10-197-54-29 ~]$