EC2 Cluster Issues


(NickHaggmark) #1

Hi Guys,

I realize that this question has been asked and answered many times,
however I'm having a bear of a time with it and I was hoping someone
might be able to help me out.

I'm currently running 2 ubuntu instances in EC2 with Elasticsearch
0.19.3 and I cannot get them to speak to one another. They are both
part of one security group (exposing ports 22, 9200, 9300), both have
the aws 1.5 plugin.

I'm using a very bare bones configuration (between the two lines:

#######################################
cluster.name: smm

path.data: /usr/local/var/data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch

bootstrap.mlockall: true

cloud:
aws:
access_key: XXXXXXXXXXXX
secret_key: YYYYYYYYYYYY

discovery:
type: ec2
ping_timeout: 30s
#######################################

Here are the logs from start up:

[2012-05-31 22:29:27,082][INFO ][plugins ] [Bogeyman]
loaded [cloud-aws], sites [head]
[2012-05-31 22:29:30,520][DEBUG][discovery.zen.ping.multicast]
[Bogeyman] using group [224.2.2.4], with port [54328], ttl [3], and
address [null]
[2012-05-31 22:29:30,524][DEBUG][discovery.zen.ping.unicast]
[Bogeyman] using initial hosts [], with concurrent_connects [10]
[2012-05-31 22:29:30,525][DEBUG][discovery.ec2 ] [Bogeyman]
using ping.timeout [3s]
[2012-05-31 22:29:30,530][DEBUG][discovery.zen.elect ] [Bogeyman]
using minimum_master_nodes [-1]
[2012-05-31 22:29:30,532][DEBUG][discovery.zen.fd ] [Bogeyman]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:30,535][DEBUG][discovery.zen.fd ] [Bogeyman]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:31,533][DEBUG][discovery.ec2 ] [Bogeyman]
using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group
[true], availability_zones [[]]
[2012-05-31 22:29:34,522][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: initialized
[2012-05-31 22:29:34,523][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: starting ...
[2012-05-31 22:29:34,698][INFO ][transport ] [Bogeyman]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
10.168.71.28:9300]}
[2012-05-31 22:29:34,945][TRACE][discovery ] [Bogeyman]
waiting for 30s for the initial state to be set by the discovery
[2012-05-31 22:29:35,945][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:37,597][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,250][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:39,251][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,252][DEBUG][discovery.ec2 ] [Bogeyman]
ping responses: {none}
[2012-05-31 22:29:39,256][INFO ][cluster.service ] [Bogeyman]
new_master [Bogeyman][thRSc8zgTQmE5unwlp84Qw][inet[/
10.168.71.28:9300]], reason: zen-disco-join (elected_as_master)
[2012-05-31 22:29:39,479][TRACE][discovery ] [Bogeyman]
initial state set from discovery
[2012-05-31 22:29:39,479][INFO ][discovery ] [Bogeyman]
smm/thRSc8zgTQmE5unwlp84Qw
[2012-05-31 22:29:39,526][INFO ][http ] [Bogeyman]
bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
10.168.71.28:9200]}
[2012-05-31 22:29:39,526][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: started
[2012-05-31 22:29:39,604][INFO ][gateway ] [Bogeyman]
recovered [0] indices into cluster_state

Any suggestions or help for a relative newbie would be most
appreciated!!!

Thanks!

Nick


(Patrick) #2

Sorry if I'm covering bases that you may have already covered, but bare
with me :slight_smile:

Patrick


patrick eefy net

On Thu, May 31, 2012 at 4:36 PM, NickHaggmark nhaggmark@gmail.com wrote:

Hi Guys,

I realize that this question has been asked and answered many times,
however I'm having a bear of a time with it and I was hoping someone
might be able to help me out.

I'm currently running 2 ubuntu instances in EC2 with Elasticsearch
0.19.3 and I cannot get them to speak to one another. They are both
part of one security group (exposing ports 22, 9200, 9300), both have
the aws 1.5 plugin.

I'm using a very bare bones configuration (between the two lines:

#######################################
cluster.name: smm

path.data: /usr/local/var/data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch

bootstrap.mlockall: true

cloud:
aws:
access_key: XXXXXXXXXXXX
secret_key: YYYYYYYYYYYY

discovery:
type: ec2
ping_timeout: 30s
#######################################

Here are the logs from start up:

[2012-05-31 22:29:27,082][INFO ][plugins ] [Bogeyman]
loaded [cloud-aws], sites [head]
[2012-05-31 22:29:30,520][DEBUG][discovery.zen.ping.multicast]
[Bogeyman] using group [224.2.2.4], with port [54328], ttl [3], and
address [null]
[2012-05-31 22:29:30,524][DEBUG][discovery.zen.ping.unicast]
[Bogeyman] using initial hosts [], with concurrent_connects [10]
[2012-05-31 22:29:30,525][DEBUG][discovery.ec2 ] [Bogeyman]
using ping.timeout [3s]
[2012-05-31 22:29:30,530][DEBUG][discovery.zen.elect ] [Bogeyman]
using minimum_master_nodes [-1]
[2012-05-31 22:29:30,532][DEBUG][discovery.zen.fd ] [Bogeyman]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:30,535][DEBUG][discovery.zen.fd ] [Bogeyman]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:31,533][DEBUG][discovery.ec2 ] [Bogeyman]
using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group
[true], availability_zones [[]]
[2012-05-31 22:29:34,522][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: initialized
[2012-05-31 22:29:34,523][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: starting ...
[2012-05-31 22:29:34,698][INFO ][transport ] [Bogeyman]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
10.168.71.28:9300]}
[2012-05-31 22:29:34,945][TRACE][discovery ] [Bogeyman]
waiting for 30s for the initial state to be set by the discovery
[2012-05-31 22:29:35,945][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:37,597][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,250][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:39,251][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,252][DEBUG][discovery.ec2 ] [Bogeyman]
ping responses: {none}
[2012-05-31 22:29:39,256][INFO ][cluster.service ] [Bogeyman]
new_master [Bogeyman][thRSc8zgTQmE5unwlp84Qw][inet[/
10.168.71.28:9300]], reason: zen-disco-join (elected_as_master)
[2012-05-31 22:29:39,479][TRACE][discovery ] [Bogeyman]
initial state set from discovery
[2012-05-31 22:29:39,479][INFO ][discovery ] [Bogeyman]
smm/thRSc8zgTQmE5unwlp84Qw
[2012-05-31 22:29:39,526][INFO ][http ] [Bogeyman]
bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
10.168.71.28:9200]}
[2012-05-31 22:29:39,526][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: started
[2012-05-31 22:29:39,604][INFO ][gateway ] [Bogeyman]
recovered [0] indices into cluster_state

Any suggestions or help for a relative newbie would be most
appreciated!!!

Thanks!

Nick


(NickHaggmark) #3

Thanks for the response! I figured out my problem last night.
Basically, I'm operating in the us-west-1 region and I didn't realize
that it defaulted to us-east. Once I specified my region explicitly,
auto detection worked like a charm.

Thanks!

Nick

On May 31, 5:28 pm, Patrick patr...@eefy.net wrote:

Sorry if I'm covering bases that you may have already covered, but bare
with me :slight_smile:

  • Have you confirmed you can ping the nodes from each side?
  • Are they in the same region, and/or zone?
  • Have you confirmed the same AWS key on both sides?
  • Have you tried to have them directly discover one another (alahttp://www.elasticsearch.org/guide/reference/modules/discovery/zen.ht...

Patrick
----------------------------------------http://about.me/patrick.ancillotti
patrick eefy net

On Thu, May 31, 2012 at 4:36 PM, NickHaggmark nhaggm...@gmail.com wrote:

Hi Guys,

I realize that this question has been asked and answered many times,
however I'm having a bear of a time with it and I was hoping someone
might be able to help me out.

I'm currently running 2 ubuntu instances in EC2 with Elasticsearch
0.19.3 and I cannot get them to speak to one another. They are both
part of one security group (exposing ports 22, 9200, 9300), both have
the aws 1.5 plugin.

I'm using a very bare bones configuration (between the two lines:

#######################################
cluster.name: smm

path.data: /usr/local/var/data/elasticsearch
path.logs: /usr/local/var/log/elasticsearch

bootstrap.mlockall: true

cloud:
aws:
access_key: XXXXXXXXXXXX
secret_key: YYYYYYYYYYYY

discovery:
type: ec2
ping_timeout: 30s
#######################################

Here are the logs from start up:

[2012-05-31 22:29:27,082][INFO ][plugins ] [Bogeyman]
loaded [cloud-aws], sites [head]
[2012-05-31 22:29:30,520][DEBUG][discovery.zen.ping.multicast]
[Bogeyman] using group [224.2.2.4], with port [54328], ttl [3], and
address [null]
[2012-05-31 22:29:30,524][DEBUG][discovery.zen.ping.unicast]
[Bogeyman] using initial hosts [], with concurrent_connects [10]
[2012-05-31 22:29:30,525][DEBUG][discovery.ec2 ] [Bogeyman]
using ping.timeout [3s]
[2012-05-31 22:29:30,530][DEBUG][discovery.zen.elect ] [Bogeyman]
using minimum_master_nodes [-1]
[2012-05-31 22:29:30,532][DEBUG][discovery.zen.fd ] [Bogeyman]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:30,535][DEBUG][discovery.zen.fd ] [Bogeyman]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-05-31 22:29:31,533][DEBUG][discovery.ec2 ] [Bogeyman]
using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group
[true], availability_zones [[]]
[2012-05-31 22:29:34,522][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: initialized
[2012-05-31 22:29:34,523][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: starting ...
[2012-05-31 22:29:34,698][INFO ][transport ] [Bogeyman]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
10.168.71.28:9300]}
[2012-05-31 22:29:34,945][TRACE][discovery ] [Bogeyman]
waiting for 30s for the initial state to be set by the discovery
[2012-05-31 22:29:35,945][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:35,945][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:37,597][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:37,597][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,250][TRACE][discovery.ec2 ] [Bogeyman]
building dynamic unicast discovery nodes...
[2012-05-31 22:29:39,251][DEBUG][discovery.ec2 ] [Bogeyman]
using dynamic discovery nodes []
[2012-05-31 22:29:39,252][DEBUG][discovery.ec2 ] [Bogeyman]
ping responses: {none}
[2012-05-31 22:29:39,256][INFO ][cluster.service ] [Bogeyman]
new_master [Bogeyman][thRSc8zgTQmE5unwlp84Qw][inet[/
10.168.71.28:9300]], reason: zen-disco-join (elected_as_master)
[2012-05-31 22:29:39,479][TRACE][discovery ] [Bogeyman]
initial state set from discovery
[2012-05-31 22:29:39,479][INFO ][discovery ] [Bogeyman]
smm/thRSc8zgTQmE5unwlp84Qw
[2012-05-31 22:29:39,526][INFO ][http ] [Bogeyman]
bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
10.168.71.28:9200]}
[2012-05-31 22:29:39,526][INFO ][node ] [Bogeyman]
{0.19.3}[17859]: started
[2012-05-31 22:29:39,604][INFO ][gateway ] [Bogeyman]
recovered [0] indices into cluster_state

Any suggestions or help for a relative newbie would be most
appreciated!!!

Thanks!

Nick


(system) #4