EC2 discovery issue

Ridvan_Gyundogan · July 4, 2011, 8:21pm

Hi,
I am trying to run 2 ec2 instances according to the following article
http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

Paul_Loy · July 4, 2011, 8:58pm

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan ridvansg@gmail.com wrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Paul_Loy · July 4, 2011, 9:01pm

You also need to specify IP addresses as the EC2 plugin searches other
instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy keteracel@gmail.com wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan ridvansg@gmail.comwrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Karl_Baum · July 4, 2011, 9:59pm

If we specify ip addresses, there probably is no advantage over zen discovery.

I vaguely remember that on head you don't need ip addresses for ec2 discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy keteracel@gmail.com wrote:

You also need to specify IP addresses as the EC2 plugin searches other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy keteracel@gmail.com wrote:
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan ridvansg@gmail.com wrote:
Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Paul_Loy · July 4, 2011, 10:37pm

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum karl.baum@gmail.com wrote:

If we specify ip addresses, there probably is no advantage over zen
discovery.

I vaguely remember that on head you don't need ip addresses for ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy keteracel@gmail.com wrote:

You also need to specify IP addresses as the EC2 plugin searches other
instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy < keteracel@gmail.com
keteracel@gmail.com> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan < ridvansg@gmail.com
ridvansg@gmail.com> wrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
paul@keteracel.com paul@keteracel.com
http://uk.linkedin.com/in/paulloy http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com paul@keteracel.com
http://uk.linkedin.com/in/paulloy http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

kimchy · July 5, 2011, 10:57am

Even without master (where you just say ec2), most cloud platform have common network interfaces names, and you can still use a single configuration file by providing the network interface name (check the network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.baum@gmail.com (mailto:karl.baum@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over zen discovery.

I vaguely remember that on head you don't need ip addresses for ec2 discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <keteracel@gmail.com (mailto:keteracel@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin searches other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <keteracel@gmail.com (mailto:keteracel@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <ridvansg@gmail.com (mailto:ridvansg@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
paul@keteracel.com (mailto:paul@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com (mailto:paul@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com (mailto:paul@keteracel.com)
http://uk.linkedin.com/in/paulloy

Ridvan_Gyundogan · July 5, 2011, 11:32am

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?
Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Even without master (where you just say ec2), most cloud platform have common network interfaces names, and you can still use a single configuration file by providing the network interface name (check the network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com (mailto:karl.b...@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over zen discovery.

I vaguely remember that on head you don't need ip addresses for ec2 discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin searches other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

Ridvan_Gyundogan · July 5, 2011, 12:14pm

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0:
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan ridva...@gmail.com wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Even without master (where you just say ec2), most cloud platform have common network interfaces names, and you can still use a single configuration file by providing the network interface name (check the network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com (mailto:karl.b...@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over zen discovery.

I vaguely remember that on head you don't need ip addresses for ec2 discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin searches other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the following article
Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

Paul_Loy · July 5, 2011, 12:26pm

network:
bind_host: ec2

?

On Tue, Jul 5, 2011 at 1:14 PM, Ridvan Gyundogan ridvansg@gmail.com wrote:

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0:
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan ridva...@gmail.com wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Even without master (where you just say ec2), most cloud platform
have common network interfaces names, and you can still use a single
configuration file by providing the network interface name (check the
network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com(mailto:
karl.b...@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over zen
discovery.

I vaguely remember that on head you don't need ip addresses for ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:
ketera...@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin searches
other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com(mailto:
ketera...@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <
ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the
following article

Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I
start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join
(elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a
cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Ridvan_Gyundogan · July 5, 2011, 2:58pm

Hi Paul thanks for your help, I still have an issue.
Now I get:
{elasticsearch/0.16.2}: Startup Failed ...

BindTransportException[Failed to resolve host [null]]
IOException[Failed to find network interface for [ec2]]

To summarize my elasticsearch.yml now looks like the following:
cluster:
name: perftests
discovery:
type: ec2
cloud:
aws:
access_key: XXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

network:
bind_host: ec2

On Jul 5, 3:26 pm, Paul Loy ketera...@gmail.com wrote:

network:
bind_host: ec2

?

On Tue, Jul 5, 2011 at 1:14 PM, Ridvan Gyundogan ridva...@gmail.com wrote:

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0:
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan ridva...@gmail.com wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Even without master (where you just say ec2), most cloud platform
have common network interfaces names, and you can still use a single
configuration file by providing the network interface name (check the
network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com(mailto:
karl.b...@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over zen
discovery.

I vaguely remember that on head you don't need ip addresses for ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:
ketera...@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin searches
other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com(mailto:
ketera...@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <
ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the
following article

Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I
start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join
(elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a
cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.comhttp://uk.linkedin.com/in/paulloy

Paul_Loy · July 5, 2011, 3:12pm

Is this with master? I think that should be 0.17.0 not 0.16.2

On Tue, Jul 5, 2011 at 3:58 PM, Ridvan Gyundogan ridvansg@gmail.com wrote:

Hi Paul thanks for your help, I still have an issue.
Now I get:
{elasticsearch/0.16.2}: Startup Failed ...

BindTransportException[Failed to resolve host [null]]
IOException[Failed to find network interface for [ec2]]

To summarize my elasticsearch.yml now looks like the following:
cluster:
name: perftests
discovery:
type: ec2
cloud:
aws:
access_key: XXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

network:
bind_host: ec2

On Jul 5, 3:26 pm, Paul Loy ketera...@gmail.com wrote:

network:
bind_host: ec2

?

On Tue, Jul 5, 2011 at 1:14 PM, Ridvan Gyundogan ridva...@gmail.com
wrote:

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0:
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan ridva...@gmail.com wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Even without master (where you just say ec2), most cloud platform
have common network interfaces names, and you can still use a single
configuration file by providing the network interface name (check the
network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com
(mailto:
karl.b...@gmail.com)> wrote:

If we specify ip addresses, there probably is no advantage over
zen
discovery.

I vaguely remember that on head you don't need ip addresses for
ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com(mailto:
ketera...@gmail.com)> wrote:

You also need to specify IP addresses as the EC2 plugin
searches
other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <
ketera...@gmail.com(mailto:
ketera...@gmail.com)> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <
ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the
following article

Elasticsearch Platform — Find real-time answers at scale | Elastic
.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one
I
start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]},
publish_address
{inet[/
10.234.149.155:9300 (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ]
[Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join
(elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]},
publish_address
{inet[/
10.234.149.155:9200 (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a
cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.comhttp://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

kimchy · July 6, 2011, 1:11am

network.host: ec2 is only available in master. Setting to non_loopback is not preferable, you should explicitly use the interface name matching the internal ip.

On Tuesday, July 5, 2011 at 6:12 PM, Paul Loy wrote:

Is this with master? I think that should be 0.17.0 not 0.16.2

On Tue, Jul 5, 2011 at 3:58 PM, Ridvan Gyundogan <ridvansg@gmail.com (mailto:ridvansg@gmail.com)> wrote:

Hi Paul thanks for your help, I still have an issue.
Now I get:
{elasticsearch/0.16.2}: Startup Failed ...

BindTransportException[Failed to resolve host [null]]
IOException[Failed to find network interface for [ec2]]

To summarize my elasticsearch.yml now looks like the following:
cluster:
name: perftests
discovery:
type: ec2
cloud:
aws:
access_key: XXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

network:
bind_host: ec2

On Jul 5, 3:26 pm, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

network:
bind_host: ec2

?

On Tue, Jul 5, 2011 at 1:14 PM, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0 (http://0.0.0.0):
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300 (http://10.234.189.8:9300)]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200 (http://10.234.189.8:9200)]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon <shay.ba...@elasticsearch.com (mailto:shay.ba...@elasticsearch.com)> wrote:

Even without master (where you just say ec2), most cloud platform
have common network interfaces names, and you can still use a single
configuration file by providing the network interface name (check the
network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com (mailto:karl.b...@gmail.com)(mailto:
karl.b...@gmail.com (mailto:karl.b...@gmail.com))> wrote:

If we specify ip addresses, there probably is no advantage over zen
discovery.

I vaguely remember that on head you don't need ip addresses for ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com) (mailto:
ketera...@gmail.com (mailto:ketera...@gmail.com))> wrote:

You also need to specify IP addresses as the EC2 plugin searches
other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)(mailto:
ketera...@gmail.com (mailto:ketera...@gmail.com))> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <
ridva...@gmail.com (mailto:ridva...@gmail.com) (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the
following article

Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I
start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/
10.234.149.155:9300 (http://10.234.149.155:9300) (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join
(elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/
10.234.149.155:9200 (http://10.234.149.155:9200) (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a
cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.comhttp://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com (mailto:paul@keteracel.com)
http://uk.linkedin.com/in/paulloy

Ridvan_Gyundogan · July 6, 2011, 11:26am

Hi Shay,
I tried both:
network:
host: ec2
network:
bind_host: ec2

I always get:
Startup Failed ...

BindTransportException[Failed to resolve host [null]]
IOException[Failed to find network interface for [ec2]]

I start only 1 node at the moment so sure it should be the master.

Any ideas?

On Jul 6, 4:11 am, Shay Banon shay.ba...@elasticsearch.com wrote:

network.host: ec2 is only available in master. Setting to non_loopback is not preferable, you should explicitly use the interface name matching the internal ip.

On Tuesday, July 5, 2011 at 6:12 PM, Paul Loy wrote:

Is this with master? I think that should be 0.17.0 not 0.16.2

On Tue, Jul 5, 2011 at 3:58 PM, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Hi Paul thanks for your help, I still have an issue.
Now I get:
{elasticsearch/0.16.2}: Startup Failed ...

BindTransportException[Failed to resolve host [null]]
IOException[Failed to find network interface for [ec2]]

To summarize my elasticsearch.yml now looks like the following:
cluster:
name: perftests
discovery:
type: ec2
cloud:
aws:
access_key: XXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

network:
bind_host: ec2

On Jul 5, 3:26 pm, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)> wrote:

network:
bind_host: ec2

?

On Tue, Jul 5, 2011 at 1:14 PM, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

I added the following to my elasticsearch.yml and now I am not bound
to 0.0.0.0 (http://0.0.0.0):
network:
bind_host: non_loopback:ipv4

And the log of the second node now looks like this:
[2011-07-05 12:05:50,944][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initializing ...
[2011-07-05 12:05:50,966][INFO ][plugins ]
[Powderkeg] loaded [cloud-aws]
[2011-07-05 12:05:55,795][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: initialized
[2011-07-05 12:05:55,796][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: starting ...
[2011-07-05 12:05:55,929][INFO ][transport ]
[Powderkeg] bound_address {inet[/10.234.189.8:9300]}, publish_address
{inet[/10.234.189.8:9300 (http://10.234.189.8:9300)]}
[2011-07-05 12:06:00,819][INFO ][cluster.service ]
[Powderkeg] new_master [Powderkeg][9GIGXIA4Tf6jBEWT7TpG9A][inet[/
10.234.189.8:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-05 12:06:00,859][INFO ][discovery ]
[Powderkeg] perftests/9GIGXIA4Tf6jBEWT7TpG9A
[2011-07-05 12:06:00,913][INFO ][gateway ]
[Powderkeg] recovered [0] indices into cluster_state
[2011-07-05 12:06:00,914][INFO ][http ]
[Powderkeg] bound_address {inet[/10.234.189.8:9200]}, publish_address
{inet[/10.234.189.8:9200 (http://10.234.189.8:9200)]}
[2011-07-05 12:06:00,914][INFO ][node ]
[Powderkeg] {elasticsearch/0.16.2}[988]: started

but still it doesn't find the master.
Any ideas?

On Jul 5, 2:32 pm, Ridvan Gyundogan <ridva...@gmail.com (mailto:ridva...@gmail.com)> wrote:

Thanks to all, but I still don't completely get it:

Where should I put ec2 ?

Do I need to add something else except ec2 to my config/
elasticsearch.yml file?

On Jul 5, 1:57 pm, Shay Banon <shay.ba...@elasticsearch.com (mailto:shay.ba...@elasticsearch.com)> wrote:

Even without master (where you just say ec2), most cloud platform
have common network interfaces names, and you can still use a single
configuration file by providing the network interface name (check the
network settings docs).

On Tuesday, July 5, 2011 at 1:37 AM, Paul Loy wrote:

Yeah, in master you can use ec2 and it'll use the private ipv4.

On Mon, Jul 4, 2011 at 10:59 PM, Karl Baum <karl.b...@gmail.com (mailto:karl.b...@gmail.com)(mailto:
karl.b...@gmail.com (mailto:karl.b...@gmail.com))> wrote:

If we specify ip addresses, there probably is no advantage over zen
discovery.

I vaguely remember that on head you don't need ip addresses for ec2
discovery. Is that correct?

Thx

Sent from my iPhone

On Jul 4, 2011, at 5:01 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com) (mailto:
ketera...@gmail.com (mailto:ketera...@gmail.com))> wrote:

You also need to specify IP addresses as the EC2 plugin searches
other instances by their private IPv4 address.

On Mon, Jul 4, 2011 at 9:58 PM, Paul Loy <ketera...@gmail.com (mailto:ketera...@gmail.com)(mailto:
ketera...@gmail.com (mailto:ketera...@gmail.com))> wrote:

bound_address {inet[/0:0:0:0:0:0:0:0:9300]}

On Mon, Jul 4, 2011 at 9:21 PM, Ridvan Gyundogan <
ridva...@gmail.com (mailto:ridva...@gmail.com) (mailto:ridva...@gmail.com)> wrote:

Hi,
I am trying to run 2 ec2 instances according to the
following article

Elasticsearch Platform — Find real-time answers at scale | Elastic.

This is my config/elasticsearch.yml:

cluster:
name: perftests

discovery:
type: ec2

cloud:
aws:
access_key: XXXXXXXXX
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

and this is my log on both ec2 nodes no matter which one I
start
first:
[2011-07-04 20:11:34,732][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initializing ...
[2011-07-04 20:11:34,741][INFO ][plugins ] [Fusion]
loaded [cloud-aws]
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: initialized
[2011-07-04 20:11:38,868][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: starting ...
[2011-07-04 20:11:38,974][INFO ][transport ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/
10.234.149.155:9300 (http://10.234.149.155:9300) (http://10.234.149.155:9300)]}
[2011-07-04 20:11:46,324][INFO ][cluster.service ] [Fusion]
new_master [Fusion][1Mq8yU1hQJWmO_7ubqNt5A][inet[/
10.234.149.155:9300]], reason: zen-disco-join
(elected_as_master)
[2011-07-04 20:11:46,361][INFO ][discovery ] [Fusion]
perftests/1Mq8yU1hQJWmO_7ubqNt5A
[2011-07-04 20:11:46,391][INFO ][gateway ] [Fusion]
recovered [0] indices into cluster_state
[2011-07-04 20:11:46,395][INFO ][http ] [Fusion]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/
10.234.149.155:9200 (http://10.234.149.155:9200) (http://10.234.149.155:9200)]}
[2011-07-04 20:11:46,396][INFO ][node ] [Fusion]
{elasticsearch/0.16.2}[1223]: started

Why are they not able to find each other and run as a
cluster?

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com) (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.comhttp://uk.linkedin.com/in/paulloy

--

Paul Loy
p...@keteracel.com (mailto:p...@keteracel.com)
http://uk.linkedin.com/in/paulloy

Clinton_Gormley · July 6, 2011, 11:37am

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

clint

Ridvan_Gyundogan · July 6, 2011, 11:40am

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Ridvan_Gyundogan · July 6, 2011, 1:34pm

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan ridva...@gmail.com wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley clin...@iannounce.co.ukwrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

kimchy · July 6, 2011, 4:16pm

You need to set the discovery type to ec2: Elasticsearch Platform — Find real-time answers at scale | Elastic. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (Elasticsearch Platform — Find real-time answers at scale | Elastic), for example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Ridvan_Gyundogan · July 6, 2011, 6:35pm

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You need to set the discovery type to ec2:Elasticsearch Platform — Find real-time answers at scale | Elastic. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (Elasticsearch Platform — Find real-time answers at scale | Elastic), for example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

kimchy · July 6, 2011, 7:35pm

Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

You need to set the discovery type to ec2:Elasticsearch Platform — Find real-time answers at scale | Elastic. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (Elasticsearch Platform — Find real-time answers at scale | Elastic), for example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

Ridvan_Gyundogan · July 7, 2011, 12:10pm

Shay,
here is the gist:

gist.github.com

https://gist.github.com/anonymous/1069376

master

[2011-07-07 11:28:11,517][INFO ][node                     ] [Dillon, Maxwell] {elasticsearch/0.16.2}[987]: initializing ...
[2011-07-07 11:28:11,756][TRACE][plugins                  ] [Dillon, Maxwell] --- adding plugin [/opt/elasticsearch-0.16.2/plugins/cloud-aws]
[2011-07-07 11:28:11,808][INFO ][plugins                  ] [Dillon, Maxwell] loaded [cloud-aws]
[2011-07-07 11:28:15,011][DEBUG][cache.memory             ] [Dillon, Maxwell] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
[2011-07-07 11:28:15,028][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [cached], type [cached], keep_alive [5m]
[2011-07-07 11:28:15,033][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [index], type [cached], keep_alive [5m]
[2011-07-07 11:28:15,034][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [search], type [cached], keep_alive [5m]
[2011-07-07 11:28:15,034][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [percolate], type [cached], keep_alive [5m]
[2011-07-07 11:28:15,034][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [management], type [scaling], min [1], size [20], keep_alive [5m]
[2011-07-07 11:28:15,037][DEBUG][threadpool               ] [Dillon, Maxwell] creating thread_pool [merge], type [scaling], min [1], size [20], keep_alive [5m]

This file has been truncated. show original

slave

[2011-07-07 12:02:19,978][INFO ][node                     ] [Fenris] {elasticsearch/0.16.2}[994]: initializing ...
[2011-07-07 12:02:19,989][TRACE][plugins                  ] [Fenris] --- adding plugin [/opt/elasticsearch-0.16.2/plugins/cloud-aws]
[2011-07-07 12:02:20,012][INFO ][plugins                  ] [Fenris] loaded [cloud-aws]
[2011-07-07 12:02:25,123][DEBUG][cache.memory             ] [Fenris] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
[2011-07-07 12:02:25,208][DEBUG][threadpool               ] [Fenris] creating thread_pool [cached], type [cached], keep_alive [5m]
[2011-07-07 12:02:25,229][DEBUG][threadpool               ] [Fenris] creating thread_pool [index], type [cached], keep_alive [5m]
[2011-07-07 12:02:25,229][DEBUG][threadpool               ] [Fenris] creating thread_pool [search], type [cached], keep_alive [5m]
[2011-07-07 12:02:25,229][DEBUG][threadpool               ] [Fenris] creating thread_pool [percolate], type [cached], keep_alive [5m]
[2011-07-07 12:02:25,230][DEBUG][threadpool               ] [Fenris] creating thread_pool [management], type [scaling], min [1], size [20], keep_alive [5m]
[2011-07-07 12:02:25,241][DEBUG][threadpool               ] [Fenris] creating thread_pool [merge], type [scaling], min [1], size [20], keep_alive [5m]

This file has been truncated. show original

On Jul 6, 10:35 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

You need to set the discovery type to ec2:Elasticsearch Platform — Find real-time answers at scale | Elastic. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (Elasticsearch Platform — Find real-time answers at scale | Elastic), for example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

NoClassSettingsException[Failed to load class setting
[discovery.type] with value [cloud]]
ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform — Find real-time answers at scale | Elastic

clint