EC2 discovery issue

Hey,

First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE.

If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
master ยท GitHub

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com (http://elasticsearch.com)> wrote:

You need to set the discovery type to ec2:Elasticsearch Platform โ€” Find real-time answers at scale | Elastic. The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name (Elasticsearch Platform โ€” Find real-time answers at scale | Elastic), for example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

  • NoClassSettingsException[Failed to load class setting
    [discovery.type] with value [cloud]]
    ClassNotFoundException[cloud]

for the network.bind_host I tried eth0:ipv4 but this gives me host
notfound exception.

Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
discovery for him with the 16.2 version?

On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

clint

Hi all,

maybe related to this discussion - I just got ES working in amazon and
there was one key point, the "cloud.aws.region" setting was not optional
in my case. Without it Amazon was returning an empty list of instances.
The

page implies that one could skip this setting.

Regards,
Pavel

On 7.07.2011 17:49, Shay Banon wrote:

Hey,

First, it seems like all the logging is DEBUG mode, thats noisy, all
you need is discovery set to TRACE.

If you see, on both nodes, the output: "using dynamic discovery
nodes" is empty, meaning that it went ahead and queries aws for the
list of nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
master ยท GitHub

On Jul 6, 10:35 pm, Shay Banon <shay.ba...@elasticsearch.com
http://elasticsearch.com> wrote:

Can you set logging for discovery to TRACE in the logging file and
gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com
(http://elasticsearch.com)> wrote:

You need to set the discovery type to
ec2:Elasticsearch Platform โ€” Find real-time answers at scale | Elastic.
The blog post is outdated... .

And you don't have to use master to have discovery working on
ec2. You can set network.host to whatever value you want, for
example, the internal IP address. If you want teh same config
across nodes, you can bind to the network interface name
(http://www.elasticsearch.org/guide/reference/modules/network.html),
for example: _en0:ipv4_.
On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:
Unfortunately using the trunk is not an option at the moment
because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.
    Now I tried the "Zen" as described here:
    http://www.elasticsearch.org/blog/2010/05/11/here-comes-the-cloud.html
    my elasticsearch.yml on the slave looks like this:
    cluster:
    name: perftests
    discovery:
    type: cloud
    zen:
    ping_timeout: 30s
    cloud:
    account: 551234567890
    key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    compute:
    type: amazon
    network:
    bind_host: _non_loopback:ipv4_
    On this I get the exception:
    {elasticsearch/0.16.2}: Initialization Failed ...
    - NoClassSettingsException[Failed to load class setting
    [discovery.type] with value [cloud]]
    ClassNotFoundException[cloud]
    for the network.bind_host I tried eth0:ipv4 but this gives
    me host
    notfound exception.
    Any ideas? Can someone upload an elasticsearch.yml file
    able to do ec2
    discovery for him with the 16.2 version?
    On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com
    (http://gmail.com)> wrote:
    Ah ok thanks I will try.
        On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley
        <clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:
                I start only 1 node at the moment so sure it
                should be the master.
            By master, Shay means the version of ElasticSearch
            that is currently
            master on github, not a released version.
            You need to download and compile it yourself:
            http://www.elasticsearch.org/download/master/
            clint
1 Like

Hi Pavel,
thanks a lot for the hint.

Can you probably share your working elasticsearch.yml?

On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev pavel.penchev@gmail.comwrote:

Hi all,

maybe related to this discussion - I just got ES working in amazon and
there was one key point, the "cloud.aws.region" setting was not optional in
my case. Without it Amazon was returning an empty list of instances.
The
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic implies that one could skip this setting.

Regards,
Pavel

On 7.07.2011 17:49, Shay Banon wrote:

Hey,

First, it seems like all the logging is DEBUG mode, thats noisy, all
you need is discovery set to TRACE.

If you see, on both nodes, the output: "using dynamic discovery nodes"

is empty, meaning that it went ahead and queries aws for the list of nodes,
and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
master ยท GitHub

On Jul 6, 10:35 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you set logging for discovery to TRACE in the logging file and gist it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com (
http://elasticsearch.com)> wrote:

You need to set the discovery type to ec2:
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic.
The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You
can set network.host to whatever value you want, for example, the internal
IP address. If you want teh same config across nodes, you can bind to the
network interface name (
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic), for
example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

  • NoClassSettingsException[Failed to load class setting
    [discovery.type] with value [cloud]]
    ClassNotFoundException[cloud]

    for the network.bind_host I tried eth0:ipv4 but this gives me host
    notfound exception.

    Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
    discovery for him with the 16.2 version?

    On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (
    http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <clin...@iannounce.co.uk(
http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

clint

Pavel you are right the cloud.aws.region was the key point.

Here my elasticsearch.yml in case someone else needs this too:

cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: AAAAAAAAAAAAA
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
aws:
region: eu-west-1

network:
bind_host: eth0:ipv4

On Fri, Jul 8, 2011 at 9:14 AM, Ridvan Gyundogan ridvansg@gmail.com wrote:

Hi Pavel,
thanks a lot for the hint.

Can you probably share your working elasticsearch.yml?

On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev pavel.penchev@gmail.comwrote:

Hi all,

maybe related to this discussion - I just got ES working in amazon and
there was one key point, the "cloud.aws.region" setting was not optional in
my case. Without it Amazon was returning an empty list of instances.
The
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic implies that one could skip this setting.

Regards,
Pavel

On 7.07.2011 17:49, Shay Banon wrote:

Hey,

First, it seems like all the logging is DEBUG mode, thats noisy, all
you need is discovery set to TRACE.

If you see, on both nodes, the output: "using dynamic discovery

nodes" is empty, meaning that it went ahead and queries aws for the list of
nodes, and got none... . Maybe the groups / tags are not correct?

-shay.banon

On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote:

Shay,
here is the gist:
master ยท GitHub

On Jul 6, 10:35 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you set logging for discovery to TRACE in the logging file and gist
it?

On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote:

Thanks Shay.
Now it is bit better, at least I do not have an exception but still
the ec2 instances do not run as a cluster.
What I additionally did is the following. Enabled ping for the
security group with the ec2-authorise. Tried that the 2 machines can
ping each other.
Checked with telnet that the 2 machines can open each other's 9200 and
9300 ports. The 9301 - 9303 ports are open too but the other ports are
closed by default.
These are the 2 log files I have. First the master and next the one
which had to become a slave.At the bottom is the elasticsearch.yml
config file I use.
Any ideas what I need to change?

master:
[2011-07-06 18:14:42,458][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initializing ...
[2011-07-06 18:14:42,468][INFO ][plugins ] [Jimmy
Woo] loaded [cloud-aws]
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: initialized
[2011-07-06 18:14:48,272][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: starting ...
[2011-07-06 18:14:48,414][INFO ][transport ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9300]}, publish_address {inet[/
10.224.50.50:9300]}
[2011-07-06 18:14:52,797][INFO ][cluster.service ] [Jimmy
Woo] new_master [Jimmy Woo][T_0Y3f_UQJmfuiqYYW7Ivg][inet[/
10.224.50.50:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:14:52,845][INFO ][discovery ] [Jimmy
Woo] perftests/T_0Y3f_UQJmfuiqYYW7Ivg
[2011-07-06 18:14:52,893][INFO ][gateway ] [Jimmy
Woo] recovered [0] indices into cluster_state
[2011-07-06 18:14:52,894][INFO ][http ] [Jimmy
Woo] bound_address {inet[/10.224.50.50:9200]}, publish_address {inet[/
10.224.50.50:9200]}
[2011-07-06 18:14:52,895][INFO ][node ] [Jimmy
Woo] {elasticsearch/0.16.2}[2142]: started

slave:
[2011-07-06 18:18:07,733][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initializing ...
[2011-07-06 18:18:07,741][INFO ][plugins ] [Machine
Teen] loaded [cloud-aws]
[2011-07-06 18:18:11,994][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: initialized
[2011-07-06 18:18:11,995][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: starting ...
[2011-07-06 18:18:12,105][INFO ][transport ] [Machine
Teen] bound_address {inet[/10.234.227.72:9300]}, publish_address
{inet[/10.234.227.72:9300]}
[2011-07-06 18:18:16,242][INFO ][cluster.service ] [Machine
Teen] new_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ_IA][inet[/
10.234.227.72:9300]], reason: zen-disco-join (elected_as_master)
[2011-07-06 18:18:16,278][INFO ][discovery ] [Machine
Teen] perftests/4vSbOGJqRx6q3Vi5HrJ_IA
[2011-07-06 18:18:16,305][INFO ][gateway ] [Machine
Teen] recovered [0] indices into cluster_state
[2011-07-06 18:18:16,312][INFO ][http ] [Machine
Teen] bound_address {inet[/10.234.227.72:9200]}, publish_address
{inet[/10.234.227.72:9200]}
[2011-07-06 18:18:16,313][INFO ][node ] [Machine
Teen] {elasticsearch/0.16.2}[1251]: started

elasticsearch.yml:
cluster:
name: perftests

discovery:
type: ec2
zen:
ping_timeout: 30s

cloud:
account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID",
not the "Account Number"
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon

network:
bind_host: eth0:ipv4

On Jul 6, 7:16 pm, Shay Banon <shay.ba...@elasticsearch.com (
http://elasticsearch.com)> wrote:

You need to set the discovery type to ec2:
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic.
The blog post is outdated... .

And you don't have to use master to have discovery working on ec2. You
can set network.host to whatever value you want, for example, the internal
IP address. If you want teh same config across nodes, you can bind to the
network interface name (
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic), for
example: en0:ipv4.

On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote:

Unfortunately using the trunk is not an option at the moment because
there are some maven dependencies to the 16.2 jar in our project.
We will upgrade to 0.17 once it is officially released.

Now I tried the "Zen" as described here:
Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

my elasticsearch.yml on the slave looks like this:
cluster:
name: perftests

discovery:
type: cloud
zen:
ping_timeout: 30s
cloud:
account: 551234567890
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
compute:
type: amazon
network:
bind_host: non_loopback:ipv4

On this I get the exception:
{elasticsearch/0.16.2}: Initialization Failed ...

  • NoClassSettingsException[Failed to load class setting
    [discovery.type] with value [cloud]]
    ClassNotFoundException[cloud]

    for the network.bind_host I tried eth0:ipv4 but this gives me host
    notfound exception.

    Any ideas? Can someone upload an elasticsearch.yml file able to do ec2
    discovery for him with the 16.2 version?

    On Jul 6, 2:40 pm, Ridvan Gyundogan <ridva...@gmail.com (
    http://gmail.com)> wrote:

Ah ok thanks I will try.

On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley <
clin...@iannounce.co.uk (http://iannounce.co.uk)>wrote:

I start only 1 node at the moment so sure it should be the master.

By master, Shay means the version of Elasticsearch that is currently
master on github, not a released version.

You need to download and compile it yourself:

Elasticsearch Platform โ€” Find real-time answers at scale | Elastic

clint

Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Hi Alex, thanks for the notes.

  1. It was required for the europe cloud.
  2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files.
For example you use cloud.aws.access_key and In mine it is cloud.account.

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow apiggott@ikanow.com wrote:

Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Probably the region is hte problem then, if you set it to the euro one, does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access key across the different aws services used in ES. Its the preferable way to set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.

  1. It was required for the europe cloud.
  2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files.
For example you use cloud.aws.access_key and In mine it is cloud.account.

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow <apiggott@ikanow.com (mailto:apiggott@ikanow.com)> wrote:

Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9 (http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9)), I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Yes with cloud.aws.region = eu-west

On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Probably the region is hte problem then, if you set it to the euro one,
does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access
key across the different aws services used in ES. Its the preferable way to
set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.

  1. It was required for the europe cloud.
  2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files.
For example you use cloud.aws.access_key and In mine it is cloud.account.

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow apiggott@ikanow.comwrote:

Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9),
I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Sorry, I mean with cloud.aws.region = eu-west-1 it works.

On Sun, Jul 10, 2011 at 11:05 AM, Ridvan Gyundogan ridvansg@gmail.comwrote:

Yes with cloud.aws.region = eu-west

On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Probably the region is hte problem then, if you set it to the euro one,
does it work then? Set cloud.aws.region to eu-west.

The cloud.aws.account is a global setting that defines the account access
key across the different aws services used in ES. Its the preferable way to
set it, though one can set explicitly the ec2 access key if needed.

On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote:

Hi Alex, thanks for the notes.

  1. It was required for the europe cloud.
  2. Thanks I added it to my file and I see it is waiting 30s.

I se some other differences between the 2 elasticsearch.yml files.
For example you use cloud.aws.access_key and In mine it is
cloud.account.

On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow apiggott@ikanow.comwrote:

Two quick and minor addenda to this discussion

1] I have a working ec2 cluster without having set the
"cloud.aws.region" parameter. (I'm in us-east-1, not sure if that
makes any difference.) Though I suppose I'll add it now!

(I also didn't need the "network.bind_host")

2] From a previous EC2 configuration discussion (http://
groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/
2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9),
I
think the "ping_timeout" parameter you're looking for is
"discovery.ec2.ping_timeout".

Thanks Pavel, You've really helped me out with this one, I've been busting my head on it for a few days but adding the region parameter did the trick.