Elasticsearch on AWS OpsWorks


(Stuart Warren) #1

Hi there,

I've been struggling for a few days to get a basic cluster going using AWS
OpsWorks using the new plugin.
Admitedly I'm pretty new to the Idea of Chef/AWS, but I have setup a
private cluster using Puppet so I'm not a complete newbie to Elasticsearch.

Nodes have installed, have started, I can access port 9200 on each, but
they don't find each other using the EC2 API (status: 503) and hence don't
form a cluster like they do in a normal environment.

Is anyone able to spot what I may be doing wrong, or simply provide their
chef cookbooks and JSON config for using Elasticsearch > v1.0?

Config
The following is how I've configured everything currently, happy to provide
any further details.

I've based my config on this AWS blog post


I have a git repo with a bunch of submodules hosting my cookbooks

My stack is called "elasticsearch-test"
It's set to use the eu-west-1 region (Ireland)
Using Chef 11.4

My Layer is called ES-Search-Test
My custom chef recipes:
Setup - java::default
Deploy - elasticsearch elasticsearch::aws elasticsearch::nginx

I have 2 instances in the same availability zone

I have an App: elasticsearch
Same git repo url
And a deployment with this Custom Chef JSON http://pastebin.com/P6vdWBGL

Observations
Each node gets the tag "opsworks:stack=elasticsearch-test" so that is what
I'm using as a filter
Each node can telnet to the others 9300 port so I don't think it is a
Security groups/communications issue
Each node can run aws ec2 describe-instances --filters Name=tag:opsworks:stack,Values=elasticsearch-test without specifying keys
and get the nodes, so I don't think it is a profiles/API issue.
Nginx does not yet work either, but that is a secondary priority at the
moment.

Config
Here is the generated config from one of the nodes
http://pastebin.com/QNEKrf9N

Logs
This is my log after a restart
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] version[1.1.0], pid[3349],
build[2181e11/2014-03-25T15:59:51Z]
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] initializing ...
[2014-04-04 10:19:32,631][INFO ][plugins ]
[amefurikozo.localdomain] loaded [cloud-aws], sites []
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] initialized
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] starting ...
[2014-04-04 10:19:38,805][INFO ][transport ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9300]},
publish_address {inet[/172.31.x.x:9300]}
[2014-04-04 10:20:08,853][WARN ][discovery ]
[amefurikozo.localdomain] waited for 30s and no initial state was set by
the discovery
[2014-04-04 10:20:08,853][INFO ][discovery ]
[amefurikozo.localdomain] elasticsearch-test-dsi/z2CyXhZnTuWVTuD_07v5lQ
[2014-04-04 10:20:08,863][INFO ][http ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9200]},
publish_address {inet[/172.31.x.x:9200]}
[2014-04-04 10:20:08,961][INFO ][node ]
[amefurikozo.localdomain] started

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

Some thoughts. Not sure if it will help though.

discovery.ec2.tag.opsworks::stack: elasticsearch-test

I never saw that form for tags "opsworks::stack".

I can't see your credentials:

cloud.aws.access_key: YOURKEY
cloud.aws. secret_key: YOURSECRET

May be you just removed those lines when pasting instead of replacing key/secret with dummy values?

You did not set cloud.aws.region so it fall back to us-east-1 by default.

If you defined security groups, you should add them in discovery.ec2.groups

Have a look at https://github.com/elasticsearch/elasticsearch-cloud-aws#ec2-discovery

Hope this help

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 4 avril 2014 à 12:34:07, Stuart Warren (stuartwarren83@gmail.com) a écrit:

Hi there,

I've been struggling for a few days to get a basic cluster going using AWS OpsWorks using the new plugin.
Admitedly I'm pretty new to the Idea of Chef/AWS, but I have setup a private cluster using Puppet so I'm not a complete newbie to Elasticsearch.

Nodes have installed, have started, I can access port 9200 on each, but they don't find each other using the EC2 API (status: 503) and hence don't form a cluster like they do in a normal environment.

Is anyone able to spot what I may be doing wrong, or simply provide their chef cookbooks and JSON config for using Elasticsearch > v1.0?

Config
The following is how I've configured everything currently, happy to provide any further details.

I've based my config on this AWS blog post http://blogs.aws.amazon.com/application-management/post/Tx3MEVKS0A4G7R5/Deploying-Elasticsearch-with-OpsWorks
I have a git repo with a bunch of submodules hosting my cookbooks https://github.com/stuart-warren/opsworks-cookbooks

My stack is called "elasticsearch-test"
It's set to use the eu-west-1 region (Ireland)
Using Chef 11.4

My Layer is called ES-Search-Test
My custom chef recipes:
Setup - java::default
Deploy - elasticsearch elasticsearch::aws elasticsearch::nginx

I have 2 instances in the same availability zone

I have an App: elasticsearch
Same git repo url
And a deployment with this Custom Chef JSON http://pastebin.com/P6vdWBGL

Observations
Each node gets the tag "opsworks:stack=elasticsearch-test" so that is what I'm using as a filter
Each node can telnet to the others 9300 port so I don't think it is a Security groups/communications issue
Each node can run aws ec2 describe-instances --filters Name=tag:opsworks:stack,Values=elasticsearch-test without specifying keys and get the nodes, so I don't think it is a profiles/API issue.
Nginx does not yet work either, but that is a secondary priority at the moment.

Config
Here is the generated config from one of the nodes
http://pastebin.com/QNEKrf9N

Logs
This is my log after a restart
[2014-04-04 10:19:32,552][INFO ][node ] [amefurikozo.localdomain] version[1.1.0], pid[3349], build[2181e11/2014-03-25T15:59:51Z]
[2014-04-04 10:19:32,552][INFO ][node ] [amefurikozo.localdomain] initializing ...
[2014-04-04 10:19:32,631][INFO ][plugins ] [amefurikozo.localdomain] loaded [cloud-aws], sites []
[2014-04-04 10:19:38,683][INFO ][node ] [amefurikozo.localdomain] initialized
[2014-04-04 10:19:38,683][INFO ][node ] [amefurikozo.localdomain] starting ...
[2014-04-04 10:19:38,805][INFO ][transport ] [amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/172.31.x.x:9300]}
[2014-04-04 10:20:08,853][WARN ][discovery ] [amefurikozo.localdomain] waited for 30s and no initial state was set by the discovery
[2014-04-04 10:20:08,853][INFO ][discovery ] [amefurikozo.localdomain] elasticsearch-test-dsi/z2CyXhZnTuWVTuD_07v5lQ
[2014-04-04 10:20:08,863][INFO ][http ] [amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/172.31.x.x:9200]}
[2014-04-04 10:20:08,961][INFO ][node ] [amefurikozo.localdomain] started

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.533e8fa8.6b8b4567.52ee%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(Stuart Warren) #3

Ah you legend!

I was sure I had fixed up the tag name, it should only have a single colon.
You also pointed out the default region issue, it was in the json, but
within the discovery block so the Chef cookbook didn't handle it and hence
didn't make it into the YAML

Here is my working Chef JSON http://pastebin.com/78LePYUv

Cheers!

On 4 April 2014 11:55, David Pilato david@pilato.fr wrote:

Some thoughts. Not sure if it will help though.

discovery.ec2.tag.opsworks::stack: elasticsearch-test

I never saw that form for tags "opsworks::stack".

I can't see your credentials:

cloud.aws.access_key: YOURKEY
cloud.aws. secret_key: YOURSECRET

May be you just removed those lines when pasting instead of replacing
key/secret with dummy values?

You did not set cloud.aws.region so it fall back to us-east-1 by default.

If you defined security groups, you should add them in discovery.ec2.groups

Have a look at
https://github.com/elasticsearch/elasticsearch-cloud-aws#ec2-discovery

Hope this help

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 4 avril 2014 à 12:34:07, Stuart Warren (stuartwarren83@gmail.com) a
écrit:

Hi there,

I've been struggling for a few days to get a basic cluster going using AWS
OpsWorks using the new plugin.
Admitedly I'm pretty new to the Idea of Chef/AWS, but I have setup a
private cluster using Puppet so I'm not a complete newbie to Elasticsearch.

Nodes have installed, have started, I can access port 9200 on each, but
they don't find each other using the EC2 API (status: 503) and hence don't
form a cluster like they do in a normal environment.

Is anyone able to spot what I may be doing wrong, or simply provide their
chef cookbooks and JSON config for using Elasticsearch > v1.0?

Config
The following is how I've configured everything currently, happy to
provide any further details.

I've based my config on this AWS blog post
http://blogs.aws.amazon.com/application-management/post/Tx3MEVKS0A4G7R5/Deploying-Elasticsearch-with-OpsWorks
I have a git repo with a bunch of submodules hosting my cookbooks
https://github.com/stuart-warren/opsworks-cookbooks

My stack is called "elasticsearch-test"
It's set to use the eu-west-1 region (Ireland)
Using Chef 11.4

My Layer is called ES-Search-Test
My custom chef recipes:
Setup - java::default
Deploy - elasticsearch elasticsearch::aws elasticsearch::nginx

I have 2 instances in the same availability zone

I have an App: elasticsearch
Same git repo url
And a deployment with this Custom Chef JSON http://pastebin.com/P6vdWBGL

Observations
Each node gets the tag "opsworks:stack=elasticsearch-test" so that is what
I'm using as a filter
Each node can telnet to the others 9300 port so I don't think it is a
Security groups/communications issue
Each node can run aws ec2 describe-instances --filters Name=tag:opsworks:stack,Values=elasticsearch-test without specifying keys
and get the nodes, so I don't think it is a profiles/API issue.
Nginx does not yet work either, but that is a secondary priority at the
moment.

Config
Here is the generated config from one of the nodes
http://pastebin.com/QNEKrf9N

Logs
This is my log after a restart
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] version[1.1.0], pid[3349],
build[2181e11/2014-03-25T15:59:51Z]
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] initializing ...
[2014-04-04 10:19:32,631][INFO ][plugins ]
[amefurikozo.localdomain] loaded [cloud-aws], sites []
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] initialized
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] starting ...
[2014-04-04 10:19:38,805][INFO ][transport ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9300]},
publish_address {inet[/172.31.x.x:9300]}
[2014-04-04 10:20:08,853][WARN ][discovery ]
[amefurikozo.localdomain] waited for 30s and no initial state was set by
the discovery
[2014-04-04 10:20:08,853][INFO ][discovery ]
[amefurikozo.localdomain] elasticsearch-test-dsi/z2CyXhZnTuWVTuD_07v5lQ
[2014-04-04 10:20:08,863][INFO ][http ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9200]},
publish_address {inet[/172.31.x.x:9200]}
[2014-04-04 10:20:08,961][INFO ][node ]
[amefurikozo.localdomain] started

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/m_1Pdu6J2gg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.533e8fa8.6b8b4567.52ee%40MacBook-Air-de-David.localhttps://groups.google.com/d/msgid/elasticsearch/etPan.533e8fa8.6b8b4567.52ee%40MacBook-Air-de-David.local?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADHbheUy4tQPN4tKwtUD8x-UVYtm4rsK_5rDTC8Hb2rjZUi%2BpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Stuart Warren) #4

Also not using access keys at the moment because the server is given an IAM
role with read access to the API aws-opsworks-ec2-role.

On 4 April 2014 14:12, Stuart Warren stuartwarren83@gmail.com wrote:

Ah you legend!

I was sure I had fixed up the tag name, it should only have a single colon.
You also pointed out the default region issue, it was in the json, but
within the discovery block so the Chef cookbook didn't handle it and hence
didn't make it into the YAML

Here is my working Chef JSON http://pastebin.com/78LePYUv

Cheers!

On 4 April 2014 11:55, David Pilato david@pilato.fr wrote:

Some thoughts. Not sure if it will help though.

discovery.ec2.tag.opsworks::stack: elasticsearch-test

I never saw that form for tags "opsworks::stack".

I can't see your credentials:

cloud.aws.access_key: YOURKEY
cloud.aws. secret_key: YOURSECRET

May be you just removed those lines when pasting instead of replacing
key/secret with dummy values?

You did not set cloud.aws.region so it fall back to us-east-1 by default.

If you defined security groups, you should add them in
discovery.ec2.groups

Have a look at
https://github.com/elasticsearch/elasticsearch-cloud-aws#ec2-discovery

Hope this help

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 4 avril 2014 à 12:34:07, Stuart Warren (stuartwarren83@gmail.com) a
écrit:

Hi there,

I've been struggling for a few days to get a basic cluster going using
AWS OpsWorks using the new plugin.
Admitedly I'm pretty new to the Idea of Chef/AWS, but I have setup a
private cluster using Puppet so I'm not a complete newbie to Elasticsearch.

Nodes have installed, have started, I can access port 9200 on each, but
they don't find each other using the EC2 API (status: 503) and hence don't
form a cluster like they do in a normal environment.

Is anyone able to spot what I may be doing wrong, or simply provide their
chef cookbooks and JSON config for using Elasticsearch > v1.0?

Config
The following is how I've configured everything currently, happy to
provide any further details.

I've based my config on this AWS blog post
http://blogs.aws.amazon.com/application-management/post/Tx3MEVKS0A4G7R5/Deploying-Elasticsearch-with-OpsWorks
I have a git repo with a bunch of submodules hosting my cookbooks
https://github.com/stuart-warren/opsworks-cookbooks

My stack is called "elasticsearch-test"
It's set to use the eu-west-1 region (Ireland)
Using Chef 11.4

My Layer is called ES-Search-Test
My custom chef recipes:
Setup - java::default
Deploy - elasticsearch elasticsearch::aws elasticsearch::nginx

I have 2 instances in the same availability zone

I have an App: elasticsearch
Same git repo url
And a deployment with this Custom Chef JSON http://pastebin.com/P6vdWBGL

Observations
Each node gets the tag "opsworks:stack=elasticsearch-test" so that is
what I'm using as a filter
Each node can telnet to the others 9300 port so I don't think it is a
Security groups/communications issue
Each node can run aws ec2 describe-instances --filters Name=tag:opsworks:stack,Values=elasticsearch-test without specifying keys
and get the nodes, so I don't think it is a profiles/API issue.
Nginx does not yet work either, but that is a secondary priority at the
moment.

Config
Here is the generated config from one of the nodes
http://pastebin.com/QNEKrf9N

Logs
This is my log after a restart
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] version[1.1.0], pid[3349],
build[2181e11/2014-03-25T15:59:51Z]
[2014-04-04 10:19:32,552][INFO ][node ]
[amefurikozo.localdomain] initializing ...
[2014-04-04 10:19:32,631][INFO ][plugins ]
[amefurikozo.localdomain] loaded [cloud-aws], sites []
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] initialized
[2014-04-04 10:19:38,683][INFO ][node ]
[amefurikozo.localdomain] starting ...
[2014-04-04 10:19:38,805][INFO ][transport ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9300]},
publish_address {inet[/172.31.x.x:9300]}
[2014-04-04 10:20:08,853][WARN ][discovery ]
[amefurikozo.localdomain] waited for 30s and no initial state was set by
the discovery
[2014-04-04 10:20:08,853][INFO ][discovery ]
[amefurikozo.localdomain] elasticsearch-test-dsi/z2CyXhZnTuWVTuD_07v5lQ
[2014-04-04 10:20:08,863][INFO ][http ]
[amefurikozo.localdomain] bound_address {inet[/0.0.0.0:9200]},
publish_address {inet[/172.31.x.x:9200]}
[2014-04-04 10:20:08,961][INFO ][node ]
[amefurikozo.localdomain] started

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/48dfb888-9e61-42d6-982b-b76feaa795cd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/m_1Pdu6J2gg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.533e8fa8.6b8b4567.52ee%40MacBook-Air-de-David.localhttps://groups.google.com/d/msgid/elasticsearch/etPan.533e8fa8.6b8b4567.52ee%40MacBook-Air-de-David.local?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADHbheW_%3DZw%2BFBZnTW_8gHmcW3tj%2BMyTNh%2BkyQVyG1FRck%2BN3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5