Ec2 instance cluster help


(Mehtryx) #1

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks


(Gustavo Maia) #2

Did you install ec2 plugin ?

2011/11/3 Mehtryx keith@benedicthome.com:

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks

--
Gustavo Maia


(Anthony Ikeda) #3

I'm use the tarball distribution of the DSE 1.0 and there hasn't been any
indication of installing an ec2 plugin.

Is this for hector or Cassandra?

On Thu, Nov 3, 2011 at 12:53 PM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Did you install ec2 plugin ?

2011/11/3 Mehtryx keith@benedicthome.com:

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks

--
Gustavo Maia


(Anthony Ikeda) #4

Sorry wrong thread.

On Thu, Nov 3, 2011 at 1:27 PM, Anthony Ikeda
anthony.ikeda.dev@gmail.comwrote:

I'm use the tarball distribution of the DSE 1.0 and there hasn't been any
indication of installing an ec2 plugin.

Is this for hector or Cassandra?

On Thu, Nov 3, 2011 at 12:53 PM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Did you install ec2 plugin ?

2011/11/3 Mehtryx keith@benedicthome.com:

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks

--
Gustavo Maia


(Mehtryx) #5

I followed the instructions as per here:
http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html

And I installed the cloud-aws plugin...if that is what you mean by ec2
plugin, you can see in the output on the gist that it sees the cloud-
aws plugin.

Thanks

On Nov 3, 3:53 pm, Gustavo Maia gustavobbm...@gmail.com wrote:

Did you install ec2 plugin ?

2011/11/3 Mehtryx ke...@benedicthome.com:

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks

--
Gustavo Maia


(Mehtryx) #6

Interesting enough. My s3 bucket is not populating either. Though
I've indexed 11K documents this morning.

I may try to remove and reinstall this plugin, any gotchas to that?

On Nov 4, 8:43 am, Mehtryx ke...@benedicthome.com wrote:

I followed the instructions as per here:http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec...

And I installed the cloud-aws plugin...if that is what you mean by ec2
plugin, you can see in the output on the gist that it sees the cloud-
aws plugin.

Thanks

On Nov 3, 3:53 pm, Gustavo Maia gustavobbm...@gmail.com wrote:

Did you install ec2 plugin ?

2011/11/3 Mehtryx ke...@benedicthome.com:

Hi,

I've been trying to get my ec2 instances to talk as a cluster for a
bit, it's the last stage nearly to having things configured and imaged
for us and it seems to be the most defiant.

A copy of my config (sana keys) and server output is located here:
https://gist.github.com/1336876

I have the two instances running with the same config, both are in the
same region (US-east-1) but unable to communicate.

Any suggestions?

Thanks

--
Gustavo Maia


(Mehtryx) #7

In the spirit of ever trying to move forward I've done a few things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.

  2. I have reinstalled the plugin ( doing a ./bin/plugin -remove cloud-
    aws and then the install again ), this has not rectified the issue
    for me, I again have things configured as per instructions so I would
    suspect if everything worked as described...I should see my cluster.
    I'll keep looking if there is no one here able to point me to a
    mistake then I'll submit a bug on github for it, as I can only assume
    something in es is broken then.

Cheers and thanks!


(Shay Banon) #8

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx keith@benedicthome.com wrote:

In the spirit of ever trying to move forward I've done a few things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.

  2. I have reinstalled the plugin ( doing a ./bin/plugin -remove cloud-
    aws and then the install again ), this has not rectified the issue
    for me, I again have things configured as per instructions so I would
    suspect if everything worked as described...I should see my cluster.
    I'll keep looking if there is no one here able to point me to a
    mistake then I'll submit a bug on github for it, as I can only assume
    something in es is broken then.

Cheers and thanks!


(Mehtryx) #9

Hi Shay,

Sorry I missed your response here.

I have discovery set to trace. I'm copying up the output to this
gist: git://gist.github.com/1367485.git and my logging.yml to this
gist: git://gist.github.com/1367493.git

If I'm missing something let me know :slight_smile: Thanks!

Keith

On Nov 7, 10:18 pm, Shay Banon kim...@gmail.com wrote:

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx ke...@benedicthome.com wrote:

In the spirit of ever trying to move forward I've done a few things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.
  1. I have reinstalled the plugin ( doing a ./bin/plugin -remove cloud-
    aws and then the install again ), this has not rectified the issue
    for me, I again have things configured as per instructions so I would
    suspect if everything worked as described...I should see my cluster.
    I'll keep looking if there is no one here able to point me to a
    mistake then I'll submit a bug on github for it, as I can only assume
    something in es is broken then.

Cheers and thanks!


(Shay Banon) #10

It looks like you provided the log for only one instance, and also, it does
not look like its configured with TRACE, but with DEBUG.

On Tue, Nov 15, 2011 at 6:23 PM, Mehtryx keith@benedicthome.com wrote:

Hi Shay,

Sorry I missed your response here.

I have discovery set to trace. I'm copying up the output to this
gist: git://gist.github.com/1367485.git and my logging.yml to this
gist: git://gist.github.com/1367493.git

If I'm missing something let me know :slight_smile: Thanks!

Keith

On Nov 7, 10:18 pm, Shay Banon kim...@gmail.com wrote:

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It
should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx ke...@benedicthome.com wrote:

In the spirit of ever trying to move forward I've done a few things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.
  1. I have reinstalled the plugin ( doing a ./bin/plugin -remove cloud-
    aws and then the install again ), this has not rectified the issue
    for me, I again have things configured as per instructions so I would
    suspect if everything worked as described...I should see my cluster.
    I'll keep looking if there is no one here able to point me to a
    mistake then I'll submit a bug on github for it, as I can only assume
    something in es is broken then.

Cheers and thanks!


(Mehtryx) #11

Shay,

I have the other instance running and there were no log entries at the
time I started the second instance...that is why I only provided the
one. Let me do this.

I'm going to build two new separate instances ( as I need the one
running now for a prototype and cant be restarting it while our dev's
use the data)....k made (time lapse an hour there, both manually setup
and configured with duplicate configuration options.)

Output from 1st server: https://gist.github.com/1370248

Output from second server: https://gist.github.com/1370242

Configurations : https://gist.github.com/1370255
(Note I seperated them by a line of ********* in the file)

One last thing you mentioned that it looks like I only have
debug....I'm confused in the file I linked before and the
configurations above the entry discovery: TRACE
is present... I'm not following you on how then I'm suppose to set it
to trace any further than that.

Thanks for the help.

On Nov 16, 9:04 am, Shay Banon kim...@gmail.com wrote:

It looks like you provided the log for only one instance, and also, it does
not look like its configured with TRACE, but with DEBUG.

On Tue, Nov 15, 2011 at 6:23 PM, Mehtryx ke...@benedicthome.com wrote:

Hi Shay,

Sorry I missed your response here.

I have discovery set to trace. I'm copying up the output to this
gist: git://gist.github.com/1367485.git and my logging.yml to this
gist: git://gist.github.com/1367493.git

If I'm missing something let me know :slight_smile: Thanks!

Keith

On Nov 7, 10:18 pm, Shay Banon kim...@gmail.com wrote:

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It
should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx ke...@benedicthome.com wrote:

In the spirit of ever trying to move forward I've done a few things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.
  1. I have reinstalled the plugin ( doing a ./bin/plugin -remove cloud-
    aws and then the install again ), this has not rectified the issue
    for me, I again have things configured as per instructions so I would
    suspect if everything worked as described...I should see my cluster.
    I'll keep looking if there is no one here able to point me to a
    mistake then I'll submit a bug on github for it, as I can only assume
    something in es is broken then.

Cheers and thanks!


(Shay Banon) #12

Based on the log, it does not seem like its using the aws ec2 discovery at
all. Can you gist your configuration again?

On Wed, Nov 16, 2011 at 4:59 PM, Mehtryx keith@benedicthome.com wrote:

Shay,

I have the other instance running and there were no log entries at the
time I started the second instance...that is why I only provided the
one. Let me do this.

I'm going to build two new separate instances ( as I need the one
running now for a prototype and cant be restarting it while our dev's
use the data)....k made (time lapse an hour there, both manually setup
and configured with duplicate configuration options.)

Output from 1st server: https://gist.github.com/1370248

Output from second server: https://gist.github.com/1370242

Configurations : https://gist.github.com/1370255
(Note I seperated them by a line of ********* in the file)

One last thing you mentioned that it looks like I only have
debug....I'm confused in the file I linked before and the
configurations above the entry discovery: TRACE
is present... I'm not following you on how then I'm suppose to set it
to trace any further than that.

Thanks for the help.

On Nov 16, 9:04 am, Shay Banon kim...@gmail.com wrote:

It looks like you provided the log for only one instance, and also, it
does
not look like its configured with TRACE, but with DEBUG.

On Tue, Nov 15, 2011 at 6:23 PM, Mehtryx ke...@benedicthome.com wrote:

Hi Shay,

Sorry I missed your response here.

I have discovery set to trace. I'm copying up the output to this
gist: git://gist.github.com/1367485.git and my logging.yml to this
gist: git://gist.github.com/1367493.git

If I'm missing something let me know :slight_smile: Thanks!

Keith

On Nov 7, 10:18 pm, Shay Banon kim...@gmail.com wrote:

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It
should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx ke...@benedicthome.com
wrote:

In the spirit of ever trying to move forward I've done a few
things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.
  1. I have reinstalled the plugin ( doing a ./bin/plugin -remove
    cloud-

aws and then the install again ), this has not rectified the issue
for me, I again have things configured as per instructions so I
would

suspect if everything worked as described...I should see my
cluster.

I'll keep looking if there is no one here able to point me to a
mistake then I'll submit a bug on github for it, as I can only
assume

something in es is broken then.

Cheers and thanks!


(Mehtryx) #13

Sure thing, in the test cluster I am trying to set up the
configurations are identical so I've made one gist : https://gist.github.com/1373291

Note I limited the discovery originally to only servers with the tag
"es_cluster" : "pd-search" but for this test I left that commented out
as you will see.

On Nov 17, 8:06 am, Shay Banon kim...@gmail.com wrote:

Based on the log, it does not seem like its using the aws ec2 discovery at
all. Can you gist your configuration again?

On Wed, Nov 16, 2011 at 4:59 PM, Mehtryx ke...@benedicthome.com wrote:

Shay,

I have the other instance running and there were no log entries at the
time I started the second instance...that is why I only provided the
one. Let me do this.

I'm going to build two new separate instances ( as I need the one
running now for a prototype and cant be restarting it while our dev's
use the data)....k made (time lapse an hour there, both manually setup
and configured with duplicate configuration options.)

Output from 1st server:https://gist.github.com/1370248

Output from second server:https://gist.github.com/1370242

Configurations : https://gist.github.com/1370255
(Note I seperated them by a line of ********* in the file)

One last thing you mentioned that it looks like I only have
debug....I'm confused in the file I linked before and the
configurations above the entry discovery: TRACE
is present... I'm not following you on how then I'm suppose to set it
to trace any further than that.

Thanks for the help.

On Nov 16, 9:04 am, Shay Banon kim...@gmail.com wrote:

It looks like you provided the log for only one instance, and also, it
does
not look like its configured with TRACE, but with DEBUG.

On Tue, Nov 15, 2011 at 6:23 PM, Mehtryx ke...@benedicthome.com wrote:

Hi Shay,

Sorry I missed your response here.

I have discovery set to trace. I'm copying up the output to this
gist: git://gist.github.com/1367485.git and my logging.yml to this
gist: git://gist.github.com/1367493.git

If I'm missing something let me know :slight_smile: Thanks!

Keith

On Nov 7, 10:18 pm, Shay Banon kim...@gmail.com wrote:

Heya,

Can you set in the logging(.yml) config file "discovery" to TRACE
(similar to how action is set to DEBUG), and gist the logs again? It
should
print what list of servers it gets from the amazon API.

-shay.banon

On Fri, Nov 4, 2011 at 8:32 PM, Mehtryx ke...@benedicthome.com
wrote:

In the spirit of ever trying to move forward I've done a few
things.

  1. I am not using s3....it seems the preference is EBS for speed of
    recovery and persistence anyway and ot wasn't working for me even
    though I've configured as documented.
  1. I have reinstalled the plugin ( doing a ./bin/plugin -remove
    cloud-

aws and then the install again ), this has not rectified the issue
for me, I again have things configured as per instructions so I
would

suspect if everything worked as described...I should see my
cluster.

I'll keep looking if there is no one here able to point me to a
mistake then I'll submit a bug on github for it, as I can only
assume

something in es is broken then.

Cheers and thanks!


(vinod) #14

The discovery key should be at the top level and not under 'cloud'.

On Nov 17, 6:43 am, Mehtryx ke...@benedicthome.com wrote:

Sure thing, in the test cluster I am trying to set up the
configurations are identical so I've made one gist :https://gist.github.com/1373291

Note I limited the discovery originally to only servers with the tag
"es_cluster" : "pd-search" but for this test I left that commented out
as you will see.


(Shay Banon) #15

Right, I added a comment with the correct configuration.

On Fri, Nov 18, 2011 at 11:53 PM, vinod vinod.chandru@gmail.com wrote:

The discovery key should be at the top level and not under 'cloud'.

On Nov 17, 6:43 am, Mehtryx ke...@benedicthome.com wrote:

Sure thing, in the test cluster I am trying to set up the
configurations are identical so I've made one gist :
https://gist.github.com/1373291

Note I limited the discovery originally to only servers with the tag
"es_cluster" : "pd-search" but for this test I left that commented out
as you will see.


(Mehtryx) #16

Thanks, sigh that was harder to find but simple enough of a problem

I am working great, ty for your time and patience Shay.

Keith

On Nov 20, 3:45 am, Shay Banon kim...@gmail.com wrote:

Right, I added a comment with the correct configuration.

On Fri, Nov 18, 2011 at 11:53 PM, vinod vinod.chan...@gmail.com wrote:

The discovery key should be at the top level and not under 'cloud'.

On Nov 17, 6:43 am, Mehtryx ke...@benedicthome.com wrote:

Sure thing, in the test cluster I am trying to set up the
configurations are identical so I've made one gist :
https://gist.github.com/1373291

Note I limited the discovery originally to only servers with the tag
"es_cluster" : "pd-search" but for this test I left that commented out
as you will see.


(system) #17