ec2 discovery

Karl_Baum · June 22, 2011, 2:24pm

I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

Paul_Loy · June 22, 2011, 3:10pm

For 0.16.2 and below I think you may have to specify the tcp.host as the
private ipv4 address. This is what the ec2 plugin will add to zen so if tcp
has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:

I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Karl_Baum · June 22, 2011, 3:26pm

Hi Paul. I am a bit confused because my elasticsearch does not supply any ip addresses.. just my ec2 account information.

I am running into a bit of trouble with my logs. Somehow i have disabled logging to a file in an attempt to turn trace on for discovery.

https://gist.github.com/4a6e846163315c0cb9b3

Thanks for your help!

-karl

On Jun 22, 2011, at 11:10 AM, Paul Loy wrote:

For 0.16.2 and below I think you may have to specify the tcp.host as the private ipv4 address. This is what the ec2 plugin will add to zen so if tcp has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:
I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Paul_Loy · June 23, 2011, 11:00am

That's what I mean. You need to specify, in your config, the private ipv4
address as the tcp host. I think that's the publish_host here:

On Wed, Jun 22, 2011 at 4:26 PM, Karl Baum karl.baum@gmail.com wrote:

Hi Paul. I am a bit confused because my elasticsearch does not supply any
ip addresses.. just my ec2 account information.

I am running into a bit of trouble with my logs. Somehow i have disabled
logging to a file in an attempt to turn trace on for discovery.

https://gist.github.com/4a6e846163315c0cb9b3

Thanks for your help!

-karl

On Jun 22, 2011, at 11:10 AM, Paul Loy wrote:

For 0.16.2 and below I think you may have to specify the tcp.host as the
private ipv4 address. This is what the ec2 plugin will add to zen so if tcp
has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:

I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Karl_Baum · June 23, 2011, 11:24am

Hmmm.. if that is the case why not just use zen discovery?

Thx!
On Jun 23, 2011, at 7:00 AM, Paul Loy wrote:

That's what I mean. You need to specify, in your config, the private ipv4 address as the tcp host. I think that's the publish_host here: Elasticsearch Platform — Find real-time answers at scale | Elastic

On Wed, Jun 22, 2011 at 4:26 PM, Karl Baum karl.baum@gmail.com wrote:
Hi Paul. I am a bit confused because my elasticsearch does not supply any ip addresses.. just my ec2 account information.

I am running into a bit of trouble with my logs. Somehow i have disabled logging to a file in an attempt to turn trace on for discovery.

https://gist.github.com/4a6e846163315c0cb9b3

Thanks for your help!

-karl

On Jun 22, 2011, at 11:10 AM, Paul Loy wrote:

For 0.16.2 and below I think you may have to specify the tcp.host as the private ipv4 address. This is what the ec2 plugin will add to zen so if tcp has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:
I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Paul_Loy · June 23, 2011, 11:34am

it is a pain in 0.16.2 that you have to have specific config per node.
However, in the next release of Elasticsearch you will get special values
that will query EC2 meta-data to obtain the ip address, such as _
ec2:privateIp_

You do already get benefits of using the ec2 plugin. You get Zen + the
initial hosts set for you from EC2 instance information.

On Thu, Jun 23, 2011 at 12:24 PM, Karl Baum karl.baum@gmail.com wrote:

Hmmm.. if that is the case why not just use zen discovery?

Thx!
On Jun 23, 2011, at 7:00 AM, Paul Loy wrote:

That's what I mean. You need to specify, in your config, the private ipv4
address as the tcp host. I think that's the publish_host here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Wed, Jun 22, 2011 at 4:26 PM, Karl Baum karl.baum@gmail.com wrote:

Hi Paul. I am a bit confused because my elasticsearch does not supply any
ip addresses.. just my ec2 account information.

I am running into a bit of trouble with my logs. Somehow i have disabled
logging to a file in an attempt to turn trace on for discovery.

https://gist.github.com/4a6e846163315c0cb9b3

Thanks for your help!

-karl

On Jun 22, 2011, at 11:10 AM, Paul Loy wrote:

For 0.16.2 and below I think you may have to specify the tcp.host as the
private ipv4 address. This is what the ec2 plugin will add to zen so if tcp
has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:

I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

Paul_Loy · June 23, 2011, 11:35am

So, yeah, if you use master, you can get this already!

On Thu, Jun 23, 2011 at 12:34 PM, Paul Loy keteracel@gmail.com wrote:

it is a pain in 0.16.2 that you have to have specific config per node.
However, in the next release of Elasticsearch you will get special values
that will query EC2 meta-data to obtain the ip address, such as _
ec2:privateIp_

You do already get benefits of using the ec2 plugin. You get Zen + the
initial hosts set for you from EC2 instance information.

On Thu, Jun 23, 2011 at 12:24 PM, Karl Baum karl.baum@gmail.com wrote:

Hmmm.. if that is the case why not just use zen discovery?

Thx!
On Jun 23, 2011, at 7:00 AM, Paul Loy wrote:

That's what I mean. You need to specify, in your config, the private ipv4
address as the tcp host. I think that's the publish_host here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Wed, Jun 22, 2011 at 4:26 PM, Karl Baum karl.baum@gmail.com wrote:

Hi Paul. I am a bit confused because my elasticsearch does not supply
any ip addresses.. just my ec2 account information.

I am running into a bit of trouble with my logs. Somehow i have disabled
logging to a file in an attempt to turn trace on for discovery.

https://gist.github.com/4a6e846163315c0cb9b3

Thanks for your help!

-karl

On Jun 22, 2011, at 11:10 AM, Paul Loy wrote:

For 0.16.2 and below I think you may have to specify the tcp.host as the
private ipv4 address. This is what the ec2 plugin will add to zen so if tcp
has not bound to that address they will not find each other.

what do you see in the logs?

On Wed, Jun 22, 2011 at 3:24 PM, Karl karl.baum@gmail.com wrote:

I am attempting to get ec2 discovery working but thus far i think i
have been unsuccessful. My config file looks like:

https://gist.github.com/563690af25d54f5708b2

I have two instances running on two different ec2 instances, but when
i run cluster health, it gives back only 1 node:

{ "active_primary_shards" : 6,
"active_shards" : 6,
"cluster_name" : "emailsync_elasticsearch",
"initializing_shards" : 0,
"number_of_data_nodes" : 1,
"number_of_nodes" : 1,
"relocating_shards" : 0,
"status" : "yellow",
"timed_out" : false,
"unassigned_shards" : 6
}

Anything obvious i am getting wrong?

Thx!

-karl

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy

--

Paul Loy
paul@keteracel.com
http://uk.linkedin.com/in/paulloy