Discovery.ec2.host_type:public_dns not working as expected


(John Chang) #1

I have two nodes up in ec2 configured as follows:
discovery:
type: ec2
initial_state_timeout: 120s
ec2:
host_type: public_dns
ports: 9300
groups: dev_private

I cannot connect to them running tests from nodes on my dev machine, trying to connect as follows:
NodeBuilder nb = NodeBuilder.nodeBuilder().client(true);
nb.settings().put("discovery.zen.ping.multicast.enabled", "false");
nb.settings().put("discovery.zen.ping.unicast.hosts", "elasticsearch1.dev5.cloud.test.com:9300,elasticsearch2.dev5.cloud.test.com:9300");
nb.settings().put("cluster.name", "dev5");
this.indexClient = nb.node().client();

It still seems to try to join the nodes on private ip (the ip mentioned below, 10.198.109.171, is a private ip for one of the nodes):

12:42:14,757 WARN [elasticsearch[Fury][cached]-pool-3-thread-1] [discovery.zen] - [Fury] failed to connect to master [[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Freakshow][inet[/10.198.109.171:9300]]
Caused by: org.elasticsearch.transport.ConnectTransportException: [Freakshow][inet[/10.198.109.171:9300]]
12:42:19,275 WARN [elasticsearch[Fury][cached]-pool-3-thread-1] [discovery.zen] - [Fury] failed to connect to master [[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]]], retrying...

It does that over and over until it gives up:

12:42:39,845 WARN [main] [discovery] - [Fury] waited for 30s and no initial state was set by the discovery


(Shay Banon) #2

Hi,

Can you check in the logs of the nodes what is the publish_host? This is
the one you want if you want to be able to connect to them. Currently, the
ec2 discovery is not tied to the definition of what is the host the nodes
will be connected to, so you will need to configure it as well.

Note, you don't have to configure it differently for each node. Usually,
en1 will be the public one (if memory serves), so you can configure the
network.host setting to en1. More info here:
http://www.elasticsearch.com/docs/elasticsearch/modules/node/network/.

-shay.banon

On Mon, Oct 18, 2010 at 9:47 PM, John Chang jchangkihtest2@gmail.comwrote:

I have two nodes up in ec2 configured as follows:
discovery:
type: ec2
initial_state_timeout: 120s
ec2:
host_type: public_dns
ports: 9300
groups: dev_private

I cannot connect to them running tests from nodes on my dev machine, trying
to connect as follows:
NodeBuilder nb = NodeBuilder.nodeBuilder().client(true);
nb.settings().put("discovery.zen.ping.multicast.enabled",
"false");
nb.settings().put("discovery.zen.ping.unicast.hosts",
"elasticsearch1.dev5.cloud.test.com:9300,
elasticsearch2.dev5.cloud.test.com:9300");
nb.settings().put("cluster.name", "dev5");
this.indexClient = nb.node().client();

It still seems to try to join the nodes on private ip (the ip mentioned
below, 10.198.109.171, is a private ip for one of the nodes):

12:42:14,757 WARN [elasticsearch[Fury][cached]-pool-3-thread-1]
[discovery.zen] - [Fury] failed to connect to master

[[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300
]]],
retrying...
org.elasticsearch.transport.ConnectTransportException:
[Freakshow][inet[/10.198.109.171:9300]]
Caused by: org.elasticsearch.transport.ConnectTransportException:
[Freakshow][inet[/10.198.109.171:9300]]
12:42:19,275 WARN [elasticsearch[Fury][cached]-pool-3-thread-1]
[discovery.zen] - [Fury] failed to connect to master

[[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300
]]],
retrying...

It does that over and over until it gives up:

12:42:39,845 WARN [main] [discovery] - [Fury] waited for 30s and no
initial
state was set by the discovery

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1727789.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(John Chang) #3

I'm sorry, I'm not sure I understand your response, but at any rate I think I failed for two reasons to give you a correct understanding of my situation.

We are trying to connect to two ec2-hosted elasticsearch nodes from outside the cloud. We don't have access to ec2's private IP space from our dev network, so we need the elasticsearch discovery to publish the public dns name of the instances (eg ec2-204-236-195-66.compute-1.amazonaws.com), not an IP address. That public DNS name is the only way to access the elasticsearch nodes from both
inside and outside the cloud. My dev machine outside the cloud is the one that produced the ConnectTransportException log errors copied above.

Also, the config of the nodes in the ec2 cloud I gave you was wrong. This is what I should have given you:
discovery:
host_type: public_dns
type: zen
zen:
ping:
unicast:
hosts: elasticsearch1.dev5.cloud.test.com[9300],elasticsearch2.dev5.cloud.test.com[9300]
enabled: true
multicast:
enabled: false


(Shay Banon) #4

Can you try and connect using the transport client and not the node client?

On Tue, Oct 19, 2010 at 12:18 AM, John Chang jchangkihtest2@gmail.comwrote:

I'm sorry, I'm not sure I understand your response, but at any rate I think
I
failed for two reasons to give you a correct understanding of my situation.

We are trying to connect to two ec2-hosted elasticsearch nodes from outside
the cloud. We don't have access to ec2's private IP space from our dev
network, so we need the elasticsearch discovery to publish the public dns
name of the instances (eg ec2-204-236-195-66.compute-1.amazonaws.com), not
an IP address. That public DNS name is the only way to access the
elasticsearch nodes from both
inside and outside the cloud. My dev machine outside the cloud is the one
that produced the ConnectTransportException log errors copied above.

Also, the config of the nodes in the ec2 cloud I gave you was wrong. This
is what I should have given you:
discovery:
host_type: public_dns
type: zen
zen:
ping:
unicast:
hosts:
elasticsearch1.dev5.cloud.test.com[9300],
elasticsearch2.dev5.cloud.test.com[9300]
enabled: true
multicast:
enabled: false

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1728737.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(John Chang) #5

Still can't connect, but the errors manifest differently.

I switched to connect the node in my dev machine (outside the ec2 cloud) this way:
indexClient = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress("", 9300))
.addTransportAddress(new InetSocketTransportAddress("", 9300));

I can get through the above lines without exception, but I see in the logs:
16:24:47,658 WARN [main] [client.transport] - [Rodstvow] Node [Jimmy Woo][06aff47a-c269-4ac0-9170-a64d4a66ced1][inet[/10.206.98.255:9300]] not part of the cluster Cluster [elasticsearch], ignoring...
16:24:47,660 WARN [main] [client.transport] - [Rodstvow] Node [Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/10.198.109.171:9300]] not part of the cluster Cluster [elasticsearch], ignoring...

10.206.98.255 and 10.198.109.171 are still the private ip's only visible inside the ec2 cloud. Looking the logs of the nodes in the cloud, I don't see any evidence that I've joined.

Then, when I search I get a
org.elasticsearch.client.transport.NoNodeAvailableException: No node available

By the way, once I get it to work with the Transport client, how do I set the cluster name?


(Shay Banon) #6

You need to set the cluster name on the transport client so it can join the
cluster you started. When you construct a new TransportClient instance, you
can pass it settings, in the settings, you can place "cluster.name" with the
cluster name that you started the nodes with.

You won't see the client joining the nodes, as the transport client is not
part of the cluster.

-shay.banon

On Tue, Oct 19, 2010 at 1:30 AM, John Chang jchangkihtest2@gmail.comwrote:

Still can't connect, but the errors manifest differently.

I switched to connect the node in my dev machine (outside the ec2 cloud)
this way:
indexClient = new TransportClient()
.addTransportAddress(new
InetSocketTransportAddress("", 9300))
.addTransportAddress(new
InetSocketTransportAddress("", 9300));

I can get through the above lines without exception, but I see in the logs:
16:24:47,658 WARN [main] [client.transport] - [Rodstvow] Node [Jimmy
Woo][06aff47a-c269-4ac0-9170-a64d4a66ced1][inet[/10.206.98.255:9300]] not
part of the cluster Cluster [elasticsearch], ignoring...
16:24:47,660 WARN [main] [client.transport] - [Rodstvow] Node
[Freakshow][594cc07f-6181-4794-87c5-7471ba31e70d][inet[/
10.198.109.171:9300]]
not part of the cluster Cluster [elasticsearch], ignoring...

10.206.98.255 and 10.198.109.171 are still the private ip's only visible
inside the ec2 cloud. Looking the logs of the nodes in the cloud, I don't
see any evidence that I've joined.

Then, when I search I get a
org.elasticsearch.client.transport.NoNodeAvailableException: No node
available

By the way, once I get it to work with the Transport client, how do I set
the cluster name?

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/discovery-ec2-host-type-public-dns-not-working-as-expected-tp1727789p1729092.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(system) #7