EC2 client node cluster discovery

Elastic Stack Elasticsearch
1

I am running elasticsearch 1.4.4 and the aws plugin 2.4.1

I am trying to run multiple clusters on the same machine with the same es
install. It's a small dataset and I'm running a qe, demo, and staging
cluster.

My java api client won't connect to a running es server after a tomcat
restart, but when I restart the es cluster it discovers the running client.
I have opened ports 9200-9400 for internal communication using a security
group that is also used for ec2 discovery

When I restart the staging tomcat instance it rejoins elastic search. When
I restart qe it does not rejoin with the warning:
"org.elasticsearch.discovery: 91 - [Peter Noble] waited for 30s and no
initial state was set by the discovery"

If I restart the qe es cluster it picks up Peter Noble right away.

Earlier today this situation was switched where the staging cluster would
have to be restarted. I think the change was after I brought down both
clusters and brought staging up first.

Additional config details:
The web servers a all seperate AWS servers in the same subnet and the 3 es
clusters are all running on the same box.

I'm passing cluster name and node name in when starting elastic search

elasticsearch --cluster.name=QE --node.name=es-qe-1

My es master has the config:

discovery.zen.ping.multicast.enabled: false
discovery.type: ec2
discovery.ec2.groups: elasticsearch
cloud.aws.protocol: http
cloud.aws.region: us-east-1
cloud.aws.access_key: ******
cloud.aws.secret_key: ******
discovery.ec2.ping_timeout: 10s

I'm using the java api from a node client

NodeBuilder nodeBuilder = nodeBuilder().clusterName(clusterName).client(true);
if (isAws) {
ImmutableSettings.Builder settings =
ImmutableSettings.settingsBuilder()
.put("cloud.aws.access_key", "")
.put("cloud.aws.secret_key", "")
.put("cloud.aws.region", "us-east-1")
.put("cloud.aws.protocol", "http")
.put("discovery.type", "ec2")
.put("discovery.ec2.groups", "elasticsearch")
.put("discovery.zen.ping.multicast.enabled", "false");
nodeBuilder = nodeBuilder.settings(settings);

Any ideas would be appreciated!

Jerry

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/090d6bab-0ace-4537-99f7-547410fa5372%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Still trying to figure out a way past this issue but I actually know what's going on now. The first cluster takes port 9300 and anything looking for a different cluster on that same machine stops when it sees 9300 is in use by a different cluster. Elasticsearch used to scan the port range but they removed that code because it's expensive to check so many ports. There is a pull request to make the number of ports to scan configurable.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/870412d3-95dd-4b5b-b42e-60dc16411c78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Seems like not many other people have this problem...

In my case this was only a problem using unicast (which cloud-aws does), so
I changed the cloud plugin to work for my case. The file that needed to be
changed was AwsEc2UnicastHostsProvider.java. Look for UnicastZenPing.LIMIT_PORTS_COUNT.
That const is hard coded to 1 in ES v1.5. I changed it to read a config
value or use the const if no config was found. This worked fine for me
with the es-1.5 2.5.0 version of the plugin.

Seems like anyone running more than one node on a server not using
multicast could have this issue.

On Friday, March 27, 2015 at 11:02:05 PM UTC-4, Jerry Pattenaude wrote:

Still trying to figure out a way past this issue but I actually know
what's going on now. The first cluster takes port 9300 and anything looking
for a different cluster on that same machine stops when it sees 9300 is in
use by a different cluster. Elasticsearch used to scan the port range but
they removed that code because it's expensive to check so many ports. There
is a pull request to make the number of ports to scan configurable.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ccafba78-3c68-40e8-9e0b-0f6618517abc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.