EC2 Discovery

Elastic Stack Elasticsearch
1

I can't seem to make my EC2 cluster of 2 nodes/instances work.

  • If I comment out the section below, I can connect to each instance
    individually and query it, I am using Sense plugin to query Elastic Search
    Instance.

  • With the sections included as below, I cannot query neither node, I get
    "Request failed to get to the server (status code: 0):"

//////////////////////// In /instance 1/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_1"

cloud:

aws:

   access_key: <MY ACCESSKEY>

   secret_key: <My SecretKEY>

discovery:

type: ec2

//////////////////////// In /instance 2/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_2"

cloud:

aws:

  access_key: <MY ACCESSKEY>

  secret_key: <My SecretKEY>

discovery:

type: ec2

I also tried with:

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2
IP:9300"]

Here is "mycluster.log" from both instances:

$ more /var/log/elasticsearch/mycluster.log

[2014-03-20 19:00:37,635][INFO ][node ] [node_1] version
[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:00:37,698][INFO ][plugins ] [node_1] loaded
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:17,898][INFO ][node ] [node_1] version
[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:01:17,961][INFO ][plugins ] [node_1] loaded
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:50,048][INFO ][node ] [node_1] version
[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:03:50,111][INFO ][plugins ] [node_1] loaded
[mapper-attachments, cloud-aws], sites []

$more /var/log/elasticsearch/mycluster.log

[2014-03-20 19:00:29,465][INFO ][node ] [node_2] version
[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node ] [node_2]initializing
...

[2014-03-20 19:00:29,528][INFO ][plugins ] [node_2] loaded
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:26,645][INFO ][node ] [node_2] version
[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node ] [node_2]initializing
...

[2014-03-20 19:01:26,709][INFO ][plugins ] [node_2] loaded
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:58,357][INFO ][node ] [node_2] version
[0.90.10], pid[2951], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:58,357][INFO ][node ] [node_2]initializing
...

[2014-03-20 19:03:58,420][INFO ][plugins ] [node_2] loaded
[mapper-attachments, cloud-aws], sites []

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/69b43625-f560-4e42-a987-2ba2310d462b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Any clues to what i am missing, i turned discovery trace on, but dont't see any useful info.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f3dce1e-53d1-4c38-804f-6262896d43d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Don't try ec2 discovery until you have tested that:

  • you can connect from one machine to another on port 9300 ( nc as client
    and server, basic networking/ firewalling)
  • run a simple aws ec2 describe instances call with the API key you plan to
    use, and you can see the machines you need there. Bonus points for
    filtering based on the rules you intense to use ( sec group, tags). This is
    to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will
run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to
ack your new nodes request for cluster info...but that would be a problem
with zen disco too.
On 21/03/2014 12:31 PM, "Raphael Miranda" raphaelmiranda@gmail.com wrote:

are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4LjvKAW9cnkrjQUR6%3Dk8FRf%3DKzmDUAUopHLVUMNc1ixOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

I am not sure if I missed something, but what you mentioned I believe I
already tried as showing in my original post.
I can connect to each machine individually and I am able ti index and query
it fine with default configuration without any zen or ec2 settings. But,
when I turned them on like I show on the post, I get this "Request failed
to get to the server (status code: 0):" when trying to query the instance.
Did you mean I should try to see if I can access one instance from the
other? This I didn't try yet.

On Friday, March 21, 2014 4:46:40 AM UTC-4, Norberto Meijome wrote:

Don't try ec2 discovery until you have tested that:

  • you can connect from one machine to another on port 9300 ( nc as client
    and server, basic networking/ firewalling)
  • run a simple aws ec2 describe instances call with the API key you plan
    to use, and you can see the machines you need there. Bonus points for
    filtering based on the rules you intense to use ( sec group, tags). This is
    to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will
run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to
ack your new nodes request for cluster info...but that would be a problem
with zen disco too.
On 21/03/2014 12:31 PM, "Raphael Miranda" <raphael...@gmail.com<javascript:>>
wrote:

are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c9e9da8-6efe-4005-8a69-c00daa6ec711%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

I am not sure if I missed something, but what you mentioned I believe I
already tried as showing in my original post.
I can connect from one instance to another.
I can connect to each machine individually and I am able to index and query
it fine with default configuration without any zen or ec2 settings. But,
when I turned them on like I show on the post, I get this "Request failed
to get to the server (status code: 0):" when trying to query the instance,
and when I do this, it won't even log anything, it is not getting that far.

On Friday, March 21, 2014 4:46:40 AM UTC-4, Norberto Meijome wrote:

Don't try ec2 discovery until you have tested that:

  • you can connect from one machine to another on port 9300 ( nc as client
    and server, basic networking/ firewalling)
  • run a simple aws ec2 describe instances call with the API key you plan
    to use, and you can see the machines you need there. Bonus points for
    filtering based on the rules you intense to use ( sec group, tags). This is
    to ensure your API keys have the correct access needed.

Once you have those basic steps working, use them on es config.

Make sure you enable ec2 discovery and disable the zen discovery ( it will
run first and likely time out and ec2 disco won't get to exec).

The other thing to watch out for is contacting nodes which are too busy to
ack your new nodes request for cluster info...but that would be a problem
with zen disco too.
On 21/03/2014 12:31 PM, "Raphael Miranda" <raphael...@gmail.com<javascript:>>
wrote:

are both machines in the same security group?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48571118-fd84-45da-9aaf-0314c936336b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Ok, we are seeing this error in the log, any clues?

Error injecting constructor, java.lang.IllegalStateException: This is a
proxy used to support circular references involving constructors. The
object we're proxying is not constructed yet. Please wait until after
injection has completed to use this object.
at org.elasticsearch.cache.NodeCache.(Unknown Source)
while locating org.elasticsearch.cache.NodeCache
Caused by: java.lang.IllegalStateException: This is a proxy used to support
circular references involving constructors.

On Thursday, March 20, 2014 3:18:11 PM UTC-4, IronMan2014 wrote:

I can't seem to make my EC2 cluster of 2 nodes/instances work.

  • If I comment out the section below, I can connect to each instance
    individually and query it, I am using Sense plugin to query Elastic Search
    Instance.

  • With the sections included as below, I cannot query neither node, I get
    "Request failed to get to the server (status code: 0):"

//////////////////////// In /instance 1/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_1"

cloud:

aws:

   access_key: <MY ACCESSKEY>

   secret_key: <My SecretKEY>

discovery:

type: ec2

//////////////////////// In /instance 2/ elasticsearch.yml////////////////
cluster.name: mycluster
node.name: "node_2"

cloud:

aws:

  access_key: <MY ACCESSKEY>

  secret_key: <My SecretKEY>

discovery:

type: ec2

I also tried with:

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2
IP:9300"]

Here is "mycluster.log" from both instances:

$ more /var/log/elasticsearch/mycluster.log

[2014-03-20 19:00:37,635][INFO ][node ] [node_1]version
[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:00:37,698][INFO ][plugins ] [node_1]loaded
[mapper-attachments, cloud-aws], sites

[2014-03-20 19:01:17,898][INFO ][node ] [node_1]version
[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:01:17,961][INFO ][plugins ] [node_1]loaded
[mapper-attachments, cloud-aws], sites

[2014-03-20 19:03:50,048][INFO ][node ] [node_1]version
[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node ] [node_1]initializing
...

[2014-03-20 19:03:50,111][INFO ][plugins ] [node_1]loaded
[mapper-attachments, cloud-aws], sites

$more /var/log/elasticsearch/mycluster.log

[2014-03-20 19:00:29,465][INFO ][node ] [node_2]version
[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node ] [node_2]initializing
...

[2014-03-20 19:00:29,528][INFO ][plugins ] [node_2]loaded
[mapper-attachments, cloud-aws], sites

[2014-03-20 19:01:26,645][INFO ][node ] [node_2]version
[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node ] [node_2]initializing
...

[2014-03-<s
...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ed56a05-1538-4a4d-91d7-8a9880ef7508%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.