EC2 Discovery problems for Elasticsearch clustering

Hi everyone,

I was able to setup two instances of elasticsearch on separate EC2
instances (t1.micro). They are backed by S3 for storage. I followed the
tutorial here
(http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/).

I'm new to both EC2 and elasticsearch. I can't get the two EC2 instances
to cluster.

In related posts - one of the first questions people ask is whether or not
you can telnet between the EC2 instances. In my case, I cannot telnet
between the two EC2 instances. Both EC2 instances use the same security
group called "elastic" which is configured the following way:

9300 sg-3bc28e0b (elastic)
22 (SSH) 0.0.0.0/0
23 (TELNET) 0.0.0.0/0
9200 0.0.0.0/0

When i try and telnet from one box to the other using the private IP, or
public DNS I get "Connection refused"

When I'm on one EC2 instance, I've tried telnetting to the other by:

Nothing works. What am i doing wrong? I have IAM setup, not sure if that
would have anything to do with it? Any ideas?

If I can't telnet from one box to the other, my understanding is that I'm
not going to have luck with elasticsearch discover (which I haven't so
far). Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday 06 June 2013 07:29:31 PM IST, Alex Ressi wrote:

Hi everyone,

I was able to setup two instances of elasticsearch on separate EC2
instances (t1.micro). They are backed by S3 for storage. I followed
the tutorial here
(http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/).

I'm new to both EC2 and elasticsearch. I can't get the two EC2
instances to cluster.

In related posts - one of the first questions people ask is whether or
not you can telnet between the EC2 instances. In my case, I cannot
telnet between the two EC2 instances. Both EC2 instances use the same
security group called "elastic" which is configured the following way:

|9300         sg-3bc28e0b (elastic)
22 (SSH)     0.0.0.0/0
23 (TELNET)  0.0.0.0/0
9200         0.0.0.0/0|

if sg-3bc28e0b be the elastic security group, the above mentioned
tabular inbound rules belong to which security group?? In case, If you
have mentioned recursively ( i dont know how amazon allows it), it is
not required as it is enforced by default.
What i am suggesting is 9300 0.0.0.0/0 being an inbound rule for
elastic SG
Tell me if there is a caveat hidden.

When i try and telnet from one box to the other using the private IP,
or public DNS I get "Connection refused"

When I'm on one EC2 instance, I've tried telnetting to the other by:

Binding it to a local IP is a smart thing to do. What does it say when
you try to connect to 9200

Nothing works. What am i doing wrong? I have IAM setup, not sure if
that would have anything to do with it? Any ideas?

If I can't telnet from one box to the other, my understanding is that
I'm not going to have luck with elasticsearch discover (which I
haven't so far). Thanks in advance!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

  • Your configuration seems fulfilling to me. If the 2 instances belong
    to the same security group. No inbound or outbound rules apply them.
    And since you are from one instance to another, It is likely you
    should be able to connect.
    And in that case I guess the inbound rules you specified are them to be
    accessed from outside EC2.

  • Here is a couple of things that you may want to check from inside the
    instance.
    -> Is the elasticsearch binding to port 9300/9200.
    -> if yes. is it on 0.0.0.0 or 10.XX.XX.XX? It shouldnot be on
    127.0.0.0. e.g. you will find a rule in netstat -ntupl lile
    X.X.X.X/9300 /java. (where X.X.X.X not belonging to 127.0.0.0 n/w)
    -> normally elasticsearch binds to two ports. Check both of em.
    -> Is ICMP enabled on your system.
    -> Check the iptables rules of the system (assuming linux). If possible
    turn it off.
    -> are u able to telnet localhost at 9300 from your instance.
    -> Check the elasticsearch logs for any misconfiguration.
    -> Run TCPDump to see whether TCP handshake is completing or not.

--saurajeet.d

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for responding Tobias

Here are some answers and some follow-up questions as well.

  1. My "elastic" security group with the inbound ports enabled are all
    "custom TCP rules." Is that the right setting or is that the problem right
    there?

  2. I changed the security group so that 9300 now references 0.0.0.0/0

  3. If i'm logged into one of my instances and try "telnet localhost 9300"
    or "telnet localhost 9200" the connection is refused.

  4. Still can't telnet to the other machine using private IP and either port
    9200 or 9300.

Here's the output when I start elasticsearch up on one instance of EC2.
Let me know if that helps and thanks for your help on this.

elasticsearch-0.19.0]$ sudo bin/elasticsearch -f
[2013-06-06 18:15:41,124][WARN ][bootstrap ] jvm uses the
client vm, make sure to run java with the server vm for best performance
by adding -server to the command line
[2013-06-06 18:15:41,133][INFO ][node ] [Eagle]
{0.19.0}[20440]: initializing ...
[2013-06-06 18:15:41,151][INFO ][plugins ] [Eagle] loaded
[cloud-aws], sites []
[2013-06-06 18:15:42,216][DEBUG][discovery.zen.ping.unicast] [Eagle] using
initial hosts [], with concurrent_connects [10]
[2013-06-06 18:15:42,217][DEBUG][discovery.ec2 ] [Eagle] using
ping.timeout [3s]
[2013-06-06 18:15:42,224][DEBUG][discovery.zen.elect ] [Eagle] using
minimum_master_nodes [-1]
[2013-06-06 18:15:42,226][DEBUG][discovery.zen.fd ] [Eagle]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 18:15:42,229][DEBUG][discovery.zen.fd ] [Eagle] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 18:15:42,444][DEBUG][discovery.ec2 ] [Eagle] using
host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true],
availability_zones [[]]
[2013-06-06 18:15:43,349][DEBUG][gateway.s3 ] [Eagle] using
bucket [intralist], region [null], chunk_size [100mb], concurrent_streams
[5]
[2013-06-06 18:15:44,223][DEBUG][gateway.s3 ] [Eagle] Latest
metadata found at index [1]
[2013-06-06 18:15:44,226][INFO ][node ] [Eagle]
{0.19.0}[20440]: initialized
[2013-06-06 18:15:44,226][INFO ][node ] [Eagle]
{0.19.0}[20440]: starting ...
[2013-06-06 18:15:44,295][INFO ][transport ] [Eagle]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.248.65.219:9300]}
[2013-06-06 18:15:44,300][TRACE][discovery ] [Eagle] waiting
for 30s for the initial state to be set by the discovery
[2013-06-06 18:15:44,586][TRACE][discovery.ec2 ] [Eagle]
building dynamic unicast discovery nodes...
[2013-06-06 18:15:44,586][DEBUG][discovery.ec2 ] [Eagle] using
dynamic discovery nodes []
[2013-06-06 18:15:46,241][TRACE][discovery.ec2 ] [Eagle]
building dynamic unicast discovery nodes...
[2013-06-06 18:15:46,241][DEBUG][discovery.ec2 ] [Eagle] using
dynamic discovery nodes []
[2013-06-06 18:15:47,894][TRACE][discovery.ec2 ] [Eagle]
building dynamic unicast discovery nodes...
[2013-06-06 18:15:47,895][DEBUG][discovery.ec2 ] [Eagle] using
dynamic discovery nodes []
[2013-06-06 18:15:47,895][DEBUG][discovery.ec2 ] [Eagle] ping
responses: {none}
[2013-06-06 18:15:47,899][INFO ][cluster.service ] [Eagle]
new_master
[Eagle][BjyfzoRgTGmkdOMQBYSZpQ][inet[/10.248.65.219:9300]]{master=true},
reason: zen-disco-join (elected_as_master)
[2013-06-06 18:15:47,934][TRACE][discovery ] [Eagle] initial
state set from discovery
[2013-06-06 18:15:47,934][INFO ][discovery ] [Eagle]
es-prod/BjyfzoRgTGmkdOMQBYSZpQ
[2013-06-06 18:15:47,935][DEBUG][gateway.s3 ] [Eagle] reading
state from gateway
org.elasticsearch.gateway.shared.SharedStorageGateway$1@2e879 ...
[2013-06-06 18:15:47,938][INFO ][http ] [Eagle]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.248.65.219:9200]}
[2013-06-06 18:15:47,939][INFO ][node ] [Eagle]
{0.19.0}[20440]: started
[2013-06-06 18:15:48,000][DEBUG][gateway.s3 ] [Eagle] read
state from gateway
org.elasticsearch.gateway.shared.SharedStorageGateway$1@2e879, took 64ms
[2013-06-06 18:15:48,013][INFO ][gateway ] [Eagle]
recovered [0] indices into cluster_state
[2013-06-06 18:15:48,014][DEBUG][gateway.s3 ] [Eagle] writing
to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@bafda6
...
[2013-06-06 18:15:48,241][DEBUG][gateway.s3 ] [Eagle] wrote
to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@bafda6,
took 227ms

On Thursday, June 6, 2013 12:22:23 PM UTC-4, Tobias Reiper wrote:

On Thursday 06 June 2013 07:29:31 PM IST, Alex Ressi wrote:

Hi everyone,

I was able to setup two instances of elasticsearch on separate EC2
instances (t1.micro). They are backed by S3 for storage. I followed
the tutorial here
(http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/).

I'm new to both EC2 and elasticsearch. I can't get the two EC2
instances to cluster.

In related posts - one of the first questions people ask is whether or
not you can telnet between the EC2 instances. In my case, I cannot
telnet between the two EC2 instances. Both EC2 instances use the same
security group called "elastic" which is configured the following way:

|9300         sg-3bc28e0b (elastic) 
22 (SSH)     0.0.0.0/0 
23 (TELNET)  0.0.0.0/0 
9200         0.0.0.0/0| <http://0.0.0.0/0%7C> 

if sg-3bc28e0b be the elastic security group, the above mentioned
tabular inbound rules belong to which security group?? In case, If you
have mentioned recursively ( i dont know how amazon allows it), it is
not required as it is enforced by default.
What i am suggesting is 9300 0.0.0.0/0 being an inbound rule for
elastic SG
Tell me if there is a caveat hidden.

When i try and telnet from one box to the other using the private IP,
or public DNS I get "Connection refused"

When I'm on one EC2 instance, I've tried telnetting to the other by:

Binding it to a local IP is a smart thing to do. What does it say when
you try to connect to 9200

Nothing works. What am i doing wrong? I have IAM setup, not sure if
that would have anything to do with it? Any ideas?

If I can't telnet from one box to the other, my understanding is that
I'm not going to have luck with elasticsearch discover (which I
haven't so far). Thanks in advance!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

  • Your configuration seems fulfilling to me. If the 2 instances belong
    to the same security group. No inbound or outbound rules apply them.
    And since you are from one instance to another, It is likely you
    should be able to connect.
    And in that case I guess the inbound rules you specified are them to be
    accessed from outside EC2.

  • Here is a couple of things that you may want to check from inside the
    instance.
    -> Is the elasticsearch binding to port 9300/9200.
    -> if yes. is it on 0.0.0.0 or 10.XX.XX.XX? It shouldnot be on
    127.0.0.0. e.g. you will find a rule in netstat -ntupl lile
    X.X.X.X/9300 /java. (where X.X.X.X not belonging to 127.0.0.0 n/w)
    -> normally elasticsearch binds to two ports. Check both of em.
    -> Is ICMP enabled on your system.
    -> Check the iptables rules of the system (assuming linux). If possible
    turn it off.
    -> are u able to telnet localhost at 9300 from your instance.
    -> Check the elasticsearch logs for any misconfiguration.
    -> Run TCPDump to see whether TCP handshake is completing or not.

--saurajeet.d

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.