# EC2 Discovery problems for Elasticsearch clustering **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2013, 1:59pm UTC](https://discuss.elastic.co/t/ec2-discovery-problems-for-elasticsearch-clustering/12297 "2013-06-06T13:59:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Alex\_Ressi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_ressi/32/2180_2.png) [@Alex\_Ressi](https://discuss.elastic.co/u/Alex_Ressi)\ **Post date:** [June 6, 2013, 1:59pm UTC](https://discuss.elastic.co/t/ec2-discovery-problems-for-elasticsearch-clustering/12297/1 "2013-06-06T13:59:31Z")
Hi everyone, I was able to setup two instances of elasticsearch on separate EC2 instances (t1.micro). They are backed by S3 for storage. I followed the tutorial here ([http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/](http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/)). I'm new to both EC2 and elasticsearch. I can't get the two EC2 instances to cluster. In related posts - one of the first questions people ask is whether or not you can telnet between the EC2 instances. In my case, I cannot telnet between the two EC2 instances. Both EC2 instances use the same security group called "elastic" which is configured the following way: 9300 sg-3bc28e0b (elastic) 22 (SSH) 0.0.0.0/0 23 (TELNET) 0.0.0.0/0 9200 0.0.0.0/0 When i try and telnet from one box to the other using the private IP, or public DNS I get "Connection refused" When I'm on one EC2 instance, I've tried telnetting to the other by: - telnet 10.xxx.65.xxx 9300 - telnet [ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com](http://ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com) 9300 Nothing works. What am i doing wrong? I have IAM setup, not sure if that would have anything to do with it? Any ideas? If I can't telnet from one box to the other, my understanding is that I'm not going to have luck with elasticsearch discover (which I haven't so far). Thanks in advance! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tobias\_Reiper](https://avatars.discourse-cdn.com/v4/letter/t/50afbb/32.png) [@Tobias\_Reiper](https://discuss.elastic.co/u/Tobias_Reiper)\ **Post date:** [June 6, 2013, 4:22pm UTC](https://discuss.elastic.co/t/ec2-discovery-problems-for-elasticsearch-clustering/12297/2 "2013-06-06T16:22:23Z")
On Thursday 06 June 2013 07:29:31 PM IST, Alex Ressi wrote: > Hi everyone, > > I was able to setup two instances of elasticsearch on separate EC2 > instances (t1.micro). They are backed by S3 for storage. I followed > the tutorial here > ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/)). > > I'm new to both EC2 and elasticsearch. I can't get the two EC2 > instances to cluster. > In related posts - one of the first questions people ask is whether or > not you can telnet between the EC2 instances. In my case, I cannot > telnet between the two EC2 instances. Both EC2 instances use the same > security group called "elastic" which is configured the following way: > > ``` > |9300 sg-3bc28e0b (elastic) > 22 (SSH) 0.0.0.0/0 > 23 (TELNET) 0.0.0.0/0 > 9200 0.0.0.0/0| > > ``` if sg-3bc28e0b be the elastic security group, the above mentioned tabular inbound rules belong to which security group?? In case, If you have mentioned recursively ( i dont know how amazon allows it), it is not required as it is enforced by default. What i am suggesting is 9300 0.0.0.0/0 being an inbound rule for elastic SG Tell me if there is a caveat hidden. > When i try and telnet from one box to the other using the private IP, > or public DNS I get "Connection refused" > > When I'm on one EC2 instance, I've tried telnetting to the other by: > > - telnet 10.xxx.65.xxx 9300 > - telnet [ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com](http://ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com) 9300 Binding it to a local IP is a smart thing to do. What does it say when you try to connect to 9200 > Nothing works. What am i doing wrong? I have IAM setup, not sure if > that would have anything to do with it? Any ideas? > > If I can't telnet from one box to the other, my understanding is that > I'm not going to have luck with elasticsearch discover (which I > haven't so far). Thanks in advance! > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). - Your configuration seems fulfilling to me. If the 2 instances belong to the same security group. No inbound or outbound rules apply them. And since you are from one instance to another, It is likely you should be able to connect. And in that case I guess the inbound rules you specified are them to be accessed from outside EC2. - Here is a couple of things that you may want to check from inside the instance. -\> Is the elasticsearch binding to port 9300/9200. -\> if yes. is it on 0.0.0.0 or 10.XX.XX.XX? It shouldnot be on 127.0.0.0. e.g. you will find a rule in netstat -ntupl lile X.X.X.X/9300 /java. (where X.X.X.X not belonging to 127.0.0.0 n/w) -\> normally elasticsearch binds to two ports. Check both of em. -\> Is ICMP enabled on your system. -\> Check the iptables rules of the system (assuming linux). If possible turn it off. -\> are u able to telnet localhost at 9300 from your instance. -\> Check the elasticsearch logs for any misconfiguration. -\> Run TCPDump to see whether TCP handshake is completing or not. --saurajeet.d -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Alex\_Ressi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_ressi/32/2180_2.png) [@Alex\_Ressi](https://discuss.elastic.co/u/Alex_Ressi)\ **Post date:** [June 6, 2013, 6:19pm UTC](https://discuss.elastic.co/t/ec2-discovery-problems-for-elasticsearch-clustering/12297/3 "2013-06-06T18:19:09Z")
Thanks for responding Tobias Here are some answers and some follow-up questions as well. 1. My "elastic" security group with the inbound ports enabled are all "custom TCP rules." Is that the right setting or is that the problem right there? 2. I changed the security group so that 9300 now references 0.0.0.0/0 3. If i'm logged into one of my instances and try "telnet localhost 9300" or "telnet localhost 9200" the connection is refused. 4. Still can't telnet to the other machine using private IP and either port 9200 or 9300. Here's the output when I start elasticsearch up on one instance of EC2. Let me know if that helps and thanks for your help on this. elasticsearch-0.19.0]$ sudo bin/elasticsearch -f [2013-06-06 18:15:41,124][WARN][bootstrap] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line [2013-06-06 18:15:41,133][INFO][node] [Eagle] {0.19.0}[20440]: initializing ... [2013-06-06 18:15:41,151][INFO][plugins] [Eagle] loaded [cloud-aws], sites [2013-06-06 18:15:42,216][DEBUG][discovery.zen.ping.unicast] [Eagle] using initial hosts , with concurrent\_connects [10] [2013-06-06 18:15:42,217][DEBUG][discovery.ec2] [Eagle] using ping.timeout [3s] [2013-06-06 18:15:42,224][DEBUG][discovery.zen.elect] [Eagle] using minimum\_master\_nodes [-1] [2013-06-06 18:15:42,226][DEBUG][discovery.zen.fd] [Eagle] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-06-06 18:15:42,229][DEBUG][discovery.zen.fd] [Eagle] [node ] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3] [2013-06-06 18:15:42,444][DEBUG][discovery.ec2] [Eagle] using host\_type [PRIVATE\_IP], tags [{}], groups [] with any\_group [true], availability\_zones [] [2013-06-06 18:15:43,349][DEBUG][gateway.s3] [Eagle] using bucket [intralist], region [null], chunk\_size [100mb], concurrent\_streams [5] [2013-06-06 18:15:44,223][DEBUG][gateway.s3] [Eagle] Latest metadata found at index [1] [2013-06-06 18:15:44,226][INFO][node] [Eagle] {0.19.0}[20440]: initialized [2013-06-06 18:15:44,226][INFO][node] [Eagle] {0.19.0}[20440]: starting ... [2013-06-06 18:15:44,295][INFO][transport] [Eagle] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/10.248.65.219:9300]} [2013-06-06 18:15:44,300][TRACE][discovery] [Eagle] waiting for 30s for the initial state to be set by the discovery [2013-06-06 18:15:44,586][TRACE][discovery.ec2] [Eagle] building dynamic unicast discovery nodes... [2013-06-06 18:15:44,586][DEBUG][discovery.ec2] [Eagle] using dynamic discovery nodes [2013-06-06 18:15:46,241][TRACE][discovery.ec2] [Eagle] building dynamic unicast discovery nodes... [2013-06-06 18:15:46,241][DEBUG][discovery.ec2] [Eagle] using dynamic discovery nodes [2013-06-06 18:15:47,894][TRACE][discovery.ec2] [Eagle] building dynamic unicast discovery nodes... [2013-06-06 18:15:47,895][DEBUG][discovery.ec2] [Eagle] using dynamic discovery nodes [2013-06-06 18:15:47,895][DEBUG][discovery.ec2] [Eagle] ping responses: {none} [2013-06-06 18:15:47,899][INFO][cluster.service] [Eagle] new\_master [Eagle][BjyfzoRgTGmkdOMQBYSZpQ][inet[/10.248.65.219:9300]]{master=true}, reason: zen-disco-join (elected\_as\_master) [2013-06-06 18:15:47,934][TRACE][discovery] [Eagle] initial state set from discovery [2013-06-06 18:15:47,934][INFO][discovery] [Eagle] es-prod/BjyfzoRgTGmkdOMQBYSZpQ [2013-06-06 18:15:47,935][DEBUG][gateway.s3] [Eagle] reading state from gateway org.elasticsearch.gateway.shared.SharedStorageGateway$1@2e879 ... [2013-06-06 18:15:47,938][INFO][http] [Eagle] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/10.248.65.219:9200]} [2013-06-06 18:15:47,939][INFO][node] [Eagle] {0.19.0}[20440]: started [2013-06-06 18:15:48,000][DEBUG][gateway.s3] [Eagle] read state from gateway org.elasticsearch.gateway.shared.SharedStorageGateway$1@2e879, took 64ms [2013-06-06 18:15:48,013][INFO][gateway] [Eagle] recovered [0] indices into cluster\_state [2013-06-06 18:15:48,014][DEBUG][gateway.s3] [Eagle] writing to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@bafda6 ... [2013-06-06 18:15:48,241][DEBUG][gateway.s3] [Eagle] wrote to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@bafda6, took 227ms On Thursday, June 6, 2013 12:22:23 PM UTC-4, Tobias Reiper wrote: > On Thursday 06 June 2013 07:29:31 PM IST, Alex Ressi wrote: > > > Hi everyone, > > > > I was able to setup two instances of elasticsearch on separate EC2 > > instances (t1.micro). They are backed by S3 for storage. I followed > > the tutorial here > > ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/)). > > > > I'm new to both EC2 and elasticsearch. I can't get the two EC2 > > instances to cluster. > > > In related posts - one of the first questions people ask is whether or > > not you can telnet between the EC2 instances. In my case, I cannot > > telnet between the two EC2 instances. Both EC2 instances use the same > > security group called "elastic" which is configured the following way: > > > > ``` > > |9300 sg-3bc28e0b (elastic) > > 22 (SSH) 0.0.0.0/0 > > 23 (TELNET) 0.0.0.0/0 > > 9200 0.0.0.0/0| > > > > ``` > > if sg-3bc28e0b be the elastic security group, the above mentioned > tabular inbound rules belong to which security group?? In case, If you > have mentioned recursively ( i dont know how amazon allows it), it is > not required as it is enforced by default. > What i am suggesting is 9300 0.0.0.0/0 being an inbound rule for > elastic SG > Tell me if there is a caveat hidden. > > > When i try and telnet from one box to the other using the private IP, > > or public DNS I get "Connection refused" > > > > When I'm on one EC2 instance, I've tried telnetting to the other by: > > > > - telnet 10.xxx.65.xxx 9300 > > - telnet [ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com](http://ec2-50-xxx-xx-127.us-west-2.compute.amazonaws.com) 9300 > > Binding it to a local IP is a smart thing to do. What does it say when > you try to connect to 9200 > > > Nothing works. What am i doing wrong? I have IAM setup, not sure if > > that would have anything to do with it? Any ideas? > > > > If I can't telnet from one box to the other, my understanding is that > > I'm not going to have luck with elasticsearch discover (which I > > haven't so far). Thanks in advance! > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > - Your configuration seems fulfilling to me. If the 2 instances belong > to the same security group. No inbound or outbound rules apply them. > And since you are from one instance to another, It is likely you > should be able to connect. > And in that case I guess the inbound rules you specified are them to be > accessed from outside EC2. > > - Here is a couple of things that you may want to check from inside the > instance. > -\> Is the elasticsearch binding to port 9300/9200. > -\> if yes. is it on 0.0.0.0 or 10.XX.XX.XX? It shouldnot be on > 127.0.0.0. e.g. you will find a rule in netstat -ntupl lile > X.X.X.X/9300 /java. (where X.X.X.X not belonging to 127.0.0.0 n/w) > -\> normally elasticsearch binds to two ports. Check both of em. > -\> Is ICMP enabled on your system. > -\> Check the iptables rules of the system (assuming linux). If possible > turn it off. > -\> are u able to telnet localhost at 9300 from your instance. > -\> Check the elasticsearch logs for any misconfiguration. > -\> Run TCPDump to see whether TCP handshake is completing or not. > > --saurajeet.d -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:32am UTC](https://discuss.elastic.co/t/ec2-discovery-problems-for-elasticsearch-clustering/12297/4 "2017-07-06T02:32:31Z")