# EC2 Discovery Not Working **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2013, 4:51pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386 "2013-03-29T16:51:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [March 29, 2013, 4:51pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/1 "2013-03-29T16:51:04Z")
So, this one is really puzzling me. I'm sure I've got something misconfigured, but I can't it out. I have two ES clusters on EC2 running 0.19.11 that work perfectly. I am trying to setup a new cluster of two nodes running 0.20.6 on EC2. The two nodes won't talk to each other unless I use unicast and set the host names directly in the config. EC2 discovery finds all of the other possible nodes correctly, but fails to connect to the other nodes on port 9300: [2013-03-29 11:04:16,642][WARN][discovery.zen.ping.unicast] [[es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] failed to send ping to [[#cloud-i-12345678-0][inet[/a.b.c.d:9300]]] org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/a.b.c.d:9300]][discovery/zen/unicast] request\_id [4] timed out after [3750ms] As far as I can tell, the problem is that the nodes are not binding to port 9300. Instead they're binding to 9302: 2013-03-29 11:04:09,250][INFO][transport] [[es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] bound\_address {inet[/0.0.0.0:9302]}, publish\_address {inet[/aa.bb.cc.dd:9302]} If have verified that the security group allows traffic on the port range 9200-9400. If I try to telnet from one of the two nodes to the other on port 9300, telnet hangs. If I try to telnet to port 9302, it works. I also tried disabling EC2 discovery and explicitly listing the nodes using port 9302 (instead of the default 9300) and that worked great: discovery.zen.ping.unicast.hosts: ["[es3.logstash.ec2.example.com:9302](http://es3.logstash.ec2.example.com:9302)","[es4.logstash.ec2.example.com:9302](http://es4.logstash.ec2.example.com:9302)"] If I try explicitly forcing the transport port to 9300 and the http port to 9200, ElasticSearch complains that the ports are already in use and shuts back down. My discovery configuration looks like the following: discovery.type: ec2 discovery.zen.minimum\_master\_nodes: 1 discovery.zen.ping.multicast.enabled: false cloud.node.auto\_attributes: true cloud.aws.region: us-east-1 discovery.ec2.groups: ElasticSearchLogstash Anything obvious that I'm missing? -Sean -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [April 1, 2013, 7:32pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/2 "2013-04-01T19:32:47Z")
Any suggestions? On Friday, March 29, 2013 9:51:04 AM UTC-7, VegHead wrote: > So, this one is really puzzling me. I'm sure I've got something > misconfigured, but I can't it out. > > I have two ES clusters on EC2 running 0.19.11 that work perfectly. I am > trying to setup a new cluster of two nodes running 0.20.6 on EC2. The two > nodes won't talk to each other unless I use unicast and set the host names > directly in the config. EC2 discovery finds all of the other possible nodes > correctly, but fails to connect to the other nodes on port 9300: > > [2013-03-29 11:04:16,642][WARN][discovery.zen.ping.unicast] [ > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] failed to send ping to > [[#cloud-i-12345678-0][inet[/a.b.c.d:9300]]] > org.elasticsearch.transport.ReceiveTimeoutTransportException: > [inet[/a.b.c.d:9300]][discovery/zen/unicast] request\_id [4] timed out > after [3750ms] > > As far as I can tell, the problem is that the nodes are not binding to > port 9300. Instead they're binding to 9302: > > 2013-03-29 11:04:09,250][INFO][transport] [ > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] bound\_address {inet[/0.0.0.0:9302]}, > publish\_address {inet[/aa.bb.cc.dd:9302]} > > If have verified that the security group allows traffic on the port range > 9200-9400. If I try to telnet from one of the two nodes to the other on > port 9300, telnet hangs. If I try to telnet to port 9302, it works. I also > tried disabling EC2 discovery and explicitly listing the nodes using port > 9302 (instead of the default 9300) and that worked great: > > discovery.zen.ping.unicast.hosts: ["[es3.logstash.ec2.example.com:9302](http://es3.logstash.ec2.example.com:9302)"," > [es4.logstash.ec2.example.com:9302](http://es4.logstash.ec2.example.com:9302)"] > > If I try explicitly forcing the transport port to 9300 and the http port > to 9200, Elasticsearch complains that the ports are already in use and > shuts back down. > > My discovery configuration looks like the following: > > discovery.type: ec2 > discovery.zen.minimum\_master\_nodes: 1 > discovery.zen.ping.multicast.enabled: false > cloud.node.auto\_attributes: true > cloud.aws.region: us-east-1 > discovery.ec2.groups: ElasticSearchLogstash > > Anything obvious that I'm missing? > > -Sean -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [April 4, 2013, 12:45pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/3 "2013-04-04T12:45:47Z")
Hi VH, stating the obvious maybe, but what else is listening on tcp/9300? ( netstatn -anp | grep LISTEN | grep 9300 ) ? from the ec2.group you show, it seems you are using Logstash - you know that in default config logstash starts its own ES cluster... which will compete for ports with your separate ES cluster... i also find it annoying but havent found a simple way to turn it off - I solved it in our logstash cluster by setting Logstash to start after ES , and it seems ok with it... let us know how it goes... Beto On Saturday, March 30, 2013 3:51:04 AM UTC+11, VegHead wrote: > So, this one is really puzzling me. I'm sure I've got something > misconfigured, but I can't it out. > > I have two ES clusters on EC2 running 0.19.11 that work perfectly. I am > trying to setup a new cluster of two nodes running 0.20.6 on EC2. The two > nodes won't talk to each other unless I use unicast and set the host names > directly in the config. EC2 discovery finds all of the other possible nodes > correctly, but fails to connect to the other nodes on port 9300: > > [2013-03-29 11:04:16,642][WARN][discovery.zen.ping.unicast] [ > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] failed to send ping to > [[#cloud-i-12345678-0][inet[/a.b.c.d:9300]]] > org.elasticsearch.transport.ReceiveTimeoutTransportException: > [inet[/a.b.c.d:9300]][discovery/zen/unicast] request\_id [4] timed out > after [3750ms] > > As far as I can tell, the problem is that the nodes are not binding to > port 9300. Instead they're binding to 9302: > > 2013-03-29 11:04:09,250][INFO][transport] [ > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] bound\_address {inet[/0.0.0.0:9302]}, > publish\_address {inet[/aa.bb.cc.dd:9302]} > > If have verified that the security group allows traffic on the port range > 9200-9400. If I try to telnet from one of the two nodes to the other on > port 9300, telnet hangs. If I try to telnet to port 9302, it works. I also > tried disabling EC2 discovery and explicitly listing the nodes using port > 9302 (instead of the default 9300) and that worked great: > > discovery.zen.ping.unicast.hosts: ["[es3.logstash.ec2.example.com:9302](http://es3.logstash.ec2.example.com:9302)"," > [es4.logstash.ec2.example.com:9302](http://es4.logstash.ec2.example.com:9302)"] > > If I try explicitly forcing the transport port to 9300 and the http port > to 9200, Elasticsearch complains that the ports are already in use and > shuts back down. > > My discovery configuration looks like the following: > > discovery.type: ec2 > discovery.zen.minimum\_master\_nodes: 1 > discovery.zen.ping.multicast.enabled: false > cloud.node.auto\_attributes: true > cloud.aws.region: us-east-1 > discovery.ec2.groups: ElasticSearchLogstash > > Anything obvious that I'm missing? > > -Sean -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [April 4, 2013, 3:47pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/4 "2013-04-04T15:47:57Z")
Gah! You're right! I'm an idiot! We already have one Logstash cluster consisting of two indexers and four ES servers running an older release of Logstash and ES 0.19.11. I was trying to setup a new Logstash cluster with the latest Logstash and ES 0.20.6. Even though I run the Logstash indexers on separate boxes from Elasticsearch, I still run the Logstash Agent on the ES boxes. Everything is automated with Chef and of course I changed the Chef recipe as well. As a result, the Logstash Agent config on the ES node was misconfigured with 3 very very important lines: elasticsearch { embedded =\> true } Running ES embedded within Logstash means ES doesn't show up in the process list. But looking at the open ports... whoops. $ netstat -anp | grep LISTEN | grep 9300 tcp 0 0 0.0.0.0:9300 0.0.0.0:\* LISTEN 4811/java $ ps ax |grep 4811 4811 ? Ssl 9:42 /usr/bin/java -Xms256M -Xmx256M -jar logstash-monolithic.jar agent --config /etc/logstash Absolutely awesome find. Thank you! -Sean On Thursday, April 4, 2013 5:45:47 AM UTC-7, Norberto Meijome wrote: > Hi VH, > stating the obvious maybe, but what else is listening on tcp/9300? ( > netstatn -anp | grep LISTEN | grep 9300 ) ? > > from the ec2.group you show, it seems you are using Logstash - you know > that in default config logstash starts its own ES cluster... which will > compete for ports with your separate ES cluster... i also find it annoying > but havent found a simple way to turn it off - I solved it in our logstash > cluster by setting Logstash to start after ES , and it seems ok with it... > > let us know how it goes... > Beto > > On Saturday, March 30, 2013 3:51:04 AM UTC+11, VegHead wrote: > > > So, this one is really puzzling me. I'm sure I've got something > > misconfigured, but I can't it out. > > > > I have two ES clusters on EC2 running 0.19.11 that work perfectly. I am > > trying to setup a new cluster of two nodes running 0.20.6 on EC2. The two > > nodes won't talk to each other unless I use unicast and set the host names > > directly in the config. EC2 discovery finds all of the other possible nodes > > correctly, but fails to connect to the other nodes on port 9300: > > > > [2013-03-29 11:04:16,642][WARN][discovery.zen.ping.unicast] [ > > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] failed to send ping to > > [[#cloud-i-12345678-0][inet[/a.b.c.d:9300]]] > > org.elasticsearch.transport.ReceiveTimeoutTransportException: > > [inet[/a.b.c.d:9300]][discovery/zen/unicast] request\_id [4] timed out > > after [3750ms] > > > > As far as I can tell, the problem is that the nodes are not binding to > > port 9300. Instead they're binding to 9302: > > > > 2013-03-29 11:04:09,250][INFO][transport] [ > > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] bound\_address {inet[/0.0.0.0:9302]}, > > publish\_address {inet[/aa.bb.cc.dd:9302]} > > > > If have verified that the security group allows traffic on the port range > > 9200-9400. If I try to telnet from one of the two nodes to the other on > > port 9300, telnet hangs. If I try to telnet to port 9302, it works. I also > > tried disabling EC2 discovery and explicitly listing the nodes using port > > 9302 (instead of the default 9300) and that worked great: > > > > discovery.zen.ping.unicast.hosts: ["[es3.logstash.ec2.example.com:9302](http://es3.logstash.ec2.example.com:9302)"," > > [es4.logstash.ec2.example.com:9302](http://es4.logstash.ec2.example.com:9302)"] > > > > If I try explicitly forcing the transport port to 9300 and the http port > > to 9200, Elasticsearch complains that the ports are already in use and > > shuts back down. > > > > My discovery configuration looks like the following: > > > > discovery.type: ec2 > > discovery.zen.minimum\_master\_nodes: 1 > > discovery.zen.ping.multicast.enabled: false > > cloud.node.auto\_attributes: true > > cloud.aws.region: us-east-1 > > discovery.ec2.groups: ElasticSearchLogstash > > > > Anything obvious that I'm missing? > > > > -Sean -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [April 4, 2013, 11:59pm UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/5 "2013-04-04T23:59:34Z")
Glad you solved it 🙂 B On Fri, Apr 5, 2013 at 2:47 AM, VegHead [organicveggie@gmail.com](mailto:organicveggie@gmail.com) wrote: > Gah! You're right! I'm an idiot! > > We already have one Logstash cluster consisting of two indexers and four > ES servers running an older release of Logstash and ES 0.19.11. I was > trying to setup a new Logstash cluster with the latest Logstash and ES > 0.20.6. Even though I run the Logstash indexers on separate boxes from > Elasticsearch, I still run the Logstash Agent on the ES boxes. Everything > is automated with Chef and of course I changed the Chef recipe as well. As > a result, the Logstash Agent config on the ES node was misconfigured with 3 > very very important lines: > > elasticsearch { > embedded =\> true > } > > Running ES embedded within Logstash means ES doesn't show up in the > process list. But looking at the open ports... whoops. > > $ netstat -anp | grep LISTEN | grep 9300 > tcp 0 0 0.0.0.0:9300 0.0.0.0:\* > LISTEN 4811/java > $ ps ax |grep 4811 > 4811 ? Ssl 9:42 /usr/bin/java -Xms256M -Xmx256M -jar > logstash-monolithic.jar agent --config /etc/logstash > > Absolutely awesome find. Thank you! > > -Sean > > On Thursday, April 4, 2013 5:45:47 AM UTC-7, Norberto Meijome wrote: > > > Hi VH, > > stating the obvious maybe, but what else is listening on tcp/9300? ( > > netstatn -anp | grep LISTEN | grep 9300 ) ? > > > > from the ec2.group you show, it seems you are using Logstash - you know > > that in default config logstash starts its own ES cluster... which will > > compete for ports with your separate ES cluster... i also find it annoying > > but havent found a simple way to turn it off - I solved it in our logstash > > cluster by setting Logstash to start after ES , and it seems ok with it... > > > > let us know how it goes... > > Beto > > > > On Saturday, March 30, 2013 3:51:04 AM UTC+11, VegHead wrote: > > > > > So, this one is really puzzling me. I'm sure I've got something > > > misconfigured, but I can't it out. > > > > > > I have two ES clusters on EC2 running 0.19.11 that work perfectly. I am > > > trying to setup a new cluster of two nodes running 0.20.6 on EC2. The two > > > nodes won't talk to each other unless I use unicast and set the host names > > > directly in the config. EC2 discovery finds all of the other possible nodes > > > correctly, but fails to connect to the other nodes on port 9300: > > > > > > [2013-03-29 11:04:16,642][WARN][discovery.zen.ping.unicast] [ > > > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] failed to send ping to > > > [[#cloud-i-12345678-0][inet[/\*\*a.b.c.d:9300]]] > > > org.elasticsearch.transport. **ReceiveTimeoutTransportExcepti** on: > > > [inet[/a.b.c.d:9300]][\*\*discovery/zen/unicast] request\_id [4] timed > > > out after [3750ms] > > > > > > As far as I can tell, the problem is that the nodes are not binding to > > > port 9300. Instead they're binding to 9302: > > > > > > 2013-03-29 11:04:09,250][INFO][transport] [ > > > [es3.logstash.ec2.example.com](http://es3.logstash.ec2.example.com)] bound\_address {inet[/0.0.0.0:9302]}, > > > publish\_address {inet[/aa.bb.cc.dd:9302]} > > > > > > If have verified that the security group allows traffic on the port > > > range 9200-9400. If I try to telnet from one of the two nodes to the other > > > on port 9300, telnet hangs. If I try to telnet to port 9302, it works. I > > > also tried disabling EC2 discovery and explicitly listing the nodes using > > > port 9302 (instead of the default 9300) and that worked great: > > > > > > discovery.zen.ping.unicast.**hosts: ["es3.logstash.ec2.example.** > > > com:9302 [http://es3.logstash.ec2.example.com:9302](http://es3.logstash.ec2.example.com:9302)","es4.logstash.ec2.\* > > > \*[example.com:9302](http://example.com:9302) [http://es4.logstash.ec2.example.com:9302](http://es4.logstash.ec2.example.com:9302)"] > > > > > > If I try explicitly forcing the transport port to 9300 and the http port > > > to 9200, Elasticsearch complains that the ports are already in use and > > > shuts back down. > > > > > > My discovery configuration looks like the following: > > > > > > discovery.type: ec2 > > > discovery.zen.minimum\_master\_\*\*nodes: 1 > > > discovery.zen.ping.multicast.\*\*enabled: false > > > cloud.node.auto\_attributes: true > > > cloud.aws.region: us-east-1 > > > discovery.ec2.groups: ElasticSearchLogstash > > > > > > Anything obvious that I'm missing? > > > > > > -Sean > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:42am UTC](https://discuss.elastic.co/t/ec2-discovery-not-working/11386/6 "2017-07-06T02:42:41Z")