# Connecting Tribe node to EC2 cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [April 7, 2014, 4:03pm UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848 "2014-04-07T16:03:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 7, 2014, 4:03pm UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/1 "2014-04-07T16:03:48Z")
Hi, I have a ES cluster built in EC2, using the cloud-aws plugin for discovery. The nodes are using their private IP addresses for communication and everything works ok. I want to run one or more tribe nodes remotely, so I point the tribe node to the public IP addresses of some of the nodes however this doesn't work as the network.publish\_host setting on each node in the cluster defaults to pointing at its private address. I realised that using the public EC2 DNS names within EC2 points to the private IP addresses of each node still so I reconfigured my cluster to use discovery.ec2.host\_type=public\_dns and also set network.publish\_host=_ec2:publicDns_ . The cluster still works as before with the traffic still using the private IP addresses. However my tribe node still complains it can't reach the private IP address of the nodes; I was expecting it to get the public ec2-X-X-X-X.....amazonaws.com name and resolve it to then gain the public IP address, which should then hopefully work. On two nodes in the cluster if I fetch [http://localhost:9200/\_nodes/node1,node2/transport?pretty](http://localhost:9200/_nodes/node1,node2/transport?pretty) on both node1 and node2 then I notice that the publish\_address for the local node is reported as "inet[ec2-X-X-X-X....amazonaws.com/10.0.0.1:9300]" but the publish\_address for the non-local node is only reported as being "inet[/10.0.0.2:9300]". Would this mean the the tribe node when connecting remotely still only gets "inet[/10.0.0.x:9300]" for each node address? Am I misunderstanding how this is supposed to work? Is there an alternative way to attach a remote tribe node to this cluster easily? Thanks Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140407160348.GX2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140407160348.GX2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 8, 2014, 7:50am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/2 "2014-04-08T07:50:15Z")
Hey Matt, I'd like to understand better what is happening here. Could you gist your elasticsearch.yml files (the ones for elasticsearch standard nodes and the tribe node one)? Of course, replace your EC2 credentials by dummy values! 🙂 Thanks! -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 7 avril 2014 à 18:03:32, Matt Dainty ([matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com)) a écrit: Hi, I have a ES cluster built in EC2, using the cloud-aws plugin for discovery. The nodes are using their private IP addresses for communication and everything works ok. I want to run one or more tribe nodes remotely, so I point the tribe node to the public IP addresses of some of the nodes however this doesn't work as the network.publish\_host setting on each node in the cluster defaults to pointing at its private address. I realised that using the public EC2 DNS names within EC2 points to the private IP addresses of each node still so I reconfigured my cluster to use discovery.ec2.host\_type=public\_dns and also set network.publish\_host=_ec2:publicDns_ . The cluster still works as before with the traffic still using the private IP addresses. However my tribe node still complains it can't reach the private IP address of the nodes; I was expecting it to get the public ec2-X-X-X-X.....amazonaws.com name and resolve it to then gain the public IP address, which should then hopefully work. On two nodes in the cluster if I fetch [http://localhost:9200/\_nodes/node1,node2/transport?pretty](http://localhost:9200/_nodes/node1,node2/transport?pretty) on both node1 and node2 then I notice that the publish\_address for the local node is reported as "inet[ec2-X-X-X-X....amazonaws.com/10.0.0.1:9300]" but the publish\_address for the non-local node is only reported as being "inet[/10.0.0.2:9300]". Would this mean the the tribe node when connecting remotely still only gets "inet[/10.0.0.x:9300]" for each node address? Am I misunderstanding how this is supposed to work? Is there an alternative way to attach a remote tribe node to this cluster easily? Thanks Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140407160348.GX2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140407160348.GX2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5343aa37.66ef438d.164d%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5343aa37.66ef438d.164d%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 8, 2014, 8:58am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/3 "2014-04-08T08:58:38Z")
- David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-08 03:50:52]: > Hey Matt, > > I'd like to understand better what is happening here. > > Could you gist your elasticsearch.yml files (the ones for elasticsearch standard nodes and the tribe node one)? > Of course, replace your EC2 credentials by dummy values! 🙂 Sure, they're up at [Elasticsearch configuration for EC2 cluster with remote tribe node · GitHub](https://gist.github.com/bodgit/10102642) The master nodes in my cluster have Elastic IP addresses assigned so they're on "well known" IP addresses, and that's what the tribe node configuration uses as the values for the zen unicast hosts. All other cluster nodes have normal public EC2 addresses. I also forgot to mention, I'm running ES 1.1.0 and cloud-aws 2.1.0. Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140408085838.GY2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140408085838.GY2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 11, 2014, 9:02am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/4 "2014-04-11T09:02:17Z")
- Matt Dainty [matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com) [2014-04-08 04:58:46]: > - David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-08 03:50:52]: > > > Hey Matt, > > > > I'd like to understand better what is happening here. > > > > Could you gist your elasticsearch.yml files (the ones for elasticsearch standard nodes and the tribe node one)? > > Of course, replace your EC2 credentials by dummy values! 🙂 > > Sure, they're up at [Elasticsearch configuration for EC2 cluster with remote tribe node · GitHub](https://gist.github.com/bodgit/10102642) > > The master nodes in my cluster have Elastic IP addresses assigned so > they're on "well known" IP addresses, and that's what the tribe node > configuration uses as the values for the zen unicast hosts. All other > cluster nodes have normal public EC2 addresses. > > I also forgot to mention, I'm running ES 1.1.0 and cloud-aws 2.1.0. Did you have any more thought on this? My only workaround would be to bring up a VPN so I can access all the nodes via their private IP addresses, but this will increase the complexity and require me to manage the VPN(s). Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411090217.GA2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411090217.GA2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 11, 2014, 10:51am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/5 "2014-04-11T10:51:39Z")
Sorry Matt I did not get a chance to look at it yet in details. Just to make sure it's not related to the jetty plugin, could you try remove it from the tribe node? Is the tribe node sharing the same security group? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 11 avril 2014 à 11:02:23, Matt Dainty ([matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com)) a écrit: - Matt Dainty [matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com) [2014-04-08 04:58:46]: > - David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-08 03:50:52]: > > > Hey Matt, > > > > I'd like to understand better what is happening here. > > > > Could you gist your elasticsearch.yml files (the ones for elasticsearch standard nodes and the tribe node one)? > > Of course, replace your EC2 credentials by dummy values! 🙂 > > Sure, they're up at [Elasticsearch configuration for EC2 cluster with remote tribe node · GitHub](https://gist.github.com/bodgit/10102642) > > The master nodes in my cluster have Elastic IP addresses assigned so > they're on "well known" IP addresses, and that's what the tribe node > configuration uses as the values for the zen unicast hosts. All other > cluster nodes have normal public EC2 addresses. > > I also forgot to mention, I'm running ES 1.1.0 and cloud-aws 2.1.0. Did you have any more thought on this? My only workaround would be to bring up a VPN so I can access all the nodes via their private IP addresses, but this will increase the complexity and require me to manage the VPN(s). Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411090217.GA2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411090217.GA2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5347c93c.515f007c.ea5a%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5347c93c.515f007c.ea5a%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 11, 2014, 11:56am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/6 "2014-04-11T11:56:08Z")
- David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-11 06:51:56]: > Sorry Matt > > I did not get a chance to look at it yet in details. > Just to make sure it's not related to the jetty plugin, could you try remove it from the tribe node? I just pared back the config on the tribe node to just the minimum, (gist updated), and it still occurs. I tried increasing the logging and I see things like: [2014-04-11 11:59:08,379][DEBUG][transport.netty] [es-tribe-01/sydney] disconnected from [[#zen\_unicast\_1#][es-tribe-01][inet[/54.206.x.x:9300]]] [2014-04-11 11:59:08,381][DEBUG][transport.netty] [es-tribe-01/sydney] disconnected from [[#zen\_unicast\_2#][es-tribe-01][inet[/54.206.x.x:9300]]] [2014-04-11 11:59:08,382][DEBUG][discovery.zen] [es-tribe-01/sydney] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> target [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}], master [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}] --\> target [[es-master-01][whb9bzt\_QV2AyiuhIyBtWw][es-master-01][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2a, data=false, master=true}], master [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}] I would expect the inet[/172.31.x.x:9300]'s to actually be inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300)] which I see for some of the nodes when I query the cluster using the /\_nodes/transport endpoint. Every node with HTTP enabled reports its http\_adddress with the EC2 DNS name included, but only the node I make the request to reports its publish\_address with the DNS included, all other nodes just contain their IP address, e.g.: # curl -s [http://elb:9200/\_nodes/transport?pretty](http://elb:9200/_nodes/transport?pretty) | grep http\_address ``` "http_address" : "inet[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200]", "http_address" : "inet[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200]", ``` # curl -s [http://elb:9200/\_nodes/transport?pretty](http://elb:9200/_nodes/transport?pretty) | grep publish\_address ``` "publish_address" : "inet[/172.31.x.x:9300]" "publish_address" : "inet[/172.31.x.x:9300]" "publish_address" : "inet[/172.31.x.x:9300]" "publish_address" : "inet[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300]" "publish_address" : "inet[/172.31.x.x:9300]" "publish_address" : "inet[/172.31.x.x:9300]" ``` (There's six nodes in the cluster but only two have HTTP enabled) I would expect all publish\_addresses to contain the DNS name. > Is the tribe node sharing the same security group? The tribe node is not actually in EC2 at all, but the security groups are set so it has full access on ports 9300-9399 to each node, which I guess is confirmed by the fact its getting as far as the list of masters with their private IP addresses via the zen unicast hosts. Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411115608.GB2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411115608.GB2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 11, 2014, 12:06pm UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/7 "2014-04-11T12:06:21Z")
And all your nodes have, right? network: publish\_host: _ec2:publicDns_ I think some nodes are only master and some others are only data nodes, right? Any chance you could put in your gist all individual generated elasticsearch.yml file? I don't understand why one successfully uses "publish\_address" : "inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300)]" And the other ones: "publish\_address" : "inet[/172.31.x.x:9300]" With the exact same parameters, it should not happen. Could you try using _ec2:publicIp_ and see how it goes? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 11 avril 2014 à 13:56:11, Matt Dainty ([matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com)) a écrit: - David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-11 06:51:56]: > Sorry Matt > > I did not get a chance to look at it yet in details. > Just to make sure it's not related to the jetty plugin, could you try remove it from the tribe node? I just pared back the config on the tribe node to just the minimum, (gist updated), and it still occurs. I tried increasing the logging and I see things like: [2014-04-11 11:59:08,379][DEBUG][transport.netty] [es-tribe-01/sydney] disconnected from [[#zen\_unicast\_1#][es-tribe-01][inet[/54.206.x.x:9300]]] [2014-04-11 11:59:08,381][DEBUG][transport.netty] [es-tribe-01/sydney] disconnected from [[#zen\_unicast\_2#][es-tribe-01][inet[/54.206.x.x:9300]]] [2014-04-11 11:59:08,382][DEBUG][discovery.zen] [es-tribe-01/sydney] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> target [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}], master [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}] --\> target [[es-master-01][whb9bzt\_QV2AyiuhIyBtWw][es-master-01][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2a, data=false, master=true}], master [[es-master-02][4rcZQu2USJuk6sSGu-by0w][es-master-02][inet[/172.31.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}] I would expect the inet[/172.31.x.x:9300]'s to actually be inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300)] which I see for some of the nodes when I query the cluster using the /\_nodes/transport endpoint. Every node with HTTP enabled reports its http\_adddress with the EC2 DNS name included, but only the node I make the request to reports its publish\_address with the DNS included, all other nodes just contain their IP address, e.g.: # curl -s [http://elb:9200/\_nodes/transport?pretty](http://elb:9200/_nodes/transport?pretty) | grep http\_address "http\_address" : "inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200)]", "http\_address" : "inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9200)]", # curl -s [http://elb:9200/\_nodes/transport?pretty](http://elb:9200/_nodes/transport?pretty) | grep publish\_address "publish\_address" : "inet[/172.31.x.x:9300]" "publish\_address" : "inet[/172.31.x.x:9300]" "publish\_address" : "inet[/172.31.x.x:9300]" "publish\_address" : "inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300)]" "publish\_address" : "inet[/172.31.x.x:9300]" "publish\_address" : "inet[/172.31.x.x:9300]" (There's six nodes in the cluster but only two have HTTP enabled) I would expect all publish\_addresses to contain the DNS name. > Is the tribe node sharing the same security group? The tribe node is not actually in EC2 at all, but the security groups are set so it has full access on ports 9300-9399 to each node, which I guess is confirmed by the fact its getting as far as the list of masters with their private IP addresses via the zen unicast hosts. Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411115608.GB2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411115608.GB2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5347dabd.41a7c4c9.ea5a%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5347dabd.41a7c4c9.ea5a%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 11, 2014, 1:01pm UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/8 "2014-04-11T13:01:18Z")
- David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-11 08:06:28]: > And all your nodes have, right? > > network: > publish\_host: _ec2:publicDns_ Yes. I'm using Puppet so every node gets the same configuration. > I think some nodes are only master and some others are only data nodes, right? Yes. The main difference is the data nodes have EBS volumes attached and mounted for storing the indices, and they're bigger instances. > Any chance you could put in your gist all individual generated elasticsearch.yml file? The only difference so far between the configurations is the data nodes have indices.memory.index\_buffer\_size set and obviously the values for node.data and node.master are flipped. All other settings are identical. > I don't understand why one successfully uses > > "publish\_address" : "inet[[ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.ap-southeast-2.compute.amazonaws.com/172.31.x.x:9300)]" > > And the other ones: > > "publish\_address" : "inet[/172.31.x.x:9300]" That's what I'm confused about too. I understand "inet[[ec2-54-206-x-x.amazonaws.com/172.31.x.x:9300](http://ec2-54-206-x-x.amazonaws.com/172.31.x.x:9300)]" is just a textual representation of the transport endpoint, but does that mean that another node will use the result of the DNS lookup of the hostname in preference to the IP address that's also reported? > With the exact same parameters, it should not happen. > Could you try using _ec2:publicIp_ and see how it goes? I tried to use the public IP instead, but I couldn't get the cluster to successfully associate, I think that's the security groups interfering so I'll have another attempt. However, as I understand it, using the public IP means traffic between cluster nodes performs worse and costs more, (Amazon classes it as intra-region traffic, as if I was transfering between availability zones). That's when I discovered that the public DNS name internally still resolves to the private IP address so I assumed using the public DNS names everywhere keeps the internal traffic using the private IP addresses and external traffic (from the tribe nodes) still also work. I've also just tried using the cloud-aws plugin on the tribe node and adapted the configuration like so: tribe: blocks: metadata: true sydney: cloud: aws: access\_key: abc123 region: ap-southeast-2 secret\_key: secret cluster: name: logstash discovery: ec2: groups: elasticsearch host\_type: public\_dns type: ec2 zen: minimum\_master\_nodes: 2 ping: multicast: enabled: false I can see it makes a query to find all the nodes in the cluster, with their public DNS names and IP addresses but then when it connects to them it starts trying to use the private IP addresses again and I'm in the same situation as before. Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411130118.GC2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411130118.GC2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 11, 2014, 3:01pm UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/9 "2014-04-11T15:01:29Z")
- David Pilato [david@pilato.fr](mailto:david@pilato.fr) [2014-04-11 08:06:28]: > Could you try using _ec2:publicIp_ and see how it goes? Ok, I can get the cluster to associate when using _ec2:publicIp_ however EC2 security groups don't work very well; I have to add the public IP address of every node in the cluster to the inbound rules of the security group that they're all in otherwise they can't connect to each other. Whereas before I just had a rule that said any traffic is allowed if it's from the same security group. I'd much rather get it working keeping the cluster traffic using the private IP addresses. With the cluster up I tried the tribe node again, now it just logs this every 20 seconds: [2014-04-11 15:38:25,497][INFO][discovery.zen] [es-tribe-01/sydney] failed to send join request to master [[es-master-02][Vii7h7O1RNy6rpGPRXVDLQ][es-master-02][inet[/54.206.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}], reason [org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.] What timeout is that referring to? I can telnet to the TCP ports on the master node and running tcpdump on each node I can see traffic going back and forth so it is connected. The master nodes are not logging anything while this is happening, I just get: [2014-04-11 15:59:02,219][DEBUG][org.apache.http.impl.conn.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS once a minute. I've tried setting discovery.zen.ping\_timeout to 10s but it doesn't help. All the other timeouts I can find seem to default to 30s which is more than enough. Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140411150129.GD2245%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140411150129.GD2245%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Dainty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_dainty/32/1075_2.png) [@Matt\_Dainty](https://discuss.elastic.co/u/Matt_Dainty)\ **Post date:** [April 14, 2014, 9:24am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/10 "2014-04-14T09:24:23Z")
- Matt Dainty [matt@bodgit-n-scarper.com](mailto:matt@bodgit-n-scarper.com) [2014-04-11 11:01:33]: > With the cluster up I tried the tribe node again, now it just logs this > every 20 seconds: > > [2014-04-11 15:38:25,497][INFO][discovery.zen] [es-tribe-01/sydney] failed to send join request to master [[es-master-02][Vii7h7O1RNy6rpGPRXVDLQ][es-master-02][inet[/54.206.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}], reason [org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.] I ran out of time to poke at this over the weekend so I left it running, however I've resumed looking at it and now it's also logging this: [2014-04-14 10:21:12,706][INFO][discovery.zen] [es-tribe-01/sydney] failed to send join request to master [[es-master-02][Vii7h7O1RNy6rpGPRXVDLQ][es-master-02][inet[/54.206.x.x:9300]]{aws\_availability\_zone=ap-southeast-2b, data=false, master=true}], reason [org.elasticsearch.transport.RemoteTransportException: [es-master-02][inet[/172.31.x.x:9300]][discovery/zen/join]; java.lang.OutOfMemoryError: unable to create new native thread] So leaving the tribe node attempting to join the cluster has caused it to exhaust its heap somehow. Anything I can grab from this to see what's happened? Matt -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140414092423.GA15419%40simulant.bodgit-n-scarper.com](https://groups.google.com/d/msgid/elasticsearch/20140414092423.GA15419%40simulant.bodgit-n-scarper.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:36am UTC](https://discuss.elastic.co/t/connecting-tribe-node-to-ec2-cluster/16848/11 "2017-07-06T01:36:01Z")