# Elasticsearch EC2 setup across multiple regions **URL:** \ **Category:** Elasticsearch\ **Created:** [September 27, 2012, 2:52pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170 "2012-09-27T14:52:08Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [September 27, 2012, 2:52pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/1 "2012-09-27T14:52:08Z")
Hi all, I'm trying to setup an elasticsearch cluster with 2 x ec2 small instances. One is based in the EU and the other in the US east region. I setup the 2 instances seperately with the same cluster name and verified they worked ok (just a request to :9200. I've opened up ports 22(ssh), 9200(http) and the range (9300 - 9400) on my security groups. I can't seem to get the 2 machines to communicate properly, both find each other and try to communicate but seem to elect no master.. - i've changed the logging.yml to add more debug info (based on hunting through existing group posts). The relevant settings on both machines is: EU machine: cluster.name: elasticsearch-demo-HS cloud.aws.access\_key: cloud.aws.secret\_key: discovery.type: ec2 discovery.zen.ping.timeout: 5m ### Note, no cloud.aws.region.... US machine: cluster.name: elasticsearch-demo-HS cloud.aws.access\_key: cloud.aws.secret\_key: cloud.aws.region: eu-west ####suggested in other forums to allow us machine to find eu machine ( [https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ](https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ) ) discovery.type: ec2 discovery.zen.ping.timeout: 5m When i start up the EU machine, it does setup and gets information back from amazon about the instances via the cloud aws plugin [2012-09-27 14:25:13,391][TRACE][discovery.zen.ping.unicast] [Legacy] [1] connecting (light) to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] It then times out on connection (as the other node in the US has not started) [2012-09-27 14:25:43,451][TRACE][discovery.zen.ping.unicast] [Legacy] [1] failed to connect to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] org.elasticsearch.transport.ConnectTransportException: [][inet[/10.96.67.186:9300]] connect\_timeout[30s] at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:585) I then start the US node: and get: [2012-09-27 14:30:42,262][TRACE][discovery.zen.ping.unicast] [Seth] [1] connecting (light) to [#cloud-i-cade1081-0][inet[/hidden:9300]] On the EU machine, i then get a: [2012-09-27 14:47:18,913][DEBUG][transport.netty] [Adaptoid] connected to node [[#cloud-i-b7eb27ca-0][inet[hidden:9300]]] [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] [1] connected to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] [1] sending to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] [2012-09-27 14:47:19,042][TRACE][discovery.zen.ping.unicast] [Adaptoid] [1] received response from [#cloud-i-b7eb27ca-0][inet[hidden:9300]]: [ping\_response{target [[Adaptoid][O1XIpLrPTE-yS1kh4OEfTw][inet[/10.240.50.61:9300]]], master [null], cluster\_name[elasticsearch-demo-HS]}, ping\_response{target [[Eliminator][RodL0FfpStSsl0shyZa8Vw][inet[/10.96.67.186:9300]]], master [null], cluster\_name[elasticsearch-demo-HS]}] When i query the \_cluster\_stats, i get: { - error: "ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];[SERVICE\_UNAVAILABLE/2/no master];]", - status: 503 } An initial problem (not looking up across internal IP addresses) was solved by changing the discovery ec2 type from private\_ip to public\_ip (communication across regions). Now it's just stumped by the fact that i can't get the 2 to talk to each other.. Any help greatly appreciated 😉 Derry -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [September 28, 2012, 7:27am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/2 "2012-09-28T07:27:19Z")
OK, so the problem seems to be that i can find the other machine but i can't connect to it as it is resolving on in the internal ip (e.g. 10.\*). Is their a configuration setting that will explicity force it to connect on external ip addresses or do i solve that by forcing external lookup in my hosts file? Just wondering if anyone else had this problem when resolving over multiple regions? Thank On Thursday, 27 September 2012 15:52:08 UTC+1, Derry O' Sullivan wrote: > Hi all, > > I'm trying to setup an elasticsearch cluster with 2 x ec2 small instances. > One is based in the EU and the other in the US east region. > > I setup the 2 instances seperately with the same cluster name and verified > they worked ok (just a request to :9200. > > I've opened up ports 22(ssh), 9200(http) and the range (9300 - 9400) on my > security groups. > > I can't seem to get the 2 machines to communicate properly, both find each > other and try to communicate but seem to elect no master.. - i've changed > the logging.yml to add more debug info (based on hunting through existing > group posts). > > The relevant settings on both machines is: > EU machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > ### Note, no cloud.aws.region.... > > US machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > cloud.aws.region: eu-west ####suggested in other forums to allow us > machine to find eu machine ( > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ) > ) > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > When i start up the EU machine, it does setup and gets information back > from amazon about the instances via the cloud aws plugin > [2012-09-27 14:25:13,391][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > connecting (light) to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > > It then times out on connection (as the other node in the US has not > started) > [2012-09-27 14:25:43,451][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > failed to connect to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > org.elasticsearch.transport.ConnectTransportException: [inet[/ > 10.96.67.186:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:585) > > I then start the US node: > and get: > [2012-09-27 14:30:42,262][TRACE][discovery.zen.ping.unicast] [Seth] [1] > connecting (light) to [#cloud-i-cade1081-0][inet[/hidden:9300]] > > On the EU machine, i then get a: > [2012-09-27 14:47:18,913][DEBUG][transport.netty] [Adaptoid] > connected to node [[#cloud-i-b7eb27ca-0][inet[hidden:9300]]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] connected to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] sending to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:19,042][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] received response from [#cloud-i-b7eb27ca-0][inet[hidden:9300]]: > [ping\_response{target > [[Adaptoid][O1XIpLrPTE-yS1kh4OEfTw][inet[/10.240.50.61:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}, ping\_response{target > [[Eliminator][RodL0FfpStSsl0shyZa8Vw][inet[/10.96.67.186:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}] > > When i query the \_cluster\_stats, i get: > { > > - error: "ClusterBlockException[blocked by: > [SERVICE\_UNAVAILABLE/1/state not recovered / > initialized];[SERVICE\_UNAVAILABLE/2/no master];]", > - status: 503 > > } > > An initial problem (not looking up across internal IP addresses) was > solved by changing the discovery ec2 type from private\_ip to public\_ip > (communication across regions). Now it's just stumped by the fact that i > can't get the 2 to talk to each other.. > > Any help greatly appreciated 😉 > > Derry -- ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [September 28, 2012, 2:30pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/3 "2012-09-28T14:30:21Z")
Derry O' Sullivan wrote: > OK, so the problem seems to be that i can find the other machine > but i can't connect to it as it is resolving on in the internal ip > (e.g. 10.\*). Is their a configuration setting that will explicity > force it to connect on external ip addresses or do i solve that by > forcing external lookup in my hosts file? You mentioned this in your other message: > An initial problem (not looking up across internal IP addresses) > was solved by changing the discovery ec2 type from private\_ip to > public\_ip (communication across regions). Now it's just stumped by > the fact that i can't get the 2 to talk to each other.. Did this setting actually not work? Does it look like this? discovery.ec2.host\_type: public\_ip Also, you've upped the zen discovery ping timeout: > discovery.zen.ping.timeout: 5m That's not going to have any effect with discovery.type set to ec2. Increase the ec2 discovery timeout instead: discovery.ec2.ping\_timeout: 5m Also note that you may have latency-related issues running a cluster across regions, but I'm really interested to hear how it works for you. -Drew -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [September 28, 2012, 3:05pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/4 "2012-09-28T15:05:17Z")
Hi Drew, Thanks for the response. On 28 September 2012 15:30, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > Derry O' Sullivan wrote: > > > OK, so the problem seems to be that i can find the other machine > > but i can't connect to it as it is resolving on in the internal ip > > (e.g. 10.\*). Is their a configuration setting that will explicity > > force it to connect on external ip addresses or do i solve that by > > forcing external lookup in my hosts file? > > You mentioned this in your other message: > > > An initial problem (not looking up across internal IP addresses) > > was solved by changing the discovery ec2 type from private\_ip to > > public\_ip (communication across regions). Now it's just stumped by > > the fact that i can't get the 2 to talk to each other.. > > Did this setting actually not work? Does it look like this? > > discovery.ec2.host\_type: public\_ip On further inspection, the host\_type did not seem to make a difference. The machines are able to find each other without a problem (whether it is public/private IP/DNS), the issue occurs when they try and connect on port 9300/whatever. The communication seems to be based on internal IP addresses (e.g. 10.X.X.X) meaning that clusters set up easily within the same sub domain (e.g. region/availability zone) but it does not work externally as 10.X._._ cannot connect to 10.Y._._ unless you manage the domains yourself using VPC/routing. My real question on this was whether someone had used the plugin to do cross-region clustering before. If it's a setting i have messed up, it'd be nice to figure it out. > Also, you've upped the zen discovery ping timeout: > > > discovery.zen.ping.timeout: 5m > > That's not going to have any effect with discovery.type set to ec2. > Increase the ec2 discovery timeout instead: > > discovery.ec2.ping\_timeout: 5m Thanks for that - i think i copied in that setting from when i was testing out some other means of connection > Also note that you may have latency-related issues running a cluster > across regions, but I'm really interested to hear how it works for > you. Completely understand that i will have a dependency on the AWS inter region connections and external factors but i would just like to test it first. I'm going to try using standard zen unicast lookup across explicit hosts and see if that works next. > -Drew > > -- -- ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [September 28, 2012, 5:27pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/5 "2012-09-28T17:27:23Z")
Derry O' Sullivan wrote: > On further inspection, the host\_type did not seem to make a > difference. The machines are able to find each other without a > problem (whether it is public/private IP/DNS), the issue occurs > when they try and connect on port 9300/whatever. The communication > seems to be based on internal IP addresses (e.g. 10.X.X.X) meaning > that clusters set up easily within the same sub domain > (e.g. region/availability zone) but it does not work externally as > 10.X._._ cannot connect to 10.Y._._ unless you manage the domains > yourself using VPC/routing. > > My real question on this was whether someone had used the plugin to > do cross-region clustering before. If it's a setting i have messed > up, it'd be nice to figure it out. The plugin defaults to private\_ip so it could just be a config issue that it's not getting flipped to public\_ip. Can you post your full config somewhere for a sanity check? -Drew -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 1, 2012, 10:35am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/6 "2012-10-01T10:35:52Z")
Hi drew, Thanks for the response. The config of my first server (eu): cluster.name: esdemohs cloud.aws.access\_key: cloud.aws.secret\_key: cloud.aws.region: us-east-1 # not sure if i have to tell discovery.type: ec2 discovery.ec2.host\_type: public\_dns discovery.ec2.ping.timeout: 5m discovery.ec2.tag.cluster: esdemohs config of my 2nd server (US): cluster.name: esdemohs cloud.aws.access\_key: cloud.aws.secret\_key: cloud.aws.region: eu-west-1 discovery.type: ec2 discovery.ec2.ping.timeout: 5m discovery.ec2.host\_type: public\_dns discovery.ec2.tag.cluster: esdemohs I start up both servers, i get a status 200 for the EU server and a status 503 for the US server (web searching shows that that is because it is not able to join the cluster. Looking at the trace output on the servers, i see (EU server): [2012-10-01 10:27:41,244][TRACE][discovery.ec2] [Demiurge] building dynamic unicast discovery nodes... [2012-10-01 10:27:41,249][TRACE][discovery.ec2] [Demiurge] adding i-b7eb27ca, address [ec2-107-22-26-95.compute-1.amazonaws.com](http://ec2-107-22-26-95.compute-1.amazonaws.com), transport\_address inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)] [2012-10-01 10:27:41,250][DEBUG][discovery.ec2] [Demiurge] using dynamic discovery nodes [[#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)]]] [2012-10-01 10:27:41,250][TRACE][discovery.zen.ping.unicast] [Demiurge] [1] sending to [#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)]] [2012-10-01 10:27:41,343][TRACE][discovery.zen.ping.unicast] [Demiurge] [1] received response from [#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)]]: [ping\_response{target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}] [2012-10-01 10:27:41,348][TRACE][discovery.ec2] [Demiurge] full ping responses: --\> target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null] [2012-10-01 10:27:41,348][DEBUG][discovery.ec2] [Demiurge] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null] [2012-10-01 10:27:41,350][TRACE][discovery.zen.ping.unicast] [Demiurge] [1] disconnecting from [#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)]] [2012-10-01 10:27:41,354][DEBUG][transport.netty] [Demiurge] disconnected from [[#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)]]] [2012-10-01 10:27:41,355][DEBUG][cluster.service] [Demiurge] processing [zen-disco-join (elected\_as\_master)]: execute [2012-10-01 10:27:41,356][TRACE][cluster.service] [Demiurge] cluster state updated: version [1], source [zen-disco-join (elected\_as\_master)] nodes: [Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]], local, master routing\_table: routing\_nodes: ---- unassigned On the US server, i get: [2012-10-01 10:29:02,398][TRACE][discovery.ec2] [Magnum I] building dynamic unicast discovery nodes... [2012-10-01 10:29:02,400][TRACE][discovery.ec2] [Magnum I] adding i-cade1081, address [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com), transport\_address inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)] [2012-10-01 10:29:02,400][DEBUG][discovery.ec2] [Magnum I] using dynamic discovery nodes [[#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]]] [2012-10-01 10:29:02,400][TRACE][discovery.zen.ping.unicast] [Magnum I] [13] sending to [#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]] [2012-10-01 10:29:02,498][TRACE][discovery.zen.ping.unicast] [Magnum I] [13] received response from [#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]]: [ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], cluster\_name[esdemohs]}] [2012-10-01 10:29:02,498][TRACE][discovery.ec2] [Magnum I] full ping responses: --\> target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] [2012-10-01 10:29:02,499][DEBUG][discovery.ec2] [Magnum I] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] [2012-10-01 10:29:02,501][TRACE][discovery.zen.ping.unicast] [Magnum I] [13] disconnecting from [#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]] [2012-10-01 10:29:02,501][DEBUG][transport.netty] [Magnum I] disconnected from [[#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]]] Going to: \_cluster/health?pretty=true on the EU server gives the server itself: { - cluster\_name: "esdemohs", - status: "green", - timed\_out: false, - number\_of\_nodes: 1, - number\_of\_data\_nodes: 1, - active\_primary\_shards: 0, - active\_shards: 0, - relocating\_shards: 0, - initializing\_shards: 0, - unassigned\_shards: 0 } on the us server, you get: { - error: "MasterNotDiscoveredException[waited for [30s]]", - status: 503 } It seems that the config or cloud.aws.region or tags does not affect the outcome, it just seems like they connect and then disconnect again. Thanks, Derry On 28 September 2012 18:27, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > Derry O' Sullivan wrote: > > > On further inspection, the host\_type did not seem to make a > > difference. The machines are able to find each other without a > > problem (whether it is public/private IP/DNS), the issue occurs > > when they try and connect on port 9300/whatever. The communication > > seems to be based on internal IP addresses (e.g. 10.X.X.X) meaning > > that clusters set up easily within the same sub domain > > (e.g. region/availability zone) but it does not work externally as > > 10.X._._ cannot connect to 10.Y._._ unless you manage the domains > > yourself using VPC/routing. > > > > My real question on this was whether someone had used the plugin to > > do cross-region clustering before. If it's a setting i have messed > > up, it'd be nice to figure it out. > > The plugin defaults to private\_ip so it could just be a config issue > that it's not getting flipped to public\_ip. Can you post your full > config somewhere for a sanity check? > > -Drew > > -- -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 1, 2012, 11:01am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/7 "2012-10-01T11:01:11Z")
I've also changed the host\_type to be public\_ip vs public\_dns and no difference. I'm able to telnet from one server to the other on port 9200 & 9300 so i think the issue is master election? Derry On Thursday, 27 September 2012 15:52:08 UTC+1, Derry O' Sullivan wrote: > Hi all, > > I'm trying to setup an elasticsearch cluster with 2 x ec2 small instances. > One is based in the EU and the other in the US east region. > > I setup the 2 instances seperately with the same cluster name and verified > they worked ok (just a request to :9200. > > I've opened up ports 22(ssh), 9200(http) and the range (9300 - 9400) on my > security groups. > > I can't seem to get the 2 machines to communicate properly, both find each > other and try to communicate but seem to elect no master.. - i've changed > the logging.yml to add more debug info (based on hunting through existing > group posts). > > The relevant settings on both machines is: > EU machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > ### Note, no cloud.aws.region.... > > US machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > cloud.aws.region: eu-west ####suggested in other forums to allow us > machine to find eu machine ( > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ) > ) > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > When i start up the EU machine, it does setup and gets information back > from amazon about the instances via the cloud aws plugin > [2012-09-27 14:25:13,391][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > connecting (light) to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > > It then times out on connection (as the other node in the US has not > started) > [2012-09-27 14:25:43,451][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > failed to connect to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > org.elasticsearch.transport.ConnectTransportException: [inet[/ > 10.96.67.186:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:585) > > I then start the US node: > and get: > [2012-09-27 14:30:42,262][TRACE][discovery.zen.ping.unicast] [Seth] [1] > connecting (light) to [#cloud-i-cade1081-0][inet[/hidden:9300]] > > On the EU machine, i then get a: > [2012-09-27 14:47:18,913][DEBUG][transport.netty] [Adaptoid] > connected to node [[#cloud-i-b7eb27ca-0][inet[hidden:9300]]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] connected to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] sending to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:19,042][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] received response from [#cloud-i-b7eb27ca-0][inet[hidden:9300]]: > [ping\_response{target > [[Adaptoid][O1XIpLrPTE-yS1kh4OEfTw][inet[/10.240.50.61:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}, ping\_response{target > [[Eliminator][RodL0FfpStSsl0shyZa8Vw][inet[/10.96.67.186:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}] > > When i query the \_cluster\_stats, i get: > { > > - error: "ClusterBlockException[blocked by: > [SERVICE\_UNAVAILABLE/1/state not recovered / > initialized];[SERVICE\_UNAVAILABLE/2/no master];]", > - status: 503 > > } > > An initial problem (not looking up across internal IP addresses) was > solved by changing the discovery ec2 type from private\_ip to public\_ip > (communication across regions). Now it's just stumped by the fact that i > can't get the 2 to talk to each other.. > > Any help greatly appreciated 😉 > > Derry -- ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [October 1, 2012, 5:54pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/8 "2012-10-01T17:54:14Z")
Derry O' Sullivan wrote: > The config of my first server (eu): > cluster.name: esdemohs > cloud.aws.access\_key: > cloud.aws.secret\_key: > cloud.aws.region: us-east-1 # not sure if i have to tell > discovery.type: ec2 > discovery.ec2.host\_type: public\_dns > discovery.ec2.ping.timeout: 5m > discovery.ec2.tag.cluster: esdemohs > > config of my 2nd server (US): > cluster.name: esdemohs > cloud.aws.access\_key: > cloud.aws.secret\_key: > cloud.aws.region: eu-west-1 > discovery.type: ec2 > discovery.ec2.ping.timeout: 5m > discovery.ec2.host\_type: public\_dns > discovery.ec2.tag.cluster: esdemohs > > I start up both servers, i get a status 200 for the EU server and a status > 503 for the US server (web searching shows that that is because it is not > able to join the cluster. [...] > It seems that the config or cloud.aws.region or tags does not affect the > outcome, it just seems like they connect and then disconnect again. No obvious issues with your config. Although it's interesting that the us-east node didn't elect itself master also (maybe you have master disabled on that one). Feels more like a security group issue. Let's try turning on TRACE logging for the us-east node. In logging.yml uncomment the line that looks like #discovery: TRACE and restart the node. That will give us some output about what it's trying to access. Look for lines like: using host\_type... filtering out instance... -Drew -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 1, 2012, 6:29pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/9 "2012-10-01T18:29:36Z")
HI Drew, Thanks again for the response. This is my entire config - everything else is the default (commented out). I didn't include the filtered out instances lines (we have a lot of other instances) but the output clearly shows a) The nodes making requests to EC2 to find the other nodes b) The nodes finding the other server in the other zone(e.g. EU finding us and US finding EU e.g: [2012-10-01 10:27:41,249][TRACE][discovery.ec2] [Demiurge] adding i-b7eb27ca, address [ec2-107-22-26-95.compute-1.amazonaws.com](http://ec2-107-22-26-95.compute-1.amazonaws.com), transport\_address inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)] [2012-10-01 10:27:41,250][DEBUG][discovery.ec2] [Demiurge] using dynamic discovery nodes [[#cloud-i-b7eb27ca-0][inet[ [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)][http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300]](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300%5D) ]] [2012-10-01 10:29:02,400][TRACE][discovery.ec2] [Magnum I] adding i-cade1081, address [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com), transport\_address inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)] [2012-10-01 10:29:02,400][DEBUG][discovery.ec2] [Magnum I] using dynamic discovery nodes [[#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D) ]] c) both nodes specifically connecting to their tag/cluster matching node in the other zone, but instantly disconnect: [2012-10-01 10:29:02,498][TRACE][discovery.zen.ping.unicast] [Magnum I] [13] received response from [#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D)]: [ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Magnum I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], cluster\_name[esdemohs]}] [2012-10-01 10:29:02,498][TRACE][discovery.ec2] [Magnum I] full ping responses: --\> target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] [2012-10-01 10:29:02,499][DEBUG][discovery.ec2] [Magnum I] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> target [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] [2012-10-01 10:29:02,501][TRACE][discovery.zen.ping.unicast] [Magnum I] [13] disconnecting from [#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]] [2012-10-01 10:29:02,501][DEBUG][transport.netty] [Magnum I] disconnected from [[#cloud-i-cade1081-0][inet[ [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D) ]] It is saying filter\_client true - would that mean that it thinks the other instance is a client? I don't have access to machine now but i'll try and get the full logs up on pastebin or something tomorrow. Thanks, D On 1 October 2012 18:54, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > Derry O' Sullivan wrote: > > > The config of my first server (eu): > > cluster.name: esdemohs > > cloud.aws.access\_key: > > cloud.aws.secret\_key: > > cloud.aws.region: us-east-1 # not sure if i have to tell > > discovery.type: ec2 > > discovery.ec2.host\_type: public\_dns > > discovery.ec2.ping.timeout: 5m > > discovery.ec2.tag.cluster: esdemohs > > > > config of my 2nd server (US): > > cluster.name: esdemohs > > cloud.aws.access\_key: > > cloud.aws.secret\_key: > > cloud.aws.region: eu-west-1 > > discovery.type: ec2 > > discovery.ec2.ping.timeout: 5m > > discovery.ec2.host\_type: public\_dns > > discovery.ec2.tag.cluster: esdemohs > > > > I start up both servers, i get a status 200 for the EU server and a > > status > > 503 for the US server (web searching shows that that is because it is not > > able to join the cluster. > > [...] > > > It seems that the config or cloud.aws.region or tags does not affect the > > outcome, it just seems like they connect and then disconnect again. > > No obvious issues with your config. Although it's interesting that > the us-east node didn't elect itself master also (maybe you have > master disabled on that one). > > Feels more like a security group issue. Let's try turning on TRACE > logging for the us-east node. In logging.yml uncomment the line that > looks like > > #discovery: TRACE > > and restart the node. That will give us some output about what it's > trying to access. Look for lines like: > > using host\_type... > filtering out instance... > > -Drew > > -- -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 2, 2012, 11:52am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/10 "2012-10-02T11:52:22Z")
Hi Drew, I tested again today. I stopped both nodes and started the first machine (EU) which just had a config of: cluster.name: esdemohs Nothing else, just the cluster itself. My logic was that i wanted to start up this node, put some data in it, verify it was all ok and then get another machine (US node) to join: I went to the root url and got a 200, also /\_cluster/nodes shows that it was ok (one machine in its own cluster): { - ok: true, - cluster\_name: "esdemohs", - nodes: { - NEv9k\_O4Q0SoUUKWXn0CXg: { - name: "Helio", - transport\_address: "inet[/10.239.74.227:9300]", - hostname: "ip-10-239-74-227", - http\_address: "inet[/10.239.74.227:9200]" } } } I then put in data, verified that it was all searchable and all was ok (e.g. just testing that the single instance was ok). I then went to the US machine and configured it like you said: cluster.name: esdemohs cloud.aws.access\_key: cloud.aws.secret\_key: cloud.aws.region: eu-west discovery.type: ec2 discovery.ec2.host\_type: public\_ip discovery.ec2.tag.cluster: esdemohs discovery.ec2.ping\_timeout: 5m discovery.zen.ping\_timeout: 5m discovery.ec2.groups: sg-08895c7f # also tried putting this in an array and using the US machines security group as well In the output, i get the cryptic line: [2012-10-02 11:18:48,306][TRACE][discovery.ec2] [Hildegarde] filtering out instance i-cade1081 based on groups [{GroupName: HS-ES-AWS, GroupId: _sg-08895c7f_, }], not part of [_sg-08895c7f_] Seems to be telling me that i can't join the security group that i'm explicitly telling it to be a part of (i expected the other way around!) 😉 If i remove the security groups, it finds the correct node (EU machine), recieves a response and then disconnects: [2012-10-02 11:11:06,554][TRACE][discovery.ec2] [Gauntlet] building dynamic unicast discovery nodes... [2012-10-02 11:11:06,555][TRACE][discovery.ec2] [Gauntlet] filtering out instance i-9f8fcfd7 based tags {cluster=esdemohs}, not part of [{Key: Name, Value: HS-Dublin, }] [2012-10-02 11:11:06,555][TRACE][discovery.ec2] [Gauntlet] filtering out instance i-fbdf8bb3 based tags {cluster=esdemohs}, not part of [{Key: Name, Value: Dublin-2Ports, }] [2012-10-02 11:11:06,671][TRACE][discovery.ec2] [Gauntlet] adding i-cade1081, address [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com), transport\_address inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)] [2012-10-02 11:11:06,674][DEBUG][discovery.ec2] [Gauntlet] using dynamic discovery nodes [[#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]]] [2012-10-02 11:11:06,678][TRACE][discovery.zen.ping.unicast] [Gauntlet] [1] connecting (light) to [#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]] [2012-10-02 11:11:06,868][DEBUG][transport.netty] [Gauntlet] connected to node [[#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]]] [2012-10-02 11:11:06,869][TRACE][discovery.zen.ping.unicast] [Gauntlet] [1] connected to [#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]] [2012-10-02 11:11:06,869][TRACE][discovery.zen.ping.unicast] [Gauntlet] [1] sending to [#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]] [2012-10-02 11:11:06,987][TRACE][discovery.zen.ping.unicast] [Gauntlet] [1] received response from [#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]]: [ping\_response{target [[Gauntlet][iuj9R7MOTrK\_Rsq\_Vc7V7w][inet[/10.214.27.146:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Helio][NEv9k\_O4Q0SoUUKWXn0CXg][inet[/10.239.74.227:9300]]], master [[Helio][NEv9k\_O4Q0SoUUKWXn0CXg][inet[/10.239.74.227:9300]]], cluster\_name[esdemohs]}] [2012-10-02 11:11:34,559][WARN][discovery] [Gauntlet] waited for 30s and no initial state was set by the discovery [2012-10-02 11:11:34,560][INFO][discovery] [Gauntlet] esdemohs/iuj9R7MOTrK\_Rsq\_Vc7V7w [2012-10-02 11:11:34,560][DEBUG][gateway] [Gauntlet] can't wait on start for (possibly) reading state from gateway, will do it asynchronously [2012-10-02 11:11:34,569][INFO][http] [Gauntlet] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ 10.214.27.146:9200]} [2012-10-02 11:11:34,570][INFO][node] [Gauntlet] {0.19.9}[6286]: started Really unsure how to bring this any further. It's almost like the nodes are being found and then decide not to do any connection. When this line occurs on the US machine (ran again, hence the different server name): [2012-10-02 11:22:28,947][TRACE][discovery.zen.ping.unicast] [Firebird] [1] received response from [#cloud-i-cade1081-0][inet[ [ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300](http://ec2-46-137-44-113.eu-west-1.compute.amazonaws.com/46.137.44.113:9300)]]: [ping\_response{target [[Gauntlet][iuj9R7MOTrK\_Rsq\_Vc7V7w][inet[/10.214.27.146:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Gauntlet][iuj9R7MOTrK\_Rsq\_Vc7V7w][inet[/10.214.27.146:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Gauntlet][iuj9R7MOTrK\_Rsq\_Vc7V7w][inet[/10.214.27.146:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Firebird][7TNInTRSSo21NnsfBdEzFg][inet[/10.214.27.146:9300]]], master [null], cluster\_name[esdemohs]}, ping\_response{target [[Helio][NEv9k\_O4Q0SoUUKWXn0CXg][inet[/10.239.74.227:9300]]], master [[Helio][NEv9k\_O4Q0SoUUKWXn0CXg][inet[/10.239.74.227:9300]]], cluster\_name[esdemohs]}] The EU machine shows: [2012-10-02 11:22:29,126][TRACE][transport.netty] [Helio] channel opened: [id: 0x502a3135, /50.16.164.109:\*41404 \*=\> /10.239.74.227:9300] I don't think outbound ports are being limited (only inbound) in AWS. The US machine does not elect it's own master as it seems to be looking up the EU-WEST region and not find anything. Any request to the server shows a status 503 (master not found) my rootLogger is already on TRACE, i didn't notice any relevant additional output. Derry On 1 October 2012 19:29, Derry O' Sullivan [derryos@gmail.com](mailto:derryos@gmail.com) wrote: > HI Drew, > > Thanks again for the response. This is my entire config - everything else > is the default (commented out). > > I didn't include the filtered out instances lines (we have a lot of other > instances) but the output clearly shows > a) The nodes making requests to EC2 to find the other nodes > b) The nodes finding the other server in the other zone(e.g. EU finding us > and US finding EU e.g: > [2012-10-01 10:27:41,249][TRACE][discovery.ec2] [Demiurge] > adding i-b7eb27ca, address [ec2-107-22-26-95.compute-1.amazonaws.com](http://ec2-107-22-26-95.compute-1.amazonaws.com), > transport\_address inet[ > [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)] > [2012-10-01 10:27:41,250][DEBUG][discovery.ec2] [Demiurge] > using dynamic discovery nodes [[#cloud-i-b7eb27ca-0][inet[ > [ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300)][http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300]](http://ec2-107-22-26-95.compute-1.amazonaws.com/107.22.26.95:9300%5D) > ]] > > [2012-10-01 10:29:02,400][TRACE][discovery.ec2] [Magnum I] > adding i-cade1081, address > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com), transport\_address > inet[ > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)] > [2012-10-01 10:29:02,400][DEBUG][discovery.ec2] [Magnum I] > using dynamic discovery nodes [[#cloud-i-cade1081-0][inet[ > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D) > ]] > > c) both nodes specifically connecting to their tag/cluster matching node > in the other zone, but instantly disconnect: > [2012-10-01 10:29:02,498][TRACE][discovery.zen.ping.unicast] [Magnum I] > [13] received response from [#cloud-i-cade1081-0][inet[ > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D)]: > [ping\_response{target [[Magnum > I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], > cluster\_name[esdemohs]}, ping\_response{target [[Magnum > I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], > cluster\_name[esdemohs]}, ping\_response{target [[Magnum > I][Xn8IL89ESJyij3H6DYjKhw][inet[/10.214.221.68:9300]]], master [null], > cluster\_name[esdemohs]}, ping\_response{target > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], > cluster\_name[esdemohs]}] > [2012-10-01 10:29:02,498][TRACE][discovery.ec2] [Magnum I] > full ping responses: > --\> target > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] > [2012-10-01 10:29:02,499][DEBUG][discovery.ec2] [Magnum I] > filtered ping responses: (filter\_client[true], filter\_data[false]) > --\> target > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]], master > [[Demiurge][EIUweoZ6QWmq1cVCSaay3w][inet[/10.234.254.251:9300]]] > [2012-10-01 10:29:02,501][TRACE][discovery.zen.ping.unicast] [Magnum I] > [13] disconnecting from [#cloud-i-cade1081-0][inet[ > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)]] > [2012-10-01 10:29:02,501][DEBUG][transport.netty] [Magnum I] > disconnected from [[#cloud-i-cade1081-0][inet[ > [ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300)][http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300]](http://ec2-176-34-212-181.eu-west-1.compute.amazonaws.com/176.34.212.181:9300%5D) > ]] > > It is saying filter\_client true - would that mean that it thinks the other > instance is a client? > > I don't have access to machine now but i'll try and get the full logs up > on pastebin or something tomorrow. > > Thanks, > > D > > On 1 October 2012 18:54, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > > > Derry O' Sullivan wrote: > > > > > The config of my first server (eu): > > > cluster.name: esdemohs > > > cloud.aws.access\_key: > > > cloud.aws.secret\_key: > > > cloud.aws.region: us-east-1 # not sure if i have to tell > > > discovery.type: ec2 > > > discovery.ec2.host\_type: public\_dns > > > discovery.ec2.ping.timeout: 5m > > > discovery.ec2.tag.cluster: esdemohs > > > > > > config of my 2nd server (US): > > > cluster.name: esdemohs > > > cloud.aws.access\_key: > > > cloud.aws.secret\_key: > > > cloud.aws.region: eu-west-1 > > > discovery.type: ec2 > > > discovery.ec2.ping.timeout: 5m > > > discovery.ec2.host\_type: public\_dns > > > discovery.ec2.tag.cluster: esdemohs > > > > > > I start up both servers, i get a status 200 for the EU server and a > > > status > > > 503 for the US server (web searching shows that that is because it is > > > not > > > able to join the cluster. > > > > [...] > > > > > It seems that the config or cloud.aws.region or tags does not affect the > > > outcome, it just seems like they connect and then disconnect again. > > > > No obvious issues with your config. Although it's interesting that > > the us-east node didn't elect itself master also (maybe you have > > master disabled on that one). > > > > Feels more like a security group issue. Let's try turning on TRACE > > logging for the us-east node. In logging.yml uncomment the line that > > looks like > > > > #discovery: TRACE > > > > and restart the node. That will give us some output about what it's > > trying to access. Look for lines like: > > > > using host\_type... > > filtering out instance... > > > > -Drew > > > > -- -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 2, 2012, 2:14pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/11 "2012-10-02T14:14:44Z")
Both machines are running Ubuntu 12.04.1 LTS, ES 0.19.9 and the latest cloud plugin (1.9.0). I've done no specific configurations on indexes etc. On Thursday, 27 September 2012 15:52:08 UTC+1, Derry O' Sullivan wrote: > Hi all, > > I'm trying to setup an elasticsearch cluster with 2 x ec2 small instances. > One is based in the EU and the other in the US east region. > > I setup the 2 instances seperately with the same cluster name and verified > they worked ok (just a request to :9200. > > I've opened up ports 22(ssh), 9200(http) and the range (9300 - 9400) on my > security groups. > > I can't seem to get the 2 machines to communicate properly, both find each > other and try to communicate but seem to elect no master.. - i've changed > the logging.yml to add more debug info (based on hunting through existing > group posts). > > The relevant settings on both machines is: > EU machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > ### Note, no cloud.aws.region.... > > US machine: > cluster.name: elasticsearch-demo-HS > cloud.aws.access\_key: > cloud.aws.secret\_key: > cloud.aws.region: eu-west ####suggested in other forums to allow us > machine to find eu machine ( > [Redirecting to Google Groups](https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ) > ) > discovery.type: ec2 > discovery.zen.ping.timeout: 5m > > When i start up the EU machine, it does setup and gets information back > from amazon about the instances via the cloud aws plugin > [2012-09-27 14:25:13,391][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > connecting (light) to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > > It then times out on connection (as the other node in the US has not > started) > [2012-09-27 14:25:43,451][TRACE][discovery.zen.ping.unicast] [Legacy] [1] > failed to connect to [#cloud-i-b7eb27ca-0][inet[/hidden:9300]] > org.elasticsearch.transport.ConnectTransportException: [inet[/ > 10.96.67.186:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:585) > > I then start the US node: > and get: > [2012-09-27 14:30:42,262][TRACE][discovery.zen.ping.unicast] [Seth] [1] > connecting (light) to [#cloud-i-cade1081-0][inet[/hidden:9300]] > > On the EU machine, i then get a: > [2012-09-27 14:47:18,913][DEBUG][transport.netty] [Adaptoid] > connected to node [[#cloud-i-b7eb27ca-0][inet[hidden:9300]]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] connected to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:18,914][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] sending to [#cloud-i-b7eb27ca-0][inet[hidden:9300]] > [2012-09-27 14:47:19,042][TRACE][discovery.zen.ping.unicast] [Adaptoid] > [1] received response from [#cloud-i-b7eb27ca-0][inet[hidden:9300]]: > [ping\_response{target > [[Adaptoid][O1XIpLrPTE-yS1kh4OEfTw][inet[/10.240.50.61:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}, ping\_response{target > [[Eliminator][RodL0FfpStSsl0shyZa8Vw][inet[/10.96.67.186:9300]]], master > [null], cluster\_name[elasticsearch-demo-HS]}] > > When i query the \_cluster\_stats, i get: > { > > - error: "ClusterBlockException[blocked by: > [SERVICE\_UNAVAILABLE/1/state not recovered / > initialized];[SERVICE\_UNAVAILABLE/2/no master];]", > - status: 503 > > } > > An initial problem (not looking up across internal IP addresses) was > solved by changing the discovery ec2 type from private\_ip to public\_ip > (communication across regions). Now it's just stumped by the fact that i > can't get the 2 to talk to each other.. > > Any help greatly appreciated 😉 > > Derry -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 4, 2012, 7:21am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/12 "2012-10-04T07:21:08Z")
For anyone who is interested, the problem was to do with my network.publish\_host setting. It was not being set correctly so both nodes could not contact each other. Once that was set correctly, all worked perfectly. Special thanks for Drew for the on/offline help! Derry On 2 October 2012 15:14, Derry O' Sullivan [derryos@gmail.com](mailto:derryos@gmail.com) wrote: > Both machines are running Ubuntu 12.04.1 LTS, ES 0.19.9 and the latest > cloud plugin (1.9.0). I've done no specific configurations on indexes etc. > > On Thursday, 27 September 2012 15:52:08 UTC+1, Derry O' Sullivan wrote: > > > Hi all, > > > > I'm trying to setup an elasticsearch cluster with 2 x ec2 small > > instances. One is based in the EU and the other in the US east region. > > > > I setup the 2 instances seperately with the same cluster name and > > verified they worked ok (just a request to :9200. > > > > I've opened up ports 22(ssh), 9200(http) and the range (9300 - 9400) on > > my security groups. > > > > I can't seem to get the 2 machines to communicate properly, both find > > each other and try to communicate but seem to elect no master.. - i've > > changed the logging.yml to add more debug info (based on hunting through > > existing group posts). > > > > The relevant settings on both machines is: > > EU machine: > > cluster.name: elasticsearch-demo-HS > > cloud.aws.access\_key: > > cloud.aws.secret\_key: > > discovery.type: ec2 > > discovery.zen.ping.timeout: 5m > > > > ### Note, no cloud.aws.region.... > > > > US machine: > > cluster.name: elasticsearch-demo-HS > > cloud.aws.access\_key: > > cloud.aws.secret\_key: > > cloud.aws.region: eu-west ####suggested in other forums to allow us > > machine to find eu machine ([https://groups.google.com/\*\*forum/#!searchin/](https://groups.google.com/**forum/#!searchin/) > > \*\*elasticsearch/aws/\*\*elasticsearch/gB2ag71gFT8/n7-\*\*Cpyg5O4cJ[https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ](https://groups.google.com/forum/#!searchin/elasticsearch/aws/elasticsearch/gB2ag71gFT8/n7-Cpyg5O4cJ) > > ) > > discovery.type: ec2 > > discovery.zen.ping.timeout: 5m > > > > When i start up the EU machine, it does setup and gets information back > > from amazon about the instances via the cloud aws plugin > > [2012-09-27 14:25:13,391][TRACE][\*\*discovery.zen.ping.unicast] [Legacy] > > [1] connecting (light) to [#cloud-i-b7eb27ca-0][inet[/\*\*hidden:9300]] > > > > It then times out on connection (as the other node in the US has not > > started) > > [2012-09-27 14:25:43,451][TRACE][\*\*discovery.zen.ping.unicast] [Legacy] > > [1] failed to connect to [#cloud-i-b7eb27ca-0][inet[/\*\*hidden:9300]] > > org.elasticsearch.transport.\*\*ConnectTransportException: [inet[/ > > 10.96.67.186:9300]] connect\_timeout[30s] > > at org.elasticsearch.transport. **netty.NettyTransport.** > > connectToChannelsLight(\*\*NettyTransport.java:585) > > > > I then start the US node: > > and get: > > [2012-09-27 14:30:42,262][TRACE][\*\*discovery.zen.ping.unicast] [Seth] > > [1] connecting (light) to [#cloud-i-cade1081-0][inet[/\*\*hidden:9300]] > > > > On the EU machine, i then get a: > > [2012-09-27 14:47:18,913][DEBUG][\*\*transport.netty] [Adaptoid] > > connected to node [[#cloud-i-b7eb27ca-0][inet[\*\*hidden:9300]]] > > [2012-09-27 14:47:18,914][TRACE][\*\*discovery.zen.ping.unicast] > > [Adaptoid] [1] connected to [#cloud-i-b7eb27ca-0][inet[\*\*hidden:9300]] > > [2012-09-27 14:47:18,914][TRACE][\*\*discovery.zen.ping.unicast] > > [Adaptoid] [1] sending to [#cloud-i-b7eb27ca-0][inet[\*\*hidden:9300]] > > [2012-09-27 14:47:19,042][TRACE][\*\*discovery.zen.ping.unicast] > > [Adaptoid] [1] received response from [#cloud-i-b7eb27ca-0][inet[\*\*hidden:9300]]: > > [ping\_response{target [[Adaptoid][O1XIpLrPTE-\*\*yS1kh4OEfTw][inet[/ > > 10.240.50.\*\*61:9300]]], master [null], cluster\_name[elasticsearch-**demo-HS]}, > > ping\_response{target [[Eliminator][RodL0FfpStSsl0shyZa8Vw][inet[/ > > 10.96.67.186:9300]]], master [null], cluster\_name[elasticsearch-** > > demo-HS]}] > > > > When i query the \_cluster\_stats, i get: > > { > > > > - error: "ClusterBlockException[blocked by: > > [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];[SERVICE\_\*\*UNAVAILABLE/2/no > > master];]", > > - status: 503 > > > > } > > > > An initial problem (not looking up across internal IP addresses) was > > solved by changing the discovery ec2 type from private\_ip to public\_ip > > (communication across regions). Now it's just stumped by the fact that i > > can't get the 2 to talk to each other.. > > > > Any help greatly appreciated 😉 > > > > Derry > > > > -- -- ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [October 4, 2012, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/13 "2012-10-04T15:42:22Z")
Derry O' Sullivan wrote: > For anyone who is interested, the problem was to do with my > network.publish\_host setting. It was not being set correctly so > both nodes could not contact each other. Once that was set > correctly, all worked perfectly. Here's a minimal 2-node zen config similar to what worked for him: ``` cluster.name: foo node.name: alice network.publish_host: 23.20.100.100 discovery.zen.ping_timeout: 60s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["54.247.55.55"] cluster.name: foo node.name: bob network.publish_host: 54.247.55.55 discovery.zen.ping_timeout: 60s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["23.20.100.100"] ``` If alice is in us-east and bob in eu-west, they won't be able to communicate by 10/8 addresses[1]. Since an individual ec2 node knows nothing about its public IP, its local interface is identified by a 10/8 address. When alice looks through the unicast list to try and join bob, she will connect to bob's internal interface (due to his auto-resolved network.bind\_host[2]) through the AWS perimeter by his public IP. During the transport handshake, bob needs to tell her how to get _back_ to him to communicate further, and that's what the network.publish\_host is for. alice also tells bob about her publish\_host, and they're off to the races.[3] Derry originally was trying to get this working with ec2 discovery and reported that correctly supplying publish\_host and his security group caused it to work as well. So if you want a simpler alternative to zen for this particular case, you should try that. -Drew Footnotes: [1] They have to talk over public nets. [2] In order to open a socket, ES has to bind to an interface. ES can use the network.bind\_host setting if you don't want it to resolve it for itself. And if you supply simply network.host without the other two, it will use that IP to bind to an interface and talk to peers. [3] Note that this technique should also work across generic NATs if you supplied appropriate host:port combinations in the unicast list and then told your router to redirect those ports to appropriate nodes. -- ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [October 4, 2012, 4:08pm UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/14 "2012-10-04T16:08:07Z")
To add to this, the EC2 plugin also works ok. It also seems to work fine without security groups too. On the timeouts, i suspect 60s is slightly high given the fact that cross-region AWS comms are actually very fast but for 'uncontrollable' latency purposes, i've left it at a minute. Done some wikipedia river testing on the setup and works perfectly with 000's of docs. On 4 October 2012 16:42, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > Derry O' Sullivan wrote: > > > For anyone who is interested, the problem was to do with my > > network.publish\_host setting. It was not being set correctly so > > both nodes could not contact each other. Once that was set > > correctly, all worked perfectly. > > Here's a minimal 2-node zen config similar to what worked for him: > > ``` > cluster.name: foo > node.name: alice > network.publish_host: 23.20.100.100 > discovery.zen.ping_timeout: 60s > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: ["54.247.55.55"] > > cluster.name: foo > node.name: bob > network.publish_host: 54.247.55.55 > discovery.zen.ping_timeout: 60s > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: ["23.20.100.100"] > > ``` > > If alice is in us-east and bob in eu-west, they won't be able to > communicate by 10/8 addresses[1]. Since an individual ec2 node knows > nothing about its public IP, its local interface is identified by a > 10/8 address. > > When alice looks through the unicast list to try and join bob, she > will connect to bob's internal interface (due to his auto-resolved > network.bind\_host[2]) through the AWS perimeter by his public IP. > During the transport handshake, bob needs to tell her how to get > _back_ to him to communicate further, and that's what the > network.publish\_host is for. alice also tells bob about her > publish\_host, and they're off to the races.[3] > > Derry originally was trying to get this working with ec2 discovery > and reported that correctly supplying publish\_host and his security > group caused it to work as well. So if you want a simpler > alternative to zen for this particular case, you should try that. > > -Drew > > Footnotes: > > [1] They have to talk over public nets. > > [2] In order to open a socket, ES has to bind to an interface. ES > can use the network.bind\_host setting if you don't want it to > resolve it for itself. And if you supply simply network.host > without the other two, it will use that IP to bind to an > interface and talk to peers. > > [3] Note that this technique should also work across generic NATs if > you supplied appropriate host:port combinations in the unicast > list and then told your router to redirect those ports to > appropriate nodes. > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/elasticsearch-ec2-setup-across-multiple-regions/9170/15 "2017-07-06T03:10:06Z")