# Another EC2 clustering question **URL:** \ **Category:** Elasticsearch\ **Created:** [January 6, 2015, 10:56pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513 "2015-01-06T22:56:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![shane\_adams](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shane\_adams](https://discuss.elastic.co/u/shane_adams)\ **Post date:** [January 6, 2015, 10:56pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/1 "2015-01-06T22:56:28Z")
Howdy - I cannot get two ec2 servers to connect to one another as a cluster. The servers are successfully discovering themselves via the supplied AWS credentials with proper permissions, however I the non-master server continually connects to and fails when joining the master. I have verified the ports are open for both TCP and UDP on both servers in the 9200-9400 port range. The specific error, found in the log below is: [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] org.elasticsearch.transport.RemoteTransportException: [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused by: org.elasticsearch.transport.NotSerializableTransportException: [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: /172.31.29.66:9300; I'm stuck after several hours, hoping someone can point me in the correct direction. Thanks! Shane _Log excerpt:_ [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to ping [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building dynamic unicast discovery nodes... [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-a220ea54, address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building dynamic unicast discovery nodes... [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding i-a220ea54, address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building dynamic unicast discovery nodes... [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding i-a220ea54, address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping responses: --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], hasJoinedOnce [true], cluster\_name[elasticsearch]} [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], hasJoinedOnce [true], cluster\_name[elasticsearch]} [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]] [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] org.elasticsearch.transport.RemoteTransportException: [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused by: org.elasticsearch.transport.NotSerializableTransportException: [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: /172.31.29.66:9300; _And here is the config file I am using on both servers:_ cluster.name: elasticsearch plugin: mandatory: cloud-aws cloud.aws.region: us-west-2 cloud: aws: access\_key: AAA secret\_key: BBB discovery: type: ec2 discovery.ec2.tag.stack: staging -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 6, 2015, 11:15pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/2 "2015-01-06T23:15:36Z")
Are both running the same ES and java versions? Can you telnet between the data and master nodes on 9300? On 7 January 2015 at 09:56, [shane@gethashed.com](mailto:shane@gethashed.com) wrote: > Howdy - I cannot get two ec2 servers to connect to one another as a > cluster. The servers are successfully discovering themselves via the > supplied AWS credentials with proper permissions, however I the non-master > server continually connects to and fails when joining the master. I have > verified the ports are open for both TCP and UDP on both servers in the > 9200-9400 port range. > > The specific error, found in the log below is: > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send join > request to master > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] org.elasticsearch.transport.RemoteTransportException: > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused by: > org.elasticsearch.transport.NotSerializableTransportException: > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > 172.31.29.66:9300; > > I'm stuck after several hours, hoping someone can point me in the correct > direction. > > Thanks! > > Shane > > _Log excerpt:_ > [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to ping > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building dynamic > unicast discovery nodes... > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-a220ea54, > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic > discovery nodes > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building dynamic > unicast discovery nodes... > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding i-a220ea54, > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic > discovery nodes > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building dynamic > unicast discovery nodes... > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding i-a220ea54, > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic > discovery nodes > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping > responses: > > --\> ping\_response{node > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > id[51], master > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping > responses: (filter\_client[true], filter\_data[false]) > > --\> ping\_response{node > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > id[51], master > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master > [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]] > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send join > request to master > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] > > org.elasticsearch.transport.RemoteTransportException: > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] > > Caused by: org.elasticsearch.transport.NotSerializableTransportException: > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > 172.31.29.66:9300; > > _And here is the config file I am using on both servers:_ > cluster.name: elasticsearch > plugin: > mandatory: cloud-aws > > cloud.aws.region: us-west-2 > cloud: > aws: > access\_key: AAA > secret\_key: BBB > discovery: > type: ec2 > discovery.ec2.tag.stack: staging > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![shane\_adams](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shane\_adams](https://discuss.elastic.co/u/shane_adams)\ **Post date:** [January 6, 2015, 11:22pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/3 "2015-01-06T23:22:26Z")
Hi Mark Thank you for your response. Yes both have the same ES version [same AMI with ES pre-installed]. I can telnet between the two servers on 9300, 9200, 9400. Running netstat shows the two machines are actively connected on port 9300. The original machine which becomes the master machine has around 2.8G with of indexes. The secondary machine I've tried with a complete copy of the index or starting with an empty index. Cheers, Shane On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > Are both running the same ES and java versions? > > Can you telnet between the data and master nodes on 9300? > > On 7 January 2015 at 09:56, [shane@gethashed.com](mailto:shane@gethashed.com) wrote: > > > Howdy - I cannot get two ec2 servers to connect to one another as a > > cluster. The servers are successfully discovering themselves via the > > supplied AWS credentials with proper permissions, however I the non-master > > server continually connects to and fails when joining the master. I have > > verified the ports are open for both TCP and UDP on both servers in the > > 9200-9400 port range. > > > > The specific error, found in the log below is: > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > join request to master > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] org.elasticsearch.transport.RemoteTransportException: > > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused > > by: org.elasticsearch.transport.NotSerializableTransportException: > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > 172.31.29.66:9300; > > > > I'm stuck after several hours, hoping someone can point me in the correct > > direction. > > > > Thanks! > > > > Shane > > > > _Log excerpt:_ > > [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to ping > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building dynamic > > unicast discovery nodes... > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding i-a220ea54, > > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > > > [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic > > discovery nodes > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building dynamic > > unicast discovery nodes... > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > > > [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding i-a220ea54, > > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > > > [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic > > discovery nodes > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building dynamic > > unicast discovery nodes... > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding i-dd78fdd1, > > address 172.31.29.66, transport\_address inet[/172.31.29.66:9300] > > > > [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding i-a220ea54, > > address 172.31.23.251, transport\_address inet[/172.31.23.251:9300] > > > > [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic > > discovery nodes > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping > > responses: > > > > --\> ping\_response{node > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > id[51], master > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping > > responses: (filter\_client[true], filter\_data[false]) > > > > --\> ping\_response{node > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > id[51], master > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master > > [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]] > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > join request to master > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] > > > > org.elasticsearch.transport.RemoteTransportException: > > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] > > > > Caused by: org.elasticsearch.transport.NotSerializableTransportException: > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > 172.31.29.66:9300; > > > > _And here is the config file I am using on both servers:_ > > cluster.name: elasticsearch > > plugin: > > mandatory: cloud-aws > > > > cloud.aws.region: us-west-2 > > cloud: > > aws: > > access\_key: AAA > > secret\_key: BBB > > discovery: > > type: ec2 > > discovery.ec2.tag.stack: staging > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAG8XaFUvz7nmpKcXQc6jJDqsXuCgwSAYg2hb9o-B%3DkYpWV5-KA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAG8XaFUvz7nmpKcXQc6jJDqsXuCgwSAYg2hb9o-B%3DkYpWV5-KA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![simpsora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simpsora/32/647_2.png) [@simpsora](https://discuss.elastic.co/u/simpsora)\ **Post date:** [January 7, 2015, 3:43am UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/4 "2015-01-07T03:43:13Z")
Shane, What's JVM version is ES running under, as Mark asked? That exception usually indicates that you're running two different JVMs in the cluster, which unfortunately is not supported due to how Java serializes exceptions. Ross On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: > Hi Mark > > Thank you for your response. > > Yes both have the same ES version [same AMI with ES pre-installed]. I > can telnet between the two servers on 9300, 9200, 9400. Running netstat > shows the two machines are actively connected on port 9300. > > The original machine which becomes the master machine has around 2.8G with > of indexes. The secondary machine I've tried with a complete copy of the > index or starting with an empty index. > > Cheers, > Shane > > On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom \<[markw...@gmail.com](mailto:markw...@gmail.com) > \\> wrote: > > > Are both running the same ES and java versions? > > > > Can you telnet between the data and master nodes on 9300? > > > > On 7 January 2015 at 09:56, \<[sh...@gethashed.com](mailto:sh...@gethashed.com) \\> wrote: > > > > > Howdy - I cannot get two ec2 servers to connect to one another as a > > > cluster. The servers are successfully discovering themselves via the > > > supplied AWS credentials with proper permissions, however I the non-master > > > server continually connects to and fails when joining the master. I have > > > verified the ports are open for both TCP and UDP on both servers in the > > > 9200-9400 port range. > > > > > > The specific error, found in the log below is: > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > join request to master > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] org.elasticsearch.transport.RemoteTransportException: > > > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused by: > > > org.elasticsearch.transport.NotSerializableTransportException: > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > 172.31.29.66:9300; > > > > > > I'm stuck after several hours, hoping someone can point me in the > > > correct direction. > > > > > > Thanks! > > > > > > Shane > > > > > > _Log excerpt:_ > > > [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to ping > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building dynamic > > > unicast discovery nodes... > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > 172.31.29.66:9300] > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > 172.31.23.251:9300] > > > > > > [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic > > > discovery nodes > > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building dynamic > > > unicast discovery nodes... > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > 172.31.29.66:9300] > > > > > > [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > 172.31.23.251:9300] > > > > > > [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic > > > discovery nodes > > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building dynamic > > > unicast discovery nodes... > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > 172.31.29.66:9300] > > > > > > [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > 172.31.23.251:9300] > > > > > > [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic > > > discovery nodes > > > [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], > > > [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping > > > responses: > > > > > > --\> ping\_response{node > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > id[51], master > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping > > > responses: (filter\_client[true], filter\_data[false]) > > > > > > --\> ping\_response{node > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > id[51], master > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master > > > [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]] > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > join request to master > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] > > > > > > Caused by: > > > org.elasticsearch.transport.NotSerializableTransportException: > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > 172.31.29.66:9300; > > > > > > _And here is the config file I am using on both servers:_ > > > cluster.name: elasticsearch > > > plugin: > > > mandatory: cloud-aws > > > > > > cloud.aws.region: us-west-2 > > > cloud: > > > aws: > > > access\_key: AAA > > > secret\_key: BBB > > > discovery: > > > type: ec2 > > > discovery.ec2.tag.stack: staging > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![shane\_adams](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shane\_adams](https://discuss.elastic.co/u/shane_adams)\ **Post date:** [January 7, 2015, 5:55pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/5 "2015-01-07T17:55:51Z")
Sorry new born at home and I'm tired, missed the JVM version request. I verified it's the same [same AMI image I created for both instances so I expected this to be the case]. java version "1.8.0\_25" I've tried reconfiguring to use unicast and I get a similar error but I need to retest to confirm 100%. Shane On Tue, Jan 6, 2015 at 7:43 PM, Ross Simpson [simpsora@gmail.com](mailto:simpsora@gmail.com) wrote: > Shane, > > What's JVM version is ES running under, as Mark asked? That exception > usually indicates that you're running two different JVMs in the cluster, > which unfortunately is not supported due to how Java serializes exceptions. > > Ross > > On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: > > > Hi Mark > > > > Thank you for your response. > > > > Yes both have the same ES version [same AMI with ES pre-installed]. I > > can telnet between the two servers on 9300, 9200, 9400. Running netstat > > shows the two machines are actively connected on port 9300. > > > > The original machine which becomes the master machine has around 2.8G > > with of indexes. The secondary machine I've tried with a complete copy of > > the index or starting with an empty index. > > > > Cheers, > > Shane > > > > On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) wrote: > > > > > Are both running the same ES and java versions? > > > > > > Can you telnet between the data and master nodes on 9300? > > > > > > On 7 January 2015 at 09:56, [sh...@gethashed.com](mailto:sh...@gethashed.com) wrote: > > > > > > > Howdy - I cannot get two ec2 servers to connect to one another as a > > > > cluster. The servers are successfully discovering themselves via the > > > > supplied AWS credentials with proper permissions, however I the non-master > > > > server continually connects to and fails when joining the master. I have > > > > verified the ports are open for both TCP and UDP on both servers in the > > > > 9200-9400 port range. > > > > > > > > The specific error, found in the log below is: > > > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > > join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]] org. > > > > elasticsearch.transport.RemoteTransportException: [es1][inet[/ > > > > 172.31.23.251:9300]][internal:discovery/zen/join] Caused by: > > > > org.elasticsearch.transport.NotSerializableTransportException: > > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > > 172.31.29.66:9300; > > > > > > > > I'm stuck after several hours, hoping someone can point me in the > > > > correct direction. > > > > > > > > Thanks! > > > > > > > > Shane > > > > > > > > _Log excerpt:_ > > > > [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to > > > > ping > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building > > > > dynamic unicast discovery nodes... > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > 172.31.29.66:9300] > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > 172.31.23.251:9300] > > > > > > > > [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > 172.31.23.251:9300]]] > > > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building > > > > dynamic unicast discovery nodes... > > > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > 172.31.29.66:9300] > > > > > > > > [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > 172.31.23.251:9300] > > > > > > > > [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > 172.31.23.251:9300]]] > > > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building > > > > dynamic unicast discovery nodes... > > > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > 172.31.29.66:9300] > > > > > > > > [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > 172.31.23.251:9300] > > > > > > > > [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > 172.31.23.251:9300]]] > > > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping > > > > responses: > > > > > > > > --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master > > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > > > [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping > > > > responses: (filter\_client[true], filter\_data[false]) > > > > > > > > --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master > > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master > > > > [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172. > > > > 31.23.251:9300]] > > > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > > join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]] > > > > > > > > org.elasticsearch.transport.RemoteTransportException: [es1][inet[/ > > > > 172.31.23.251:9300]][internal:discovery/zen/join] > > > > > > > > Caused by: org.elasticsearch.transport.NotSerializableTransportException: > > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > > 172.31.29.66:9300; > > > > > > > > _And here is the config file I am using on both servers:_ > > > > cluster.name: elasticsearch > > > > plugin: > > > > mandatory: cloud-aws > > > > > > > > cloud.aws.region: us-west-2 > > > > cloud: > > > > aws: > > > > access\_key: AAA > > > > secret\_key: BBB > > > > discovery: > > > > type: ec2 > > > > discovery.ec2.tag.stack: staging > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > topic/elasticsearch/TpFqBy5GdXA/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac% > > > 3D4BJ9pak0X1aT3vm9ZTdXQ%[40mail.gmail.com](http://40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWh5wGXyjriL4FW\_zD2s%2BZiJQHNqRKaf-rufVQVKQQJ\_Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWh5wGXyjriL4FW_zD2s%2BZiJQHNqRKaf-rufVQVKQQJ_Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![shane\_adams](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@shane\_adams](https://discuss.elastic.co/u/shane_adams)\ **Post date:** [January 7, 2015, 7:33pm UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/6 "2015-01-07T19:33:57Z")
Figured it out. One of the servers was not a member of the proper security group and port 9300 was closed. I had misconfigured ES to use 9200 for cluster transport and that's why it was giving an odd error, attempting a non-HTTP protocol against the REST API. Sorry to bug anyone. On Wed, Jan 7, 2015 at 9:55 AM, shane adams [shane@gethashed.com](mailto:shane@gethashed.com) wrote: > Sorry new born at home and I'm tired, missed the JVM version request. I > verified it's the same [same AMI image I created for both instances so I > expected this to be the case]. java version "1.8.0\_25" > > I've tried reconfiguring to use unicast and I get a similar error but I > need to retest to confirm 100%. > > Shane > > On Tue, Jan 6, 2015 at 7:43 PM, Ross Simpson [simpsora@gmail.com](mailto:simpsora@gmail.com) wrote: > > > Shane, > > > > What's JVM version is ES running under, as Mark asked? That exception > > usually indicates that you're running two different JVMs in the cluster, > > which unfortunately is not supported due to how Java serializes exceptions. > > > > Ross > > > > On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: > > > > > Hi Mark > > > > > > Thank you for your response. > > > > > > Yes both have the same ES version [same AMI with ES pre-installed]. I > > > can telnet between the two servers on 9300, 9200, 9400. Running netstat > > > shows the two machines are actively connected on port 9300. > > > > > > The original machine which becomes the master machine has around 2.8G > > > with of indexes. The secondary machine I've tried with a complete copy of > > > the index or starting with an empty index. > > > > > > Cheers, > > > Shane > > > > > > On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) wrote: > > > > > > > Are both running the same ES and java versions? > > > > > > > > Can you telnet between the data and master nodes on 9300? > > > > > > > > On 7 January 2015 at 09:56, [sh...@gethashed.com](mailto:sh...@gethashed.com) wrote: > > > > > > > > > Howdy - I cannot get two ec2 servers to connect to one another as a > > > > > cluster. The servers are successfully discovering themselves via the > > > > > supplied AWS credentials with proper permissions, however I the non-master > > > > > server continually connects to and fails when joining the master. I have > > > > > verified the ports are open for both TCP and UDP on both servers in the > > > > > 9200-9400 port range. > > > > > > > > > > The specific error, found in the log below is: > > > > > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > > > join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]] org. > > > > > elasticsearch.transport.RemoteTransportException: [es1][inet[/ > > > > > 172.31.23.251:9300]][internal:discovery/zen/join] Caused by: > > > > > org.elasticsearch.transport.NotSerializableTransportException: > > > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > > > 172.31.29.66:9300; > > > > > > > > > > I'm stuck after several hours, hoping someone can point me in the > > > > > correct direction. > > > > > > > > > > Thanks! > > > > > > > > > > Shane > > > > > > > > > > _Log excerpt:_ > > > > > [2015-01-06 22:51:52,618][TRACE][discovery.ec2] [es0] starting to > > > > > ping > > > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] building > > > > > dynamic unicast discovery nodes... > > > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > > 172.31.29.66:9300] > > > > > > > > > > [2015-01-06 22:51:52,758][TRACE][discovery.ec2] [es0] adding > > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > > 172.31.23.251:9300] > > > > > > > > > > [2015-01-06 22:51:52,758][DEBUG][discovery.ec2] [es0] using dynamic > > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > > 172.31.23.251:9300]]] > > > > > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] building > > > > > dynamic unicast discovery nodes... > > > > > > > > > > [2015-01-06 22:51:54,348][TRACE][discovery.ec2] [es0] adding > > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > > 172.31.29.66:9300] > > > > > > > > > > [2015-01-06 22:51:54,349][TRACE][discovery.ec2] [es0] adding > > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > > 172.31.23.251:9300] > > > > > > > > > > [2015-01-06 22:51:54,349][DEBUG][discovery.ec2] [es0] using dynamic > > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > > 172.31.23.251:9300]]] > > > > > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] building > > > > > dynamic unicast discovery nodes... > > > > > > > > > > [2015-01-06 22:51:55,913][TRACE][discovery.ec2] [es0] adding > > > > > i-dd78fdd1, address 172.31.29.66, transport\_address inet[/ > > > > > 172.31.29.66:9300] > > > > > > > > > > [2015-01-06 22:51:55,914][TRACE][discovery.ec2] [es0] adding > > > > > i-a220ea54, address 172.31.23.251, transport\_address inet[/ > > > > > 172.31.23.251:9300] > > > > > > > > > > [2015-01-06 22:51:55,914][DEBUG][discovery.ec2] [es0] using dynamic > > > > > discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ > > > > > 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ > > > > > 172.31.23.251:9300]]] > > > > > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] full ping > > > > > responses: > > > > > > > > > > --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master > > > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > > > > > [2015-01-06 22:51:55,922][DEBUG][discovery.ec2] [es0] filtered ping > > > > > responses: (filter\_client[true], filter\_data[false]) > > > > > > > > > > --\> ping\_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master > > > > > [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], > > > > > hasJoinedOnce [true], cluster\_name[elasticsearch]} > > > > > > > > > > [2015-01-06 22:51:55,922][TRACE][discovery.ec2] [es0] joining master > > > > > [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172. > > > > > 31.23.251:9300]] > > > > > > > > > > [2015-01-06 22:52:25,954][TRACE][discovery.ec2] [es0] failed to send > > > > > join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] > > > > > [ip-172-31-23-251][inet[/172.31.23.251:9300]]] > > > > > > > > > > org.elasticsearch.transport.RemoteTransportException: [es1][inet[/ > > > > > 172.31.23.251:9300]][internal:discovery/zen/join] > > > > > > > > > > Caused by: org.elasticsearch.transport.NotSerializableTransportException: > > > > > [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ > > > > > 172.31.29.66:9300]] connect\_timeout[30s]; connection timed out: / > > > > > 172.31.29.66:9300; > > > > > > > > > > _And here is the config file I am using on both servers:_ > > > > > cluster.name: elasticsearch > > > > > plugin: > > > > > mandatory: cloud-aws > > > > > > > > > > cloud.aws.region: us-west-2 > > > > > cloud: > > > > > aws: > > > > > access\_key: AAA > > > > > secret\_key: BBB > > > > > discovery: > > > > > type: ec2 > > > > > discovery.ec2.tag.stack: staging > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > topic/elasticsearch/TpFqBy5GdXA/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac% > > > > 3D4BJ9pak0X1aT3vm9ZTdXQ%[40mail.gmail.com](http://40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWWf5LWc66Sw1S5x2chRpMQ2TTZ4r0DbbQaA-LPUxSEFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWWf5LWc66Sw1S5x2chRpMQ2TTZ4r0DbbQaA-LPUxSEFA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:40am UTC](https://discuss.elastic.co/t/another-ec2-clustering-question/21513/7 "2017-07-06T00:40:15Z")