# EC2 discovery issue **URL:** \ **Category:** Elasticsearch\ **Created:** [July 4, 2011, 8:21pm UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772 "2011-07-04T20:21:11Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 7, 2011, 2:49pm UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/21 "2011-07-07T14:49:02Z")
Hey, First, it seems like all the logging is DEBUG mode, thats noisy, all you need is discovery set to TRACE. If you see, on both nodes, the output: "using dynamic discovery nodes" is empty, meaning that it went ahead and queries aws for the list of nodes, and got none... . Maybe the groups / tags are not correct? -shay.banon On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote: > Shay, > here is the gist: > [master · GitHub](https://gist.github.com/1069376) > > On Jul 6, 10:35 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > Can you set logging for discovery to TRACE in the logging file and gist it? > > > > On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote: > > > > > Thanks Shay. > > > Now it is bit better, at least I do not have an exception but still > > > the ec2 instances do not run as a cluster. > > > What I additionally did is the following. Enabled ping for the > > > security group with the ec2-authorise. Tried that the 2 machines can > > > ping each other. > > > Checked with telnet that the 2 machines can open each other's 9200 and > > > 9300 ports. The 9301 - 9303 ports are open too but the other ports are > > > closed by default. > > > These are the 2 log files I have. First the master and next the one > > > which had to become a slave.At the bottom is the elasticsearch.yml > > > config file I use. > > > Any ideas what I need to change? > > > > > master: > > > [2011-07-06 18:14:42,458][INFO][node] [Jimmy > > > Woo] {elasticsearch/0.16.2}[2142]: initializing ... > > > [2011-07-06 18:14:42,468][INFO][plugins] [Jimmy > > > Woo] loaded [cloud-aws] > > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > > Woo] {elasticsearch/0.16.2}[2142]: initialized > > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > > Woo] {elasticsearch/0.16.2}[2142]: starting ... > > > [2011-07-06 18:14:48,414][INFO][transport] [Jimmy > > > Woo] bound\_address {inet[/10.224.50.50:9300]}, publish\_address {inet[/ > > > 10.224.50.50:9300]} > > > [2011-07-06 18:14:52,797][INFO][cluster.service] [Jimmy > > > Woo] new\_master [Jimmy Woo][T\_0Y3f\_UQJmfuiqYYW7Ivg][inet[/ > > > 10.224.50.50:9300]], reason: zen-disco-join (elected\_as\_master) > > > [2011-07-06 18:14:52,845][INFO][discovery] [Jimmy > > > Woo] perftests/T\_0Y3f\_UQJmfuiqYYW7Ivg > > > [2011-07-06 18:14:52,893][INFO][gateway] [Jimmy > > > Woo] recovered [0] indices into cluster\_state > > > [2011-07-06 18:14:52,894][INFO][http] [Jimmy > > > Woo] bound\_address {inet[/10.224.50.50:9200]}, publish\_address {inet[/ > > > 10.224.50.50:9200]} > > > [2011-07-06 18:14:52,895][INFO][node] [Jimmy > > > Woo] {elasticsearch/0.16.2}[2142]: started > > > > > slave: > > > [2011-07-06 18:18:07,733][INFO][node] [Machine > > > Teen] {elasticsearch/0.16.2}[1251]: initializing ... > > > [2011-07-06 18:18:07,741][INFO][plugins] [Machine > > > Teen] loaded [cloud-aws] > > > [2011-07-06 18:18:11,994][INFO][node] [Machine > > > Teen] {elasticsearch/0.16.2}[1251]: initialized > > > [2011-07-06 18:18:11,995][INFO][node] [Machine > > > Teen] {elasticsearch/0.16.2}[1251]: starting ... > > > [2011-07-06 18:18:12,105][INFO][transport] [Machine > > > Teen] bound\_address {inet[/10.234.227.72:9300]}, publish\_address > > > {inet[/10.234.227.72:9300]} > > > [2011-07-06 18:18:16,242][INFO][cluster.service] [Machine > > > Teen] new\_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ\_IA][inet[/ > > > 10.234.227.72:9300]], reason: zen-disco-join (elected\_as\_master) > > > [2011-07-06 18:18:16,278][INFO][discovery] [Machine > > > Teen] perftests/4vSbOGJqRx6q3Vi5HrJ\_IA > > > [2011-07-06 18:18:16,305][INFO][gateway] [Machine > > > Teen] recovered [0] indices into cluster\_state > > > [2011-07-06 18:18:16,312][INFO][http] [Machine > > > Teen] bound\_address {inet[/10.234.227.72:9200]}, publish\_address > > > {inet[/10.234.227.72:9200]} > > > [2011-07-06 18:18:16,313][INFO][node] [Machine > > > Teen] {elasticsearch/0.16.2}[1251]: started > > > > > elasticsearch.yml: > > > cluster: > > > name: perftests > > > > > discovery: > > > type: ec2 > > > zen: > > > ping\_timeout: 30s > > > > > cloud: > > > account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID", > > > not the "Account Number" > > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > compute: > > > type: amazon > > > > > network: > > > bind\_host: _eth0:ipv4_ > > > > > On Jul 6, 7:16 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > You need to set the discovery type to ec2:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html). The blog post is outdated... . > > > > > > And you don't have to use master to have discovery working on ec2. You can set network.host to whatever value you want, for example, the internal IP address. If you want teh same config across nodes, you can bind to the network interface name ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/network.html)), for example: _en0:ipv4_. > > > > > > On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote: > > > > > > > > > Unfortunately using the trunk is not an option at the moment because > > > > > there are some maven dependencies to the 16.2 jar in our project. > > > > > We will upgrade to 0.17 once it is officially released. > > > > > > > Now I tried the "Zen" as described here: > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2010/05/11/here-comes-the-cloud.html) > > > > > > > my elasticsearch.yml on the slave looks like this: > > > > > cluster: > > > > > name: perftests > > > > > > > discovery: > > > > > type: cloud > > > > > zen: > > > > > ping\_timeout: 30s > > > > > cloud: > > > > > account: 551234567890 > > > > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > > > compute: > > > > > type: amazon > > > > > network: > > > > > bind\_host: _non\_loopback:ipv4_ > > > > > > > On this I get the exception: > > > > > {elasticsearch/0.16.2}: Initialization Failed ... > > > > > > > > > > - NoClassSettingsException[Failed to load class setting > > > > > [discovery.type] with value [cloud]] > > > > > ClassNotFoundException[cloud] > > > > > > > for the network.bind\_host I tried eth0:ipv4 but this gives me host > > > > > notfound exception. > > > > > > > Any ideas? Can someone upload an elasticsearch.yml file able to do ec2 > > > > > discovery for him with the 16.2 version? > > > > > > > On Jul 6, 2:40 pm, Ridvan Gyundogan \<[ridva...@gmail.com](mailto:ridva...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > > > > > > > > > Ah ok thanks I will try. > > > > > > > > On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk))\>wrote: > > > > > > > > > > I start only 1 node at the moment so sure it should be the master. > > > > > > > > > By master, Shay means the version of Elasticsearch that is currently > > > > > > > master on github, not a released version. > > > > > > > > > You need to download and compile it yourself: > > > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/) > > > > > > > > > clint ---
**Author:** ![Pavel\_Penchev](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@Pavel\_Penchev](https://discuss.elastic.co/u/Pavel_Penchev)\ **Post date:** [July 7, 2011, 3:38pm UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/22 "2011-07-07T15:38:01Z")
Hi all, maybe related to this discussion - I just got ES working in amazon and there was one key point, the "cloud.aws.region" setting was not optional in my case. Without it Amazon was returning an empty list of instances. The > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. page implies that one could skip this setting. Regards, Pavel On 7.07.2011 17:49, Shay Banon wrote: > Hey, > > First, it seems like all the logging is DEBUG mode, thats noisy, all > you need is discovery set to TRACE. > > If you see, on both nodes, the output: "using dynamic discovery > nodes" is empty, meaning that it went ahead and queries aws for the > list of nodes, and got none... . Maybe the groups / tags are not correct? > > -shay.banon > > On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote: > > > Shay, > > here is the gist: > > [master · GitHub](https://gist.github.com/1069376) > > > > On Jul 6, 10:35 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) > > [http://elasticsearch.com](http://elasticsearch.com)\> wrote: > > > > > Can you set logging for discovery to TRACE in the logging file and > > > gist it? > > > > > > On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote: > > > > > > > Thanks Shay. > > > > Now it is bit better, at least I do not have an exception but still > > > > the ec2 instances do not run as a cluster. > > > > What I additionally did is the following. Enabled ping for the > > > > security group with the ec2-authorise. Tried that the 2 machines can > > > > ping each other. > > > > Checked with telnet that the 2 machines can open each other's 9200 and > > > > 9300 ports. The 9301 - 9303 ports are open too but the other ports are > > > > closed by default. > > > > These are the 2 log files I have. First the master and next the one > > > > which had to become a slave.At the bottom is the elasticsearch.yml > > > > config file I use. > > > > Any ideas what I need to change? > > > > > > > master: > > > > [2011-07-06 18:14:42,458][INFO][node] [Jimmy > > > > Woo] {elasticsearch/0.16.2}[2142]: initializing ... > > > > [2011-07-06 18:14:42,468][INFO][plugins] [Jimmy > > > > Woo] loaded [cloud-aws] > > > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > > > Woo] {elasticsearch/0.16.2}[2142]: initialized > > > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > > > Woo] {elasticsearch/0.16.2}[2142]: starting ... > > > > [2011-07-06 18:14:48,414][INFO][transport] [Jimmy > > > > Woo] bound\_address {inet[/10.224.50.50:9300]}, publish\_address {inet[/ > > > > 10.224.50.50:9300]} > > > > [2011-07-06 18:14:52,797][INFO][cluster.service] [Jimmy > > > > Woo] new\_master [Jimmy Woo][T\_0Y3f\_UQJmfuiqYYW7Ivg][inet[/ > > > > 10.224.50.50:9300]], reason: zen-disco-join (elected\_as\_master) > > > > [2011-07-06 18:14:52,845][INFO][discovery] [Jimmy > > > > Woo] perftests/T\_0Y3f\_UQJmfuiqYYW7Ivg > > > > [2011-07-06 18:14:52,893][INFO][gateway] [Jimmy > > > > Woo] recovered [0] indices into cluster\_state > > > > [2011-07-06 18:14:52,894][INFO][http] [Jimmy > > > > Woo] bound\_address {inet[/10.224.50.50:9200]}, publish\_address {inet[/ > > > > 10.224.50.50:9200]} > > > > [2011-07-06 18:14:52,895][INFO][node] [Jimmy > > > > Woo] {elasticsearch/0.16.2}[2142]: started > > > > > > > slave: > > > > [2011-07-06 18:18:07,733][INFO][node] [Machine > > > > Teen] {elasticsearch/0.16.2}[1251]: initializing ... > > > > [2011-07-06 18:18:07,741][INFO][plugins] [Machine > > > > Teen] loaded [cloud-aws] > > > > [2011-07-06 18:18:11,994][INFO][node] [Machine > > > > Teen] {elasticsearch/0.16.2}[1251]: initialized > > > > [2011-07-06 18:18:11,995][INFO][node] [Machine > > > > Teen] {elasticsearch/0.16.2}[1251]: starting ... > > > > [2011-07-06 18:18:12,105][INFO][transport] [Machine > > > > Teen] bound\_address {inet[/10.234.227.72:9300]}, publish\_address > > > > {inet[/10.234.227.72:9300]} > > > > [2011-07-06 18:18:16,242][INFO][cluster.service] [Machine > > > > Teen] new\_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ\_IA][inet[/ > > > > 10.234.227.72:9300]], reason: zen-disco-join (elected\_as\_master) > > > > [2011-07-06 18:18:16,278][INFO][discovery] [Machine > > > > Teen] perftests/4vSbOGJqRx6q3Vi5HrJ\_IA > > > > [2011-07-06 18:18:16,305][INFO][gateway] [Machine > > > > Teen] recovered [0] indices into cluster\_state > > > > [2011-07-06 18:18:16,312][INFO][http] [Machine > > > > Teen] bound\_address {inet[/10.234.227.72:9200]}, publish\_address > > > > {inet[/10.234.227.72:9200]} > > > > [2011-07-06 18:18:16,313][INFO][node] [Machine > > > > Teen] {elasticsearch/0.16.2}[1251]: started > > > > > > > elasticsearch.yml: > > > > cluster: > > > > name: perftests > > > > > > > discovery: > > > > type: ec2 > > > > zen: > > > > ping\_timeout: 30s > > > > > > > cloud: > > > > account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID", > > > > not the "Account Number" > > > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > > compute: > > > > type: amazon > > > > > > > network: > > > > bind\_host: _eth0:ipv4_ > > > > > > > On Jul 6, 7:16 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) > > > > ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > > > You need to set the discovery type to > > > > > ec2:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html). > > > > > The blog post is outdated... . > > > > > > > ``` > > > > And you don't have to use master to have discovery working on > > > > ec2. You can set network.host to whatever value you want, for > > > > example, the internal IP address. If you want teh same config > > > > across nodes, you can bind to the network interface name > > > > (http://www.elasticsearch.org/guide/reference/modules/network.html), > > > > for example: _en0:ipv4_. > > > > > > > > ``` > > > > > > > ``` > > > > On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote: > > > > > > > > ``` > > > > > > > > > ``` > > > > > Unfortunately using the trunk is not an option at the moment > > > > > because > > > > > there are some maven dependencies to the 16.2 jar in our project. > > > > > We will upgrade to 0.17 once it is officially released. > > > > > > > > > > ``` > > > > > > > ``` > > > > Now I tried the "Zen" as described here: > > > > http://www.elasticsearch.org/blog/2010/05/11/here-comes-the-cloud.html > > > > > > > > ``` > > > > > > > ``` > > > > my elasticsearch.yml on the slave looks like this: > > > > cluster: > > > > name: perftests > > > > > > > > ``` > > > > > > > ``` > > > > discovery: > > > > type: cloud > > > > zen: > > > > ping_timeout: 30s > > > > cloud: > > > > account: 551234567890 > > > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > > > compute: > > > > type: amazon > > > > network: > > > > bind_host: _non_loopback:ipv4_ > > > > > > > > ``` > > > > > > > ``` > > > > On this I get the exception: > > > > {elasticsearch/0.16.2}: Initialization Failed ... > > > > - NoClassSettingsException[Failed to load class setting > > > > [discovery.type] with value [cloud]] > > > > ClassNotFoundException[cloud] > > > > > > > > ``` > > > > > > > ``` > > > > for the network.bind_host I tried eth0:ipv4 but this gives > > > > me host > > > > notfound exception. > > > > > > > > ``` > > > > > > > ``` > > > > Any ideas? Can someone upload an elasticsearch.yml file > > > > able to do ec2 > > > > discovery for him with the 16.2 version? > > > > > > > > ``` > > > > > > > ``` > > > > On Jul 6, 2:40 pm, Ridvan Gyundogan > > > (http://gmail.com)> wrote: > > > > > > > > ``` > > > > > > > > > ``` > > > > > Ah ok thanks I will try. > > > > > > > > > > ``` > > > > > > > ``` > > > > On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley > > > > wrote: > > > > > > > > ``` > > > > > > > ``` > > > > I start only 1 node at the moment so sure it > > > > should be the master. > > > > > > > > ``` > > > > > > > ``` > > > > By master, Shay means the version of ElasticSearch > > > > that is currently > > > > master on github, not a released version. > > > > > > > > ``` > > > > > > > ``` > > > > You need to download and compile it yourself: > > > > > > > > ``` > > > > > > > ``` > > > > http://www.elasticsearch.org/download/master/ > > > > > > > > ``` > > > > > > > ``` > > > > clint > > > > > > > > ``` ---
**Author:** ![Ridvan\_Gyundogan](https://avatars.discourse-cdn.com/v4/letter/r/e495f1/32.png) [@Ridvan\_Gyundogan](https://discuss.elastic.co/u/Ridvan_Gyundogan)\ **Post date:** [July 8, 2011, 6:14am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/23 "2011-07-08T06:14:50Z")
Hi Pavel, thanks a lot for the hint. Can you probably share your working elasticsearch.yml? On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev [pavel.penchev@gmail.com](mailto:pavel.penchev@gmail.com)wrote: > Hi all, > > maybe related to this discussion - I just got ES working in amazon and > there was one key point, the "cloud.aws.region" setting was not optional in > my case. Without it Amazon was returning an empty list of instances. > The > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.htmlpage) implies that one could skip this setting. > > Regards, > Pavel > > On 7.07.2011 17:49, Shay Banon wrote: > > Hey, > > First, it seems like all the logging is DEBUG mode, thats noisy, all > you need is discovery set to TRACE. > > ``` > If you see, on both nodes, the output: "using dynamic discovery nodes" > > ``` > > is empty, meaning that it went ahead and queries aws for the list of nodes, > and got none... . Maybe the groups / tags are not correct? > > -shay.banon > > On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote: > > Shay, > here is the gist: > [master · GitHub](https://gist.github.com/1069376) > > On Jul 6, 10:35 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > Can you set logging for discovery to TRACE in the logging file and gist it? > > On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote: > > Thanks Shay. > Now it is bit better, at least I do not have an exception but still > the ec2 instances do not run as a cluster. > What I additionally did is the following. Enabled ping for the > security group with the ec2-authorise. Tried that the 2 machines can > ping each other. > Checked with telnet that the 2 machines can open each other's 9200 and > 9300 ports. The 9301 - 9303 ports are open too but the other ports are > closed by default. > These are the 2 log files I have. First the master and next the one > which had to become a slave.At the bottom is the elasticsearch.yml > config file I use. > Any ideas what I need to change? > > master: > [2011-07-06 18:14:42,458][INFO][node] [Jimmy > Woo] {elasticsearch/0.16.2}[2142]: initializing ... > [2011-07-06 18:14:42,468][INFO][plugins] [Jimmy > Woo] loaded [cloud-aws] > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > Woo] {elasticsearch/0.16.2}[2142]: initialized > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > Woo] {elasticsearch/0.16.2}[2142]: starting ... > [2011-07-06 18:14:48,414][INFO][transport] [Jimmy > Woo] bound\_address {inet[/10.224.50.50:9300]}, publish\_address {inet[/ > 10.224.50.50:9300]} > [2011-07-06 18:14:52,797][INFO][cluster.service] [Jimmy > Woo] new\_master [Jimmy Woo][T\_0Y3f\_UQJmfuiqYYW7Ivg][inet[/ > 10.224.50.50:9300]], reason: zen-disco-join (elected\_as\_master) > [2011-07-06 18:14:52,845][INFO][discovery] [Jimmy > Woo] perftests/T\_0Y3f\_UQJmfuiqYYW7Ivg > [2011-07-06 18:14:52,893][INFO][gateway] [Jimmy > Woo] recovered [0] indices into cluster\_state > [2011-07-06 18:14:52,894][INFO][http] [Jimmy > Woo] bound\_address {inet[/10.224.50.50:9200]}, publish\_address {inet[/ > 10.224.50.50:9200]} > [2011-07-06 18:14:52,895][INFO][node] [Jimmy > Woo] {elasticsearch/0.16.2}[2142]: started > > slave: > [2011-07-06 18:18:07,733][INFO][node] [Machine > Teen] {elasticsearch/0.16.2}[1251]: initializing ... > [2011-07-06 18:18:07,741][INFO][plugins] [Machine > Teen] loaded [cloud-aws] > [2011-07-06 18:18:11,994][INFO][node] [Machine > Teen] {elasticsearch/0.16.2}[1251]: initialized > [2011-07-06 18:18:11,995][INFO][node] [Machine > Teen] {elasticsearch/0.16.2}[1251]: starting ... > [2011-07-06 18:18:12,105][INFO][transport] [Machine > Teen] bound\_address {inet[/10.234.227.72:9300]}, publish\_address > {inet[/10.234.227.72:9300]} > [2011-07-06 18:18:16,242][INFO][cluster.service] [Machine > Teen] new\_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ\_IA][inet[/ > 10.234.227.72:9300]], reason: zen-disco-join (elected\_as\_master) > [2011-07-06 18:18:16,278][INFO][discovery] [Machine > Teen] perftests/4vSbOGJqRx6q3Vi5HrJ\_IA > [2011-07-06 18:18:16,305][INFO][gateway] [Machine > Teen] recovered [0] indices into cluster\_state > [2011-07-06 18:18:16,312][INFO][http] [Machine > Teen] bound\_address {inet[/10.234.227.72:9200]}, publish\_address > {inet[/10.234.227.72:9200]} > [2011-07-06 18:18:16,313][INFO][node] [Machine > Teen] {elasticsearch/0.16.2}[1251]: started > > elasticsearch.yml: > cluster: > name: perftests > > discovery: > type: ec2 > zen: > ping\_timeout: 30s > > cloud: > account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID", > not the "Account Number" > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > compute: > type: amazon > > network: > bind\_host: _eth0:ipv4_ > > On Jul 6, 7:16 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ( > [http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > You need to set the discovery type to ec2: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html). > The blog post is outdated... . > > And you don't have to use master to have discovery working on ec2. You > can set network.host to whatever value you want, for example, the internal > IP address. If you want teh same config across nodes, you can bind to the > network interface name ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/network.html)), for > example: _en0:ipv4_. > > On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote: > > Unfortunately using the trunk is not an option at the moment because > there are some maven dependencies to the 16.2 jar in our project. > We will upgrade to 0.17 once it is officially released. > > Now I tried the "Zen" as described here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2010/05/11/here-comes-the-cloud.html) > > my elasticsearch.yml on the slave looks like this: > cluster: > name: perftests > > discovery: > type: cloud > zen: > ping\_timeout: 30s > cloud: > account: 551234567890 > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > compute: > type: amazon > network: > bind\_host: _non\_loopback:ipv4_ > > On this I get the exception: > {elasticsearch/0.16.2}: Initialization Failed ... > > - NoClassSettingsException[Failed to load class setting > [discovery.type] with value [cloud]] > ClassNotFoundException[cloud] > > Ah ok thanks I will try. > > On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk)( > [http://iannounce.co.uk](http://iannounce.co.uk))\>wrote: > > I start only 1 node at the moment so sure it should be the master. > > By master, Shay means the version of Elasticsearch that is currently > master on github, not a released version. > > You need to download and compile it yourself: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/) > > clint ---
**Author:** ![Ridvan\_Gyundogan](https://avatars.discourse-cdn.com/v4/letter/r/e495f1/32.png) [@Ridvan\_Gyundogan](https://discuss.elastic.co/u/Ridvan_Gyundogan)\ **Post date:** [July 8, 2011, 6:49am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/24 "2011-07-08T06:49:11Z")
Pavel you are right the cloud.aws.region was the key point. Here my elasticsearch.yml in case someone else needs this too: cluster: name: perftests discovery: type: ec2 zen: ping\_timeout: 30s cloud: account: AAAAAAAAAAAAA key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx compute: type: amazon aws: region: eu-west-1 network: bind\_host: _eth0:ipv4_ On Fri, Jul 8, 2011 at 9:14 AM, Ridvan Gyundogan [ridvansg@gmail.com](mailto:ridvansg@gmail.com) wrote: > Hi Pavel, > thanks a lot for the hint. > > Can you probably share your working elasticsearch.yml? > > On Thu, Jul 7, 2011 at 6:41 PM, Pavel Penchev [pavel.penchev@gmail.com](mailto:pavel.penchev@gmail.com)wrote: > > > Hi all, > > > > maybe related to this discussion - I just got ES working in amazon and > > there was one key point, the "cloud.aws.region" setting was not optional in > > my case. Without it Amazon was returning an empty list of instances. > > The > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.htmlpage) implies that one could skip this setting. > > > > Regards, > > Pavel > > > > On 7.07.2011 17:49, Shay Banon wrote: > > > > Hey, > > > > First, it seems like all the logging is DEBUG mode, thats noisy, all > > you need is discovery set to TRACE. > > > > ``` > > If you see, on both nodes, the output: "using dynamic discovery > > > > ``` > > > > nodes" is empty, meaning that it went ahead and queries aws for the list of > > nodes, and got none... . Maybe the groups / tags are not correct? > > > > -shay.banon > > > > On Thursday, July 7, 2011 at 3:10 PM, Ridvan Gyundogan wrote: > > > > Shay, > > here is the gist: > > [master · GitHub](https://gist.github.com/1069376) > > > > On Jul 6, 10:35 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > Can you set logging for discovery to TRACE in the logging file and gist > > it? > > > > On Wednesday, July 6, 2011 at 9:35 PM, Ridvan Gyundogan wrote: > > > > Thanks Shay. > > Now it is bit better, at least I do not have an exception but still > > the ec2 instances do not run as a cluster. > > What I additionally did is the following. Enabled ping for the > > security group with the ec2-authorise. Tried that the 2 machines can > > ping each other. > > Checked with telnet that the 2 machines can open each other's 9200 and > > 9300 ports. The 9301 - 9303 ports are open too but the other ports are > > closed by default. > > These are the 2 log files I have. First the master and next the one > > which had to become a slave.At the bottom is the elasticsearch.yml > > config file I use. > > Any ideas what I need to change? > > > > master: > > [2011-07-06 18:14:42,458][INFO][node] [Jimmy > > Woo] {elasticsearch/0.16.2}[2142]: initializing ... > > [2011-07-06 18:14:42,468][INFO][plugins] [Jimmy > > Woo] loaded [cloud-aws] > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > Woo] {elasticsearch/0.16.2}[2142]: initialized > > [2011-07-06 18:14:48,272][INFO][node] [Jimmy > > Woo] {elasticsearch/0.16.2}[2142]: starting ... > > [2011-07-06 18:14:48,414][INFO][transport] [Jimmy > > Woo] bound\_address {inet[/10.224.50.50:9300]}, publish\_address {inet[/ > > 10.224.50.50:9300]} > > [2011-07-06 18:14:52,797][INFO][cluster.service] [Jimmy > > Woo] new\_master [Jimmy Woo][T\_0Y3f\_UQJmfuiqYYW7Ivg][inet[/ > > 10.224.50.50:9300]], reason: zen-disco-join (elected\_as\_master) > > [2011-07-06 18:14:52,845][INFO][discovery] [Jimmy > > Woo] perftests/T\_0Y3f\_UQJmfuiqYYW7Ivg > > [2011-07-06 18:14:52,893][INFO][gateway] [Jimmy > > Woo] recovered [0] indices into cluster\_state > > [2011-07-06 18:14:52,894][INFO][http] [Jimmy > > Woo] bound\_address {inet[/10.224.50.50:9200]}, publish\_address {inet[/ > > 10.224.50.50:9200]} > > [2011-07-06 18:14:52,895][INFO][node] [Jimmy > > Woo] {elasticsearch/0.16.2}[2142]: started > > > > slave: > > [2011-07-06 18:18:07,733][INFO][node] [Machine > > Teen] {elasticsearch/0.16.2}[1251]: initializing ... > > [2011-07-06 18:18:07,741][INFO][plugins] [Machine > > Teen] loaded [cloud-aws] > > [2011-07-06 18:18:11,994][INFO][node] [Machine > > Teen] {elasticsearch/0.16.2}[1251]: initialized > > [2011-07-06 18:18:11,995][INFO][node] [Machine > > Teen] {elasticsearch/0.16.2}[1251]: starting ... > > [2011-07-06 18:18:12,105][INFO][transport] [Machine > > Teen] bound\_address {inet[/10.234.227.72:9300]}, publish\_address > > {inet[/10.234.227.72:9300]} > > [2011-07-06 18:18:16,242][INFO][cluster.service] [Machine > > Teen] new\_master [Machine Teen][4vSbOGJqRx6q3Vi5HrJ\_IA][inet[/ > > 10.234.227.72:9300]], reason: zen-disco-join (elected\_as\_master) > > [2011-07-06 18:18:16,278][INFO][discovery] [Machine > > Teen] perftests/4vSbOGJqRx6q3Vi5HrJ\_IA > > [2011-07-06 18:18:16,305][INFO][gateway] [Machine > > Teen] recovered [0] indices into cluster\_state > > [2011-07-06 18:18:16,312][INFO][http] [Machine > > Teen] bound\_address {inet[/10.234.227.72:9200]}, publish\_address > > {inet[/10.234.227.72:9200]} > > [2011-07-06 18:18:16,313][INFO][node] [Machine > > Teen] {elasticsearch/0.16.2}[1251]: started > > > > elasticsearch.yml: > > cluster: > > name: perftests > > > > discovery: > > type: ec2 > > zen: > > ping\_timeout: 30s > > > > cloud: > > account: ABCDEFGIJKLAAAAAAAAAA #here I use the "Access Key ID", > > not the "Account Number" > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > compute: > > type: amazon > > > > network: > > bind\_host: _eth0:ipv4_ > > > > On Jul 6, 7:16 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ( > > [http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > You need to set the discovery type to ec2: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/ec2.html). > > The blog post is outdated... . > > > > And you don't have to use master to have discovery working on ec2. You > > can set network.host to whatever value you want, for example, the internal > > IP address. If you want teh same config across nodes, you can bind to the > > network interface name ( > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/network.html)), for > > example: _en0:ipv4_. > > > > On Wednesday, July 6, 2011 at 4:34 PM, Ridvan Gyundogan wrote: > > > > Unfortunately using the trunk is not an option at the moment because > > there are some maven dependencies to the 16.2 jar in our project. > > We will upgrade to 0.17 once it is officially released. > > > > Now I tried the "Zen" as described here: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2010/05/11/here-comes-the-cloud.html) > > > > my elasticsearch.yml on the slave looks like this: > > cluster: > > name: perftests > > > > discovery: > > type: cloud > > zen: > > ping\_timeout: 30s > > cloud: > > account: 551234567890 > > key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > compute: > > type: amazon > > network: > > bind\_host: _non\_loopback:ipv4_ > > > > On this I get the exception: > > {elasticsearch/0.16.2}: Initialization Failed ... > > > > - NoClassSettingsException[Failed to load class setting > > [discovery.type] with value [cloud]] > > ClassNotFoundException[cloud] > > > > Ah ok thanks I will try. > > > > On Wed, Jul 6, 2011 at 2:37 PM, Clinton Gormley \< > > [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk))\>wrote: > > > > I start only 1 node at the moment so sure it should be the master. > > > > By master, Shay means the version of Elasticsearch that is currently > > master on github, not a released version. > > > > You need to download and compile it yourself: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/) > > > > clint ---
**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\ **Post date:** [July 8, 2011, 12:42pm UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/25 "2011-07-08T12:42:46Z")
Two quick and minor addenda to this discussion 1] I have a working ec2 cluster without having set the "cloud.aws.region" parameter. (I'm in us-east-1, not sure if that makes any difference.) Though I suppose I'll add it now! (I also didn't need the "network.bind\_host") 2] From a previous EC2 configuration discussion (http:// [groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9), I think the "ping\_timeout" parameter you're looking for is "discovery.ec2.ping\_timeout". ---
**Author:** ![Ridvan\_Gyundogan](https://avatars.discourse-cdn.com/v4/letter/r/e495f1/32.png) [@Ridvan\_Gyundogan](https://discuss.elastic.co/u/Ridvan_Gyundogan)\ **Post date:** [July 9, 2011, 8:42am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/26 "2011-07-09T08:42:03Z")
Hi Alex, thanks for the notes. 1. It was required for the europe cloud. 2. Thanks I added it to my file and I see it is waiting 30s. I se some other differences between the 2 elasticsearch.yml files. For example you use cloud.aws.access\_key and In mine it is cloud.account. On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow [apiggott@ikanow.com](mailto:apiggott@ikanow.com) wrote: > Two quick and minor addenda to this discussion > > 1] I have a working ec2 cluster without having set the > "cloud.aws.region" parameter. (I'm in us-east-1, not sure if that > makes any difference.) Though I suppose I'll add it now! > > (I also didn't need the "network.bind\_host") > > 2] From a previous EC2 configuration discussion (http:// > [groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) > 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9), I > think the "ping\_timeout" parameter you're looking for is > "discovery.ec2.ping\_timeout". ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 10, 2011, 12:31am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/27 "2011-07-10T00:31:11Z")
Probably the region is hte problem then, if you set it to the euro one, does it work then? Set cloud.aws.region to eu-west. The cloud.aws.account is a global setting that defines the account access key across the different aws services used in ES. Its the preferable way to set it, though one can set explicitly the ec2 access key if needed. On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote: > Hi Alex, thanks for the notes. > > 1. It was required for the europe cloud. > 2. Thanks I added it to my file and I see it is waiting 30s. > > I se some other differences between the 2 elasticsearch.yml files. > For example you use cloud.aws.access\_key and In mine it is cloud.account. > > On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow \<[apiggott@ikanow.com](mailto:apiggott@ikanow.com) ([mailto:apiggott@ikanow.com](mailto:apiggott@ikanow.com))\> wrote: > > > Two quick and minor addenda to this discussion > > > > 1] I have a working ec2 cluster without having set the > > "cloud.aws.region" parameter. (I'm in us-east-1, not sure if that > > makes any difference.) Though I suppose I'll add it now! > > > > (I also didn't need the "network.bind\_host") > > > > 2] From a previous EC2 configuration discussion (http:// > > [groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) > > 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9 ([http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) > > 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9)), I > > think the "ping\_timeout" parameter you're looking for is > > "discovery.ec2.ping\_timeout". ---
**Author:** ![Ridvan\_Gyundogan](https://avatars.discourse-cdn.com/v4/letter/r/e495f1/32.png) [@Ridvan\_Gyundogan](https://discuss.elastic.co/u/Ridvan_Gyundogan)\ **Post date:** [July 10, 2011, 8:05am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/28 "2011-07-10T08:05:04Z")
Yes with cloud.aws.region = eu-west On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Probably the region is hte problem then, if you set it to the euro one, > does it work then? Set cloud.aws.region to eu-west. > > The cloud.aws.account is a global setting that defines the account access > key across the different aws services used in ES. Its the preferable way to > set it, though one can set explicitly the ec2 access key if needed. > > On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote: > > Hi Alex, thanks for the notes. > > 1. It was required for the europe cloud. > 2. Thanks I added it to my file and I see it is waiting 30s. > > I se some other differences between the 2 elasticsearch.yml files. > For example you use cloud.aws.access\_key and In mine it is cloud.account. > > On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow [apiggott@ikanow.com](mailto:apiggott@ikanow.com)wrote: > > Two quick and minor addenda to this discussion > > 1] I have a working ec2 cluster without having set the > "cloud.aws.region" parameter. (I'm in us-east-1, not sure if that > makes any difference.) Though I suppose I'll add it now! > > (I also didn't need the "network.bind\_host") > > 2] From a previous EC2 configuration discussion (http:// > [groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) > 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9[http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9)), > I > think the "ping\_timeout" parameter you're looking for is > "discovery.ec2.ping\_timeout". ---
**Author:** ![Ridvan\_Gyundogan](https://avatars.discourse-cdn.com/v4/letter/r/e495f1/32.png) [@Ridvan\_Gyundogan](https://discuss.elastic.co/u/Ridvan_Gyundogan)\ **Post date:** [July 10, 2011, 8:06am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/29 "2011-07-10T08:06:49Z")
Sorry, I mean with cloud.aws.region = eu-west-1 it works. On Sun, Jul 10, 2011 at 11:05 AM, Ridvan Gyundogan [ridvansg@gmail.com](mailto:ridvansg@gmail.com)wrote: > Yes with cloud.aws.region = eu-west > > On Sun, Jul 10, 2011 at 3:31 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Probably the region is hte problem then, if you set it to the euro one, > > does it work then? Set cloud.aws.region to eu-west. > > > > The cloud.aws.account is a global setting that defines the account access > > key across the different aws services used in ES. Its the preferable way to > > set it, though one can set explicitly the ec2 access key if needed. > > > > On Saturday, July 9, 2011 at 11:42 AM, Ridvan Gyundogan wrote: > > > > Hi Alex, thanks for the notes. > > > > 1. It was required for the europe cloud. > > 2. Thanks I added it to my file and I see it is waiting 30s. > > > > I se some other differences between the 2 elasticsearch.yml files. > > For example you use cloud.aws.access\_key and In mine it is > > cloud.account. > > > > On Fri, Jul 8, 2011 at 3:42 PM, Alex at Ikanow [apiggott@ikanow.com](mailto:apiggott@ikanow.com)wrote: > > > > Two quick and minor addenda to this discussion > > > > 1] I have a working ec2 cluster without having set the > > "cloud.aws.region" parameter. (I'm in us-east-1, not sure if that > > makes any difference.) Though I suppose I'll add it now! > > > > (I also didn't need the "network.bind\_host") > > > > 2] From a previous EC2 configuration discussion (http:// > > [groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/) > > 2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9[http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show\_docid=ee243fa6ccf41ca9](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/2f824abaffacb3c1/ee243fa6ccf41ca9?show_docid=ee243fa6ccf41ca9)), > > I > > think the "ping\_timeout" parameter you're looking for is > > "discovery.ec2.ping\_timeout". ---
**Author:** ![Yaron\_Idan](https://avatars.discourse-cdn.com/v4/letter/y/ea5d25/32.png) [@Yaron\_Idan](https://discuss.elastic.co/u/Yaron_Idan)\ **Post date:** [July 13, 2016, 11:28am UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/30 "2016-07-13T11:28:47Z")
Thanks Pavel, You've really helped me out with this one, I've been busting my head on it for a few days but adding the region parameter did the trick. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:36pm UTC](https://discuss.elastic.co/t/ec2-discovery-issue/4772/31 "2017-07-05T22:36:06Z")
[Previous page](https://discuss.elastic.co/t/ec2-discovery-issue/4772.md?page=1)