# Elasticsearch not dynamically discovering ec2 instances **URL:** \ **Category:** Elasticsearch\ **Created:** [December 29, 2013, 5:41pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048 "2013-12-29T17:41:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Jay\_Zalowitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_zalowitz/32/1793_2.png) [@Jay\_Zalowitz](https://discuss.elastic.co/u/Jay_Zalowitz)\ **Post date:** [December 29, 2013, 5:41pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/1 "2013-12-29T17:41:20Z")
I am currently trying to set up an elasticsearch ami that looks across a specific security groups instances and trys to add itself to the cluster, as I need to be able to grow quickly as I run out of memory and speed up requests as we launch. ES is not reaching out to instances in the security group, any advice? elasticsearch.yaml ``` cluster.name: fetch cloud.aws.access_key: hidden cloud.aws.secret_key: hidden discovery.type: ec2 discovery.ec2.ping_timeout: 1m discovery.zen.ping_timeout: 1m discovery.ec2.groups: ElasticSearch gateway.type: s3 gateway.s3.bucket: bucketname gateway.recover_after_nodes: 2 gateway.recover_after_time: 1m ``` Log: ``` [2013-12-29 04:40:02,245][INFO][node] [Tutinax the Mountain-Mover] version[0.90.9], pid[1287], build[a968646/2013-12-23T10:35:28Z] [2013-12-29 04:40:02,246][INFO][node] [Tutinax the Mountain-Mover] initializing ... [2013-12-29 04:40:02,261][INFO][plugins] [Tutinax the Mountain-Mover] loaded [cloud-aws], sites [] [2013-12-29 04:40:05,738][WARN][gateway.s3] [Tutinax the Mountain-Mover] shared gateway has been deprecated, please use the (default) local gateway [2013-12-29 04:40:05,738][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] using bucket [bucketname], region [null], chunk_size [100mb], concurrent_streams [5] [2013-12-29 04:40:06,103][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] Latest metadata found at index [2] [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] initialized [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] starting ... [2013-12-29 04:40:06,278][INFO][transport] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.207.xx.xx:9301]} [2013-12-29 04:40:36,288][WARN][discovery] [Tutinax the Mountain-Mover] waited for 30s and no initial state was set by the discovery [2013-12-29 04:40:36,288][INFO][discovery] [Tutinax the Mountain-Mover] fetch/Eb5OByyGTsuwLrtRbJNGjw [2013-12-29 04:40:36,288][DEBUG][gateway] [Tutinax the Mountain-Mover] can't wait on start for (possibly) reading state from gateway, will do it asynchronously [2013-12-29 04:40:36,294][INFO][http] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.207.xx.xx:9201]} [2013-12-29 04:40:36,295][INFO][node] [Tutinax the Mountain-Mover] started ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f83873a7-804d-4c99-964f-3b6353ee9849%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f83873a7-804d-4c99-964f-3b6353ee9849%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Frederick\_Cheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frederick_cheung/32/1874_2.png) [@Frederick\_Cheung](https://discuss.elastic.co/u/Frederick_Cheung)\ **Post date:** [December 29, 2013, 6:36pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/2 "2013-12-29T18:36:04Z")
On Sunday, December 29, 2013 5:41:20 PM UTC, Jay Zalowitz wrote: > I am currently trying to set up an elasticsearch ami that looks across a > specific security groups instances and trys to add itself to the cluster, > as I need to be able to grow quickly as I run out of memory and speed up > requests as we launch. > > ES is not reaching out to instances in the security group, any advice? Have you set the EC2 region? Fred > elasticsearch.yaml > > ``` > cluster.name: fetch > cloud.aws.access_key: hidden > cloud.aws.secret_key: hidden > discovery.type: ec2 > discovery.ec2.ping_timeout: 1m > discovery.zen.ping_timeout: 1m > discovery.ec2.groups: ElasticSearch > gateway.type: s3 > gateway.s3.bucket: bucketname > gateway.recover_after_nodes: 2 > gateway.recover_after_time: 1m > > ``` > > Log: > > ``` > [2013-12-29 04:40:02,245][INFO][node] [Tutinax the Mountain-Mover] version[0.90.9], pid[1287], build[a968646/2013-12-23T10:35:28Z] > [2013-12-29 04:40:02,246][INFO][node] [Tutinax the Mountain-Mover] initializing ... > [2013-12-29 04:40:02,261][INFO][plugins] [Tutinax the Mountain-Mover] loaded [cloud-aws], sites [] > [2013-12-29 04:40:05,738][WARN][gateway.s3] [Tutinax the Mountain-Mover] shared gateway has been deprecated, please use the (default) local gateway > [2013-12-29 04:40:05,738][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] using bucket [bucketname], region [null], chunk_size [100mb], concurrent_streams [5] > [2013-12-29 04:40:06,103][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] Latest metadata found at index [2] > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] initialized > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] starting ... > [2013-12-29 04:40:06,278][INFO][transport] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.207.xx.xx:9301]} > [2013-12-29 04:40:36,288][WARN][discovery] [Tutinax the Mountain-Mover] waited for 30s and no initial state was set by the discovery > [2013-12-29 04:40:36,288][INFO][discovery] [Tutinax the Mountain-Mover] fetch/Eb5OByyGTsuwLrtRbJNGjw > [2013-12-29 04:40:36,288][DEBUG][gateway] [Tutinax the Mountain-Mover] can't wait on start for (possibly) reading state from gateway, will do it asynchronously > [2013-12-29 04:40:36,294][INFO][http] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.207.xx.xx:9201]} > [2013-12-29 04:40:36,295][INFO][node] [Tutinax the Mountain-Mover] started > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f22be1db-6d26-4402-b46a-9e72396cf153%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f22be1db-6d26-4402-b46a-9e72396cf153%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jay\_Zalowitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_zalowitz/32/1793_2.png) [@Jay\_Zalowitz](https://discuss.elastic.co/u/Jay_Zalowitz)\ **Post date:** [December 29, 2013, 6:48pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/3 "2013-12-29T18:48:17Z")
Is that a requirement for it to work? Otherwise id rather default to the security group. On Sunday, December 29, 2013 10:36:04 AM UTC-8, Frederick Cheung wrote: > On Sunday, December 29, 2013 5:41:20 PM UTC, Jay Zalowitz wrote: > > > I am currently trying to set up an elasticsearch ami that looks across a > > specific security groups instances and trys to add itself to the cluster, > > as I need to be able to grow quickly as I run out of memory and speed up > > requests as we launch. > > > > ES is not reaching out to instances in the security group, any advice? > > Have you set the EC2 region? > > Fred > > > elasticsearch.yaml > > > > ``` > > cluster.name: fetch > > cloud.aws.access_key: hidden > > cloud.aws.secret_key: hidden > > discovery.type: ec2 > > discovery.ec2.ping_timeout: 1m > > discovery.zen.ping_timeout: 1m > > discovery.ec2.groups: ElasticSearch > > gateway.type: s3 > > gateway.s3.bucket: bucketname > > gateway.recover_after_nodes: 2 > > gateway.recover_after_time: 1m > > > > ``` > > > > Log: > > > > ``` > > [2013-12-29 04:40:02,245][INFO][node] [Tutinax the Mountain-Mover] version[0.90.9], pid[1287], build[a968646/2013-12-23T10:35:28Z] > > [2013-12-29 04:40:02,246][INFO][node] [Tutinax the Mountain-Mover] initializing ... > > [2013-12-29 04:40:02,261][INFO][plugins] [Tutinax the Mountain-Mover] loaded [cloud-aws], sites [] > > [2013-12-29 04:40:05,738][WARN][gateway.s3] [Tutinax the Mountain-Mover] shared gateway has been deprecated, please use the (default) local gateway > > [2013-12-29 04:40:05,738][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] using bucket [bucketname], region [null], chunk_size [100mb], concurrent_streams [5] > > [2013-12-29 04:40:06,103][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] Latest metadata found at index [2] > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] initialized > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] starting ... > > [2013-12-29 04:40:06,278][INFO][transport] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.207.xx.xx:9301]} > > [2013-12-29 04:40:36,288][WARN][discovery] [Tutinax the Mountain-Mover] waited for 30s and no initial state was set by the discovery > > [2013-12-29 04:40:36,288][INFO][discovery] [Tutinax the Mountain-Mover] fetch/Eb5OByyGTsuwLrtRbJNGjw > > [2013-12-29 04:40:36,288][DEBUG][gateway] [Tutinax the Mountain-Mover] can't wait on start for (possibly) reading state from gateway, will do it asynchronously > > [2013-12-29 04:40:36,294][INFO][http] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.207.xx.xx:9201]} > > [2013-12-29 04:40:36,295][INFO][node] [Tutinax the Mountain-Mover] started > > > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3b95b9d8-30af-4cf8-a5da-cb290743583a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3b95b9d8-30af-4cf8-a5da-cb290743583a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Frederick\_Cheung](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frederick_cheung/32/1874_2.png) [@Frederick\_Cheung](https://discuss.elastic.co/u/Frederick_Cheung)\ **Post date:** [December 29, 2013, 7:29pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/4 "2013-12-29T19:29:14Z")
On Sunday, December 29, 2013 6:48:17 PM UTC, Jay Zalowitz wrote: > Is that a requirement for it to work? Otherwise id rather default to the > security group. I believe so - ec2 regions are almost entirely separate from each other- for example you can have identically named groups in multiple regions Fred > On Sunday, December 29, 2013 10:36:04 AM UTC-8, Frederick Cheung wrote: > > > On Sunday, December 29, 2013 5:41:20 PM UTC, Jay Zalowitz wrote: > > > > > I am currently trying to set up an elasticsearch ami that looks across a > > > specific security groups instances and trys to add itself to the cluster, > > > as I need to be able to grow quickly as I run out of memory and speed up > > > requests as we launch. > > > > > > ES is not reaching out to instances in the security group, any advice? > > > > Have you set the EC2 region? > > > > Fred > > > > > elasticsearch.yaml > > > > > > ``` > > > cluster.name: fetch > > > cloud.aws.access_key: hidden > > > cloud.aws.secret_key: hidden > > > discovery.type: ec2 > > > discovery.ec2.ping_timeout: 1m > > > discovery.zen.ping_timeout: 1m > > > discovery.ec2.groups: ElasticSearch > > > gateway.type: s3 > > > gateway.s3.bucket: bucketname > > > gateway.recover_after_nodes: 2 > > > gateway.recover_after_time: 1m > > > > > > ``` > > > > > > Log: > > > > > > ``` > > > [2013-12-29 04:40:02,245][INFO][node] [Tutinax the Mountain-Mover] version[0.90.9], pid[1287], build[a968646/2013-12-23T10:35:28Z] > > > [2013-12-29 04:40:02,246][INFO][node] [Tutinax the Mountain-Mover] initializing ... > > > [2013-12-29 04:40:02,261][INFO][plugins] [Tutinax the Mountain-Mover] loaded [cloud-aws], sites [] > > > [2013-12-29 04:40:05,738][WARN][gateway.s3] [Tutinax the Mountain-Mover] shared gateway has been deprecated, please use the (default) local gateway > > > [2013-12-29 04:40:05,738][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] using bucket [bucketname], region [null], chunk_size [100mb], concurrent_streams [5] > > > [2013-12-29 04:40:06,103][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] Latest metadata found at index [2] > > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] initialized > > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] starting ... > > > [2013-12-29 04:40:06,278][INFO][transport] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.207.xx.xx:9301]} > > > [2013-12-29 04:40:36,288][WARN][discovery] [Tutinax the Mountain-Mover] waited for 30s and no initial state was set by the discovery > > > [2013-12-29 04:40:36,288][INFO][discovery] [Tutinax the Mountain-Mover] fetch/Eb5OByyGTsuwLrtRbJNGjw > > > [2013-12-29 04:40:36,288][DEBUG][gateway] [Tutinax the Mountain-Mover] can't wait on start for (possibly) reading state from gateway, will do it asynchronously > > > [2013-12-29 04:40:36,294][INFO][http] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.207.xx.xx:9201]} > > > [2013-12-29 04:40:36,295][INFO][node] [Tutinax the Mountain-Mover] started > > > > > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3eb48964-e61f-4744-b9bc-4e6756529122%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3eb48964-e61f-4744-b9bc-4e6756529122%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jay\_Zalowitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_zalowitz/32/1793_2.png) [@Jay\_Zalowitz](https://discuss.elastic.co/u/Jay_Zalowitz)\ **Post date:** [December 29, 2013, 8:31pm UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/5 "2013-12-29T20:31:55Z")
That didnt help.. I went all the way down to just: cluster.name: fetch cloud.aws.access\_key: hidden cloud.aws.secret\_key: hidden discovery.type: ec2 and it still didn't work. On Sunday, December 29, 2013 11:29:14 AM UTC-8, Frederick Cheung wrote: > On Sunday, December 29, 2013 6:48:17 PM UTC, Jay Zalowitz wrote: > > > Is that a requirement for it to work? Otherwise id rather default to the > > security group. > > I believe so - ec2 regions are almost entirely separate from each other- > for example you can have identically named groups in multiple regions > > Fred > > > On Sunday, December 29, 2013 10:36:04 AM UTC-8, Frederick Cheung wrote: > > > > > On Sunday, December 29, 2013 5:41:20 PM UTC, Jay Zalowitz wrote: > > > > > > > I am currently trying to set up an elasticsearch ami that looks across > > > > a specific security groups instances and trys to add itself to the cluster, > > > > as I need to be able to grow quickly as I run out of memory and speed up > > > > requests as we launch. > > > > > > > > ES is not reaching out to instances in the security group, any advice? > > > > > > Have you set the EC2 region? > > > > > > Fred > > > > > > > elasticsearch.yaml > > > > > > > > ``` > > > > cluster.name: fetch > > > > cloud.aws.access_key: hidden > > > > cloud.aws.secret_key: hidden > > > > discovery.type: ec2 > > > > discovery.ec2.ping_timeout: 1m > > > > discovery.zen.ping_timeout: 1m > > > > discovery.ec2.groups: ElasticSearch > > > > gateway.type: s3 > > > > gateway.s3.bucket: bucketname > > > > gateway.recover_after_nodes: 2 > > > > gateway.recover_after_time: 1m > > > > > > > > ``` > > > > > > > > Log: > > > > > > > > ``` > > > > [2013-12-29 04:40:02,245][INFO][node] [Tutinax the Mountain-Mover] version[0.90.9], pid[1287], build[a968646/2013-12-23T10:35:28Z] > > > > [2013-12-29 04:40:02,246][INFO][node] [Tutinax the Mountain-Mover] initializing ... > > > > [2013-12-29 04:40:02,261][INFO][plugins] [Tutinax the Mountain-Mover] loaded [cloud-aws], sites [] > > > > [2013-12-29 04:40:05,738][WARN][gateway.s3] [Tutinax the Mountain-Mover] shared gateway has been deprecated, please use the (default) local gateway > > > > [2013-12-29 04:40:05,738][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] using bucket [bucketname], region [null], chunk_size [100mb], concurrent_streams [5] > > > > [2013-12-29 04:40:06,103][DEBUG][gateway.s3] [Tutinax the Mountain-Mover] Latest metadata found at index [2] > > > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] initialized > > > > [2013-12-29 04:40:06,120][INFO][node] [Tutinax the Mountain-Mover] starting ... > > > > [2013-12-29 04:40:06,278][INFO][transport] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.207.xx.xx:9301]} > > > > [2013-12-29 04:40:36,288][WARN][discovery] [Tutinax the Mountain-Mover] waited for 30s and no initial state was set by the discovery > > > > [2013-12-29 04:40:36,288][INFO][discovery] [Tutinax the Mountain-Mover] fetch/Eb5OByyGTsuwLrtRbJNGjw > > > > [2013-12-29 04:40:36,288][DEBUG][gateway] [Tutinax the Mountain-Mover] can't wait on start for (possibly) reading state from gateway, will do it asynchronously > > > > [2013-12-29 04:40:36,294][INFO][http] [Tutinax the Mountain-Mover] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/10.207.xx.xx:9201]} > > > > [2013-12-29 04:40:36,295][INFO][node] [Tutinax the Mountain-Mover] started > > > > > > > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8d106950-67be-41a7-bc08-9b65ae733031%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8d106950-67be-41a7-bc08-9b65ae733031%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-not-dynamically-discovering-ec2-instances/15048/6 "2017-07-06T01:59:06Z")