# Elasticsearch cluster on AWS. Article **URL:** \ **Category:** Elasticsearch\ **Created:** [August 14, 2014, 5:13pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292 "2014-08-14T17:13:33Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Pavel\_P](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavel_p/32/1092_2.png) [@Pavel\_P](https://discuss.elastic.co/u/Pavel_P)\ **Post date:** [August 14, 2014, 5:13pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/1 "2014-08-14T17:13:33Z")
Hi everyone, Below you can find one big article, summing up all my experience of building the cluster on AWS. When I started I had no information at all, but I found the needed pieces in different places, including this user group. With your help I succeeded, and want to share the knowledge, that newcomers would find everything in one place. Elasticsearch cluster on AWS. Part 1 - preparing the environment. [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) Hope it would help someone! Regards, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 14, 2014, 10:19pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/2 "2014-08-14T22:19:59Z")
Nice and indepth, thanks for posting. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 15 August 2014 03:13, Pavel P [pavel@kredito.de](mailto:pavel@kredito.de) wrote: > Hi everyone, > > Below you can find one big article, summing up all my experience of > building the cluster on AWS. > When I started I had no information at all, but I found the needed pieces > in different places, including this user group. > > With your help I succeeded, and want to share the knowledge, that > newcomers would find everything in one place. > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > Hope it would help someone! > > Regards, > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37fa850d-2b0f-4495-9765-24b1f4019a19%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aJT3i\_0T\_6R%3DL%2B2NC0SQZ\_Xet8kUi%2B37nhdvpF7-eybg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aJT3i_0T_6R%3DL%2B2NC0SQZ_Xet8kUi%2B37nhdvpF7-eybg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrej](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Andrej](https://discuss.elastic.co/u/Andrej)\ **Post date:** [August 15, 2014, 11:25am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/3 "2014-08-15T11:25:26Z")
Nice collection, well presented, thanks! One note, you can probably restrict your permissions even further, instead of ReadOnly I use just the following 5 rules (may be even that is too much) : { "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeAvailabilityZones", "ec2:DescribeInstances", "ec2:DescribeRegions", "ec2:DescribeSecurityGroups", "ec2:DescribeTags" ], "Resource": "\*" } ] } Greets, Andrej Am Donnerstag, 14. August 2014 19:13:34 UTC+2 schrieb Pavel P: > Hi everyone, > > Below you can find one big article, summing up all my experience of > building the cluster on AWS. > When I started I had no information at all, but I found the needed pieces > in different places, including this user group. > > With your help I succeeded, and want to share the knowledge, that > newcomers would find everything in one place. > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > Hope it would help someone! > > Regards, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Pavel\_P](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavel_p/32/1092_2.png) [@Pavel\_P](https://discuss.elastic.co/u/Pavel_P)\ **Post date:** [August 15, 2014, 11:35am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/4 "2014-08-15T11:35:14Z")
Thanks Andrej, Would have it in mind! On Fri, Aug 15, 2014 at 2:25 PM, Andrej Rosenheinrich \< [andrej.rosenheinrich@unister.de](mailto:andrej.rosenheinrich@unister.de)\> wrote: > Nice collection, well presented, thanks! > > One note, you can probably restrict your permissions even further, instead > of ReadOnly I use just the following 5 rules (may be even that is too much) > : > > { > "Statement": [ > { > "Effect": "Allow", > "Action": [ > "ec2:DescribeAvailabilityZones", > "ec2:DescribeInstances", > "ec2:DescribeRegions", > "ec2:DescribeSecurityGroups", > "ec2:DescribeTags" > ], > "Resource": "\*" > } > ] > } > > Greets, > Andrej > > Am Donnerstag, 14. August 2014 19:13:34 UTC+2 schrieb Pavel P: > > > Hi everyone, > > > > Below you can find one big article, summing up all my experience of > > building the cluster on AWS. > > When I started I had no information at all, but I found the needed pieces > > in different places, including this user group. > > > > With your help I succeeded, and want to share the knowledge, that > > newcomers would find everything in one place. > > > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > > > Hope it would help someone! > > > > Regards, > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3a76bdfb-1d95-4b32-8769-116641011c1b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- _Pavel Polyakov_ Software Engineer - PHP team E-mail: [pavel@kredito.de](mailto:pavel@kredito.de) Skype: pavel.polyakov.x1 [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) Kreditech Holding SSL GmbH Am Sandtorkai 50, 20457 Hamburg, Germany Office phone: +49 (0)40 - 605905-60 Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller Company registration: Hamburg HRB122027 [www.kreditech.com](http://www.kreditech.com) [Kreditech](http://facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFVUaqO3DzJQgqrhn9Vs7e29BwjVQZFk5R7QT2Kn4FSPCHxw%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFVUaqO3DzJQgqrhn9Vs7e29BwjVQZFk5R7QT2Kn4FSPCHxw%3Dg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![davidski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidski/32/44884_2.png) [@davidski](https://discuss.elastic.co/u/davidski)\ **Post date:** [August 15, 2014, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/5 "2014-08-15T12:32:54Z")
Thanks for collecting this information together! A couple points for tweaking: 1. Instead of hard coding the IAM credentials into the file, associate the instances with an IAM role. cloud-aws will use those automatically and AWS will handle key rotation for you. 2. You are launching all the instances into the same availability zone. That greatly reduces the ability of the cluster to tolerate an AWS outage. Stick each of your three nodes in a different availability zone and you'll be much better off. 3. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. 4. I encourage AWS hosts _not_ to be named. Users should plan for hosts to come and go. This means no-unique host names and hard coded IPs. AWS is ephemeral infrastructure and ES, as a cluster app, is very happy playing in this space. and the big one... 1. Your security group looks to open ES to the world. DON'T DO THIS! There's been a tremendous amount of angst recently from ES clusters getting owned via open tcp/9200 and these security groups look to open your cluster to the entire internet. There's no need for that. cloud-aws will work with private IPs just fine. David On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote: > Hi everyone, > > Below you can find one big article, summing up all my experience of > building the cluster on AWS. > When I started I had no information at all, but I found the needed pieces > in different places, including this user group. > > With your help I succeeded, and want to share the knowledge, that > newcomers would find everything in one place. > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > Hope it would help someone! > > Regards, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Pavel\_P](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavel_p/32/1092_2.png) [@Pavel\_P](https://discuss.elastic.co/u/Pavel_P)\ **Post date:** [August 15, 2014, 12:39pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/6 "2014-08-15T12:39:12Z")
@David Severski Thanks for the input. I've actually encountered the issue, when my security group was closed for the world and the private IPs were not stated in the security group rules. I had no idea why the could-aws does not connect to the hosts, because the public IPs were there. I've stated that issue in the article. However I agree with you, the cluster should not be available from the world. On Fri, Aug 15, 2014 at 3:32 PM, David Severski [david@severski.net](mailto:david@severski.net) wrote: > Thanks for collecting this information together! A couple points for > tweaking: > > 1. Instead of hard coding the IAM credentials into the file, associate the > instances with an IAM role. cloud-aws will use those automatically and AWS > will handle key rotation for you. > 2. You are launching all the instances into the same availability zone. > That greatly reduces the ability of the cluster to tolerate an AWS outage. > Stick each of your three nodes in a different availability zone and you'll > be much better off. > 3. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. > 4. I encourage AWS hosts _not_ to be named. Users should plan for hosts to > come and go. This means no-unique host names and hard coded IPs. AWS is > ephemeral infrastructure and ES, as a cluster app, is very happy playing in > this space. > > and the big one... > > 1. Your security group looks to open ES to the world. DON'T DO THIS! > There's been a tremendous amount of angst recently from ES clusters getting > owned via open tcp/9200 and these security groups look to open your cluster > to the entire internet. There's no need for that. cloud-aws will work with > private IPs just fine. > > David > > On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote: > > > Hi everyone, > > > > Below you can find one big article, summing up all my experience of > > building the cluster on AWS. > > When I started I had no information at all, but I found the needed pieces > > in different places, including this user group. > > > > With your help I succeeded, and want to share the knowledge, that > > newcomers would find everything in one place. > > > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > > > Hope it would help someone! > > > > Regards, > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- _Pavel Polyakov_ Software Engineer - PHP team E-mail: [pavel@kredito.de](mailto:pavel@kredito.de) Skype: pavel.polyakov.x1 [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) Kreditech Holding SSL GmbH Am Sandtorkai 50, 20457 Hamburg, Germany Office phone: +49 (0)40 - 605905-60 Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller Company registration: Hamburg HRB122027 [www.kreditech.com](http://www.kreditech.com) [Kreditech](http://facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFVUaqOWpEd5k9%3Dn5H2yu9kxjSpSJkO05jc%2BjxZNZbnm5AFN%2Bw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFVUaqOWpEd5k9%3Dn5H2yu9kxjSpSJkO05jc%2BjxZNZbnm5AFN%2Bw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Pavel\_P](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavel_p/32/1092_2.png) [@Pavel\_P](https://discuss.elastic.co/u/Pavel_P)\ **Post date:** [August 15, 2014, 12:58pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/7 "2014-08-15T12:58:49Z")
@David Severski One more question > 1. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. What do you mean by that? On Fri, Aug 15, 2014 at 3:39 PM, Pavel P [pavel@kredito.de](mailto:pavel@kredito.de) wrote: > @David Severski > > Thanks for the input. > > I've actually encountered the issue, when my security group was closed for > the world and the private IPs were not stated in the security group rules. > I had no idea why the could-aws does not connect to the hosts, because the > public IPs were there. > I've stated that issue in the article. > > However I agree with you, the cluster should not be available from the > world. > > On Fri, Aug 15, 2014 at 3:32 PM, David Severski [david@severski.net](mailto:david@severski.net) > wrote: > > > Thanks for collecting this information together! A couple points for > > tweaking: > > > > 1. Instead of hard coding the IAM credentials into the file, associate > > the instances with an IAM role. cloud-aws will use those automatically and > > AWS will handle key rotation for you. > > 2. You are launching all the instances into the same availability zone. > > That greatly reduces the ability of the cluster to tolerate an AWS outage. > > Stick each of your three nodes in a different availability zone and you'll > > be much better off. > > 3. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. > > 4. I encourage AWS hosts _not_ to be named. Users should plan for hosts > > to come and go. This means no-unique host names and hard coded IPs. AWS is > > ephemeral infrastructure and ES, as a cluster app, is very happy playing in > > this space. > > > > and the big one... > > > > 1. Your security group looks to open ES to the world. DON'T DO THIS! > > There's been a tremendous amount of angst recently from ES clusters getting > > owned via open tcp/9200 and these security groups look to open your cluster > > to the entire internet. There's no need for that. cloud-aws will work with > > private IPs just fine. > > > > David > > > > On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote: > > > > > Hi everyone, > > > > > > Below you can find one big article, summing up all my experience of > > > building the cluster on AWS. > > > When I started I had no information at all, but I found the needed > > > pieces in different places, including this user group. > > > > > > With your help I succeeded, and want to share the knowledge, that > > > newcomers would find everything in one place. > > > > > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > > > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > > > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > > > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > > > > > Hope it would help someone! > > > > > > Regards, > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe](https://groups.google.com/d/topic/elasticsearch/NU2pktgTkDc/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fdcf7c19-f097-4eda-9078-852f24b2acd6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > _Pavel Polyakov_ > > Software Engineer - PHP team > > E-mail: [pavel@kredito.de](mailto:pavel@kredito.de) > Skype: pavel.polyakov.x1 > > [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) > Kreditech Holding SSL GmbH > Am Sandtorkai 50, 20457 Hamburg, Germany > Office phone: +49 (0)40 - 605905-60 > Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller > Company registration: Hamburg HRB122027 > > [www.kreditech.com](http://www.kreditech.com) > [Kreditech](http://facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) > > [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) > > This e-mail contains confidential and/or legally protected information. If > you are not the intended recipient or if you have received this e-mail by > error please notify the sender immediately and destroy this e-mail. Any > unauthorized review, copying, disclosure or distribution of the material in > this e-mail is strictly forbidden. The contents of this e-mail is legally > binding only if it is confirmed by letter or fax. The sending of e-mails to > us does not have any period-protecting effect. Thank you for your > cooperation. -- _Pavel Polyakov_ Software Engineer - PHP team E-mail: [pavel@kredito.de](mailto:pavel@kredito.de) Skype: pavel.polyakov.x1 [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) Kreditech Holding SSL GmbH Am Sandtorkai 50, 20457 Hamburg, Germany Office phone: +49 (0)40 - 605905-60 Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller Company registration: Hamburg HRB122027 [www.kreditech.com](http://www.kreditech.com) [Kreditech](http://facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) [https://www.facebook.com/kreditech](https://www.facebook.com/kreditech) This e-mail contains confidential and/or legally protected information. If you are not the intended recipient or if you have received this e-mail by error please notify the sender immediately and destroy this e-mail. Any unauthorized review, copying, disclosure or distribution of the material in this e-mail is strictly forbidden. The contents of this e-mail is legally binding only if it is confirmed by letter or fax. The sending of e-mails to us does not have any period-protecting effect. Thank you for your cooperation. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFVUaqP2M6hSMTeT8Zpzj%3DZUo05W1J%3D3ZZ8VOGWSG4h0R9eeWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFVUaqP2M6hSMTeT8Zpzj%3DZUo05W1J%3D3ZZ8VOGWSG4h0R9eeWQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrej](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Andrej](https://discuss.elastic.co/u/Andrej)\ **Post date:** [August 15, 2014, 2:09pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/8 "2014-08-15T14:09:52Z")
David, you are of course right with 2), but one thing to concider is that you pay for incoming and outgoing traffic between different availability zones. Am Freitag, 15. August 2014 14:32:54 UTC+2 schrieb David Severski: > Thanks for collecting this information together! A couple points for > tweaking: > > 1. Instead of hard coding the IAM credentials into the file, associate the > instances with an IAM role. cloud-aws will use those automatically and AWS > will handle key rotation for you. > 2. You are launching all the instances into the same availability zone. > That greatly reduces the ability of the cluster to tolerate an AWS outage. > Stick each of your three nodes in a different availability zone and you'll > be much better off. > 3. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. > 4. I encourage AWS hosts _not_ to be named. Users should plan for hosts to > come and go. This means no-unique host names and hard coded IPs. AWS is > ephemeral infrastructure and ES, as a cluster app, is very happy playing in > this space. > > and the big one... > > 1. Your security group looks to open ES to the world. DON'T DO THIS! > There's been a tremendous amount of angst recently from ES clusters getting > owned via open tcp/9200 and these security groups look to open your cluster > to the entire internet. There's no need for that. cloud-aws will work with > private IPs just fine. > > David > > On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote: > > > Hi everyone, > > > > Below you can find one big article, summing up all my experience of > > building the cluster on AWS. > > When I started I had no information at all, but I found the needed pieces > > in different places, including this user group. > > > > With your help I succeeded, and want to share the knowledge, that > > newcomers would find everything in one place. > > > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > > > Hope it would help someone! > > > > Regards, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aa245f74-5ae1-46ad-b01d-415735345356%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa245f74-5ae1-46ad-b01d-415735345356%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![davidski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidski/32/44884_2.png) [@davidski](https://discuss.elastic.co/u/davidski)\ **Post date:** [August 15, 2014, 6:29pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/9 "2014-08-15T18:29:49Z")
Sort of. If you use private IPs on both side the rate for transfer between availability zones (not regions) is 0.00 per GB. Zero cents. Just another reason not to use public IPs on your ES instances. David On Friday, August 15, 2014 7:09:52 AM UTC-7, Andrej Rosenheinrich wrote: > David, you are of course right with 2), but one thing to concider is that > you pay for incoming and outgoing traffic between different availability > zones. > > Am Freitag, 15. August 2014 14:32:54 UTC+2 schrieb David Severski: > > > Thanks for collecting this information together! A couple points for > > tweaking: > > > > 1. Instead of hard coding the IAM credentials into the file, associate > > the instances with an IAM role. cloud-aws will use those automatically and > > AWS will handle key rotation for you. > > 2. You are launching all the instances into the same availability zone. > > That greatly reduces the ability of the cluster to tolerate an AWS outage. > > Stick each of your three nodes in a different availability zone and you'll > > be much better off. > > 3. EC2-Classic is deprecated. Demonstrating use of VPC would be helpful. > > 4. I encourage AWS hosts _not_ to be named. Users should plan for hosts > > to come and go. This means no-unique host names and hard coded IPs. AWS is > > ephemeral infrastructure and ES, as a cluster app, is very happy playing in > > this space. > > > > and the big one... > > > > 1. Your security group looks to open ES to the world. DON'T DO THIS! > > There's been a tremendous amount of angst recently from ES clusters getting > > owned via open tcp/9200 and these security groups look to open your cluster > > to the entire internet. There's no need for that. cloud-aws will work with > > private IPs just fine. > > > > David > > > > On Thursday, August 14, 2014 10:13:34 AM UTC-7, Pavel P wrote: > > > > > Hi everyone, > > > > > > Below you can find one big article, summing up all my experience of > > > building the cluster on AWS. > > > When I started I had no information at all, but I found the needed > > > pieces in different places, including this user group. > > > > > > With your help I succeeded, and want to share the knowledge, that > > > newcomers would find everything in one place. > > > > > > Elasticsearch cluster on AWS. Part 1 - preparing the environment. > > > [http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/](http://pavelpolyakov.com/2014/08/13/elasticsearch-cluster-on-aws-part-1-preparing-environment/) > > > Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. > > > [http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/](http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-part-2-configuring-the-elasticsearch/) > > > > > > Hope it would help someone! > > > > > > Regards, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/518c2feb-8c7c-4bef-9a1c-659fea2c1211%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/518c2feb-8c7c-4bef-9a1c-659fea2c1211%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:08am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-on-aws-article/19292/10 "2017-07-06T01:08:14Z")