# Best practice - architecture feedback/opinion needed **URL:** \ **Category:** Elasticsearch\ **Created:** [February 14, 2015, 5:01pm UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168 "2015-02-14T17:01:15Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![dna\_lor](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@dna\_lor](https://discuss.elastic.co/u/dna_lor)\ **Post date:** [February 14, 2015, 5:01pm UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168/1 "2015-02-14T17:01:15Z")
I am evaluating ELK for the past 2 weeks in a testing environment, and i am very pleased with the result. right now i want to move it to staging, so i want to make sure i have the best practice/advised setup which i hope can get your feedback/opinion about expected usage: - up to 20 GB of logs are sent from logstash to elasticsearch every day (continuously 24/7) - 15 days worth of data should be stored in elasticsearch for search/graph. - logs older than 15 days should be be deleted - Daily incremental backup to AWS-S3 - 7 kibana users with average of 9 graphs per page/saved templates. always on 9/7 - 1 kibana user with no graph, just a live "tail" of specific types. 24/7 - Cronjobs curls directly to elasticsearch to perform different tasks (these are negligible ) I am considering the below setup, please correct me if i am wrong: - First Option: Logstash -\> Load balancer - \> - Elasticsearch instance (4 GB ram/2 cores) - Elasticsearch instance (4 GB ram/2 cores) - Elasticsearch instance (4 GB ram/2 cores) Logstash - \>- Elasticsearch instance (4 GB ram/2 cores) # Master Elasticsearch instance (4 GB ram/2 cores) #auto discovered by master - Third Option Logstash -\> Elasticsearch (15 GB ram/4 cores) (Still haven't figured out how to solve the yellow status color) Any advice on point of failures,in the above setup would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 16, 2015, 7:37am UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168/2 "2015-02-16T07:37:42Z")
With one node you can never allocate replicas, which is why it's in a yellow state. I'd go with setup #1 personally, but you probably want more RAM, say 4GB heap. Then set all to be master eligible so that you get some level of protection against node loss. Make sure you also use curator for managing retention and snapshot+restore. On 15 February 2015 at 04:01, dna lor [dnalor.ah@gmail.com](mailto:dnalor.ah@gmail.com) wrote: > I am evaluating ELK for the past 2 weeks in a testing environment, and i > am very pleased with the result. > > right now i want to move it to staging, so i want to make sure i have the > best practice/advised setup which i hope can get your feedback/opinion about > > expected usage: > > - up to 20 GB of logs are sent from logstash to elasticsearch every > day (continuously 24/7) > - 15 days worth of data should be stored in elasticsearch for > search/graph. > - logs older than 15 days should be be deleted > - Daily incremental backup to AWS-S3 > - 7 kibana users with average of 9 graphs per page/saved templates. > always on 9/7 > - 1 kibana user with no graph, just a live "tail" of specific types. > 24/7 > - Cronjobs curls directly to elasticsearch to perform different tasks > (these are negligible ) > > I am considering the below setup, please correct me if i am wrong: > > - First Option: > > Logstash -\> Load balancer - \> > > - Elasticsearch instance (4 GB ram/2 cores) > > - Elasticsearch instance (4 GB ram/2 cores) > > - Elasticsearch instance (4 GB ram/2 cores) > > Logstash - \>- Elasticsearch instance (4 GB ram/2 cores) # Master > Elasticsearch instance (4 GB ram/2 cores) #auto discovered by master > > - Third Option > > Logstash -\> Elasticsearch (15 GB ram/4 cores) (Still haven't figured out > how to solve the yellow status color) > > Any advice on point of failures,in the above setup would be greatly > appreciated. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8CqYCvts1V61RFJsEAJikyk%2BWmdprKuWWKyYJrG-S0Tw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8CqYCvts1V61RFJsEAJikyk%2BWmdprKuWWKyYJrG-S0Tw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dna\_lor](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@dna\_lor](https://discuss.elastic.co/u/dna_lor)\ **Post date:** [February 17, 2015, 6:38am UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168/3 "2015-02-17T06:38:56Z")
Thanks for your feedback Mark, So option 1 is preferable, can i get clarification on the points you have raised just so i understand things properly ? 1. Increase ram heap to 4 GB; instead of using 3 instances, i can switch to using 3 instances with 7.5 each, and assign 6 GB heap on both. would that be the same ? 2. All ES instances will be set as master eligible; means there will be identical data on all instances, correct? 1. In this case, i set my load balancer to use round robin on all related ES and they will sync in between ? Thanks for helping out. On Monday, February 16, 2015 at 9:38:11 AM UTC+2, Mark Walkom wrote: > With one node you can never allocate replicas, which is why it's in a > yellow state. > > I'd go with setup #1 personally, but you probably want more RAM, say 4GB > heap. Then set all to be master eligible so that you get some level of > protection against node loss. > > Make sure you also use curator for managing retention and snapshot+restore. > > On 15 February 2015 at 04:01, dna lor \<[dnal...@gmail.com](mailto:dnal...@gmail.com) \\> > wrote: > > > I am evaluating ELK for the past 2 weeks in a testing environment, and i > > am very pleased with the result. > > > > right now i want to move it to staging, so i want to make sure i have the > > best practice/advised setup which i hope can get your feedback/opinion about > > > > expected usage: > > > > - up to 20 GB of logs are sent from logstash to elasticsearch every > > day (continuously 24/7) > > - 15 days worth of data should be stored in elasticsearch for > > search/graph. > > - logs older than 15 days should be be deleted > > - Daily incremental backup to AWS-S3 > > - 7 kibana users with average of 9 graphs per page/saved templates. > > always on 9/7 > > - 1 kibana user with no graph, just a live "tail" of specific types. > > 24/7 > > - Cronjobs curls directly to elasticsearch to perform different tasks > > (these are negligible ) > > > > I am considering the below setup, please correct me if i am wrong: > > > > - First Option: > > > > Logstash -\> Load balancer - \> > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > Logstash - \>- Elasticsearch instance (4 GB ram/2 cores) # Master > > Elasticsearch instance (4 GB ram/2 cores) #auto discovered by master > > > > - Third Option > > > > Logstash -\> Elasticsearch (15 GB ram/4 cores) (Still haven't figured out > > how to solve the yellow status color) > > > > Any advice on point of failures,in the above setup would be greatly > > appreciated. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 17, 2015, 6:56am UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168/4 "2015-02-17T06:56:46Z")
1 - You want to have 50% system memory for heap, the other 50% for caching. So 4GB heap can be done with 7.5GB in system, but you don't really want to go higher. 2 - No. See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. But if all nodes are in the same cluster then you can round robin and it will shard the data between them. On 17 February 2015 at 17:38, dna lor [dnalor.ah@gmail.com](mailto:dnalor.ah@gmail.com) wrote: > Thanks for your feedback Mark, > > So option 1 is preferable, can i get clarification on the points you have > raised just so i understand things properly ? > > 1. Increase ram heap to 4 GB; instead of using 3 instances, i can > switch to using 3 instances with 7.5 each, and assign 6 GB heap on both. > would that be the same ? > 2. All ES instances will be set as master eligible; means there will > be identical data on all instances, correct? > 1. In this case, i set my load balancer to use round robin on all > related ES and they will sync in between ? > > Thanks for helping out. > > On Monday, February 16, 2015 at 9:38:11 AM UTC+2, Mark Walkom wrote: > > > With one node you can never allocate replicas, which is why it's in a > > yellow state. > > > > I'd go with setup #1 personally, but you probably want more RAM, say 4GB > > heap. Then set all to be master eligible so that you get some level of > > protection against node loss. > > > > Make sure you also use curator for managing retention and > > snapshot+restore. > > > > On 15 February 2015 at 04:01, dna lor [dnal...@gmail.com](mailto:dnal...@gmail.com) wrote: > > > > > I am evaluating ELK for the past 2 weeks in a testing environment, and i > > > am very pleased with the result. > > > > > > right now i want to move it to staging, so i want to make sure i have > > > the best practice/advised setup which i hope can get your feedback/opinion > > > about > > > > > > expected usage: > > > > > > - up to 20 GB of logs are sent from logstash to elasticsearch every > > > day (continuously 24/7) > > > - 15 days worth of data should be stored in elasticsearch for > > > search/graph. > > > - logs older than 15 days should be be deleted > > > - Daily incremental backup to AWS-S3 > > > - 7 kibana users with average of 9 graphs per page/saved templates. > > > always on 9/7 > > > - 1 kibana user with no graph, just a live "tail" of specific types. > > > 24/7 > > > - Cronjobs curls directly to elasticsearch to perform different > > > tasks (these are negligible ) > > > > > > I am considering the below setup, please correct me if i am wrong: > > > > > > - First Option: > > > > > > Logstash -\> Load balancer - \> > > > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > > > - Elasticsearch instance (4 GB ram/2 cores) > > > > > > Logstash - \>- Elasticsearch instance (4 GB ram/2 cores) # Master > > > Elasticsearch instance (4 GB ram/2 cores) #auto discovered by master > > > > > > - Third Option > > > > > > Logstash -\> Elasticsearch (15 GB ram/4 cores) (Still haven't figured out > > > how to solve the yellow status color) > > > > > > Any advice on point of failures,in the above setup would be greatly > > > appreciated. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a7c881a3-fb54-4c5c-b497-1a69becbe6a8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b0ed0756-c625-46ea-a541-0ffd3c6df213%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZoTrn1TpstNA7wgRv8qjLL65gqOUgM390Ni7G39bJDQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8ZoTrn1TpstNA7wgRv8qjLL65gqOUgM390Ni7G39bJDQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/best-practice-architecture-feedback-opinion-needed/22168/5 "2017-07-06T00:32:15Z")