# $es\_heap\_size **URL:** \ **Category:** Elasticsearch\ **Created:** [February 8, 2013, 9:08am UTC](https://discuss.elastic.co/t/es-heap-size/10664 "2013-02-08T09:08:56Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 9:08am UTC](https://discuss.elastic.co/t/es-heap-size/10664/1 "2013-02-08T09:08:56Z")
Hi Guys, If i Set $ES\_HEAP\_SIZE to 64g should i see that the system is reserving 64GB of memory using htop? Regards Shawn -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 10:22am UTC](https://discuss.elastic.co/t/es-heap-size/10664/2 "2013-02-08T10:22:36Z")
On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote: > Hi Guys, > > If i Set $ES\_HEAP\_SIZE to 64g should i see that the system is > reserving 64GB of memory using htop? you should. note that using more than 30GB for the heap will prevent the jvm from using compressed pointers, and will increase GC load. you probably don't want to do this clint > Regards > Shawn > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 10:27am UTC](https://discuss.elastic.co/t/es-heap-size/10664/3 "2013-02-08T10:27:29Z")
What is i set it exactly to 30GB do i still get this problem. On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote: > > > Hi Guys, > > > > If i Set $ES\_HEAP\_SIZE to 64g should i see that the system is > > reserving 64GB of memory using htop? > > you should. note that using more than 30GB for the heap will prevent > the jvm from using compressed pointers, and will increase GC load. you > probably don't want to do this > > clint > > > Regards > > Shawn > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 10:32am UTC](https://discuss.elastic.co/t/es-heap-size/10664/4 "2013-02-08T10:32:38Z")
On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote: > What is i set it exactly to 30GB do i still get this problem. ? > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote: > \> Hi Guys, > \> > \> > \> If i Set $ES\_HEAP\_SIZE to 64g should i see that the system > is > \> reserving 64GB of memory using htop? > > ``` > you should. note that using more than 30GB for the heap will > prevent > the jvm from using compressed pointers, and will increase GC > load. you > probably don't want to do this > > clint > > > > > > > Regards > > Shawn > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails > from it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 10:33am UTC](https://discuss.elastic.co/t/es-heap-size/10664/5 "2013-02-08T10:33:45Z")
If i set the ES\_HEAP\_SIZE to 30GB would i avoid the compressed pointers problem? On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote: > > > What is i set it exactly to 30GB do i still get this problem. > > ? > > > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley wrote: > > On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie wrote: > > \> Hi Guys, > > \> > > \> > > \> If i Set $ES\_HEAP\_SIZE to 64g should i see that the system > > is > > \> reserving 64GB of memory using htop? > > > > ``` > > you should. note that using more than 30GB for the heap will > > prevent > > the jvm from using compressed pointers, and will increase GC > > load. you > > probably don't want to do this > > > > clint > > > > > > > > > > > Regards > > > Shawn > > > > > > -- > > > You received this message because you are subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 10:35am UTC](https://discuss.elastic.co/t/es-heap-size/10664/6 "2013-02-08T10:35:54Z")
On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote: > If i set the ES\_HEAP\_SIZE to 30GB would i avoid the compressed > pointers problem? yes 🙂 don't forget that you should leave about 50% of your RAM for kernel filesystem cache as well clint > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote: > \> What is i set it exactly to 30GB do i still get this > problem. > > ``` > ? > > > > > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley > wrote: > > On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie > wrote: > > > Hi Guys, > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g should i see that > the system > > is > > > reserving 64GB of memory using htop? > > > > you should. note that using more than 30GB for the > heap will > > prevent > > the jvm from using compressed pointers, and will > increase GC > > load. you > > probably don't want to do this > > > > clint > > > > > > > > > > > Regards > > > Shawn > > > > > > -- > > > You received this message because you are > subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving > emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails > from it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 10:59am UTC](https://discuss.elastic.co/t/es-heap-size/10664/7 "2013-02-08T10:59:50Z")
Shouldn't be problematic as the server has 128GB of RAM. So if i set that ES\_HEAP\_SIZE to 30GB and run Elasticsearch using top should i see elastic search using up 30GB of ram? Regards Shawn On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote: > > > If i set the ES\_HEAP\_SIZE to 30GB would i avoid the compressed > > pointers problem? > > yes 🙂 > > don't forget that you should leave about 50% of your RAM for kernel > filesystem cache as well > > clint > > > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley wrote: > > On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie wrote: > > \> What is i set it exactly to 30GB do i still get this > > problem. > > > > ``` > > ? > > > > > > > > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton Gormley > > wrote: > > > On Fri, 2013-02-08 at 01:08 -0800, Shawn Ritchie > > wrote: > > > > Hi Guys, > > > > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g should i see that > > the system > > > is > > > > reserving 64GB of memory using htop? > > > > > > you should. note that using more than 30GB for the > > heap will > > > prevent > > > the jvm from using compressed pointers, and will > > increase GC > > > load. you > > > probably don't want to do this > > > > > > clint > > > > > > > > > > > > > > > Regards > > > > Shawn > > > > > > > > -- > > > > You received this message because you are > > subscribed to the > > > Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving > > emails > > > from it, send > > > > an email to elasticsearc...@googlegroups.com. > > > > For more options, visit > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 11:11am UTC](https://discuss.elastic.co/t/es-heap-size/10664/8 "2013-02-08T11:11:24Z")
On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote: > Shouldn't be problematic as the server has 128GB of RAM. So if i set > that ES\_HEAP\_SIZE to 30GB and run Elasticsearch using top should i > see Elasticsearch using up 30GB of ram? yes with 128GB of RAM, you may consider running two instance of ES on the same machine, but just make sure that you use "awareness" to ensure that you don't have the same primary and replica shards on a single box. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. clint > Regards > Shawn > > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote: > \> If i set the ES\_HEAP\_SIZE to 30GB would i avoid the > compressed > \> pointers problem? > > ``` > yes :) > > don't forget that you should leave about 50% of your RAM for > kernel > filesystem cache as well > > clint > > > > > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley > wrote: > > On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie > wrote: > > > What is i set it exactly to 30GB do i still get > this > > problem. > > > > ? > > > > > > > > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton > Gormley > > wrote: > > > On Fri, 2013-02-08 at 01:08 -0800, Shawn > Ritchie > > wrote: > > > > Hi Guys, > > > > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g should i > see that > > the system > > > is > > > > reserving 64GB of memory using htop? > > > > > > you should. note that using more than > 30GB for the > > heap will > > > prevent > > > the jvm from using compressed pointers, > and will > > increase GC > > > load. you > > > probably don't want to do this > > > > > > clint > > > > > > > > > > > > > > > Regards > > > > Shawn > > > > > > > > -- > > > > You received this message because you > are > > subscribed to the > > > Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop > receiving > > emails > > > from it, send > > > > an email to > elasticsearc...@googlegroups.com. > > > > For more options, visit > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > -- > > > You received this message because you are > subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving > emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails > from it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 11:14am UTC](https://discuss.elastic.co/t/es-heap-size/10664/9 "2013-02-08T11:14:33Z")
Would I get greater throughput though using 2 instances? Regards Shawn On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote: > > > Shouldn't be problematic as the server has 128GB of RAM. So if i set > > that ES\_HEAP\_SIZE to 30GB and run Elasticsearch using top should i > > see Elasticsearch using up 30GB of ram? > > yes > > with 128GB of RAM, you may consider running two instance of ES on the > same machine, but just make sure that you use "awareness" to ensure that > you don't have the same primary and replica shards on a single box. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html) > > clint > > > Regards > > Shawn > > > > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley wrote: > > On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie wrote: > > \> If i set the ES\_HEAP\_SIZE to 30GB would i avoid the > > compressed > > \> pointers problem? > > > > ``` > > yes :) > > > > don't forget that you should leave about 50% of your RAM for > > kernel > > filesystem cache as well > > > > clint > > > > > > > > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton Gormley > > wrote: > > > On Fri, 2013-02-08 at 02:27 -0800, Shawn Ritchie > > wrote: > > > > What is i set it exactly to 30GB do i still get > > this > > > problem. > > > > > > ? > > > > > > > > > > > On Friday, 8 February 2013 11:22:36 UTC+1, Clinton > > Gormley > > > wrote: > > > > On Fri, 2013-02-08 at 01:08 -0800, Shawn > > Ritchie > > > wrote: > > > > > Hi Guys, > > > > > > > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g should i > > see that > > > the system > > > > is > > > > > reserving 64GB of memory using htop? > > > > > > > > you should. note that using more than > > 30GB for the > > > heap will > > > > prevent > > > > the jvm from using compressed pointers, > > and will > > > increase GC > > > > load. you > > > > probably don't want to do this > > > > > > > > clint > > > > > > > > > > > > > > > > > > > Regards > > > > > Shawn > > > > > > > > > > -- > > > > > You received this message because you > > are > > > subscribed to the > > > > Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop > > receiving > > > emails > > > > from it, send > > > > > an email to > > elasticsearc...@googlegroups.com. > > > > > For more options, visit > > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > You received this message because you are > > subscribed to the > > > Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving > > emails > > > from it, send > > > > an email to elasticsearc...@googlegroups.com. > > > > For more options, visit > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 11:21am UTC](https://discuss.elastic.co/t/es-heap-size/10664/10 "2013-02-08T11:21:06Z")
On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote: > Would I get greater throughput though using 2 instances? they'd be sharing CPU and IO but you'd be making use of all of your RAM (as opposed to wasting half of it). the answer is: I think so, but not entirely sure 🙂 try it and see - let us know what you find clint > Regards > Shawn > > On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote: > \> Shouldn't be problematic as the server has 128GB of RAM. So > if i set > \> that ES\_HEAP\_SIZE to 30GB and run Elasticsearch using top > should i > \> see Elasticsearch using up 30GB of ram? > > ``` > yes > > with 128GB of RAM, you may consider running two instance of ES > on the > same machine, but just make sure that you use "awareness" to > ensure that > you don't have the same primary and replica shards on a single > box. > > http://www.elasticsearch.org/guide/reference/modules/cluster.html > > clint > > > > > > > Regards > > Shawn > > > > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley > wrote: > > On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie > wrote: > > > If i set the ES_HEAP_SIZE to 30GB would i avoid > the > > compressed > > > pointers problem? > > > > yes :) > > > > don't forget that you should leave about 50% of your > RAM for > > kernel > > filesystem cache as well > > > > clint > > > > > > > > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton > Gormley > > wrote: > > > On Fri, 2013-02-08 at 02:27 -0800, Shawn > Ritchie > > wrote: > > > > What is i set it exactly to 30GB do i > still get > > this > > > problem. > > > > > > ? > > > > > > > > > > > On Friday, 8 February 2013 11:22:36 UTC > +1, Clinton > > Gormley > > > wrote: > > > > On Fri, 2013-02-08 at 01:08 > -0800, Shawn > > Ritchie > > > wrote: > > > > > Hi Guys, > > > > > > > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g > should i > > see that > > > the system > > > > is > > > > > reserving 64GB of memory using > htop? > > > > > > > > you should. note that using > more than > > 30GB for the > > > heap will > > > > prevent > > > > the jvm from using compressed > pointers, > > and will > > > increase GC > > > > load. you > > > > probably don't want to do this > > > > > > > > clint > > > > > > > > > > > > > > > > > > > Regards > > > > > Shawn > > > > > > > > > > -- > > > > > You received this message > because you > > are > > > subscribed to the > > > > Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group > and stop > > receiving > > > emails > > > > from it, send > > > > > an email to > > elasticsearc...@googlegroups.com. > > > > > For more options, visit > > > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > You received this message because you > are > > subscribed to the > > > Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop > receiving > > emails > > > from it, send > > > > an email to > elasticsearc...@googlegroups.com. > > > > For more options, visit > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > -- > > > You received this message because you are > subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving > emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails > from it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 11:27am UTC](https://discuss.elastic.co/t/es-heap-size/10664/11 "2013-02-08T11:27:34Z")
Hmm, i will need some help with this, so what you are implying i run 2 instances of Elasticsearch with different configs, Set one instance with node.rack\_id: rack\_one cluster.routing.allocation.awareness.attributes: rack\_id node.master: true node.data: true index.number\_of\_shards: 4 index.number\_of\_replicas: 0 set another instance with node.rack\_id: rack\_two cluster.routing.allocation.awareness.attributes: rack\_two node.master: false node.data: true index.number\_of\_shards: 4 index.number\_of\_replicas: 0 and this will automatically 2 shards on each instance? Regards Shawn On Friday, 8 February 2013 12:21:06 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 03:14 -0800, Shawn Ritchie wrote: > > > Would I get greater throughput though using 2 instances? > > they'd be sharing CPU and IO but you'd be making use of all of your RAM > (as opposed to wasting half of it). > > the answer is: I think so, but not entirely sure 🙂 > > try it and see - let us know what you find > > clint > > > Regards > > Shawn > > > > On Friday, 8 February 2013 12:11:24 UTC+1, Clinton Gormley wrote: > > On Fri, 2013-02-08 at 02:59 -0800, Shawn Ritchie wrote: > > \> Shouldn't be problematic as the server has 128GB of RAM. So > > if i set > > \> that ES\_HEAP\_SIZE to 30GB and run Elasticsearch using top > > should i > > \> see Elasticsearch using up 30GB of ram? > > > > ``` > > yes > > > > with 128GB of RAM, you may consider running two instance of ES > > on the > > same machine, but just make sure that you use "awareness" to > > ensure that > > you don't have the same primary and replica shards on a single > > box. > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html) > > > ``` > > clint > > > > > > > > > > > Regards > > > Shawn > > > > > > On Friday, 8 February 2013 11:35:54 UTC+1, Clinton Gormley > > wrote: > > > On Fri, 2013-02-08 at 02:33 -0800, Shawn Ritchie > > wrote: > > > > If i set the ES_HEAP_SIZE to 30GB would i avoid > > the > > > compressed > > > > pointers problem? > > > > > > yes :) > > > > > > don't forget that you should leave about 50% of your > > RAM for > > > kernel > > > filesystem cache as well > > > > > > clint > > > > > > > > > > > On Friday, 8 February 2013 11:32:38 UTC+1, Clinton > > Gormley > > > wrote: > > > > On Fri, 2013-02-08 at 02:27 -0800, Shawn > > Ritchie > > > wrote: > > > > > What is i set it exactly to 30GB do i > > still get > > > this > > > > problem. > > > > > > > > ? > > > > > > > > > > > > > > On Friday, 8 February 2013 11:22:36 UTC > > +1, Clinton > > > Gormley > > > > wrote: > > > > > On Fri, 2013-02-08 at 01:08 > > -0800, Shawn > > > Ritchie > > > > wrote: > > > > > > Hi Guys, > > > > > > > > > > > > > > > > > > If i Set $ES_HEAP_SIZE to 64g > > should i > > > see that > > > > the system > > > > > is > > > > > > reserving 64GB of memory using > > htop? > > > > > > > > > > you should. note that using > > more than > > > 30GB for the > > > > heap will > > > > > prevent > > > > > the jvm from using compressed > > pointers, > > > and will > > > > increase GC > > > > > load. you > > > > > probably don't want to do this > > > > > > > > > > clint > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > Shawn > > > > > > > > > > > > -- > > > > > > You received this message > > because you > > > are > > > > subscribed to the > > > > > Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group > > and stop > > > receiving > > > > emails > > > > > from it, send > > > > > > an email to > > > elasticsearc...@googlegroups.com. > > > > > > For more options, visit > > > > > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > You received this message because you > > are > > > subscribed to the > > > > Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop > > receiving > > > emails > > > > from it, send > > > > > an email to > > elasticsearc...@googlegroups.com. > > > > > For more options, visit > > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > You received this message because you are > > subscribed to the > > > Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving > > emails > > > from it, send > > > > an email to elasticsearc...@googlegroups.com. > > > > For more options, visit > > > https://groups.google.com/groups/opt_out. > > > > > > > > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the > > Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails > > from it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 11:53am UTC](https://discuss.elastic.co/t/es-heap-size/10664/12 "2013-02-08T11:53:57Z")
On Fri, 2013-02-08 at 03:27 -0800, Shawn Ritchie wrote: > Hmm, i will need some help with this, > > so what you are implying i run 2 instances of Elasticsearch with > different configs, OK, first question is: do you have just one server or more? If you have just one server then you are at risk. If that server dies, you will lose your cluster and (eg if the disks die) lose your data. With just one server, the "awareness" doesn't matter - if the server dies, you're screwed. If you have more than one server then Elasticsearch puts a primary shard on one server and a replica of the primary on the other server. That way, if you lose one server, all of your data is safe and the cluster can keep on running. If you are running two nodes on one server and two nodes on another server, then you need to use awareness to ensure that the primary and replica shards are not placed on the same server. So the two nodes running on the same server would have the same rack ID. That way Elasticsearch knows to place the primary shards on one rack, and the replica shards on the other rack. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 12:02pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/13 "2013-02-08T12:02:36Z")
Hi, Basically we only have 1 machine for Elastic Search, and for now are not interested in replication, for now we are only interested in maximising performance. So if I understood we do not need the awareness field! and the 2 elastic search instances will take care of 2 shards each automatically? Also each instance i'm guessing needs to be configured with different ports? and which instance will i be querying? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 12:06pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/14 "2013-02-08T12:06:38Z")
> Basically we only have 1 machine for Elastic Search, and for now are > not interested in replication, for now we are only interested in > maximising performance. OK > So if I understood we do not need the awareness field! and the 2 > Elasticsearch instances will take care of 2 shards each > automatically? Yes > Also each instance i'm guessing needs to be configured with different > ports? and which instance will i be querying? Each node will try to connect to port 9300 and if that is not available, try 9301, 9302 etc. This is the port for communication between nodes (and for the java clients), so make sure that you have 9300 and 9301 open in your firewall. Similarly, each node will bind to 9200, 9201.... for the HTTP interface. This is automatic - you don't need to configure anything except for opening the ports on your firewall. You can talk to either node (http on 9200 or 9201, java on 9300, 9301) clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 12:07pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/15 "2013-02-08T12:07:27Z")
or do i only need to set node.max\_local\_storage\_nodes: 2 and it will automatically setup 2 instances? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 12:09pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/16 "2013-02-08T12:09:10Z")
On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote: > or do i only need to set node.max\_local\_storage\_nodes: 2 and it will > automatically setup 2 instances? No, you need to start two instances. the above setting just stops you from starting more than X instances on the same server (or at least from the same directory) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 12:14pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/17 "2013-02-08T12:14:22Z")
Last question 😃 How do i lunch and instance with a specific config file? As i find it quite ironic that you specify the config file directory in the config file itself? On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote: > > > or do i only need to set node.max\_local\_storage\_nodes: 2 and it will > > automatically setup 2 instances? > > No, you need to start two instances. the above setting just stops you > from starting more than X instances on the same server (or at least from > the same directory) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 12:21pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/18 "2013-02-08T12:21:48Z")
On Fri, 2013-02-08 at 04:14 -0800, Shawn Ritchie wrote: > Last question 😃 > > How do i lunch and instance with a specific config file? As i find it > quite ironic that you specify the config file directory in the config > file itself? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > On Friday, 8 February 2013 13:09:10 UTC+1, Clinton Gormley wrote: > On Fri, 2013-02-08 at 04:07 -0800, Shawn Ritchie wrote: > \> or do i only need to set node.max\_local\_storage\_nodes: 2 and > it will > \> automatically setup 2 instances? > > ``` > No, you need to start two instances. the above setting just > stops you > from starting more than X instances on the same server (or at > least from > the same directory) > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kayngee](https://avatars.discourse-cdn.com/v4/letter/k/43a26b/32.png) [@kayngee](https://discuss.elastic.co/u/kayngee)\ **Post date:** [February 8, 2013, 12:27pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/19 "2013-02-08T12:27:08Z")
Already read that post, but from what i understood or misunderstood, is its making the assumption you will have 1 instance of elastic search running on a machine. What i'd like to do is with 1 elastic search installation is lunch 2 instance of elastic search with different /config /data /log node name. Or is it that multiple versions of elastic search on the same machine run @ a directory level, that is 2 instances which are sharing the /config /data and /log directries together with the node name? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 8, 2013, 12:40pm UTC](https://discuss.elastic.co/t/es-heap-size/10664/20 "2013-02-08T12:40:19Z")
On Fri, 2013-02-08 at 04:27 -0800, Shawn Ritchie wrote: > Already read that post, but from what i understood or misunderstood, > is its making the assumption you will have 1 instance of elastic > search running on a machine. > > What i'd like to do is with 1 Elasticsearch installation is lunch 2 > instance of Elasticsearch with different /config /data /log node > name. > > Or is it that multiple versions of Elasticsearch on the same machine > run @ a directory level, that is 2 instances which are sharing > the /config /data and /log directries together with the node name? You can run multiple instances with the same paths (including logging and data). If you just want to specify a different node name, then you could do so on the command line: ./bin/elasticsearch -Des.node.name=node\_1 ./bin/elasticsearch -Des.node.name=node\_2 If you want to change more than that, you could specify a specific config file: ./bin/elasticsearch -Des.config=/path/to/config/file\_1 ./bin/elasticsearch -Des.config=/path/to/config/file\_2 clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). [Next page](https://discuss.elastic.co/t/es-heap-size/10664.md?page=2)