# MIN/MAX memory allocation and mlockall **URL:** \ **Category:** Elasticsearch\ **Created:** [July 29, 2013, 5:36am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978 "2013-07-29T05:36:52Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 29, 2013, 5:36am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978/1 "2013-07-29T05:36:52Z")
We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0\_25 and Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of physical). However during startup we were noticing that ES would continually core dump. We ended up having to set; ES\_MIN\_MEM=64g ES\_MAX\_MEM=256g Where before we were using; ES\_HEAP\_SIZE=256g Just to get ES to start and connect to the existing cluster. This does appear to be a java bug occurring when trying to allocate the complete xms. Our main concern is if there be any issues if we continue to set "bootstrap.mlockall: true", as this setup doesn't align with the official documentation? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 29, 2013, 7:13am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978/2 "2013-07-29T07:13:04Z")
No lunch is for free. I would be nice to get the core dump for analysis, so it is not possible to help. So I have only general hints, where YMMV. First, check if your Linux gives you enough resources for mmap() with ulimit -l (for the mlockall call by JNA). Second, please note, that you must also reserve swap for the size of your memory space you demand with Xms (which in the end is something like a sequence of simple malloc()'s on fragments of contiguous space). So, check if your system offers 256G contiguous space, and if the swap area is at least 256G (plus some GB for non-heap memory). When you start the JVM process with Xms, Linux must be able to move other pages out of the way to swap, if you already had pages allocated which are swappable. If you don't, Linux let the process crash. By default, Linux reserves 25% of total RAM for it (and this matches your 64G setting). Third, personally I'm curious how you analyzed that only such a large heap memory is feasible for JVM apps, like ES. Have you run other apps with such a setting succesfully? I must admit I doubt it. For example, managing caches in such a large heap is a nightmare, and I predict it will slow down your ES significantly. Also, GC will get slow, compared to less aggressive heap setting, like 8GB or 16GB. Jörg On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0\_25 and > Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of > physical). > > However during startup we were noticing that ES would continually core > dump. We ended up having to set; > > ES\_MIN\_MEM=64g > ES\_MAX\_MEM=256g > > Where before we were using; > > ES\_HEAP\_SIZE=256g > > Just to get ES to start and connect to the existing cluster. > > This does appear to be a java bug occurring when trying to allocate the > complete xms. Our main concern is if there be any issues if we continue to > set "bootstrap.mlockall: true", as this setup doesn't align with the > official documentation? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 29, 2013, 8:46am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978/3 "2013-07-29T08:46:53Z")
ulimit is pretty high, 512000. The total filesystem is 10TB and most of that is empty, so a contiguous 256GB shouldn't be a problem. We have 4G of swap as we were of the impression that given the large amount of RAM it wouldn't need a lot of swap, just enough to allow the system to operate and occasionally use. This was based on readings such as > **[9.15.5. Recommended Partitioning Scheme Red Hat Enterprise Linux 6 | Red...](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/s2-diskpartrecommend-x86)** > > Access Red Hat’s knowledge, guidance, and support through your subscription. . However based on your comments I am guessing that setting the xms to 256G with such a small swap is probably why it wouldn't initially start. These are new machines and we are experimenting with heap sizes etc, we're happy to get comments on how feasible it is to run with such a large heap size and if there are other methods we could use to allow ES efficient access to the RAM. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 July 2013 17:13, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > No lunch is for free. > > I would be nice to get the core dump for analysis, so it is not possible > to help. > > So I have only general hints, where YMMV. > > First, check if your Linux gives you enough resources for mmap() with > ulimit -l (for the mlockall call by JNA). > > Second, please note, that you must also reserve swap for the size of your > memory space you demand with Xms (which in the end is something like a > sequence of simple malloc()'s on fragments of contiguous space). So, check > if your system offers 256G contiguous space, and if the swap area is at > least 256G (plus some GB for non-heap memory). When you start the JVM > process with Xms, Linux must be able to move other pages out of the way to > swap, if you already had pages allocated which are swappable. If you don't, > Linux let the process crash. By default, Linux reserves 25% of total RAM > for it (and this matches your 64G setting). > > Third, personally I'm curious how you analyzed that only such a large heap > memory is feasible for JVM apps, like ES. Have you run other apps with such > a setting succesfully? I must admit I doubt it. For example, managing > caches in such a large heap is a nightmare, and I predict it will slow down > your ES significantly. Also, GC will get slow, compared to less aggressive > heap setting, like 8GB or 16GB. > > Jörg > > On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > > > We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0\_25 and > > Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of > > physical). > > > > However during startup we were noticing that ES would continually core > > dump. We ended up having to set; > > > > ES\_MIN\_MEM=64g > > ES\_MAX\_MEM=256g > > > > Where before we were using; > > > > ES\_HEAP\_SIZE=256g > > > > Just to get ES to start and connect to the existing cluster. > > > > This does appear to be a java bug occurring when trying to allocate the > > complete xms. Our main concern is if there be any issues if we continue to > > set "bootstrap.mlockall: true", as this setup doesn't align with the > > official documentation? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 29, 2013, 11:23am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978/4 "2013-07-29T11:23:52Z")
ulimit -l value of 512000 kb (=512M) is by far not enough for using mlockall option. Set it to unlimited. The filesystem does not matter at all for contiguous RAM. The fragmentation of the RAM is managed by the kernel. The easiest method to regain contiguous RAM is reboot - but I assume you use only ES on the machine so there should be no problem. Jörg On Mon, Jul 29, 2013 at 10:46 AM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > ulimit is pretty high, 512000. The total filesystem is 10TB and most of > that is empty, so a contiguous 256GB shouldn't be a problem. > > We have 4G of swap as we were of the impression that given the large > amount of RAM it wouldn't need a lot of swap, just enough to allow the > system to operate and occasionally use. This was based on readings such as > [9.15.5. Recommended Partitioning Scheme Red Hat Enterprise Linux 6 | Red Hat Customer Portal](https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s2-diskpartrecommend-x86.html) > . > However based on your comments I am guessing that setting the xms to 256G > with such a small swap is probably why it wouldn't initially start. > > These are new machines and we are experimenting with heap sizes etc, we're > happy to get comments on how feasible it is to run with such a large heap > size and if there are other methods we could use to allow ES efficient > access to the RAM. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 29 July 2013 17:13, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com)wrote: > > > No lunch is for free. > > > > I would be nice to get the core dump for analysis, so it is not possible > > to help. > > > > So I have only general hints, where YMMV. > > > > First, check if your Linux gives you enough resources for mmap() with > > ulimit -l (for the mlockall call by JNA). > > > > Second, please note, that you must also reserve swap for the size of your > > memory space you demand with Xms (which in the end is something like a > > sequence of simple malloc()'s on fragments of contiguous space). So, check > > if your system offers 256G contiguous space, and if the swap area is at > > least 256G (plus some GB for non-heap memory). When you start the JVM > > process with Xms, Linux must be able to move other pages out of the way to > > swap, if you already had pages allocated which are swappable. If you don't, > > Linux let the process crash. By default, Linux reserves 25% of total RAM > > for it (and this matches your 64G setting). > > > > Third, personally I'm curious how you analyzed that only such a large > > heap memory is feasible for JVM apps, like ES. Have you run other apps with > > such a setting succesfully? I must admit I doubt it. For example, managing > > caches in such a large heap is a nightmare, and I predict it will slow down > > your ES significantly. Also, GC will get slow, compared to less aggressive > > heap setting, like 8GB or 16GB. > > > > Jörg > > > > On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > > > > > We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0\_25 and > > > Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of > > > physical). > > > > > > However during startup we were noticing that ES would continually core > > > dump. We ended up having to set; > > > > > > ES\_MIN\_MEM=64g > > > ES\_MAX\_MEM=256g > > > > > > Where before we were using; > > > > > > ES\_HEAP\_SIZE=256g > > > > > > Just to get ES to start and connect to the existing cluster. > > > > > > This does appear to be a java bug occurring when trying to allocate the > > > complete xms. Our main concern is if there be any issues if we continue to > > > set "bootstrap.mlockall: true", as this setup doesn't align with the > > > official documentation? > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:24am UTC](https://discuss.elastic.co/t/min-max-memory-allocation-and-mlockall/12978/5 "2017-07-06T02:24:02Z")