# Killing swap once and for all, file descriptors too, linux and windows **URL:** \ **Category:** Elasticsearch\ **Created:** [December 6, 2013, 7:43am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745 "2013-12-06T07:43:03Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [December 6, 2013, 7:43am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/1 "2013-12-06T07:43:03Z")
So, for ES, swap is a bad thing, right? I'm running, currently, a mixed Windows and Linux environment. All running 0.90.5. On linux (RHEL): Locked memory has been set to unlimited. mlockall is (supposed to be) on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). Limits applied to the Linux process: Max open files 65536 65536 files Max locked memory unlimited unlimited bytes I still see swap space getting used, just a few megs at first, and I've seen it get as far up as 800mb in elastichq The only way I can get swap to turn off for ES is to turn off swap for the whole system - which seems kinda overkill. File descriptor limits on linux seem fine What am I missing? I feel like there's probably one little thing that will make it all click into place and I've just glazed over it. On Windows There is no mlockall. I briefly tried disabling the page file and rebooting the server, but when ES came back up, it had its same 32GB swap usage show up in elastichq as it did when the page file was around. ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more instability - rivers dropping, response times to a simple query taking 5-10 minutes, etc, at the recommended 30GB. I still get to enjoy those things on at 24GB, but not quite as often. There is also, apparently, no functional way to check file descriptors in use, or how many the process is allowed to open. I just get back the -1 unknown value. Do file descriptor counts not matter on Windows? Is swap not a bad word on Windows as far as ES is concerned? If it is, since it sounds like there are at least a few people out there running production ES windows servers, how do I manage it properly? Ideally, we'll be moving the Windows server to RHEL, but there may be enough red tape in the way that it will take a while. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 6, 2013, 8:09am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/2 "2013-12-06T08:09:07Z")
Hey, I would not recommend running a mixed environment, as debugging will become pretty much PITA when debugging different performance statistics of operating systems - might be something for the really adventurous of us. Now back to topic: If you enabled mlockall, but the elasticsearch process is still swapping (can you please verify by checking top or the /proc file system, I have no idea how elastichq is measuring this to be honest, maybe Roy can help here?), mlockall seems not enabled. Can you check the elasticsearch log file on startup, maybe there is an mlockall error written out, which means, that it is not enabled (we recently added, if the mlockall was successful on startup, so you can see it in the nodes info, but it is not yet part of a 0.90 release). --Alex On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > So, for ES, swap is a bad thing, right? > I'm running, currently, a mixed Windows and Linux environment. All running > 0.90.5. > On linux (RHEL): > Locked memory has been set to unlimited. mlockall is (supposed to be) on. > ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > Limits applied to the Linux process: > > Max open files 65536 65536 files > > Max locked memory unlimited unlimited bytes > I still see swap space getting used, just a few megs at first, and I've > seen it get as far up as 800mb in elastichq > > The only way I can get swap to turn off for ES is to turn off swap for the > whole system - which seems kinda overkill. File descriptor limits on linux > seem fine > > What am I missing? I feel like there's probably one little thing that will > make it all click into place and I've just glazed over it. > > On Windows > There is no mlockall. I briefly tried disabling the page file and > rebooting the server, but when ES came back up, it had its same 32GB swap > usage show up in elastichq as it did when the page file was around. > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > instability - rivers dropping, response times to a simple query taking 5-10 > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > at 24GB, but not quite as often. > There is also, apparently, no functional way to check file descriptors in > use, or how many the process is allowed to open. I just get back the -1 > unknown value. Do file descriptor counts not matter on Windows? > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > since it sounds like there are at least a few people out there running > production ES windows servers, how do I manage it properly? > > Ideally, we'll be moving the Windows server to RHEL, but there may be > enough red tape in the way that it will take a while. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [December 6, 2013, 8:24am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/3 "2013-12-06T08:24:33Z")
At the moment, the only reason for the mixed environment is to provide an easy migration path for when we burn down the windows node and reimage to RHEL. If we can force the Windows node to be as stable as its linux counterparts, we may just stick with it, though. I am indeed getting an mlock error [2013-12-05 21:25:34,439][WARN][common.jna] Unknown mlockall error 0 ulimit -l unlimited has been set, which is the only suggestion I've been able to find for this particular error. Found this thread, [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE) but it looks like they didn't find a solution. Thanks, Josh On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hey, > > I would not recommend running a mixed environment, as debugging will become pretty much PITA when debugging different performance statistics of operating systems - might be something for the really adventurous of us. > > Now back to topic: If you enabled mlockall, but the elasticsearch process is still swapping (can you please verify by checking top or the /proc file system, I have no idea how elastichq is measuring this to be honest, maybe Roy can help here?), mlockall seems not enabled. Can you check the elasticsearch log file on startup, maybe there is an mlockall error written out, which means, that it is not enabled (we recently added, if the mlockall was successful on startup, so you can see it in the nodes info, but it is not yet part of a 0.90 release). > > --Alex > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > So, for ES, swap is a bad thing, right? > I'm running, currently, a mixed Windows and Linux environment. All running 0.90.5. > On linux (RHEL): > Locked memory has been set to unlimited. mlockall is (supposed to be) on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > Limits applied to the Linux process: > > Max open files 65536 65536 files > > Max locked memory unlimited unlimited bytes > > I still see swap space getting used, just a few megs at first, and I've seen it get as far up as 800mb in elastichq > > The only way I can get swap to turn off for ES is to turn off swap for the whole system - which seems kinda overkill. File descriptor limits on linux seem fine > > What am I missing? I feel like there's probably one little thing that will make it all click into place and I've just glazed over it. > > On Windows > There is no mlockall. I briefly tried disabling the page file and rebooting the server, but when ES came back up, it had its same 32GB swap usage show up in elastichq as it did when the page file was around. ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more instability - rivers dropping, response times to a simple query taking 5-10 minutes, etc, at the recommended 30GB. I still get to enjoy those things on at 24GB, but not quite as often. > There is also, apparently, no functional way to check file descriptors in use, or how many the process is allowed to open. I just get back the -1 unknown value. Do file descriptor counts not matter on Windows? > > Is swap not a bad word on Windows as far as ES is concerned? If it is, since it sounds like there are at least a few people out there running production ES windows servers, how do I manage it properly? > > Ideally, we'll be moving the Windows server to RHEL, but there may be enough red tape in the way that it will take a while. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 6, 2013, 9:13am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/4 "2013-12-06T09:13:34Z")
Hey, ok, so mlockall is configured but not set on startup - at least you know why it is swapping. How are you starting elasticsearch? Do you use the RPM? If so, I guess you set MAX\_LOCKED\_MEMORY=unlimited in there? If not, can you try? Another issue might be (depending on how ES gets started), that the memlock setting is configured for the root user, but not for the elasticsearch one (wild guessing here). --Alex On Fri, Dec 6, 2013 at 9:24 AM, Joshua Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > At the moment, the only reason for the mixed environment is to provide an > easy migration path for when we burn down the windows node and reimage to > RHEL. If we can force the Windows node to be as stable as its linux > counterparts, we may just stick with it, though. > > I am indeed getting an mlock error > [2013-12-05 21:25:34,439][WARN][common.jna] Unknown > mlockall error 0 > ulimit -l unlimited has been set, which is the only suggestion I've been > able to find for this particular error. > > Found this thread, > [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE) but it > looks like they didn't find a solution. > Thanks, > Josh > > On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > > Hey, > > I would not recommend running a mixed environment, as debugging will > become pretty much PITA when debugging different performance statistics of > operating systems - might be something for the really adventurous of us. > > Now back to topic: If you enabled mlockall, but the elasticsearch process > is still swapping (can you please verify by checking top or the /proc file > system, I have no idea how elastichq is measuring this to be honest, maybe > Roy can help here?), mlockall seems not enabled. Can you check the > elasticsearch log file on startup, maybe there is an mlockall error written > out, which means, that it is not enabled (we recently added, if the > mlockall was successful on startup, so you can see it in the nodes info, > but it is not yet part of a 0.90 release). > > --Alex > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > > > So, for ES, swap is a bad thing, right? > > I'm running, currently, a mixed Windows and Linux environment. All > > running 0.90.5. > > On linux (RHEL): > > Locked memory has been set to unlimited. mlockall is (supposed to be) on. > > ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > Limits applied to the Linux process: > > > > Max open files 65536 65536 files > > > > Max locked memory unlimited unlimited bytes > > I still see swap space getting used, just a few megs at first, and I've > > seen it get as far up as 800mb in elastichq > > > > The only way I can get swap to turn off for ES is to turn off swap for > > the whole system - which seems kinda overkill. File descriptor limits on > > linux seem fine > > > > What am I missing? I feel like there's probably one little thing that > > will make it all click into place and I've just glazed over it. > > > > On Windows > > There is no mlockall. I briefly tried disabling the page file and > > rebooting the server, but when ES came back up, it had its same 32GB swap > > usage show up in elastichq as it did when the page file was around. > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > instability - rivers dropping, response times to a simple query taking 5-10 > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > at 24GB, but not quite as often. > > There is also, apparently, no functional way to check file descriptors in > > use, or how many the process is allowed to open. I just get back the -1 > > unknown value. Do file descriptor counts not matter on Windows? > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > since it sounds like there are at least a few people out there running > > production ES windows servers, how do I manage it properly? > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > enough red tape in the way that it will take a while. > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8Stu1MmcahTwYSPR4Fv-0TzOF9kk9nUp0b8w88HE%2BNGQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8Stu1MmcahTwYSPR4Fv-0TzOF9kk9nUp0b8w88HE%2BNGQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 6, 2013, 9:14am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/5 "2013-12-06T09:14:46Z")
Wondering if this article could help you (windows)? [http://www.oracle.com/technetwork/java/javase/tech/largememory-jsp-137182.html](http://www.oracle.com/technetwork/java/javase/tech/largememory-jsp-137182.html) -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 6 décembre 2013 at 09:24:33, Joshua Harrison ([hijakk@gmail.com](mailto:hijakk@gmail.com)) a écrit: At the moment, the only reason for the mixed environment is to provide an easy migration path for when we burn down the windows node and reimage to RHEL. If we can force the Windows node to be as stable as its linux counterparts, we may just stick with it, though. I am indeed getting an mlock error [2013-12-05 21:25:34,439][WARN][common.jna] Unknown mlockall error 0 ulimit -l unlimited has been set, which is the only suggestion I've been able to find for this particular error. Found this thread, [https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE](https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE) but it looks like they didn't find a solution. Thanks, Josh On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: Hey, I would not recommend running a mixed environment, as debugging will become pretty much PITA when debugging different performance statistics of operating systems - might be something for the really adventurous of us. Now back to topic: If you enabled mlockall, but the elasticsearch process is still swapping (can you please verify by checking top or the /proc file system, I have no idea how elastichq is measuring this to be honest, maybe Roy can help here?), mlockall seems not enabled. Can you check the elasticsearch log file on startup, maybe there is an mlockall error written out, which means, that it is not enabled (we recently added, if the mlockall was successful on startup, so you can see it in the nodes info, but it is not yet part of a 0.90 release). --Alex On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: So, for ES, swap is a bad thing, right? I'm running, currently, a mixed Windows and Linux environment. All running 0.90.5. On linux (RHEL): Locked memory has been set to unlimited. mlockall is (supposed to be) on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). Limits applied to the Linux process: Max open files 65536 65536 files Max locked memory unlimited unlimited bytes I still see swap space getting used, just a few megs at first, and I've seen it get as far up as 800mb in elastichq The only way I can get swap to turn off for ES is to turn off swap for the whole system - which seems kinda overkill. File descriptor limits on linux seem fine What am I missing? I feel like there's probably one little thing that will make it all click into place and I've just glazed over it. On Windows There is no mlockall. I briefly tried disabling the page file and rebooting the server, but when ES came back up, it had its same 32GB swap usage show up in elastichq as it did when the page file was around. ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more instability - rivers dropping, response times to a simple query taking 5-10 minutes, etc, at the recommended 30GB. I still get to enjoy those things on at 24GB, but not quite as often. There is also, apparently, no functional way to check file descriptors in use, or how many the process is allowed to open. I just get back the -1 unknown value. Do file descriptor counts not matter on Windows? Is swap not a bad word on Windows as far as ES is concerned? If it is, since it sounds like there are at least a few people out there running production ES windows servers, how do I manage it properly? Ideally, we'll be moving the Windows server to RHEL, but there may be enough red tape in the way that it will take a while. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52a19586.1fbfe8e0.bd3d%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52a19586.1fbfe8e0.bd3d%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [December 6, 2013, 7:20pm UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/6 "2013-12-06T19:20:09Z")
Yep, using the RPM. I ended up following the settings in here: [Limits are not consumed using Systemd (ulimit -n / ulimit -l) by skymeyer · Pull Request #3355 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/3355) and now top says I have a swap usage of 0, agreeing with /proc/pid/smaps - though elastichq still reports swap usage, so I'm not sure what to make of that. So for those of you that run into this in the future, try this: After applying fix Removing limits from /etc/security/limits.conf #elasticsearch soft nofile 65535 #elasticsearch hard nofile 65535 #elasticsearch - memlock unlimited /etc/sysconfig/elasticsearch: # Maximum number of open files MAX\_OPEN\_FILES=65535 # Maximum amount of locked memory MAX\_LOCKED\_MEMORY=unlimited /etc/elasticsearch/elasticsearch.yml: bootstrap.mlockall: true Actual fix in /etc/systemd/system/elasticsearch.service [Service] ... LimitMEMLOCK=infinity LimitNOFILE=65535 Note: run the following command after altering the above file: systemctl --system daemon-reload Result max\_open\_files and no mlockall error: /etc/rc.d/init.d/elasticsearch restart [2013-07-18 11:20:21,476][INFO][bootstrap] max\_open\_files [65511] [2013-07-18 11:20:21,616][INFO][node] [node1] {0.90.2}[28558]: initializing ... On Friday, December 6, 2013 1:13:34 AM UTC-8, Alexander Reelsen wrote: > Hey, > > ok, so mlockall is configured but not set on startup - at least you know > why it is swapping. > > How are you starting elasticsearch? Do you use the RPM? If so, I guess you > set MAX\_LOCKED\_MEMORY=unlimited in there? If not, can you try? > Another issue might be (depending on how ES gets started), that the > memlock setting is configured for the root user, but not for the > elasticsearch one (wild guessing here). > > --Alex > > On Fri, Dec 6, 2013 at 9:24 AM, Joshua Harrison \<[hij...@gmail.com](mailto:hij...@gmail.com)\ > > > wrote: > > > At the moment, the only reason for the mixed environment is to provide an > > easy migration path for when we burn down the windows node and reimage to > > RHEL. If we can force the Windows node to be as stable as its linux > > counterparts, we may just stick with it, though. > > > > I am indeed getting an mlock error > > [2013-12-05 21:25:34,439][WARN][common.jna] Unknown > > mlockall error 0 > > ulimit -l unlimited has been set, which is the only suggestion I've been > > able to find for this particular error. > > > > Found this thread, > > [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE) but it > > looks like they didn't find a solution. > > Thanks, > > Josh > > > > On Dec 6, 2013, at 12:09 AM, Alexander Reelsen \<[a...@spinscale.de](mailto:a...@spinscale.de)\\> > > wrote: > > > > Hey, > > > > I would not recommend running a mixed environment, as debugging will > > become pretty much PITA when debugging different performance statistics of > > operating systems - might be something for the really adventurous of us. > > > > Now back to topic: If you enabled mlockall, but the elasticsearch process > > is still swapping (can you please verify by checking top or the /proc file > > system, I have no idea how elastichq is measuring this to be honest, maybe > > Roy can help here?), mlockall seems not enabled. Can you check the > > elasticsearch log file on startup, maybe there is an mlockall error written > > out, which means, that it is not enabled (we recently added, if the > > mlockall was successful on startup, so you can see it in the nodes info, > > but it is not yet part of a 0.90 release). > > > > --Alex > > > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison \<[hij...@gmail.com](mailto:hij...@gmail.com)\ > > > > > wrote: > > > > > So, for ES, swap is a bad thing, right? > > > I'm running, currently, a mixed Windows and Linux environment. All > > > running 0.90.5. > > > On linux (RHEL): > > > Locked memory has been set to unlimited. mlockall is (supposed to be) > > > on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > > > Limits applied to the Linux process: > > > > > > Max open files 65536 65536 files > > > > > > Max locked memory unlimited unlimited bytes > > > I still see swap space getting used, just a few megs at first, and I've > > > seen it get as far up as 800mb in elastichq > > > > > > The only way I can get swap to turn off for ES is to turn off swap for > > > the whole system - which seems kinda overkill. File descriptor limits on > > > linux seem fine > > > > > > What am I missing? I feel like there's probably one little thing that > > > will make it all click into place and I've just glazed over it. > > > > > > On Windows > > > There is no mlockall. I briefly tried disabling the page file and > > > rebooting the server, but when ES came back up, it had its same 32GB swap > > > usage show up in elastichq as it did when the page file was around. > > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > > instability - rivers dropping, response times to a simple query taking 5-10 > > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > > at 24GB, but not quite as often. > > > There is also, apparently, no functional way to check file descriptors > > > in use, or how many the process is allowed to open. I just get back the -1 > > > unknown value. Do file descriptor counts not matter on Windows? > > > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > > since it sounds like there are at least a few people out there running > > > production ES windows servers, how do I manage it properly? > > > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > > enough red tape in the way that it will take a while. > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 7, 2013, 3:07pm UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/7 "2013-12-07T15:07:00Z")
Hey Josh, glad it is working now. Can you have a look at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. tell me, if we can improve the docs somehow (or also mentioning it differently in the sysconfig file maybe). Any pointer how we can make this more failureproof? Mentioning it more obvious somewhere? (Except that I should have thought about systemd earlier 😉 Any help is much appreciated! --Alex On Fri, Dec 6, 2013 at 8:20 PM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > Yep, using the RPM. > I ended up following the settings in here: > [Limits are not consumed using Systemd (ulimit -n / ulimit -l) by skymeyer · Pull Request #3355 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/3355) and now top says > I have a swap usage of 0, agreeing with /proc/pid/smaps - though elastichq > still reports swap usage, so I'm not sure what to make of that. > So for those of you that run into this in the future, try this: > After applying fix > > Removing limits from /etc/security/limits.conf > > #elasticsearch soft nofile 65535 > #elasticsearch hard nofile 65535 > #elasticsearch - memlock unlimited > > /etc/sysconfig/elasticsearch: > > # Maximum number of open files > > MAX\_OPEN\_FILES=65535 > > # Maximum amount of locked memory > > MAX\_LOCKED\_MEMORY=unlimited > > /etc/elasticsearch/elasticsearch.yml: > > bootstrap.mlockall: true > > Actual fix in /etc/systemd/system/elasticsearch.service > > [Service] > ... > LimitMEMLOCK=infinity > LimitNOFILE=65535 > > Note: run the following command after altering the above file: > > systemctl --system daemon-reload > > Result max\_open\_files and no mlockall error: > > /etc/rc.d/init.d/elasticsearch restart > [2013-07-18 11:20:21,476][INFO][bootstrap] max\_open\_files [65511] > [2013-07-18 11:20:21,616][INFO][node] [node1] {0.90.2}[28558]: initializing ... > > On Friday, December 6, 2013 1:13:34 AM UTC-8, Alexander Reelsen wrote: > > > Hey, > > > > ok, so mlockall is configured but not set on startup - at least you know > > why it is swapping. > > > > How are you starting elasticsearch? Do you use the RPM? If so, I guess > > you set MAX\_LOCKED\_MEMORY=unlimited in there? If not, can you try? > > Another issue might be (depending on how ES gets started), that the > > memlock setting is configured for the root user, but not for the > > elasticsearch one (wild guessing here). > > > > --Alex > > > > On Fri, Dec 6, 2013 at 9:24 AM, Joshua Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > > > > At the moment, the only reason for the mixed environment is to provide > > > an easy migration path for when we burn down the windows node and reimage > > > to RHEL. If we can force the Windows node to be as stable as its linux > > > counterparts, we may just stick with it, though. > > > > > > I am indeed getting an mlock error > > > [2013-12-05 21:25:34,439][WARN][common.jna] Unknown > > > mlockall error 0 > > > ulimit -l unlimited has been set, which is the only suggestion I've been > > > able to find for this particular error. > > > > > > Found this thread, [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/) > > > elasticsearch/0CaBak7sdRE but it looks like they didn't find a solution. > > > Thanks, > > > Josh > > > > > > On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [a...@spinscale.de](mailto:a...@spinscale.de) > > > wrote: > > > > > > Hey, > > > > > > I would not recommend running a mixed environment, as debugging will > > > become pretty much PITA when debugging different performance statistics of > > > operating systems - might be something for the really adventurous of us. > > > > > > Now back to topic: If you enabled mlockall, but the elasticsearch > > > process is still swapping (can you please verify by checking top or the > > > /proc file system, I have no idea how elastichq is measuring this to be > > > honest, maybe Roy can help here?), mlockall seems not enabled. Can you > > > check the elasticsearch log file on startup, maybe there is an mlockall > > > error written out, which means, that it is not enabled (we recently added, > > > if the mlockall was successful on startup, so you can see it in the nodes > > > info, but it is not yet part of a 0.90 release). > > > > > > --Alex > > > > > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > > > > > > So, for ES, swap is a bad thing, right? > > > > I'm running, currently, a mixed Windows and Linux environment. All > > > > running 0.90.5. > > > > On linux (RHEL): > > > > Locked memory has been set to unlimited. mlockall is (supposed to be) > > > > on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > > > > > Limits applied to the Linux process: > > > > > > > > Max open files 65536 65536 > > > > files > > > > > > > > Max locked memory unlimited unlimited > > > > bytes > > > > I still see swap space getting used, just a few megs at first, and I've > > > > seen it get as far up as 800mb in elastichq > > > > > > > > The only way I can get swap to turn off for ES is to turn off swap for > > > > the whole system - which seems kinda overkill. File descriptor limits on > > > > linux seem fine > > > > > > > > What am I missing? I feel like there's probably one little thing that > > > > will make it all click into place and I've just glazed over it. > > > > > > > > On Windows > > > > There is no mlockall. I briefly tried disabling the page file and > > > > rebooting the server, but when ES came back up, it had its same 32GB swap > > > > usage show up in elastichq as it did when the page file was around. > > > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > > > instability - rivers dropping, response times to a simple query taking 5-10 > > > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > > > at 24GB, but not quite as often. > > > > There is also, apparently, no functional way to check file descriptors > > > > in use, or how many the process is allowed to open. I just get back the -1 > > > > unknown value. Do file descriptor counts not matter on Windows? > > > > > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > > > since it sounds like there are at least a few people out there running > > > > production ES windows servers, how do I manage it properly? > > > > > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > > > enough red tape in the way that it will take a while. > > > > > > > > Thanks! > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > topic/elasticsearch/JWtzbphQEVQ/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_% > > > 3DtNaChf5kGg%[40mail.gmail.com](http://40mail.gmail.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%[40gmail.com](http://40gmail.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [December 7, 2013, 6:18pm UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/8 "2013-12-07T18:18:17Z")
I'd probably mention it in the default elasticsearch.yml file with a line in the description of mlockall like: If running as a linux service from an RPM build it (may be/is) necessary to set LimitMEMLOCK=infinity in /etc/systemd/system/elasticsearch.service Mostly because all of the documentation, suggestions and stackoverflow answers just say "turn on mlockall", having this near that may be helpful. Alternatively, I don't know if that error 0 happens in other circumstances, it probably does, but it could be worth injecting something like the above into the log when that error happens as a possible solution. Finally, I figured out that the heap usage reported in elasticHQ and other similar tools appears to be the swap usage of the entire host system - not of ES. Since swap is a bad thing for ES, would it be possible to get a property in \_node that actually shows what ES itself is using? Thanks Alex! -Josh On Dec 7, 2013, at 7:07 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > Hey Josh, > > glad it is working now. Can you have a look at [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_linux.html) and tell me, if we can improve the docs somehow (or also mentioning it differently in the sysconfig file maybe). Any pointer how we can make this more failureproof? Mentioning it more obvious somewhere? (Except that I should have thought about systemd earlier 😉 > > Any help is much appreciated! > > --Alex > > On Fri, Dec 6, 2013 at 8:20 PM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > Yep, using the RPM. > I ended up following the settings in here: [Limits are not consumed using Systemd (ulimit -n / ulimit -l) by skymeyer · Pull Request #3355 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/3355) and now top says I have a swap usage of 0, agreeing with /proc/pid/smaps - though elastichq still reports swap usage, so I'm not sure what to make of that. > So for those of you that run into this in the future, try this: > After applying fix > > Removing limits from /etc/security/limits.conf > > #elasticsearch soft nofile 65535 > #elasticsearch hard nofile 65535 > #elasticsearch - memlock unlimited > /etc/sysconfig/elasticsearch: > > # Maximum number of open files > > MAX\_OPEN\_FILES=65535 > > # Maximum amount of locked memory > > MAX\_LOCKED\_MEMORY=unlimited > /etc/elasticsearch/elasticsearch.yml: > > bootstrap.mlockall: true > Actual fix in /etc/systemd/system/elasticsearch.service > > [Service] > ... > LimitMEMLOCK=infinity > LimitNOFILE=65535 > Note: run the following command after altering the above file: > > systemctl --system daemon-reload > Result max\_open\_files and no mlockall error: > > /etc/rc.d/init.d/elasticsearch restart > [2013-07-18 11:20:21,476][INFO][bootstrap] max\_open\_files [65511] > [2013-07-18 11:20:21,616][INFO][node] [node1] {0.90.2}[28558]: initializing ... > > On Friday, December 6, 2013 1:13:34 AM UTC-8, Alexander Reelsen wrote: > Hey, > > ok, so mlockall is configured but not set on startup - at least you know why it is swapping. > > How are you starting elasticsearch? Do you use the RPM? If so, I guess you set MAX\_LOCKED\_MEMORY=unlimited in there? If not, can you try? > Another issue might be (depending on how ES gets started), that the memlock setting is configured for the root user, but not for the elasticsearch one (wild guessing here). > > --Alex > > On Fri, Dec 6, 2013 at 9:24 AM, Joshua Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > At the moment, the only reason for the mixed environment is to provide an easy migration path for when we burn down the windows node and reimage to RHEL. If we can force the Windows node to be as stable as its linux counterparts, we may just stick with it, though. > > I am indeed getting an mlock error > [2013-12-05 21:25:34,439][WARN][common.jna] Unknown mlockall error 0 > ulimit -l unlimited has been set, which is the only suggestion I've been able to find for this particular error. > > Found this thread, [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/0CaBak7sdRE) but it looks like they didn't find a solution. > Thanks, > Josh > > On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [a...@spinscale.de](mailto:a...@spinscale.de) wrote: > > > Hey, > > > > I would not recommend running a mixed environment, as debugging will become pretty much PITA when debugging different performance statistics of operating systems - might be something for the really adventurous of us. > > > > Now back to topic: If you enabled mlockall, but the elasticsearch process is still swapping (can you please verify by checking top or the /proc file system, I have no idea how elastichq is measuring this to be honest, maybe Roy can help here?), mlockall seems not enabled. Can you check the elasticsearch log file on startup, maybe there is an mlockall error written out, which means, that it is not enabled (we recently added, if the mlockall was successful on startup, so you can see it in the nodes info, but it is not yet part of a 0.90 release). > > > > --Alex > > > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > So, for ES, swap is a bad thing, right? > > I'm running, currently, a mixed Windows and Linux environment. All running 0.90.5. > > On linux (RHEL): > > Locked memory has been set to unlimited. mlockall is (supposed to be) on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > Limits applied to the Linux process: > > > > Max open files 65536 65536 files > > > > Max locked memory unlimited unlimited bytes > > > > I still see swap space getting used, just a few megs at first, and I've seen it get as far up as 800mb in elastichq > > > > The only way I can get swap to turn off for ES is to turn off swap for the whole system - which seems kinda overkill. File descriptor limits on linux seem fine > > > > What am I missing? I feel like there's probably one little thing that will make it all click into place and I've just glazed over it. > > > > On Windows > > There is no mlockall. I briefly tried disabling the page file and rebooting the server, but when ES came back up, it had its same 32GB swap usage show up in elastichq as it did when the page file was around. ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more instability - rivers dropping, response times to a simple query taking 5-10 minutes, etc, at the recommended 30GB. I still get to enjoy those things on at 24GB, but not quite as often. > > There is also, apparently, no functional way to check file descriptors in use, or how many the process is allowed to open. I just get back the -1 unknown value. Do file descriptor counts not matter on Windows? > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, since it sounds like there are at least a few people out there running production ES windows servers, how do I manage it properly? > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be enough red tape in the way that it will take a while. > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_%3DtNaChf5kGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER_%3DtNaChf5kGg%40mail.gmail.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%40gmail.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1CE5CD83-FA46-400B-8C03-91A1A4FEBBAC%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/1CE5CD83-FA46-400B-8C03-91A1A4FEBBAC%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Roy\_Russo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roy_russo/32/44835_2.png) [@Roy\_Russo](https://discuss.elastic.co/u/Roy_Russo)\ **Post date:** [December 7, 2013, 6:45pm UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/9 "2013-12-07T18:45:33Z")
> I have no idea how elastichq is measuring this to be honest, maybe Roy can > help here? The swap space alert is using the following formula, taken from the endpoint (\_cluster/nodes/stats?all=1) formula:"stats.os.swap.used\_in\_bytes / 1024 / 1024", upper\_limit:["1", "1"], comment:"Any use of swap by the JVM, no matter how small, can greatly impact the speed of the garbage collector." It's merely a warning, as the comment states. If my formula is incorrect, or needs tuning, please let me know and I will change it accordingly. Honestly, this warning appears on every instance I deploy on windows. Haven't looked on Linux, so maybe my acceptable threshold is incorrect? On Friday, December 6, 2013 3:09:07 AM UTC-5, Alexander Reelsen wrote: > Hey, > > I would not recommend running a mixed environment, as debugging will > become pretty much PITA when debugging different performance statistics of > operating systems - might be something for the really adventurous of us. > > Now back to topic: If you enabled mlockall, but the elasticsearch process > is still swapping (can you please verify by checking top or the /proc file > system, I have no idea how elastichq is measuring this to be honest, maybe > Roy can help here?), mlockall seems not enabled. Can you check the > elasticsearch log file on startup, maybe there is an mlockall error written > out, which means, that it is not enabled (we recently added, if the > mlockall was successful on startup, so you can see it in the nodes info, > but it is not yet part of a 0.90 release). > > --Alex > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison \<[hij...@gmail.com](mailto:hij...@gmail.com)\ > > > wrote: > > > So, for ES, swap is a bad thing, right? > > I'm running, currently, a mixed Windows and Linux environment. All > > running 0.90.5. > > On linux (RHEL): > > Locked memory has been set to unlimited. mlockall is (supposed to be) on. > > ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > Limits applied to the Linux process: > > > > Max open files 65536 65536 files > > > > Max locked memory unlimited unlimited bytes > > I still see swap space getting used, just a few megs at first, and I've > > seen it get as far up as 800mb in elastichq > > > > The only way I can get swap to turn off for ES is to turn off swap for > > the whole system - which seems kinda overkill. File descriptor limits on > > linux seem fine > > > > What am I missing? I feel like there's probably one little thing that > > will make it all click into place and I've just glazed over it. > > > > On Windows > > There is no mlockall. I briefly tried disabling the page file and > > rebooting the server, but when ES came back up, it had its same 32GB swap > > usage show up in elastichq as it did when the page file was around. > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > instability - rivers dropping, response times to a simple query taking 5-10 > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > at 24GB, but not quite as often. > > There is also, apparently, no functional way to check file descriptors in > > use, or how many the process is allowed to open. I just get back the -1 > > unknown value. Do file descriptor counts not matter on Windows? > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > since it sounds like there are at least a few people out there running > > production ES windows servers, how do I manage it properly? > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > enough red tape in the way that it will take a while. > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36b476c6-ff27-47e8-ae03-f2a3ea6a3bf5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36b476c6-ff27-47e8-ae03-f2a3ea6a3bf5%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 7, 2013, 7:54pm UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/10 "2013-12-07T19:54:24Z")
ElasticHQ uses ES node stats, and ES node stats uses sigar, and sigar uses the file /proc/meminfo. If you want to find out if ES JVM is using swap, you can use cat /proc//status | grep VmSwap where is the process ID of the ES JVM. On each Linux system, using swap is a decision of the kernel and this must not necessarily mean a bad thing. To persuade the kernel in its decisions not to swap any more, the swappiness sysctl -a | grep swappiness should be set to another than the default value of 60. See also > **[Swappiness](https://en.wikipedia.org/wiki/Swappiness)** > > Swappiness is a Linux kernel parameter that controls the relative weight given to swapping out of runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value causes the kernel to avoid swapping; a higher value causes the kernel to try to use swap space. The default value is 60; setting it higher will increase performance of "hot" processes at the cost of making a return to inactive "cold" ones take a long pause,... Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHqmFOJ2skLkKwao\_nJz65Nf0MBZ5991okQTChdgS8hpg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHqmFOJ2skLkKwao_nJz65Nf0MBZ5991okQTChdgS8hpg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 9, 2013, 8:07am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/11 "2013-12-09T08:07:21Z")
Hey Roy, this seems swap of the OS (judging by a quick peek in the sigar lib) and thus is not bound to the JVM process of the running elasticsearch instance. If there are other process on the system, which swap out, that should be fine in many cases. --Alex On Sat, Dec 7, 2013 at 7:45 PM, Roy Russo [royrusso@gmail.com](mailto:royrusso@gmail.com) wrote: > > I have no idea how elastichq is measuring this to be honest, maybe Roy > > can help here? > > The swap space alert is using the following formula, taken from the > endpoint (\_cluster/nodes/stats?all=1) > > formula:"stats.os.swap.used\_in\_bytes / 1024 / 1024", > upper\_limit:["1", "1"], > comment:"Any use of swap by the JVM, no matter how small, can greatly > impact the speed of the garbage collector." > > It's merely a warning, as the comment states. If my formula is incorrect, or needs tuning, please let me know and I will change it accordingly. Honestly, this warning appears on every instance I deploy on windows. Haven't looked on Linux, so maybe my acceptable threshold is incorrect? > > On Friday, December 6, 2013 3:09:07 AM UTC-5, Alexander Reelsen wrote: > > > Hey, > > > > I would not recommend running a mixed environment, as debugging will > > become pretty much PITA when debugging different performance statistics of > > operating systems - might be something for the really adventurous of us. > > > > Now back to topic: If you enabled mlockall, but the elasticsearch process > > is still swapping (can you please verify by checking top or the /proc file > > system, I have no idea how elastichq is measuring this to be honest, maybe > > Roy can help here?), mlockall seems not enabled. Can you check the > > elasticsearch log file on startup, maybe there is an mlockall error written > > out, which means, that it is not enabled (we recently added, if the > > mlockall was successful on startup, so you can see it in the nodes info, > > but it is not yet part of a 0.90 release). > > > > --Alex > > > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > > > > So, for ES, swap is a bad thing, right? > > > I'm running, currently, a mixed Windows and Linux environment. All > > > running 0.90.5. > > > On linux (RHEL): > > > Locked memory has been set to unlimited. mlockall is (supposed to be) > > > on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > > > Limits applied to the Linux process: > > > > > > Max open files 65536 65536 files > > > > > > Max locked memory unlimited unlimited bytes > > > I still see swap space getting used, just a few megs at first, and I've > > > seen it get as far up as 800mb in elastichq > > > > > > The only way I can get swap to turn off for ES is to turn off swap for > > > the whole system - which seems kinda overkill. File descriptor limits on > > > linux seem fine > > > > > > What am I missing? I feel like there's probably one little thing that > > > will make it all click into place and I've just glazed over it. > > > > > > On Windows > > > There is no mlockall. I briefly tried disabling the page file and > > > rebooting the server, but when ES came back up, it had its same 32GB swap > > > usage show up in elastichq as it did when the page file was around. > > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > > instability - rivers dropping, response times to a simple query taking 5-10 > > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > > at 24GB, but not quite as often. > > > There is also, apparently, no functional way to check file descriptors > > > in use, or how many the process is allowed to open. I just get back the -1 > > > unknown value. Do file descriptor counts not matter on Windows? > > > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > > since it sounds like there are at least a few people out there running > > > production ES windows servers, how do I manage it properly? > > > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > > enough red tape in the way that it will take a while. > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a% > > > [40googlegroups.com](http://40googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/36b476c6-ff27-47e8-ae03-f2a3ea6a3bf5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36b476c6-ff27-47e8-ae03-f2a3ea6a3bf5%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_7GMAcUf8xNGH7-ccAkPrG3e6Kt%3DY6kKcaCmNbe5gOkg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_7GMAcUf8xNGH7-ccAkPrG3e6Kt%3DY6kKcaCmNbe5gOkg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 9, 2013, 8:10am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/12 "2013-12-09T08:10:25Z")
Hey Josh, I think your mentions might make sense to dedicate a whole section in the reference guide to this, in order to make things more clear. As I fell over this already myself, I added a process information in the nodes info output, which tells, if the mlockall call was successful (so there is no need anymore to look into logfiles). Still, adding more documentation makes sense, I'l try to put it somewhere into the documentation section over the week. Thanks for your feedback! --Alex On Sat, Dec 7, 2013 at 7:18 PM, Joshua Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > I'd probably mention it in the default elasticsearch.yml file with a line > in the description of mlockall like: > If running as a linux service from an RPM build it (may be/is) necessary > to set LimitMEMLOCK=infinity in /etc/systemd/system/elasticsearch.service > Mostly because all of the documentation, suggestions and stackoverflow > answers just say "turn on mlockall", having this near that may be helpful. > > Alternatively, I don't know if that error 0 happens in other > circumstances, it probably does, but it could be worth injecting something > like the above into the log when that error happens as a possible solution. > > Finally, I figured out that the heap usage reported in elasticHQ and other > similar tools appears to be the swap usage of the entire host system - not > of ES. Since swap is a bad thing for ES, would it be possible to get a > property in \_node that actually shows what ES itself is using? > Thanks Alex! > -Josh > > On Dec 7, 2013, at 7:07 AM, Alexander Reelsen [alr@spinscale.de](mailto:alr@spinscale.de) wrote: > > Hey Josh, > > glad it is working now. Can you have a look at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_linux.htmland) tell me, if we can improve the docs somehow (or also mentioning it > differently in the sysconfig file maybe). Any pointer how we can make this > more failureproof? Mentioning it more obvious somewhere? (Except that I > should have thought about systemd earlier 😉 > > Any help is much appreciated! > > --Alex > > On Fri, Dec 6, 2013 at 8:20 PM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > > > Yep, using the RPM. > > I ended up following the settings in here: > > [Limits are not consumed using Systemd (ulimit -n / ulimit -l) by skymeyer · Pull Request #3355 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/pull/3355) and now top > > says I have a swap usage of 0, agreeing with /proc/pid/smaps - though > > elastichq still reports swap usage, so I'm not sure what to make of that. > > So for those of you that run into this in the future, try this: > > After applying fix > > > > Removing limits from /etc/security/limits.conf > > > > #elasticsearch soft nofile 65535 > > #elasticsearch hard nofile 65535 > > #elasticsearch - memlock unlimited > > > > /etc/sysconfig/elasticsearch: > > > > # Maximum number of open files > > > > MAX\_OPEN\_FILES=65535 > > > > # Maximum amount of locked memory > > > > MAX\_LOCKED\_MEMORY=unlimited > > > > /etc/elasticsearch/elasticsearch.yml: > > > > bootstrap.mlockall: true > > > > Actual fix in /etc/systemd/system/elasticsearch.service > > > > [Service] > > ... > > LimitMEMLOCK=infinity > > LimitNOFILE=65535 > > > > Note: run the following command after altering the above file: > > > > systemctl --system daemon-reload > > > > Result max\_open\_files and no mlockall error: > > > > /etc/rc.d/init.d/elasticsearch restart > > [2013-07-18 11:20:21,476][INFO][bootstrap] max\_open\_files [65511] > > [2013-07-18 11:20:21,616][INFO][node] [node1] {0.90.2}[28558]: initializing ... > > > > On Friday, December 6, 2013 1:13:34 AM UTC-8, Alexander Reelsen wrote: > > > > > Hey, > > > > > > ok, so mlockall is configured but not set on startup - at least you know > > > why it is swapping. > > > > > > How are you starting elasticsearch? Do you use the RPM? If so, I guess > > > you set MAX\_LOCKED\_MEMORY=unlimited in there? If not, can you try? > > > Another issue might be (depending on how ES gets started), that the > > > memlock setting is configured for the root user, but not for the > > > elasticsearch one (wild guessing here). > > > > > > --Alex > > > > > > On Fri, Dec 6, 2013 at 9:24 AM, Joshua Harrison [hij...@gmail.com](mailto:hij...@gmail.com)wrote: > > > > > > > At the moment, the only reason for the mixed environment is to provide > > > > an easy migration path for when we burn down the windows node and reimage > > > > to RHEL. If we can force the Windows node to be as stable as its linux > > > > counterparts, we may just stick with it, though. > > > > > > > > I am indeed getting an mlock error > > > > [2013-12-05 21:25:34,439][WARN][common.jna] Unknown > > > > mlockall error 0 > > > > ulimit -l unlimited has been set, which is the only suggestion I've > > > > been able to find for this particular error. > > > > > > > > Found this thread, [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/) > > > > elasticsearch/0CaBak7sdRE but it looks like they didn't find a > > > > solution. > > > > Thanks, > > > > Josh > > > > > > > > On Dec 6, 2013, at 12:09 AM, Alexander Reelsen [a...@spinscale.de](mailto:a...@spinscale.de) > > > > wrote: > > > > > > > > Hey, > > > > > > > > I would not recommend running a mixed environment, as debugging will > > > > become pretty much PITA when debugging different performance statistics of > > > > operating systems - might be something for the really adventurous of us. > > > > > > > > Now back to topic: If you enabled mlockall, but the elasticsearch > > > > process is still swapping (can you please verify by checking top or the > > > > /proc file system, I have no idea how elastichq is measuring this to be > > > > honest, maybe Roy can help here?), mlockall seems not enabled. Can you > > > > check the elasticsearch log file on startup, maybe there is an mlockall > > > > error written out, which means, that it is not enabled (we recently added, > > > > if the mlockall was successful on startup, so you can see it in the nodes > > > > info, but it is not yet part of a 0.90 release). > > > > > > > > --Alex > > > > > > > > On Fri, Dec 6, 2013 at 8:43 AM, Josh Harrison [hij...@gmail.com](mailto:hij...@gmail.com) wrote: > > > > > > > > > So, for ES, swap is a bad thing, right? > > > > > I'm running, currently, a mixed Windows and Linux environment. All > > > > > running 0.90.5. > > > > > On linux (RHEL): > > > > > Locked memory has been set to unlimited. mlockall is (supposed to be) > > > > > on. ES\_HEAP\_SIZE is set to 12gb (24gb of ram on server). > > > > > > > > > > Limits applied to the Linux process: > > > > > > > > > > Max open files 65536 65536 > > > > > files > > > > > > > > > > Max locked memory unlimited unlimited > > > > > bytes > > > > > I still see swap space getting used, just a few megs at first, and > > > > > I've seen it get as far up as 800mb in elastichq > > > > > > > > > > The only way I can get swap to turn off for ES is to turn off swap for > > > > > the whole system - which seems kinda overkill. File descriptor limits on > > > > > linux seem fine > > > > > > > > > > What am I missing? I feel like there's probably one little thing that > > > > > will make it all click into place and I've just glazed over it. > > > > > > > > > > On Windows > > > > > There is no mlockall. I briefly tried disabling the page file and > > > > > rebooting the server, but when ES came back up, it had its same 32GB swap > > > > > usage show up in elastichq as it did when the page file was around. > > > > > ES\_HEAP\_SIZE is set to 24GB, 64GB on the server but I was getting even more > > > > > instability - rivers dropping, response times to a simple query taking 5-10 > > > > > minutes, etc, at the recommended 30GB. I still get to enjoy those things on > > > > > at 24GB, but not quite as often. > > > > > There is also, apparently, no functional way to check file descriptors > > > > > in use, or how many the process is allowed to open. I just get back the -1 > > > > > unknown value. Do file descriptor counts not matter on Windows? > > > > > > > > > > Is swap not a bad word on Windows as far as ES is concerned? If it is, > > > > > since it sounds like there are at least a few people out there running > > > > > production ES windows servers, how do I manage it properly? > > > > > > > > > > Ideally, we'll be moving the Windows server to RHEL, but there may be > > > > > enough red tape in the way that it will take a while. > > > > > > > > > > Thanks! > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/be6818aa-6593-47ab-93cf-76104da7406a% > > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > topic/elasticsearch/JWtzbphQEVQ/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAGCwEM\_TKRqAkHHxft7Q%2BDFKHXj6hGjnGDSmER\_% > > > > 3DtNaChf5kGg%[40mail.gmail.com](http://40mail.gmail.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/A50D1C97-C6C6-43D7-8CEB-2B29DC863B3F%[40gmail.com](http://40gmail.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5eaf1820-72e4-4dd9-89ae-01f0bef815d5%40googlegroups.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/JWtzbphQEVQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_oDpVxp5bfXKzZTub4f9YqvRacqcvYTVU2idEm9mtGOQ%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1CE5CD83-FA46-400B-8C03-91A1A4FEBBAC%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/1CE5CD83-FA46-400B-8C03-91A1A4FEBBAC%40gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9sR9B4ZKRqvipXzAj634Vdtv5vVGT%2Bzon1%2BCrzp9ijvA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9sR9B4ZKRqvipXzAj634Vdtv5vVGT%2Bzon1%2BCrzp9ijvA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/killing-swap-once-and-for-all-file-descriptors-too-linux-and-windows/14745/13 "2017-07-06T02:02:35Z")