# ES/Lucene eating up entire memory! **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2015, 8:22pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972 "2015-03-29T20:22:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Yogesh\_Kansal](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@Yogesh\_Kansal](https://discuss.elastic.co/u/Yogesh_Kansal)\ **Post date:** [March 29, 2015, 8:22pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/1 "2015-03-29T20:22:59Z")
Hi, I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when I do "top", the ES process shows the VIRT memory to be around 34g. That would be I assume the max mapped memory. The %MEM though always hovers around 10% However, within a few days post-reboot, the memory used keeps going up. From 10g to almost 50g (as shown in the third line) because of which my other dbs start behaving badly. Below is the snapshot of "top". Despite the fact that VIRT and %MEM still hover around the same 34g and 10% respectively. Please help me understand where is my memory going over time! My one guess is that Lucene is eating it up. How do I remedy it? Thanks-in-advance! [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![rmuir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmuir/32/44949_2.png) [@rmuir](https://discuss.elastic.co/u/rmuir)\ **Post date:** [March 29, 2015, 9:38pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/2 "2015-03-29T21:38:45Z")
Do you know what virtual memory is? You have terabytes of it. On Sun, Mar 29, 2015 at 4:22 PM, Yogesh [bindasyogesh@gmail.com](mailto:bindasyogesh@gmail.com) wrote: > Hi, > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run > the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when I > do "top", the ES process shows the VIRT memory to be around 34g. That would > be I assume the max mapped memory. The %MEM though always hovers around 10% > > However, within a few days post-reboot, the memory used keeps going up. > From 10g to almost 50g (as shown in the third line) because of which my > other dbs start behaving badly. Below is the snapshot of "top". Despite the > fact that VIRT and %MEM still hover around the same 34g and 10% > respectively. > > Please help me understand where is my memory going over time! My one guess > is that Lucene is eating it up. How do I remedy it? > > Thanks-in-advance! > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6eaac271-9b6b-4d6e-84b3-2c1194e0796b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOdYfZWCV0c-bMsg1\_x%3DtYdB9tYjXHwY%2ByuknxuWJGYP99\_uTA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOdYfZWCV0c-bMsg1_x%3DtYdB9tYjXHwY%2ByuknxuWJGYP99_uTA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Uwe\_Schindler](https://avatars.discourse-cdn.com/v4/letter/u/a8b319/32.png) [@Uwe\_Schindler](https://discuss.elastic.co/u/Uwe_Schindler)\ **Post date:** [March 29, 2015, 9:49pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/3 "2015-03-29T21:49:13Z")
You should read: [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) Maybe this allows you to figure out what's going on! VIRT means nothing about consumption, you should look at RES. Thanks, Uwe Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > Hi, > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I run > the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when I > do "top", the ES process shows the VIRT memory to be around 34g. That would > be I assume the max mapped memory. The %MEM though always hovers around 10% > > However, within a few days post-reboot, the memory used keeps going up. > From 10g to almost 50g (as shown in the third line) because of which my > other dbs start behaving badly. Below is the snapshot of "top". Despite the > fact that VIRT and %MEM still hover around the same 34g and 10% > respectively. > > Please help me understand where is my memory going over time! My one guess > is that Lucene is eating it up. How do I remedy it? > > Thanks-in-advance! > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yogesh\_Kansal](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@Yogesh\_Kansal](https://discuss.elastic.co/u/Yogesh_Kansal)\ **Post date:** [March 31, 2015, 10:24am UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/4 "2015-03-31T10:24:52Z")
Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't indicate RAM consumption. What I am concerned about is the 3rd row which shows memory and indicates that out of the total 50g, 43g is in use. Once this number crosses 45g, my other databases start behaving badly. Problem is, even after I kill all the processes, this doesn't go down. (Attaching snapshot of top after killing all processes). Right now what I do is reboot the system every three days which is the time it takes to gradually fill the memory with something (I have no clue what that is). Though I think the max file descriptors wouldn't be the culprit for this? I haven't changed that yet. On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler [uwe.h.schindler@gmail.com](mailto:uwe.h.schindler@gmail.com) wrote: > You should read: > [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) > > Maybe this allows you to figure out what's going on! VIRT means nothing > about consumption, you should look at RES. > > Thanks, > Uwe > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > Hi, > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > would be I assume the max mapped memory. The %MEM though always hovers > > around 10% > > > > However, within a few days post-reboot, the memory used keeps going up. > > From 10g to almost 50g (as shown in the third line) because of which my > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > fact that VIRT and %MEM still hover around the same 34g and 10% > > respectively. > > > > Please help me understand where is my memory going over time! My one > > guess is that Lucene is eating it up. How do I remedy it? > > > > Thanks-in-advance! > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADM0w%3Di5%2B\_Cd-swP3f58jm-cE%2B7ULsq6QwfafQjsmka47h3fkg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADM0w%3Di5%2B_Cd-swP3f58jm-cE%2B7ULsq6QwfafQjsmka47h3fkg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![aaronmefford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaronmefford/32/460_2.png) [@aaronmefford](https://discuss.elastic.co/u/aaronmefford)\ **Post date:** [March 31, 2015, 4:27pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/5 "2015-03-31T16:27:57Z")
You need to read up a bit on how memory is allocated in Linux. In an Elasticsearch or Database server, this seems to be both, you want that free column to be 0. All available free memory should be used to cache files. In your snapshot you have 35GB of file cache listed under the cached heading. Memory listed under cached is essentially free memory that is temporarily being used to cache files until it is otherwise requested. This is how Linux makes efficient use of your memory, leveraging free memory for file cache, but still having it available when you need it. As such when determining if your box is out of memory you need to sum free and cached. This is precisely the reason that it is recommended that Elasticsearch only be allocated 50% of the memory on the box for heap. In your case where you have databases running, it should be 50% of the memory you have available for Elasticsearch. For that matter you should apply the same basic rule( 50%) to your database unless it has specifically some other file caching mechanism. For instance, you have 50GB of ram, assuming MySQL and Elasticsearch, and you want to equally divide the ram, 25GB to each. Elasticsearch then would be allowed to use 25GB, 12GB should be allocated to heap, the balance left to the OS for file caching on behalf of Elasticsearch. Assuming MySQL, with MyIsam, the same would be done 12GB to MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if you are using InnoDB things are different but that is way outside the scope of this discussion. So that you have 35GB of files being cached is a very good thing. It means that you have a large amount of your data cached. It means you have ample free memory, well beyond the 12GB a 50/50 split would demand. The 12GB of free you have now probably came from the processes that you killed, I think you meant this was Elasticsearch, though you were not specific. The one concern I see looking at your top, is that you have a large swap, and that some of it has been used. This is a sign that at some point you had memory pressure, the only sign I see from your snapshot. That pressure was not significant, but any swapping will destroy the performance of a database, or Elasticsearch. In many cases people go to the extreme of disabling swap entirely, as performance during swapping will be so poor, that it will be unusable. Further by the time you were to even put a dent in the size of that swap you will have wanted to reboot your box. My approach is to keep a small swap available, so that I can see if the system ever got to a point that it needed it, and to potentially buy a moment of time. If you are experiencing database slowdowns, this screenshot does not illustrate that it is due to memory issues. Rather I would suspect disk IO instead based on this information. On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: > Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't indicate > RAM consumption. > > What I am concerned about is the 3rd row which shows memory and indicates > that out of the total 50g, 43g is in use. Once this number crosses 45g, my > other databases start behaving badly. > > Problem is, even after I kill all the processes, this doesn't go down. > (Attaching snapshot of top after killing all processes). Right now what I > do is reboot the system every three days which is the time it takes to > gradually fill the memory with something (I have no clue what that is). > > Though I think the max file descriptors wouldn't be the culprit for this? > I haven't changed that yet. > > On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler \<[uwe.h.s...@gmail.com](mailto:uwe.h.s...@gmail.com) > \\> wrote: > > > You should read: > > [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) > > > > Maybe this allows you to figure out what's going on! VIRT means nothing > > about consumption, you should look at RES. > > > > Thanks, > > Uwe > > > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > > > Hi, > > > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > > would be I assume the max mapped memory. The %MEM though always hovers > > > around 10% > > > > > > However, within a few days post-reboot, the memory used keeps going up. > > > From 10g to almost 50g (as shown in the third line) because of which my > > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > > fact that VIRT and %MEM still hover around the same 34g and 10% > > > respectively. > > > > > > Please help me understand where is my memory going over time! My one > > > guess is that Lucene is eating it up. How do I remedy it? > > > > > > Thanks-in-advance! > > > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 31, 2015, 8:37pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/6 "2015-03-31T20:37:09Z")
Also, we don't recommend running ES alongside other apps. As you can see contention is an issue and you will have to pay the price there. On 1 April 2015 at 03:27, Aaron Mefford [aaron@mefford.org](mailto:aaron@mefford.org) wrote: > You need to read up a bit on how memory is allocated in Linux. > > In an Elasticsearch or Database server, this seems to be both, you want > that free column to be 0. All available free memory should be used to > cache files. In your snapshot you have 35GB of file cache listed under the > cached heading. Memory listed under cached is essentially free memory that > is temporarily being used to cache files until it is otherwise requested. > This is how Linux makes efficient use of your memory, leveraging free > memory for file cache, but still having it available when you need it. As > such when determining if your box is out of memory you need to sum free and > cached. > > This is precisely the reason that it is recommended that Elasticsearch > only be allocated 50% of the memory on the box for heap. In your case > where you have databases running, it should be 50% of the memory you have > available for Elasticsearch. For that matter you should apply the same > basic rule( 50%) to your database unless it has specifically some other > file caching mechanism. For instance, you have 50GB of ram, assuming MySQL > and Elasticsearch, and you want to equally divide the ram, 25GB to each. > Elasticsearch then would be allowed to use 25GB, 12GB should be allocated > to heap, the balance left to the OS for file caching on behalf of > Elasticsearch. Assuming MySQL, with MyIsam, the same would be done 12GB to > MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if > you are using InnoDB things are different but that is way outside the scope > of this discussion. > > So that you have 35GB of files being cached is a very good thing. It > means that you have a large amount of your data cached. It means you have > ample free memory, well beyond the 12GB a 50/50 split would demand. The > 12GB of free you have now probably came from the processes that you killed, > I think you meant this was Elasticsearch, though you were not specific. > > The one concern I see looking at your top, is that you have a large swap, > and that some of it has been used. This is a sign that at some point you > had memory pressure, the only sign I see from your snapshot. That pressure > was not significant, but any swapping will destroy the performance of a > database, or Elasticsearch. In many cases people go to the extreme of > disabling swap entirely, as performance during swapping will be so poor, > that it will be unusable. Further by the time you were to even put a dent > in the size of that swap you will have wanted to reboot your box. My > approach is to keep a small swap available, so that I can see if the system > ever got to a point that it needed it, and to potentially buy a moment of > time. > > If you are experiencing database slowdowns, this screenshot does not > illustrate that it is due to memory issues. Rather I would suspect disk IO > instead based on this information. > > On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: > > > Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't > > indicate RAM consumption. > > > > What I am concerned about is the 3rd row which shows memory and indicates > > that out of the total 50g, 43g is in use. Once this number crosses 45g, my > > other databases start behaving badly. > > > > Problem is, even after I kill all the processes, this doesn't go down. > > (Attaching snapshot of top after killing all processes). Right now what I > > do is reboot the system every three days which is the time it takes to > > gradually fill the memory with something (I have no clue what that is). > > > > Though I think the max file descriptors wouldn't be the culprit for this? > > I haven't changed that yet. > > > > On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler [uwe.h.s...@gmail.com](mailto:uwe.h.s...@gmail.com) > > wrote: > > > > > You should read: [http://blog.thetaphi.de/2012/07/use-lucenes-](http://blog.thetaphi.de/2012/07/use-lucenes-) > > > mmapdirectory-on-64bit.html > > > > > > Maybe this allows you to figure out what's going on! VIRT means nothing > > > about consumption, you should look at RES. > > > > > > Thanks, > > > Uwe > > > > > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > > > > > Hi, > > > > > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > > > would be I assume the max mapped memory. The %MEM though always hovers > > > > around 10% > > > > > > > > However, within a few days post-reboot, the memory used keeps going up. > > > > From 10g to almost 50g (as shown in the third line) because of which my > > > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > > > fact that VIRT and %MEM still hover around the same 34g and 10% > > > > respectively. > > > > > > > > Please help me understand where is my memory going over time! My one > > > > guess is that Lucene is eating it up. How do I remedy it? > > > > > > > > Thanks-in-advance! > > > > > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > topic/elasticsearch/kTDNDJwxOzA/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-9%3DPXrTjZz\_AHL\_1%2Bv65cy7YLPQ7A7WfZP%2B1swWNLFRQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-9%3DPXrTjZz_AHL_1%2Bv65cy7YLPQ7A7WfZP%2B1swWNLFRQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yogesh\_Kansal](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@Yogesh\_Kansal](https://discuss.elastic.co/u/Yogesh_Kansal)\ **Post date:** [April 3, 2015, 4:52pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/7 "2015-04-03T16:52:57Z")
Thanks Aaron. Your post was very informative. Can you recommend any blogposts, articles etc. where I could read more on this topic? Thanks again for your help. On Tuesday, March 31, 2015 at 9:57:58 PM UTC+5:30, Aaron Mefford wrote: > You need to read up a bit on how memory is allocated in Linux. > > In an Elasticsearch or Database server, this seems to be both, you want > that free column to be 0. All available free memory should be used to > cache files. In your snapshot you have 35GB of file cache listed under the > cached heading. Memory listed under cached is essentially free memory that > is temporarily being used to cache files until it is otherwise requested. > This is how Linux makes efficient use of your memory, leveraging free > memory for file cache, but still having it available when you need it. As > such when determining if your box is out of memory you need to sum free and > cached. > > This is precisely the reason that it is recommended that Elasticsearch > only be allocated 50% of the memory on the box for heap. In your case > where you have databases running, it should be 50% of the memory you have > available for Elasticsearch. For that matter you should apply the same > basic rule( 50%) to your database unless it has specifically some other > file caching mechanism. For instance, you have 50GB of ram, assuming MySQL > and Elasticsearch, and you want to equally divide the ram, 25GB to each. > Elasticsearch then would be allowed to use 25GB, 12GB should be allocated > to heap, the balance left to the OS for file caching on behalf of > Elasticsearch. Assuming MySQL, with MyIsam, the same would be done 12GB to > MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if > you are using InnoDB things are different but that is way outside the scope > of this discussion. > > So that you have 35GB of files being cached is a very good thing. It > means that you have a large amount of your data cached. It means you have > ample free memory, well beyond the 12GB a 50/50 split would demand. The > 12GB of free you have now probably came from the processes that you killed, > I think you meant this was Elasticsearch, though you were not specific. > > The one concern I see looking at your top, is that you have a large swap, > and that some of it has been used. This is a sign that at some point you > had memory pressure, the only sign I see from your snapshot. That pressure > was not significant, but any swapping will destroy the performance of a > database, or Elasticsearch. In many cases people go to the extreme of > disabling swap entirely, as performance during swapping will be so poor, > that it will be unusable. Further by the time you were to even put a dent > in the size of that swap you will have wanted to reboot your box. My > approach is to keep a small swap available, so that I can see if the system > ever got to a point that it needed it, and to potentially buy a moment of > time. > > If you are experiencing database slowdowns, this screenshot does not > illustrate that it is due to memory issues. Rather I would suspect disk IO > instead based on this information. > > On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: > > > Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't > > indicate RAM consumption. > > > > What I am concerned about is the 3rd row which shows memory and indicates > > that out of the total 50g, 43g is in use. Once this number crosses 45g, my > > other databases start behaving badly. > > > > Problem is, even after I kill all the processes, this doesn't go down. > > (Attaching snapshot of top after killing all processes). Right now what I > > do is reboot the system every three days which is the time it takes to > > gradually fill the memory with something (I have no clue what that is). > > > > Though I think the max file descriptors wouldn't be the culprit for this? > > I haven't changed that yet. > > > > On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler [uwe.h.s...@gmail.com](mailto:uwe.h.s...@gmail.com) > > wrote: > > > > > You should read: > > > [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) > > > > > > Maybe this allows you to figure out what's going on! VIRT means nothing > > > about consumption, you should look at RES. > > > > > > Thanks, > > > Uwe > > > > > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > > > > > Hi, > > > > > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > > > would be I assume the max mapped memory. The %MEM though always hovers > > > > around 10% > > > > > > > > However, within a few days post-reboot, the memory used keeps going up. > > > > From 10g to almost 50g (as shown in the third line) because of which my > > > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > > > fact that VIRT and %MEM still hover around the same 34g and 10% > > > > respectively. > > > > > > > > Please help me understand where is my memory going over time! My one > > > > guess is that Lucene is eating it up. How do I remedy it? > > > > > > > > Thanks-in-advance! > > > > > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe). > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ae78aa3c-d951-4af5-bf02-8630eaa92363%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ae78aa3c-d951-4af5-bf02-8630eaa92363%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yogesh\_Kansal](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@Yogesh\_Kansal](https://discuss.elastic.co/u/Yogesh_Kansal)\ **Post date:** [April 3, 2015, 5:03pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/8 "2015-04-03T17:03:40Z")
Thanks Mark. I separated out the processes on different servers, so now ES has a full server. I made the changes yesterday and it seems to be stable since then. On Wednesday, April 1, 2015 at 2:07:41 AM UTC+5:30, Mark Walkom wrote: > Also, we don't recommend running ES alongside other apps. As you can see > contention is an issue and you will have to pay the price there. > > On 1 April 2015 at 03:27, Aaron Mefford \<[aa...@mefford.org](mailto:aa...@mefford.org) \\> > wrote: > > > You need to read up a bit on how memory is allocated in Linux. > > > > In an Elasticsearch or Database server, this seems to be both, you want > > that free column to be 0. All available free memory should be used to > > cache files. In your snapshot you have 35GB of file cache listed under the > > cached heading. Memory listed under cached is essentially free memory that > > is temporarily being used to cache files until it is otherwise requested. > > This is how Linux makes efficient use of your memory, leveraging free > > memory for file cache, but still having it available when you need it. As > > such when determining if your box is out of memory you need to sum free and > > cached. > > > > This is precisely the reason that it is recommended that Elasticsearch > > only be allocated 50% of the memory on the box for heap. In your case > > where you have databases running, it should be 50% of the memory you have > > available for Elasticsearch. For that matter you should apply the same > > basic rule( 50%) to your database unless it has specifically some other > > file caching mechanism. For instance, you have 50GB of ram, assuming MySQL > > and Elasticsearch, and you want to equally divide the ram, 25GB to each. > > Elasticsearch then would be allowed to use 25GB, 12GB should be allocated > > to heap, the balance left to the OS for file caching on behalf of > > Elasticsearch. Assuming MySQL, with MyIsam, the same would be done 12GB to > > MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if > > you are using InnoDB things are different but that is way outside the scope > > of this discussion. > > > > So that you have 35GB of files being cached is a very good thing. It > > means that you have a large amount of your data cached. It means you have > > ample free memory, well beyond the 12GB a 50/50 split would demand. The > > 12GB of free you have now probably came from the processes that you killed, > > I think you meant this was Elasticsearch, though you were not specific. > > > > The one concern I see looking at your top, is that you have a large swap, > > and that some of it has been used. This is a sign that at some point you > > had memory pressure, the only sign I see from your snapshot. That pressure > > was not significant, but any swapping will destroy the performance of a > > database, or Elasticsearch. In many cases people go to the extreme of > > disabling swap entirely, as performance during swapping will be so poor, > > that it will be unusable. Further by the time you were to even put a dent > > in the size of that swap you will have wanted to reboot your box. My > > approach is to keep a small swap available, so that I can see if the system > > ever got to a point that it needed it, and to potentially buy a moment of > > time. > > > > If you are experiencing database slowdowns, this screenshot does not > > illustrate that it is due to memory issues. Rather I would suspect disk IO > > instead based on this information. > > > > On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: > > > > > Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't > > > indicate RAM consumption. > > > > > > What I am concerned about is the 3rd row which shows memory and > > > indicates that out of the total 50g, 43g is in use. Once this number > > > crosses 45g, my other databases start behaving badly. > > > > > > Problem is, even after I kill all the processes, this doesn't go down. > > > (Attaching snapshot of top after killing all processes). Right now what I > > > do is reboot the system every three days which is the time it takes to > > > gradually fill the memory with something (I have no clue what that is). > > > > > > Though I think the max file descriptors wouldn't be the culprit for > > > this? I haven't changed that yet. > > > > > > On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler [uwe.h.s...@gmail.com](mailto:uwe.h.s...@gmail.com) > > > wrote: > > > > > > > You should read: [http://blog.thetaphi.de/2012/07/use-lucenes-](http://blog.thetaphi.de/2012/07/use-lucenes-) > > > > mmapdirectory-on-64bit.html > > > > > > > > Maybe this allows you to figure out what's going on! VIRT means nothing > > > > about consumption, you should look at RES. > > > > > > > > Thanks, > > > > Uwe > > > > > > > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > > > > > > > Hi, > > > > > > > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > > > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > > > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > > > > would be I assume the max mapped memory. The %MEM though always hovers > > > > > around 10% > > > > > > > > > > However, within a few days post-reboot, the memory used keeps going > > > > > up. From 10g to almost 50g (as shown in the third line) because of which my > > > > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > > > > fact that VIRT and %MEM still hover around the same 34g and 10% > > > > > respectively. > > > > > > > > > > Please help me understand where is my memory going over time! My one > > > > > guess is that Lucene is eating it up. How do I remedy it? > > > > > > > > > > Thanks-in-advance! > > > > > > > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > topic/elasticsearch/kTDNDJwxOzA/unsubscribe. > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11d2d973-bbe8-49e3-a2ba-b9ad2c3df0b3%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4df4d368-1113-4718-9a14-3bf5abd10ea0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4df4d368-1113-4718-9a14-3bf5abd10ea0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![aaronmefford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaronmefford/32/460_2.png) [@aaronmefford](https://discuss.elastic.co/u/aaronmefford)\ **Post date:** [April 3, 2015, 7:54pm UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/9 "2015-04-03T19:54:15Z")
First of all the what, not so much the why: > **[Heap: Sizing and Swapping | Elasticsearch: The Definitive Guide \[master\] |...](https://www.elastic.co/guide/en/elasticsearch/guide/master/heap-sizing.html)** In order of depth, first one provides light coverage of the topic: [http://linux-mm.org/Low\_On\_Memory](http://linux-mm.org/Low_On_Memory) [http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm](http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm) [http://www.tldp.org/LDP/tlk/mm/memory.html](http://www.tldp.org/LDP/tlk/mm/memory.html) > **[understand.pdf](https://www.kernel.org/doc/gorman/pdf/understand.pdf)** > > 1834.76 KB This is also pretty good but deals more with memory management in ES not out. [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html) On Friday, April 3, 2015 at 10:52:57 AM UTC-6, Yogesh wrote: > Thanks Aaron. Your post was very informative. > Can you recommend any blogposts, articles etc. where I could read more on > this topic? > > Thanks again for your help. > > On Tuesday, March 31, 2015 at 9:57:58 PM UTC+5:30, Aaron Mefford wrote: > > > You need to read up a bit on how memory is allocated in Linux. > > > > In an Elasticsearch or Database server, this seems to be both, you want > > that free column to be 0. All available free memory should be used to > > cache files. In your snapshot you have 35GB of file cache listed under the > > cached heading. Memory listed under cached is essentially free memory that > > is temporarily being used to cache files until it is otherwise requested. > > This is how Linux makes efficient use of your memory, leveraging free > > memory for file cache, but still having it available when you need it. As > > such when determining if your box is out of memory you need to sum free and > > cached. > > > > This is precisely the reason that it is recommended that Elasticsearch > > only be allocated 50% of the memory on the box for heap. In your case > > where you have databases running, it should be 50% of the memory you have > > available for Elasticsearch. For that matter you should apply the same > > basic rule( 50%) to your database unless it has specifically some other > > file caching mechanism. For instance, you have 50GB of ram, assuming MySQL > > and Elasticsearch, and you want to equally divide the ram, 25GB to each. > > Elasticsearch then would be allowed to use 25GB, 12GB should be allocated > > to heap, the balance left to the OS for file caching on behalf of > > Elasticsearch. Assuming MySQL, with MyIsam, the same would be done 12GB to > > MySQL, 12GB to the OS for file system caching of the MyISAM tables. Now if > > you are using InnoDB things are different but that is way outside the scope > > of this discussion. > > > > So that you have 35GB of files being cached is a very good thing. It > > means that you have a large amount of your data cached. It means you have > > ample free memory, well beyond the 12GB a 50/50 split would demand. The > > 12GB of free you have now probably came from the processes that you killed, > > I think you meant this was Elasticsearch, though you were not specific. > > > > The one concern I see looking at your top, is that you have a large swap, > > and that some of it has been used. This is a sign that at some point you > > had memory pressure, the only sign I see from your snapshot. That pressure > > was not significant, but any swapping will destroy the performance of a > > database, or Elasticsearch. In many cases people go to the extreme of > > disabling swap entirely, as performance during swapping will be so poor, > > that it will be unusable. Further by the time you were to even put a dent > > in the size of that swap you will have wanted to reboot your box. My > > approach is to keep a small swap available, so that I can see if the system > > ever got to a point that it needed it, and to potentially buy a moment of > > time. > > > > If you are experiencing database slowdowns, this screenshot does not > > illustrate that it is due to memory issues. Rather I would suspect disk IO > > instead based on this information. > > > > On Tuesday, March 31, 2015 at 4:25:40 AM UTC-6, Yogesh wrote: > > > > > Thanks Uwe. As I mentioned earlier, I did guess that VIRT doesn't > > > indicate RAM consumption. > > > > > > What I am concerned about is the 3rd row which shows memory and > > > indicates that out of the total 50g, 43g is in use. Once this number > > > crosses 45g, my other databases start behaving badly. > > > > > > Problem is, even after I kill all the processes, this doesn't go down. > > > (Attaching snapshot of top after killing all processes). Right now what I > > > do is reboot the system every three days which is the time it takes to > > > gradually fill the memory with something (I have no clue what that is). > > > > > > Though I think the max file descriptors wouldn't be the culprit for > > > this? I haven't changed that yet. > > > > > > On Mon, Mar 30, 2015 at 3:19 AM, Uwe Schindler [uwe.h.s...@gmail.com](mailto:uwe.h.s...@gmail.com) > > > wrote: > > > > > > > You should read: > > > > [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) > > > > > > > > Maybe this allows you to figure out what's going on! VIRT means nothing > > > > about consumption, you should look at RES. > > > > > > > > Thanks, > > > > Uwe > > > > > > > > Am Sonntag, 29. März 2015 22:23:00 UTC+2 schrieb Yogesh: > > > > > > > > > Hi, > > > > > > > > > > I have a single node ES setup (50GB memory, 500GB disk, 4 cores) and I > > > > > run the Twitter river on it. I've set the ES\_HEAP\_SIZE to 5g. However, when > > > > > I do "top", the ES process shows the VIRT memory to be around 34g. That > > > > > would be I assume the max mapped memory. The %MEM though always hovers > > > > > around 10% > > > > > > > > > > However, within a few days post-reboot, the memory used keeps going > > > > > up. From 10g to almost 50g (as shown in the third line) because of which my > > > > > other dbs start behaving badly. Below is the snapshot of "top". Despite the > > > > > fact that VIRT and %MEM still hover around the same 34g and 10% > > > > > respectively. > > > > > > > > > > Please help me understand where is my memory going over time! My one > > > > > guess is that Lucene is eating it up. How do I remedy it? > > > > > > > > > > Thanks-in-advance! > > > > > > > > > > [https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png](https://lh3.googleusercontent.com/-zD9y4f2Eqqk/VRhdtX2XtTI/AAAAAAAAAN8/aq8-wxm2bBg/s1600/top.png) > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/kTDNDJwxOzA/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c6e834ab-77c4-4a99-9307-b6b3baf0d232%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9f5160f2-994e-4f08-99bf-0f0af181916e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f5160f2-994e-4f08-99bf-0f0af181916e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:21am UTC](https://discuss.elastic.co/t/es-lucene-eating-up-entire-memory/22972/10 "2017-07-06T00:21:45Z")