# ES vs. Lucene memory **URL:** \ **Category:** Elasticsearch\ **Created:** [November 26, 2014, 6:53pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959 "2014-11-26T18:53:03Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![BradVido](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradvido/32/741_2.png) [@BradVido](https://discuss.elastic.co/u/BradVido)\ **Post date:** [November 26, 2014, 6:53pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/1 "2014-11-26T18:53:03Z")
I've read the recommendations for ES\_HEAP\_SIZE [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which basically state to set -Xms and -Xmx to 50% physical RAM. It says the rest should be left for Lucene to use (OS filesystem caching). But I'm confused on how Lucene uses that. Doesn't Lucene run in the same JVM as ES? So they would share the same max heap setting of 50%. Can someone clear this up? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 26, 2014, 7:00pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/2 "2014-11-26T19:00:34Z")
Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps files and then iterates over their content inteligently. That means most of its actual storage is "off heap" (its a java buzz-phrase). Anyway, Linux will serve reads from mmaped files from its page cache. That is why you want to leave linux a whole bunch of unused memory. Nik On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyvidovic@gmail.com](mailto:bradyvidovic@gmail.com) wrote: > I've read the recommendations for ES\_HEAP\_SIZE > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > basically state to set -Xms and -Xmx to 50% physical RAM. > It says the rest should be left for Lucene to use (OS filesystem caching). > But I'm confused on how Lucene uses that. Doesn't Lucene run in the same > JVM as ES? So they would share the same max heap setting of 50%. > > Can someone clear this up? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2aPWDzg9PZcHpQyKbFiK\_p-385NR7z8PNsnpdmu%3D\_f4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2aPWDzg9PZcHpQyKbFiK_p-385NR7z8PNsnpdmu%3D_f4Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![BradVido](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradvido/32/741_2.png) [@BradVido](https://discuss.elastic.co/u/BradVido)\ **Post date:** [November 26, 2014, 7:42pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/3 "2014-11-26T19:42:43Z")
I see, but I'm running on Windows. Is the behavior similar, or does this not exist on Windows? On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett wrote: > Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps > files and then iterates over their content inteligently. That means most > of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > Linux will serve reads from mmaped files from its page cache. That is why > you want to leave linux a whole bunch of unused memory. > > Nik > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido \<[bradyv...@gmail.com](mailto:bradyv...@gmail.com) > \\> wrote: > > > I've read the recommendations for ES\_HEAP\_SIZE > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > basically state to set -Xms and -Xmx to 50% physical RAM. > > It says the rest should be left for Lucene to use (OS filesystem > > caching). > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the same > > JVM as ES? So they would share the same max heap setting of 50%. > > > > Can someone clear this up? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 26, 2014, 7:44pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/4 "2014-11-26T19:44:24Z")
I imagine all operating systems have some kind of disk caching. I just happen to be used to linux. On Wed, Nov 26, 2014 at 2:42 PM, BradVido [bradyvidovic@gmail.com](mailto:bradyvidovic@gmail.com) wrote: > I see, but I'm running on Windows. Is the behavior similar, or does this > not exist on Windows? > > On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett wrote: > > > Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps > > files and then iterates over their content inteligently. That means most > > of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > > Linux will serve reads from mmaped files from its page cache. That is why > > you want to leave linux a whole bunch of unused memory. > > > > Nik > > > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > I've read the recommendations for ES\_HEAP\_SIZE > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > > basically state to set -Xms and -Xmx to 50% physical RAM. > > > It says the rest should be left for Lucene to use (OS filesystem > > > caching). > > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the same > > > JVM as ES? So they would share the same max heap setting of 50%. > > > > > > Can someone clear this up? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [November 26, 2014, 8:51pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/5 "2014-11-26T20:51:00Z")
Indeed the behaviour is the same on Windows and Linux: memory that is not used by processes is used by the operating system in order to cache the hottest parts of the file system. The reason why the docs say that the rest should be left to Lucene is that most disk accesses that elasticsearch performs are done through Lucene. On Wed, Nov 26, 2014 at 8:44 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > I imagine all operating systems have some kind of disk caching. I just > happen to be used to linux. > > On Wed, Nov 26, 2014 at 2:42 PM, BradVido [bradyvidovic@gmail.com](mailto:bradyvidovic@gmail.com) wrote: > > > I see, but I'm running on Windows. Is the behavior similar, or does this > > not exist on Windows? > > > > On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett wrote: > > > > > Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps > > > files and then iterates over their content inteligently. That means most > > > of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > > > Linux will serve reads from mmaped files from its page cache. That is why > > > you want to leave linux a whole bunch of unused memory. > > > > > > Nik > > > > > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > > > I've read the recommendations for ES\_HEAP\_SIZE > > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > > > basically state to set -Xms and -Xmx to 50% physical RAM. > > > > It says the rest should be left for Lucene to use (OS filesystem > > > > caching). > > > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the > > > > same JVM as ES? So they would share the same max heap setting of 50%. > > > > > > > > Can someone clear this up? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j54pntydsUmd%3DAHcz0aRSd8YuD70XhukCwcghkFeG2ugA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j54pntydsUmd%3DAHcz0aRSd8YuD70XhukCwcghkFeG2ugA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liu\_wei](https://avatars.discourse-cdn.com/v4/letter/l/cab0a1/32.png) [@liu\_wei](https://discuss.elastic.co/u/liu_wei)\ **Post date:** [February 26, 2015, 6:14am UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/6 "2015-02-26T06:14:51Z")
Is there anyway to actually see how much memory is used by file system cache? A strange problem I see is, my machine has 32GB in total, i gave 16GB to ES. But very often the total OS memory usage will reach 100%, but task manager only shows around 12~13GB of the java exe. But when i kill the java process, memory usage dropped to only 20%. I can't explain why this is happening. Not sure if it's the file system cache. On Wednesday, November 26, 2014 at 12:51:11 PM UTC-8, Adrien Grand wrote: > Indeed the behaviour is the same on Windows and Linux: memory that is not > used by processes is used by the operating system in order to cache the > hottest parts of the file system. The reason why the docs say that the rest > should be left to Lucene is that most disk accesses that elasticsearch > performs are done through Lucene. > > On Wed, Nov 26, 2014 at 8:44 PM, Nikolas Everett \<[nik...@gmail.com](mailto:nik...@gmail.com) > \\> wrote: > > > I imagine all operating systems have some kind of disk caching. I just > > happen to be used to linux. > > > > On Wed, Nov 26, 2014 at 2:42 PM, BradVido \<[bradyv...@gmail.com](mailto:bradyv...@gmail.com) > > \\> wrote: > > > > > I see, but I'm running on Windows. Is the behavior similar, or does this > > > not exist on Windows? > > > > > > On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett wrote: > > > > > > > Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps > > > > files and then iterates over their content inteligently. That means most > > > > of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > > > > Linux will serve reads from mmaped files from its page cache. That is why > > > > you want to leave linux a whole bunch of unused memory. > > > > > > > > Nik > > > > > > > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > > > > > I've read the recommendations for ES\_HEAP\_SIZE > > > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > > > > basically state to set -Xms and -Xmx to 50% physical RAM. > > > > > It says the rest should be left for Lucene to use (OS filesystem > > > > > caching). > > > > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the > > > > > same JVM as ES? So they would share the same max heap setting of 50%. > > > > > > > > > > Can someone clear this up? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 26, 2015, 6:57am UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/7 "2015-02-26T06:57:36Z")
Definitely sounds like FS cache to me. On 26 February 2015 at 17:14, liu wei [liu.liuwei.wei@gmail.com](mailto:liu.liuwei.wei@gmail.com) wrote: > Is there anyway to actually see how much memory is used by file system > cache? > A strange problem I see is, my machine has 32GB in total, i gave 16GB to > ES. But very often the total OS memory usage will reach 100%, but task > manager only shows around 12~13GB of the java exe. But when i kill the java > process, memory usage dropped to only 20%. I can't explain why this is > happening. Not sure if it's the file system cache. > > On Wednesday, November 26, 2014 at 12:51:11 PM UTC-8, Adrien Grand wrote: > > > Indeed the behaviour is the same on Windows and Linux: memory that is not > > used by processes is used by the operating system in order to cache the > > hottest parts of the file system. The reason why the docs say that the rest > > should be left to Lucene is that most disk accesses that elasticsearch > > performs are done through Lucene. > > > > On Wed, Nov 26, 2014 at 8:44 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com) > > wrote: > > > > > I imagine all operating systems have some kind of disk caching. I just > > > happen to be used to linux. > > > > > > On Wed, Nov 26, 2014 at 2:42 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > > > I see, but I'm running on Windows. Is the behavior similar, or does > > > > this not exist on Windows? > > > > > > > > On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett wrote: > > > > > > > > > Lucene runs in the same JVM as Elasticsearch but (by default) it mmaps > > > > > files and then iterates over their content inteligently. That means most > > > > > of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > > > > > Linux will serve reads from mmaped files from its page cache. That is why > > > > > you want to leave linux a whole bunch of unused memory. > > > > > > > > > > Nik > > > > > > > > > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > > > > > > > I've read the recommendations for ES\_HEAP\_SIZE > > > > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > > > > > basically state to set -Xms and -Xmx to 50% physical RAM. > > > > > > It says the rest should be left for Lucene to use (OS filesystem > > > > > > caching). > > > > > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the > > > > > > same JVM as ES? So they would share the same max heap setting of 50%. > > > > > > > > > > > > Can someone clear this up? > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40goo > > > > > > [glegroups.com](http://glegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_ > > > Ek3tfAkuPA08Q1GYbdH2Rg%[40mail.gmail.com](http://40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_EzmESRXkZxgDf-sfcEOJqUyEbo3CJ7vwUO9b6ggDReA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_EzmESRXkZxgDf-sfcEOJqUyEbo3CJ7vwUO9b6ggDReA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liu\_wei](https://avatars.discourse-cdn.com/v4/letter/l/cab0a1/32.png) [@liu\_wei](https://discuss.elastic.co/u/liu_wei)\ **Post date:** [February 26, 2015, 8:12pm UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/8 "2015-02-26T20:12:15Z")
I used procexp and VMMap to double check, ya, i think they are file system cache. Is there anyway to control the size of file system cache? Coz now it's easily driving up OS memory consumption. When it's reaching 100%, the node would fail to respond... On Wednesday, February 25, 2015 at 10:58:10 PM UTC-8, Mark Walkom wrote: > Definitely sounds like FS cache to me. > > On 26 February 2015 at 17:14, liu wei \<[liu.liu...@gmail.com](mailto:liu.liu...@gmail.com) \\> > wrote: > > > Is there anyway to actually see how much memory is used by file system > > cache? > > A strange problem I see is, my machine has 32GB in total, i gave 16GB to > > ES. But very often the total OS memory usage will reach 100%, but task > > manager only shows around 12~13GB of the java exe. But when i kill the java > > process, memory usage dropped to only 20%. I can't explain why this is > > happening. Not sure if it's the file system cache. > > > > On Wednesday, November 26, 2014 at 12:51:11 PM UTC-8, Adrien Grand wrote: > > > > > Indeed the behaviour is the same on Windows and Linux: memory that is > > > not used by processes is used by the operating system in order to cache the > > > hottest parts of the file system. The reason why the docs say that the rest > > > should be left to Lucene is that most disk accesses that elasticsearch > > > performs are done through Lucene. > > > > > > On Wed, Nov 26, 2014 at 8:44 PM, Nikolas Everett [nik...@gmail.com](mailto:nik...@gmail.com) > > > wrote: > > > > > > > I imagine all operating systems have some kind of disk caching. I just > > > > happen to be used to linux. > > > > > > > > On Wed, Nov 26, 2014 at 2:42 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) wrote: > > > > > > > > > I see, but I'm running on Windows. Is the behavior similar, or does > > > > > this not exist on Windows? > > > > > > > > > > On Wednesday, November 26, 2014 1:01:02 PM UTC-6, Nikolas Everett > > > > > wrote: > > > > > > > > > > > Lucene runs in the same JVM as Elasticsearch but (by default) it > > > > > > mmaps files and then iterates over their content inteligently. That means > > > > > > most of its actual storage is "off heap" (its a java buzz-phrase). Anyway, > > > > > > Linux will serve reads from mmaped files from its page cache. That is why > > > > > > you want to leave linux a whole bunch of unused memory. > > > > > > > > > > > > Nik > > > > > > > > > > > > On Wed, Nov 26, 2014 at 1:53 PM, BradVido [bradyv...@gmail.com](mailto:bradyv...@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > I've read the recommendations for ES\_HEAP\_SIZE > > > > > > > [http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html) which > > > > > > > basically state to set -Xms and -Xmx to 50% physical RAM. > > > > > > > It says the rest should be left for Lucene to use (OS filesystem > > > > > > > caching). > > > > > > > But I'm confused on how Lucene uses that. Doesn't Lucene run in the > > > > > > > same JVM as ES? So they would share the same max heap setting of 50%. > > > > > > > > > > > > > > Can someone clear this up? > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d) > > > > > > > 4-400a-b636-e150a1261be4%[40googlegroups.com](http://40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4c0b4045-74d4-400a-b636-e150a1261be4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b7c67641-6a83-428c-91d2-10bbb6d17f0f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_ > > > > Ek3tfAkuPA08Q1GYbdH2Rg%[40mail.gmail.com](http://40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD\_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3g0g7gv79a6DchCCjrbhD_Ek3tfAkuPA08Q1GYbdH2Rg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > Adrien Grand > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/24871d7a-e99c-4905-aac4-793816dca96b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5f1d2268-3a1a-4643-a882-37b4732c4df9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5f1d2268-3a1a-4643-a882-37b4732c4df9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:29am UTC](https://discuss.elastic.co/t/es-vs-lucene-memory/20959/9 "2017-07-06T00:29:48Z")