# ES Memory consuption **URL:** \ **Category:** Elasticsearch\ **Created:** [March 20, 2013, 6:00pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228 "2013-03-20T18:00:30Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 20, 2013, 6:00pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/1 "2013-03-20T18:00:30Z")
Hi, I'm having troubles managing the memory on my ES server. I have 3 nodes with heap size 4GB each. However, when i start indexing my memory start growing proportionally to the size of my index. In detail, if I run "sar -r" i see that memory used doesn't go more than 12GB which is ok, but my kbcached continous growing. How can I limit that? I've tried some stuff, but since I'm newbie on this I must be missing something here. Here an example of my configurations by now (some might look unnecessary or simple despair...) 🙂 index.translog.flush\_threshold\_ops: 20000 index.store.type: niofs index.store.fs.memory.enabled: false index.cache.field.type: soft cache.memory.direct: false gateway.type: local Can anyone help me please? Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 20, 2013, 6:14pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/2 "2013-03-20T18:14:00Z")
Did you run into troubles? Are there messages in the logs? What is your challenge? Why do you want to limit what kind of cache? It is not very clear how to help - please, do not "press buttons" out of despair, it will not help without understanding what is going on. Jörg Am 20.03.13 19:00, schrieb Delmar: > Hi, > > I'm having troubles managing the memory on my ES server. I have 3 > nodes with heap size 4GB each. However, when i start indexing my > memory start growing proportionally to the size of my index. In > detail, if I run "sar -r" i see that memory used doesn't go more than > 12GB which is ok, but my kbcached continous growing. How can I limit that? > > I've tried some stuff, but since I'm newbie on this I must be missing > something here. > Here an example of my configurations by now (some might look > unnecessary or simple despair...) 🙂 > > index.translog.flush\_threshold\_ops: 20000 > index.store.type: niofs > index.store.fs.memory.enabled: false > index.cache.field.type: soft > cache.memory.direct: false > gateway.type: local > > Can anyone help me please? > > ## Thanks. > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 20, 2013, 6:17pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/3 "2013-03-20T18:17:42Z")
I'm trying to understand, that's why I tried different aproaches. My problem is that i want to limit the size of the cache because there might be running other applications on the same server that might need resources also. I don't want ES to own or try to own all the RAM resources of my server. Nevertheless, is it possible to limit it? If so, how can i do it? Thank's Quarta-feira, 20 de Março de 2013 18:14:00 UTC, Jörg Prante escreveu: > Did you run into troubles? Are there messages in the logs? What is your > challenge? Why do you want to limit what kind of cache? > > It is not very clear how to help - please, do not "press buttons" out of > despair, it will not help without understanding what is going on. > > Jörg > > Am 20.03.13 19:00, schrieb Delmar: > > > Hi, > > > > I'm having troubles managing the memory on my ES server. I have 3 > > nodes with heap size 4GB each. However, when i start indexing my > > memory start growing proportionally to the size of my index. In > > detail, if I run "sar -r" i see that memory used doesn't go more than > > 12GB which is ok, but my kbcached continous growing. How can I limit > > that? > > > > I've tried some stuff, but since I'm newbie on this I must be missing > > something here. > > Here an example of my configurations by now (some might look > > unnecessary or simple despair...) 🙂 > > > > index.translog.flush\_threshold\_ops: 20000 > > index.store.type: niofs > > index.store.fs.memory.enabled: false > > index.cache.field.type: soft > > cache.memory.direct: false > > gateway.type: local > > > > Can anyone help me please? > > > > ## Thanks. > > > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 20, 2013, 6:42pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/4 "2013-03-20T18:42:34Z")
Just like any other JVM, best method is limiting the heap (ES uses an environment variable $ES\_HEAP\_SIZE), and optionally, limit the direct buffer memory with $ES\_DIRECT\_SIZE. Limiting the buffer decreases performance and should not be necessary, because the OS will take really good care of it, it will be distributed among the running processes at demand. See also bin/elasticsearch.in.sh for memory setup of the ES JVM startup procedure. Jörg Am 20.03.13 19:17, schrieb Delmar: > I'm trying to understand, that's why I tried different aproaches. > My problem is that i want to limit the size of the cache because there > might be running other applications on the same server that might need > resources also. I don't want ES to own or try to own all the RAM > resources of my server. Nevertheless, is it possible to limit it? If > so, how can i do it? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 20, 2013, 6:52pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/5 "2013-03-20T18:52:57Z")
I've already limited the heap size to 4GB each, but not using the environment variable $ES\_HEAP\_SIZE. I changed it directly in the bin/elasticsearch script. Regarding the $ES\_DIRECT\_SIZE, shouldn't my configurations on conf/elasticsearch.yml (namely the cache.memory.direct: false) or my bin/elasticsearch configurations -XX:MaxDirectMemorySize have worked? Nonetheless, I will give a try to your recommendations and see if it works. Thanks. 2013/3/20 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > Just like any other JVM, best method is limiting the heap (ES uses an > environment variable $ES\_HEAP\_SIZE), and optionally, limit the direct > buffer memory with $ES\_DIRECT\_SIZE. Limiting the buffer decreases > performance and should not be necessary, because the OS will take really > good care of it, it will be distributed among the running processes at > demand. > > See also bin/elasticsearch.in.sh for memory setup of the ES JVM startup > procedure. > > Jörg > > Am 20.03.13 19:17, schrieb Delmar: > > I'm trying to understand, that's why I tried different aproaches. > > > My problem is that i want to limit the size of the cache because there > > might be running other applications on the same server that might need > > resources also. I don't want ES to own or try to own all the RAM resources > > of my server. Nevertheless, is it possible to limit it? If so, how can i do > > it? > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > topic/elasticsearch/\*\*Rt0Tej458ks/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Hugo Delmar Cabral -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 20, 2013, 7:42pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/6 "2013-03-20T19:42:13Z")
cache.memory.direct is just switching the JVM between using the direct buffer memory of the OS (true) or the heap (false) for node-level caching. Memory usage is the same. It is advisable to use the default value which is true, so field cache elements are exposed very efficiently to the garbage collection (they are not moved around). Jörg Am 20.03.13 19:52, schrieb Hugo Delmar Cabral: > I've already limited the heap size to 4GB each, but not using the > environment variable $ES\_HEAP\_SIZE. I changed it directly in the > bin/elasticsearch script. > Regarding the $ES\_DIRECT\_SIZE, shouldn't my configurations on > conf/elasticsearch.yml (namely the cache.memory.direct: false) or my > bin/elasticsearch configurations -XX:MaxDirectMemorySize have worked? > Nonetheless, I will give a try to your recommendations and see if it > works. > Thanks. > > 2013/3/20 Jörg Prante \<[joergprante@gmail.com](mailto:joergprante@gmail.com) > [mailto:joergprante@gmail.com](mailto:joergprante@gmail.com)\> > > ``` > Just like any other JVM, best method is limiting the heap (ES uses > an environment variable $ES_HEAP_SIZE), and optionally, limit the > direct buffer memory with $ES_DIRECT_SIZE. Limiting the buffer > decreases performance and should not be necessary, because the OS > will take really good care of it, it will be distributed among the > running processes at demand. > > See also bin/elasticsearch.in.sh for > memory setup of the ES JVM startup procedure. > > Jörg > > Am 20.03.13 19:17, schrieb Delmar: > > I'm trying to understand, that's why I tried different aproaches. > My problem is that i want to limit the size of the cache > because there might be running other applications on the same > server that might need resources also. I don't want ES to own > or try to own all the RAM resources of my server. > Nevertheless, is it possible to limit it? If so, how can i do it? > > -- > You received this message because you are subscribed to a topic in > the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US. > To unsubscribe from this group and all its topics, send an email > to elasticsearch+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/groups/opt_out. > > ``` > > ## -- Hugo Delmar Cabral > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 20, 2013, 7:49pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/7 "2013-03-20T19:49:34Z")
Ok, got it. Unfortunately your previous suggestions didn't work! Do you have any other suggestions? The problem is not the heap size, it is the cache that continuous growing accordingly to the size of my index until consumes all the available RAM of the server! 2013/3/20 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > cache.memory.direct is just switching the JVM between using the direct > buffer memory of the OS (true) or the heap (false) for node-level caching. > Memory usage is the same. It is advisable to use the default value which is > true, so field cache elements are exposed very efficiently to the garbage > collection (they are not moved around). > > Jörg > > Am 20.03.13 19:52, schrieb Hugo Delmar Cabral: > > > I've already limited the heap size to 4GB each, but not using the > > environment variable $ES\_HEAP\_SIZE. I changed it directly in the > > bin/elasticsearch script. > > Regarding the $ES\_DIRECT\_SIZE, shouldn't my configurations on > > conf/elasticsearch.yml (namely the cache.memory.direct: false) or my > > bin/elasticsearch configurations -XX:MaxDirectMemorySize have worked? > > Nonetheless, I will give a try to your recommendations and see if it > > works. > > Thanks. > > > > 2013/3/20 Jörg Prante \<[joergprante@gmail.com](mailto:joergprante@gmail.com) \ > [joergprante@gmail.com](mailto:joergprante@gmail.com)\>\*\*\> > > > > ``` > > Just like any other JVM, best method is limiting the heap (ES uses > > an environment variable $ES_HEAP_SIZE), and optionally, limit the > > direct buffer memory with $ES_DIRECT_SIZE. Limiting the buffer > > decreases performance and should not be necessary, because the OS > > will take really good care of it, it will be distributed among the > > running processes at demand. > > > > See also bin/elasticsearch.in.sh for > > > > memory setup of the ES JVM startup procedure. > > > > Jörg > > > > Am 20.03.13 19:17, schrieb Delmar: > > > > I'm trying to understand, that's why I tried different aproaches. > > My problem is that i want to limit the size of the cache > > because there might be running other applications on the same > > server that might need resources also. I don't want ES to own > > or try to own all the RAM resources of my server. > > Nevertheless, is it possible to limit it? If so, how can i do it? > > > > -- You received this message because you are subscribed to a > > > > ``` > > > > topic in > > the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/\*\*topic/elasticsearch/](https://groups.google.com/d/**topic/elasticsearch/)\*\* > > Rt0Tej458ks/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US) > > . > > To unsubscribe from this group and all its topics, send an email > > to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > \ > \*\*\>. > > > > ``` > > For more options, visit https://groups.google.com/**groups/opt_out > > > > ``` > > > > . > > > > ## -- Hugo Delmar Cabral > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > topic/elasticsearch/\*\*Rt0Tej458ks/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Hugo Delmar Cabral -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 20, 2013, 7:53pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/8 "2013-03-20T19:53:57Z")
Well, the cache is supposed to cover all the RAM, as it's managed by the OS, it must be assumed to be a correct decision. So what exactly are you doing - do you send queries, do you index? What are the numbers that worry you? Sorry but I can't figure it what the problem is. Jörg Am 20.03.13 20:49, schrieb Hugo Delmar Cabral: > The problem is not the heap size, it is the cache that continuous > growing accordingly to the size of my index until consumes all the > available RAM of the server! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 21, 2013, 10:16am UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/9 "2013-03-21T10:16:13Z")
I'm indexing, but since I'm trying to figure out the resources needed to install an Elasticsearch in a production environment, it's hard to do figure out the needs in terms of RAM. Also, the usage or RAM/cache influences the performance of the Elasticsearch itself, regarding throughput and latency. Summarizing, I'm trying to conduct performance and sizing of ES, but it becomes hard when you don't control some variables... 2013/3/20 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) > Well, the cache is supposed to cover all the RAM, as it's managed by the > OS, it must be assumed to be a correct decision. So what exactly are you > doing - do you send queries, do you index? What are the numbers that worry > you? Sorry but I can't figure it what the problem is. > > Jörg > > Am 20.03.13 20:49, schrieb Hugo Delmar Cabral: > > The problem is not the heap size, it is the cache that continuous growing > > > accordingly to the size of my index until consumes all the available RAM of > > the server! > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > topic/elasticsearch/\*\*Rt0Tej458ks/unsubscribe?hl=en-\*\*US[https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Rt0Tej458ks/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Hugo Delmar Cabral -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 21, 2013, 2:52pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/10 "2013-03-21T14:52:17Z")
Any other thoughts on this guys? Thanks. Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu: > Hi, > > I'm having troubles managing the memory on my ES server. I have 3 nodes > with heap size 4GB each. However, when i start indexing my memory start > growing proportionally to the size of my index. In detail, if I run "sar > -r" i see that memory used doesn't go more than 12GB which is ok, but > my kbcached continous growing. How can I limit that? > > I've tried some stuff, but since I'm newbie on this I must be missing > something here. > Here an example of my configurations by now (some might look unnecessary > or simple despair...) 🙂 > > index.translog.flush\_threshold\_ops: 20000 > index.store.type: niofs > index.store.fs.memory.enabled: false > index.cache.field.type: soft > cache.memory.direct: false > gateway.type: local > > Can anyone help me please? > > Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [March 21, 2013, 3:58pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/11 "2013-03-21T15:58:36Z")
Are you simply trying to determine a "worst case" performance for ES? It's unclear to me why you would want to limit the file OS cache (Direct Buffer). The underlying OS will do a good job managing, freely giving memory to new processes that need them (e.g. a new Apache process). If memory contention with other processes is a serious concern (e.g. running ES on the same host as an busy web stack) and you are worried about performance, it is advisable to give ES it's own dedicated box. This is similar advice for any data store - it is often preferable to run the datastore/database on a dedicated machine to fully take advantage of memory, reduce context switching, etc. -Zach On Thursday, March 21, 2013 10:52:17 AM UTC-4, Delmar wrote: > Any other thoughts on this guys? > Thanks. > > Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu: > > > Hi, > > > > I'm having troubles managing the memory on my ES server. I have 3 nodes > > with heap size 4GB each. However, when i start indexing my memory start > > growing proportionally to the size of my index. In detail, if I run "sar > > -r" i see that memory used doesn't go more than 12GB which is ok, but > > my kbcached continous growing. How can I limit that? > > > > I've tried some stuff, but since I'm newbie on this I must be missing > > something here. > > Here an example of my configurations by now (some might look unnecessary > > or simple despair...) 🙂 > > > > index.translog.flush\_threshold\_ops: 20000 > > index.store.type: niofs > > index.store.fs.memory.enabled: false > > index.cache.field.type: soft > > cache.memory.direct: false > > gateway.type: local > > > > Can anyone help me please? > > > > Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 21, 2013, 4:27pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/12 "2013-03-21T16:27:18Z")
I think you care for the wrong part of the system. You can control indexing throughput, but not by intentionally limiting RAM memory. Indexing in RAM is very fast, and the bottleneck are the disks. Even the fastest SSD disk subsystem is 1000x slower than memory. Check if your disks are fast enough for huge peaks of writes caused by segment merging, this will limit your overall performance. Slow disk IO can be compensated by store throttling described here (see "Store level throttling"): > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Jörg Am 21.03.13 11:16, schrieb Hugo Delmar Cabral: > Also, the usage or RAM/cache influences the performance of the elastic > search itself, regarding throughput and latency. Summarizing, I'm > trying to conduct performance and sizing of ES, but it becomes hard > when you don't control some variables... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Delmar](https://avatars.discourse-cdn.com/v4/letter/d/c67d28/32.png) [@Delmar](https://discuss.elastic.co/u/Delmar)\ **Post date:** [March 21, 2013, 4:31pm UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/13 "2013-03-21T16:31:18Z")
Thanks guys. I understand and agree with your thoughts. My question was to understand if we could control it or not! Because for my tests it would be helpful to understand the impact of costs in RAM vs. gain in throughput and latency. But considering your answers I believe this is out of user control, meaning that it is internally managed by the ES/OS. Cheers Quarta-feira, 20 de Março de 2013 18:00:30 UTC, Delmar escreveu: > Hi, > > I'm having troubles managing the memory on my ES server. I have 3 nodes > with heap size 4GB each. However, when i start indexing my memory start > growing proportionally to the size of my index. In detail, if I run "sar > -r" i see that memory used doesn't go more than 12GB which is ok, but > my kbcached continous growing. How can I limit that? > > I've tried some stuff, but since I'm newbie on this I must be missing > something here. > Here an example of my configurations by now (some might look unnecessary > or simple despair...) 🙂 > > index.translog.flush\_threshold\_ops: 20000 > index.store.type: niofs > index.store.fs.memory.enabled: false > index.cache.field.type: soft > cache.memory.direct: false > gateway.type: local > > Can anyone help me please? > > Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/es-memory-consuption/11228/14 "2017-07-06T02:45:18Z")