# ES seen to allocate more memory outside ES\_HEAP\_SIZE using "top" **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2013, 3:32pm UTC](https://discuss.elastic.co/t/es-seen-to-allocate-more-memory-outside-es-heap-size-using-top/11822 "2013-05-05T15:32:18Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![subin](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@subin](https://discuss.elastic.co/u/subin)\ **Post date:** [May 5, 2013, 3:32pm UTC](https://discuss.elastic.co/t/es-seen-to-allocate-more-memory-outside-es-heap-size-using-top/11822/1 "2013-05-05T15:32:18Z")
I have a test Logstash setup feeding ES 2800 log/s. Total server memory is 8G. ES\_HEAP\_SIZE is set to 3072m. After 6 hours into import ES - as from 'top' - is seen to use 4.1G while BigDesk only reports a "used" of 1.1G from the total HEAP of "committed" of 2.9G. Please find the attached image. Why is this wide variation in statistics? I'm unable to make up the real memory usage of ES. Additional info that may help: ES = v0.90.0 OS = Debian 6 Java = OpenJDK 6 Issue-relevant ES settings: index.number\_of\_shards: 1 index.number\_of\_replicas: 0 index.translog.flush\_threshold\_ops: 50000 indices.memory.index\_buffer\_size: 40% indices.store.throttle.type: merge indices.store.throttle.max\_bytes\_per\_sec: 5mb index.fielddata.cache: soft index.store.type: mmapfs index.fielddata.cache.size: 20% bootstrap.mlockall: true ES\_HEAP\_SIZE=3072m ES\_DIRECT\_SIZE=3072m MAX\_OPEN\_FILES=65535 MAX\_LOCKED\_MEMORY=unlimited Only 1 index template for Logstash indices only: ``` curl -XPUT http://localhost:9200/_template/loggerstash ``` [http://10.0.4.24:9200/\_template/loggerstash](http://10.0.4.24:9200/_template/loggerstash) -d ' { ``` "template" : "logstash*", "settings" : { "number_of_shards" : 1, "index.refresh_interval" : "10s", "index.store.compress.stored" : true } }' ``` If "top" is right, how to tackle this ever growing memory usage? If not, how to deduce the real memory usage? Re, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [May 29, 2013, 9:13pm UTC](https://discuss.elastic.co/t/es-seen-to-allocate-more-memory-outside-es-heap-size-using-top/11822/2 "2013-05-29T21:13:40Z")
Is the memory usage really 'growing' ?? Linux uses all free memory as buffer cache to speed up disk reading, and the memory is freed/used when a process actually needs it. The TOP is showing higher memory number because Linux probably is caching some ES stuff in memory . Just from experience, I would say don't worry about TOP showing little higher memory for the process than ES\_HEAP\_SIZE. 1.1 GB is the current heap being used, while 3 GB is the max memory committed to jvm . I'd say treat the ES\_HEAP\_SIZE as your true memory usage, and think about increasing memory if you heap\_used gets close to heap\_committed. On Sunday, May 5, 2013 11:32:18 AM UTC-4, vims wrote: > I have a test Logstash setup feeding ES 2800 log/s. Total server memory > is 8G. ES\_HEAP\_SIZE is set to 3072m. After 6 hours > into import ES - as from 'top' - is seen to use 4.1G while BigDesk only > reports a "used" of 1.1G from the total HEAP of > "committed" of 2.9G. Please find the attached image. Why is this wide > variation in statistics? I'm unable to make up the > real memory usage of ES. Additional info that may help: > > ES = v0.90.0 > OS = Debian 6 > Java = OpenJDK 6 > > Issue-relevant ES settings: > > index.number\_of\_shards: 1 > index.number\_of\_replicas: 0 > > index.translog.flush\_threshold\_ops: 50000 > indices.memory.index\_buffer\_size: 40% > > indices.store.throttle.type: merge > indices.store.throttle.max\_bytes\_per\_sec: 5mb > > index.fielddata.cache: soft > index.store.type: mmapfs > index.fielddata.cache.size: 20% > > bootstrap.mlockall: true > ES\_HEAP\_SIZE=3072m > ES\_DIRECT\_SIZE=3072m > MAX\_OPEN\_FILES=65535 > MAX\_LOCKED\_MEMORY=unlimited > > Only 1 index template for Logstash indices only: > > ``` > curl -XPUT http://localhost:9200/_template/loggerstash-d ' { > > "template" : > > ``` > > "logstash\*", > "settings" : { "number\_of\_shards" : 1, > "index.refresh\_interval" : "10s", > "index.store.compress.stored" : true } > }' > > If "top" is right, how to tackle this ever growing memory usage? If not, > how to deduce the real memory usage? > > Re, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 29, 2013, 9:40pm UTC](https://discuss.elastic.co/t/es-seen-to-allocate-more-memory-outside-es-heap-size-using-top/11822/3 "2013-05-29T21:40:22Z")
Note, the heap size and the direct memory are different parameters, you have assigned 2 \* 3072m to ES which is total of ~6 G. In most cases you do not need to set ES\_DIRECT\_SIZE because OS is handling it well enough. Jörg Am 05.05.13 17:32, schrieb sub: > ES\_HEAP\_SIZE=3072m > ES\_DIRECT\_SIZE=3072m -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:34am UTC](https://discuss.elastic.co/t/es-seen-to-allocate-more-memory-outside-es-heap-size-using-top/11822/4 "2017-07-06T02:34:01Z")