# Elasticsearch high heap usage **URL:** \ **Category:** Elasticsearch\ **Created:** [March 16, 2015, 3:39am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678 "2015-03-16T03:39:54Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![chris85lang](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chris85lang](https://discuss.elastic.co/u/chris85lang)\ **Post date:** [March 16, 2015, 3:39am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/1 "2015-03-16T03:39:54Z")
Hello, We have a 2 node elasticsearch cluster which is used by logstash to store log files. The current input is around 100 documents (logs) per second wit a size of around 50kb - 150kb. Compared to what i have read so far this is not a high amount but we experience already a high heap usage 70% form the total of 11GB heap size, the system has in total 32GB RAM. CPU and IO are totally fine. Any suggestion highly appreciated! Cheers Chris [http://i.imgur.com/muodrOO.png](http://i.imgur.com/muodrOO.png) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 16, 2015, 4:35am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/2 "2015-03-16T04:35:41Z")
Those are reasonably large documents. You also seem to have a lot of shards for the data. What sort of data is it, are you using doc values, how are you bucketing data (ie time series indices)? On 15 March 2015 at 20:39, [chris85lang@googlemail.com](mailto:chris85lang@googlemail.com) wrote: > Hello, > > We have a 2 node elasticsearch cluster which is used by logstash to store > log files. The current input is around 100 documents (logs) per second wit > a size of around 50kb - 150kb. > Compared to what i have read so far this is not a high amount but we > experience already a high heap usage 70% form the total of 11GB heap size, > the system has in total 32GB RAM. CPU and IO are totally fine. > > Any suggestion highly appreciated! > > Cheers > Chris > > [http://i.imgur.com/muodrOO.png](http://i.imgur.com/muodrOO.png) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2kqt5%3DBcVJ9fJRP1GCfr3Aagt3sybBjHJEyN6iWok9w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2kqt5%3DBcVJ9fJRP1GCfr3Aagt3sybBjHJEyN6iWok9w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 16, 2015, 8:35am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/3 "2015-03-16T08:35:49Z")
This is not high. The JVM always uses the whole heap to avoid garbage collection as much as possible. In ES, a threshold is set to 75% before CMS garbage collection kicks in. Jörg On Mon, Mar 16, 2015 at 4:39 AM, [chris85lang@googlemail.com](mailto:chris85lang@googlemail.com) wrote: > Hello, > > We have a 2 node elasticsearch cluster which is used by logstash to store > log files. The current input is around 100 documents (logs) per second wit > a size of around 50kb - 150kb. > Compared to what i have read so far this is not a high amount but we > experience already a high heap usage 70% form the total of 11GB heap size, > the system has in total 32GB RAM. CPU and IO are totally fine. > > Any suggestion highly appreciated! > > Cheers > Chris > > [http://i.imgur.com/muodrOO.png](http://i.imgur.com/muodrOO.png) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG3cOg7NHTfq%2BH69gFWyw6VARyoikzdAoh7W1Tp\_yRhcg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG3cOg7NHTfq%2BH69gFWyw6VARyoikzdAoh7W1Tp_yRhcg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![chris85lang](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chris85lang](https://discuss.elastic.co/u/chris85lang)\ **Post date:** [March 17, 2015, 3:29am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/4 "2015-03-17T03:29:23Z")
Hello Mark, Thanks for your answer! We are using the default values, so no doc\_values. I did some research about it and it sounds very interesting and helpful to keep the heap usage lower. How can I add doc\_values: true to the index template so that the new daily based generated indexes using this feature. Cheers Chris On Monday, March 16, 2015 at 11:36:13 AM UTC+7, Mark Walkom wrote: > Those are reasonably large documents. You also seem to have a lot of > shards for the data. > > What sort of data is it, are you using doc values, how are you bucketing > data (ie time series indices)? > > On 15 March 2015 at 20:39, \<[chris...@googlemail.com](mailto:chris...@googlemail.com) \\> wrote: > > > Hello, > > > > We have a 2 node elasticsearch cluster which is used by logstash to store > > log files. The current input is around 100 documents (logs) per second wit > > a size of around 50kb - 150kb. > > Compared to what i have read so far this is not a high amount but we > > experience already a high heap usage 70% form the total of 11GB heap size, > > the system has in total 32GB RAM. CPU and IO are totally fine. > > > > Any suggestion highly appreciated! > > > > Cheers > > Chris > > > > [http://i.imgur.com/muodrOO.png](http://i.imgur.com/muodrOO.png) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 17, 2015, 9:01pm UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/5 "2015-03-17T21:01:06Z")
Take a look at [http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html) On 16 March 2015 at 20:29, [chris85lang@googlemail.com](mailto:chris85lang@googlemail.com) wrote: > Hello Mark, > > Thanks for your answer! We are using the default values, so no doc\_values. > I did some research about it and it sounds very interesting and helpful to > keep the heap usage lower. > How can I add doc\_values: true to the index template so that the new daily > based generated indexes using this feature. > > Cheers > Chris > > On Monday, March 16, 2015 at 11:36:13 AM UTC+7, Mark Walkom wrote: > > > Those are reasonably large documents. You also seem to have a lot of > > shards for the data. > > > > What sort of data is it, are you using doc values, how are you bucketing > > data (ie time series indices)? > > > > On 15 March 2015 at 20:39, [chris...@googlemail.com](mailto:chris...@googlemail.com) wrote: > > > > > Hello, > > > > > > We have a 2 node elasticsearch cluster which is used by logstash to > > > store log files. The current input is around 100 documents (logs) per > > > second wit a size of around 50kb - 150kb. > > > Compared to what i have read so far this is not a high amount but we > > > experience already a high heap usage 70% form the total of 11GB heap size, > > > the system has in total 32GB RAM. CPU and IO are totally fine. > > > > > > Any suggestion highly appreciated! > > > > > > Cheers > > > Chris > > > > > > [http://i.imgur.com/muodrOO.png](http://i.imgur.com/muodrOO.png) > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2679cad-72ec-472f-a009-a6c9e2abbb9d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bdd6d405-61b4-449d-b1ce-3f95bdf8b0c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X89%3D9nd1AVZGvu7TWSM7%2BA\_dNsVo%3D9Z0YcxNT%3DsNc6msQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X89%3D9nd1AVZGvu7TWSM7%2BA_dNsVo%3D9Z0YcxNT%3DsNc6msQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:26am UTC](https://discuss.elastic.co/t/elasticsearch-high-heap-usage/22678/6 "2017-07-06T00:26:06Z")