# Reducing heap usage dedicated to indices.segments.terms\_memory\_in\_bytes...? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 13, 2016, 3:56pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899 "2016-12-13T15:56:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![diranged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diranged/32/4065_2.png) [@diranged](https://discuss.elastic.co/u/diranged)\ **Post date:** [December 13, 2016, 3:56pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/1 "2016-12-13T15:56:55Z")
We have a heterogeneous cluster setup where a large set of powerful `hot` data nodes ingest daily data live, and store about 5 days worth of data (~15tb or so). Then after 5 days we migrate the data to a smaller set of less powerful but larger-in-storage `warm` nodes. These warm nodes maintain data for 25 days before we destroy it. Our `warm` nodes are `d2.2xl` instances (60gb memory, 12TB local storage) and we dedicate half the memory to ES (so 30gb). Each node is storing roughly ~5tb of data, and we have 14 nodes right now. Over 14 nodes we have 25 days of indices (`* 2` because we store two types of data each day). This nets out to about `700 shards` and about `70tb` of raw data across the machines. From a performance perspective this is fine because this data is generally not searched frequently, so we don't mind if the queries are slow. The problem we've got is that the JVM on each of these machines is running right at about _80%_ used, and there is no garbage collection jigsaw pattern .. the machines just hold steady at 80%. After digging, we realized that it looks like we're storing an average of `22gb` of data in memory for terms: ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9490f47362796c19c1031b4cb010f172895eef9c.png) So, the ultimate question here .. is it possible for me to tune ES to store fewer terms in the JVM heap? Or is this a simple limitation .. the more data we have, the more heap size we simply _have_ to have available to us? Can we not make some kind of performance tradeoff here? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 13, 2016, 4:01pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/2 "2016-12-13T16:01:35Z")
Which version of Elasticsearch are you using? ---
**Author:** ![stlava](https://avatars.discourse-cdn.com/v4/letter/s/5fc32e/32.png) [@stlava](https://discuss.elastic.co/u/stlava)\ **Post date:** [December 14, 2016, 5:31pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/3 "2016-12-14T17:31:24Z")
I work with @diranged and wanted to followup and say we're currently on 2.4.2. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 14, 2016, 5:42pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/4 "2016-12-14T17:42:00Z")
What does your mappings look like? Do you have a lot of analyzed fields? Are you using doc\_values for not\_analyzed strings (which is the default in ES 2.4.2)? ---
**Author:** ![diranged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diranged/32/4065_2.png) [@diranged](https://discuss.elastic.co/u/diranged)\ **Post date:** [December 15, 2016, 5:54pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/5 "2016-12-15T17:54:25Z")
Our mapping is pretty dynamic .. and indeed we have a _ton_ of fields. On an average day, we generate probably 700-1000 unique fields on a given index. While I can't share a full actual generated mapping for an index, here's our template: ```auto { "order": 0, "template": "*", "settings": { "index": { "codec": "best_compression", "routing": { "allocation": { "require": { "box_type": "hot" } } }, "refresh_interval": "30s", "number_of_shards": "14", "number_of_replicas": "1" } }, "mappings": { "event": { "dynamic_templates": [ { "string_fields": { "mapping": { "fielddata": { "format": "disabled" }, "index": "analyzed", "omit_norms": true, "type": "string", "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } } }, "match_mapping_type": "string", "match": "*" } } ], "properties": { "processing_time": { "index": "not_analyzed", "omit_norms": true, "type": "date", "doc_values": true }, "arrival_time": { "index": "not_analyzed", "omit_norms": true, "type": "date", "doc_values": true }, "timestamp": { "index": "not_analyzed", "omit_norms": true, "type": "date", "doc_values": true } }, "_all": { "enabled": true } } }, "aliases": {} } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 15, 2016, 6:18pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/6 "2016-12-15T18:18:36Z")
> [@diranged](#): > > ``` > "string_fields": { > "mapping": { > "fielddata": { > "format": "disabled" > }, > "index": "analyzed", > "omit_norms": true, > "type": "string", > "fields": { > "raw": { > "ignore_above": 256, > "index": "not_analyzed", > "type": "string" > } > } > }, > "match_mapping_type": "string", > "match": "*" > } > > ``` It looks like the vast majority of string fields are dynamically typed, which will result in one `analyzed` and one `not_analyzed` version. You also have the `_all` field enabled, which is an analysed field. Depending on the number of string fields, this is likely to be behind your field data heap usage. The best way to reduce heap usage is probably going to be to go through your mappings and map fields that only really are used in a `not_analyzed` capacity as such. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [December 18, 2016, 6:09am UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/7 "2016-12-18T06:09:52Z")
@diranged Hi, I have a same problem. I'm wondering if you've found a solution. BTW, Could I know which plugin is used to generate the image you've attached above? Thanks. ---
**Author:** ![diranged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diranged/32/4065_2.png) [@diranged](https://discuss.elastic.co/u/diranged)\ **Post date:** [December 19, 2016, 3:36pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/8 "2016-12-19T15:36:23Z")
@Christian_Dahlqvist, thanks for the feedback. I do understand that we are analyzing almost every field, but I definitely thought that had more of an impact to disk usage and cpu time (at-indexing-time) rather than long term sustained memory use. Out of curiosity, is there a way for us to tell how much heap is being used by a particular index? We can certainly try an experiment where we stop analyzing all the fields and instead only analyze a few specific ones ... but I would like a measurable way of seeing the difference. @Brad_Jungsu_Heo, we have not yet solved the issue other than by throwing more hardware at those nodes .. and ultimately we may choose to either continue with that solution or reduce the total number of days that we store. Re: the image, we use Datadog for all of our monitoring here, so those graphs were built with Datadog. ---
**Author:** ![Brad\_Jungsu\_Heo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brad_jungsu_heo/32/13135_2.png) [@Brad\_Jungsu\_Heo](https://discuss.elastic.co/u/Brad_Jungsu_Heo)\ **Post date:** [December 20, 2016, 2:20am UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/9 "2016-12-20T02:20:42Z")
@diranged Thank you for your reply. In my case, after merging segments into 1 segment using [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html), `segments.terms_memory` was decreased by 30%. (note that this percentage depends on your index characteristics) I hope segment merging help you prevent adding more nodes (or resources, or money). ---
**Author:** ![diranged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/diranged/32/4065_2.png) [@diranged](https://discuss.elastic.co/u/diranged)\ **Post date:** [December 21, 2016, 3:37pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/10 "2016-12-21T15:37:59Z")
Thanks for that feedback -- we're still in the process of merging a ton of our old segments, so I'll check the memory usage when we've finished that process. We are going to begin a process of going through and starting to tune our mappings to be more specific rather than being wildly general purpose. This is gonna take some time though, but I'll post our results here when we have em. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2017, 3:38pm UTC](https://discuss.elastic.co/t/reducing-heap-usage-dedicated-to-indices-segments-terms-memory-in-bytes/68899/11 "2017-01-18T15:38:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.