# High memory taken by segments.terms\_memory\_in\_bytes

**URL:** <https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134>\
**Category:** Elasticsearch\
**Created:** [March 8, 2022, 6:53pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134 "2022-03-08T18:53:42Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 8, 2022, 6:53pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/1 "2022-03-08T18:53:42Z")

</div>

Hello Experts,

ES : 6.4

We have 21 data nodes cluster. Seeing high memory consumed by segments.terms\_memory\_in\_bytes which causing circuit breaker issues for us. Wanted to understand why segments are taking so much memory? Referred some old discussions, do we need to force merge shards to reduce the segments and get rid of deleted documents to reduce the term\_memory\_in\_bytes. Dynamic mapping used for indices. approx 300 fields per index, most of the fields are notanalyzed (i.e keyword fields)

Number of segments: 128431  
total number of docs: 53410044582 (as per \_cat/segments output)  
number of deleted docs: 3022763394 (as per \_cat/segments output)

Number of shards approx 7K

Seeing many segments with compound false state however not sure whether it can impact the memory or not.

Total amount of memory taken by segments is 220GB as per sum of size.memory output from \_cat/segments.

From [ES documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-segments.html)

compound (Default) If true, the segment is stored in a compound file. This means Lucene merged all files from the segment in a single file to save file descriptors.

---

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 8, 2022, 6:56pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/2 "2022-03-08T18:56:43Z")

</div>

we have 31GB (approx) memory for java heap out of which max is consumed by segments, following output showing memory consumed by segments and term\_memory in GB.

```auto
Node Name SegmentCount SegmentMemory TermMem
prodnode-01 5893 10 9
prodnode-02 7451 17 14
prodnode-03 6318 11 9
prodnode-04 6139 10 9
prodnode-05 7339 16 14
prodnode-06 6902 13 11
prodnode-07 5620 7 6
prodnode-08 7280 17 15
prodnode-09 6420 12 10
prodnode-10 5979 12 10
prodnode-11 5486 9 8
prodnode-12 5515 8 6
prodnode-13 5539 8 6
prodnode-14 5498 8 7
prodnode-15 5609 8 7
prodnode-16 6358 12 10
prodnode-17 5605 9 8
prodnode-18 5886 9 7
prodnode-19 6121 9 8
prodnode-20 5660 9 7
prodnode-21 5315 7 6

```

---

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 8, 2022, 6:59pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/3 "2022-03-08T18:59:12Z")

</div>

this is how heap utilization looks like

```auto
name id node.role heap.current heap.percent heap.max
prodnode-12 L_-j di 20gb 64 30.9gb
prodnode-05 TBOg di 21.9gb 71 30.9gb
prodnode-16 VHc3 di 14.4gb 46 30.9gb
prodnode-19 pn1M di 17.7gb 57 30.9gb
prodnode-18 D83h di 12.4gb 40 30.9gb
prodnode-15 wiGb di 13.2gb 42 30.9gb
prodnode-04 qQPY di 22gb 71 30.9gb
prodnode-09 AC3x di 20.4gb 66 30.9gb
prodnode-07 bzYO di 19.3gb 62 30.9gb
prodnode-08 ldTY di 20.3gb 65 30.9gb
prodnode-06 xcRB di 20.4gb 66 30.9gb
prodnode-21 uGbl di 14gb 45 30.9gb
prodnode-02 gmAb di 21.5gb 69 30.9gb
prodnode-20 DAWZ di 20.3gb 65 30.9gb
prodnode-11 QfHe di 20.7gb 66 30.9gb
prodnode-14 vprk di 16.7gb 54 30.9gb
prodnode-17 fpOL di 21.6gb 69 30.9gb
prodnode-13 feLL di 19gb 61 30.9gb
prodnode-10 cl1F di 20.1gb 65 30.9gb
prodnode-01 D8vH di 20.9gb 67 30.9gb
prodnode-03 y886 di 19.3gb 62 30.9gb

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 8, 2022, 7:22pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/4 "2022-03-08T19:22:45Z")

</div>

> [@Vikrant\_Aggarwal](#):
>
> ES : 6.4

This version is extremely old, literally years past [EOL](https://www.elastic.co/support/eol). You should upgrade as a matter of urgency. Newer versions are significantly more memory-efficient.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 8, 2022, 8:40pm UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/5 "2022-03-08T20:40:09Z")

</div>

Yep, your best bet is to upgrade as the entire set of 6.X releases are EOL with 8.X now GA.

---

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 9, 2022, 5:26am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/6 "2022-03-09T05:26:28Z")

</div>

Thanks for providing the suggestion. We are planning to for it, may take sometime. in the meantime we are tying to make the cluster stable. segment memory seems to be a biggest problem for us.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 9, 2022, 5:28am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/7 "2022-03-09T05:28:42Z")

</div>

What is the output from the `_cluster/stats?pretty&human` API?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 9, 2022, 5:43am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/8 "2022-03-09T05:43:29Z")

</div>

If you are no longer writing to some of the indices you can save heap usage by forcemerging down to a single segment. If all your indices however can continue to be updated or have documents deleted, the only way to reduce heap usage is to upgrade as far as i know.

---

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 9, 2022, 5:59am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/9 "2022-03-09T05:59:29Z")

</div>

```auto
  "status": "green",
  "indices": {
    "count": 2325,
    "shards": {
      "total": 6957,
      "primaries": 3469,
      "replication": 1.005477082732776,
      "index": {
        "shards": {
          "min": 2,
          "max": 21,
          "avg": 2.992258064516129
        },
        "primaries": {
          "min": 1,
          "max": 5,
          "avg": 1.492043010752688
        },
        "replication": {
          "min": 1,
          "max": 20,
          "avg": 1.0081720430107526
        }
      }
    },
    "docs": {
      "count": 26739032827,
      "deleted": 1521435325
    },
    "store": {
      "size": "84.4tb",
      "size_in_bytes": 92898167696858
    },
    "fielddata": {
      "memory_size": "32.3gb",
      "memory_size_in_bytes": 34786252884,
      "evictions": 0
    },
    "query_cache": {
      "memory_size": "4.7gb",
      "memory_size_in_bytes": 5071146612,
      "total_count": 15520344,
      "hit_count": 1900137,
      "miss_count": 13620207,
      "cache_size": 63067,
      "cache_count": 71211,
      "evictions": 8144
    },
    "completion": {
      "size": "0b",
      "size_in_bytes": 0
    },
    "segments": {
      "count": 128321,
      "memory": "217gb",
      "memory_in_bytes": 233085706881,
      "terms_memory": "183gb",
      "terms_memory_in_bytes": 196505121341,
      "stored_fields_memory": "27gb",
      "stored_fields_memory_in_bytes": 29002930872,
      "term_vectors_memory": "0b",
      "term_vectors_memory_in_bytes": 0,
      "norms_memory": "14.7mb",
      "norms_memory_in_bytes": 15464960,
      "points_memory": "6.5gb",
      "points_memory_in_bytes": 7037194208,
      "doc_values_memory": "500.6mb",
      "doc_values_memory_in_bytes": 524995500,
      "index_writer_memory": "109.6mb",
      "index_writer_memory_in_bytes": 114924356,
      "version_map_memory": "1.5mb",
      "version_map_memory_in_bytes": 1595430,
      "fixed_bit_set": "187mb",
      "fixed_bit_set_memory_in_bytes": 196151392,
      "max_unsafe_auto_id_timestamp": 1646784006999,
      "file_sizes": {}
    }
  }

```

---

<div class="post-metadata">

**Author:** ![Vikrant\_Aggarwal](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\
**Post date:** [March 9, 2022, 6:00am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/10 "2022-03-09T06:00:33Z")

</div>

thanks for suggestion. We will try to forcemerge the segments to one for indices which are not writable anymore.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 9, 2022, 6:23am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/11 "2022-03-09T06:23:51Z")

</div>

There are additional details around this available [in this webinar](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch). Note that this most likely no longer applies to recent versions due to the improvements made to memory usage since it was realeased several years ago.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 9, 2022, 10:18am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/12 "2022-03-09T10:18:11Z")

</div>

That appears to be only part of the output?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2022, 10:18am UTC](https://discuss.elastic.co/t/high-memory-taken-by-segments-terms-memory-in-bytes/299134/13 "2022-04-06T10:18:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.