High memory taken by segments.terms_memory_in_bytes

Elastic Stack Elasticsearch
1

Hello Experts,

ES : 6.4

We have 21 data nodes cluster. Seeing high memory consumed by segments.terms_memory_in_bytes which causing circuit breaker issues for us. Wanted to understand why segments are taking so much memory? Referred some old discussions, do we need to force merge shards to reduce the segments and get rid of deleted documents to reduce the term_memory_in_bytes. Dynamic mapping used for indices. approx 300 fields per index, most of the fields are notanalyzed (i.e keyword fields)

Number of segments: 128431
total number of docs: 53410044582 (as per _cat/segments output)
number of deleted docs: 3022763394 (as per _cat/segments output)

Number of shards approx 7K

Seeing many segments with compound false state however not sure whether it can impact the memory or not.

Total amount of memory taken by segments is 220GB as per sum of size.memory output from _cat/segments.

From ES documentation

compound (Default) If true, the segment is stored in a compound file. This means Lucene merged all files from the segment in a single file to save file descriptors.

2

we have 31GB (approx) memory for java heap out of which max is consumed by segments, following output showing memory consumed by segments and term_memory in GB.

Node Name  SegmentCount SegmentMemory    TermMem
prodnode-01 5893               10          9
prodnode-02 7451               17         14
prodnode-03 6318               11          9
prodnode-04 6139               10          9
prodnode-05 7339               16         14
prodnode-06 6902               13         11
prodnode-07 5620                7          6
prodnode-08 7280               17         15
prodnode-09 6420               12         10
prodnode-10 5979               12         10
prodnode-11 5486                9          8
prodnode-12 5515                8          6
prodnode-13 5539                8          6
prodnode-14 5498                8          7
prodnode-15 5609                8          7
prodnode-16 6358               12         10
prodnode-17 5605                9          8
prodnode-18 5886                9          7
prodnode-19 6121                9          8
prodnode-20 5660                9          7
prodnode-21 5315                7          6
3

this is how heap utilization looks like

name                  id   node.role heap.current heap.percent heap.max
prodnode-12        L_-j di                20gb           64   30.9gb
prodnode-05        TBOg di              21.9gb           71   30.9gb
prodnode-16        VHc3 di              14.4gb           46   30.9gb
prodnode-19        pn1M di              17.7gb           57   30.9gb
prodnode-18        D83h di              12.4gb           40   30.9gb
prodnode-15        wiGb di              13.2gb           42   30.9gb
prodnode-04        qQPY di                22gb           71   30.9gb
prodnode-09        AC3x di              20.4gb           66   30.9gb
prodnode-07        bzYO di              19.3gb           62   30.9gb
prodnode-08        ldTY di              20.3gb           65   30.9gb
prodnode-06        xcRB di              20.4gb           66   30.9gb
prodnode-21        uGbl di                14gb           45   30.9gb
prodnode-02        gmAb di              21.5gb           69   30.9gb
prodnode-20        DAWZ di              20.3gb           65   30.9gb
prodnode-11        QfHe di              20.7gb           66   30.9gb
prodnode-14        vprk di              16.7gb           54   30.9gb
prodnode-17        fpOL di              21.6gb           69   30.9gb
prodnode-13        feLL di                19gb           61   30.9gb
prodnode-10        cl1F di              20.1gb           65   30.9gb
prodnode-01        D8vH di              20.9gb           67   30.9gb
prodnode-03        y886 di              19.3gb           62   30.9gb
4

This version is extremely old, literally years past EOL. You should upgrade as a matter of urgency. Newer versions are significantly more memory-efficient.

1 Like
5

Yep, your best bet is to upgrade as the entire set of 6.X releases are EOL with 8.X now GA.

6

Thanks for providing the suggestion. We are planning to for it, may take sometime. in the meantime we are tying to make the cluster stable. segment memory seems to be a biggest problem for us.

7

What is the output from the _cluster/stats?pretty&human API?

8

If you are no longer writing to some of the indices you can save heap usage by forcemerging down to a single segment. If all your indices however can continue to be updated or have documents deleted, the only way to reduce heap usage is to upgrade as far as i know.

9 
  "status": "green",
  "indices": {
    "count": 2325,
    "shards": {
      "total": 6957,
      "primaries": 3469,
      "replication": 1.005477082732776,
      "index": {
        "shards": {
          "min": 2,
          "max": 21,
          "avg": 2.992258064516129
        },
        "primaries": {
          "min": 1,
          "max": 5,
          "avg": 1.492043010752688
        },
        "replication": {
          "min": 1,
          "max": 20,
          "avg": 1.0081720430107526
        }
      }
    },
    "docs": {
      "count": 26739032827,
      "deleted": 1521435325
    },
    "store": {
      "size": "84.4tb",
      "size_in_bytes": 92898167696858
    },
    "fielddata": {
      "memory_size": "32.3gb",
      "memory_size_in_bytes": 34786252884,
      "evictions": 0
    },
    "query_cache": {
      "memory_size": "4.7gb",
      "memory_size_in_bytes": 5071146612,
      "total_count": 15520344,
      "hit_count": 1900137,
      "miss_count": 13620207,
      "cache_size": 63067,
      "cache_count": 71211,
      "evictions": 8144
    },
    "completion": {
      "size": "0b",
      "size_in_bytes": 0
    },
    "segments": {
      "count": 128321,
      "memory": "217gb",
      "memory_in_bytes": 233085706881,
      "terms_memory": "183gb",
      "terms_memory_in_bytes": 196505121341,
      "stored_fields_memory": "27gb",
      "stored_fields_memory_in_bytes": 29002930872,
      "term_vectors_memory": "0b",
      "term_vectors_memory_in_bytes": 0,
      "norms_memory": "14.7mb",
      "norms_memory_in_bytes": 15464960,
      "points_memory": "6.5gb",
      "points_memory_in_bytes": 7037194208,
      "doc_values_memory": "500.6mb",
      "doc_values_memory_in_bytes": 524995500,
      "index_writer_memory": "109.6mb",
      "index_writer_memory_in_bytes": 114924356,
      "version_map_memory": "1.5mb",
      "version_map_memory_in_bytes": 1595430,
      "fixed_bit_set": "187mb",
      "fixed_bit_set_memory_in_bytes": 196151392,
      "max_unsafe_auto_id_timestamp": 1646784006999,
      "file_sizes": {}
    }
  }
10

thanks for suggestion. We will try to forcemerge the segments to one for indices which are not writable anymore.

11

There are additional details around this available in this webinar. Note that this most likely no longer applies to recent versions due to the improvements made to memory usage since it was realeased several years ago.

12

That appears to be only part of the output?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.