Causes of High JVM in Data and Master nodes

Elastic Stack Elasticsearch
1

Hi,

What are the possible causes for JVM to be high? Our cluster's JVM Memory Pressure in data nodes are always about 60-75%. Master JVM Memory Pressure peaks at 75% every 10 minutes.
We have ~4,000 indices in our cluster and have fielddata in our mapping. Our disk consumption however is very small.

2

Field data, mappings and cluster state all drive heap usage. What is the output of the cluster stats API?

3

Hi,

Here it is.

{
"_nodes": {
    "total": 7,
    "successful": 7,
    "failed": 0
},
"cluster_name": "<removed>",
"status": "green",
"indices": {
    "count": 5439,
    "shards": {
        "total": 30678,
        "primaries": 15339,
        "replication": 1,
        "index": {
            "shards": {
                "min": 2,
                "max": 10,
                "avg": 5.640375068946497
            },
            "primaries": {
                "min": 1,
                "max": 5,
                "avg": 2.8201875344732485
            },
            "replication": {
                "min": 1,
                "max": 1,
                "avg": 1
            }
        }
    },
    "docs": {
        "count": 79202583,
        "deleted": 5808301
    },
    "store": {
        "size_in_bytes": 141836384798,
        "throttle_time_in_millis": 0
    },
    "fielddata": {
        "memory_size_in_bytes": 101200,
        "evictions": 0
    },
    "query_cache": {
        "memory_size_in_bytes": 1013499,
        "total_count": 45995,
        "hit_count": 1372,
        "miss_count": 44623,
        "cache_size": 419,
        "cache_count": 1193,
        "evictions": 774
    },
    "completion": {
        "size_in_bytes": 0
    },
    "segments": {
        "count": 104470,
        "memory_in_bytes": 871332877,
        "terms_memory_in_bytes": 707943013,
        "stored_fields_memory_in_bytes": 64486824,
        "term_vectors_memory_in_bytes": 0,
        "norms_memory_in_bytes": 33342528,
        "points_memory_in_bytes": 2194720,
        "doc_values_memory_in_bytes": 63365792,
        "index_writer_memory_in_bytes": 0,
        "version_map_memory_in_bytes": 0,
        "fixed_bit_set_memory_in_bytes": 26151104,
        "max_unsafe_auto_id_timestamp": -1,
        "file_sizes": {}
    }
},
"nodes": {
    "count": {
        "total": 7,
        "data": 4,
        "coordinating_only": 0,
        "master": 3,
        "ingest": 4
    },
    "versions": [
        "5.5.2"
    ],
    "os": {
        "available_processors": 38,
        "allocated_processors": 38,
        "names": [
            {
                "count": 7
            }
        ],
        "mem": {
            "total_in_bytes": 160042971136,
            "free_in_bytes": 4261928960,
            "used_in_bytes": 155781042176,
            "free_percent": 3,
            "used_percent": 97
        }
    },
    "process": {
        "cpu": {
            "percent": 15
        },
        "open_file_descriptors": {
            "min": 776,
            "max": 16371,
            "avg": 9658
        }
    },
    "jvm": {
        "max_uptime_in_millis": 5552780219,
        "mem": {
            "heap_used_in_bytes": 52076067624,
            "heap_max_in_bytes": 84494385152
        },
        "threads": 1101
    },
    "fs": {
        "total_in_bytes": 6359232143360,
        "free_in_bytes": 6205748789248,
        "available_in_bytes": 5883508801536
    }
}}

Thanks!

4

Getting up to 75% heap usage periodically is normal and not necessarily something to be concerned about, at least as long as it drops back down again. One problem in your cluster is that you have far too many shards in you cluster which wastes resources. I would expect a node to have a few hundred shards at most and you are way beyond that. I would recommend you reduce this dramatically as it will cause problems if it is not already.

1 Like
5

We are periodically experiencing 75% but recently, it reaches 75% more often and drops only to ~60% which is noticeably higher than previous instances where the lowest JVM is around 50%. Will this number of shards also result to the high JVM on master node? Thanks!

6

It will result in a large cluster state which can affect cluster stability. It will also use heap, although not sure how much. Have a look at this blog post for some practical guidance.

7

Thank you so much.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.