What causes JVM memory to be high?

Elastic Stack Elasticsearch
1

Whenever I look at the stats of my cluster on Marvel, over time, JVM usage gets really high. I have a few questions:

What causes the JVM memory usage to gradually increase?

Why does the cluster tend to become unresponsive when it gets too high? I personally thought that if it has resources, it should be good to use 100% of them.

2

You should uninstall Marvel.. Actually Marvel Always create index for each index..That takes server space...so avoid Marvel then check...

3

How much of an effect can Marvel have?

I have 16 gb of ram (allocating 8 to ES_HEAP), 20 nodes, mlockall set to true and swapoff on all the machines.

4

What? No.
Use Elasticsearch Curator to clean up older data automatically if you don't need it.
Also, using Marvel can show you the cause of this.

Depends, it can be queries or increases in datasets or many things.
Can you provide more info on your cluster, sizing, version etc.

5

@warkolm

My cluster has 20 data nodes(20 primary shards, 2 replicas), 1 client and 1 master.

Elasticsearch version: 2.3.3

Marvel version: 2.3.0

JVM version: 1.8.0_91

OS version: Ubuntu 14.04

I have mlockall set to true, ran "sudo swapoff -a" on the entire cluster. I'm allocating 16 gb of ram in each node (8gb for ES_HEAP_SIZE).

When I first posted this, I was experiencing a mapper parsing exception due to a bug in the ES output plugin in logstash. This caused the GC to not work (or work incredibly slow to the point the node (and cluster) became unresponsive). As of right now, the log doesn't show me any errors other than the occasional "marvel.agent uncaught thread exception" which doesn't seem to have much of an effect.

I've been ingesting a back log of data I lost in a cluster failure for the past 12 hours. Currently at 132,942,626 documents, 124GB total data, Memory: 118GB / 171GB. My data nodes are constantly sitting at 75% JVM memory usage at this point.

I also have this settings in the log and they're still not producing any more information:
{"persistent":{},"transient":{"logger":{"index":{"indexing":{"slowlog":"DEBUG"},"search":{"slowlog":"DEBUG"}}}}}

Here's a gist of the output of: curl 'http://localhost:9200/_nodes/stats?pretty&human''
gist

Is it normal for the JVM memory usage to be constantly hitting those metrics? Ideally, I would be running queries while indexing, is that attainable? RIght now, there seems to be a high amount of memory usage by indexing/updating (I perform both).

6

Why 2 replicas?
And one master is bad as well, see Important Configuration Changes | Elasticsearch: The Definitive Guide [2.x] | Elastic

7

I will definitely look at the link and increase the number of master nodes. As for the replicas, it's in case of cluster failure? As in, a VM not being recovered after any sort of failure.

Thanks for the help by the way!

8

RAM is ok for u and ur server is only have es ..if u view indices u will see list of indices with big size unnecessary... only because of Marvel.. So delete those indices created by Marvel and check it..

9

Somehow, you are compiling lots of scripts over and over:

  "script" : {
    "compilations" : 140018,
    "cache_evictions" : 139918
  }

These classes are not ever unloaded (see https://github.com/elastic/elasticsearch/issues/18572)

    "classes" : {
      "current_loaded_count" : 151826,
      "total_loaded_count" : 151950,
      "total_unloaded_count" : 124
    }

Don't send a unique new script for every request: instead, make use of script params for the part that changes.