# JVM Memory Always at 50% and Above **URL:** \ **Category:** Elasticsearch\ **Created:** [February 25, 2018, 6:45am UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365 "2018-02-25T06:45:41Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![rajivchodisetti](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rajivchodisetti](https://discuss.elastic.co/u/rajivchodisetti)\ **Post date:** [February 25, 2018, 6:45am UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/1 "2018-02-25T06:45:41Z")
![35%20AM](https://us1.discourse-cdn.com/elastic/original/3X/2/7/27e4e474114879e2115bc561298f94c6f638e8b0.png) Hi, We have a 3 master and 6 Data Node ES setup, where data node is of config 30 GB RAM and 8 Core and where 16GB RAM is allocated for heap and rest for file system cache (i.e Lucene index caching) We use ES mostly as a time series database and we are not using druid because our use case involve updating the dimensional data i.e updating documents whenever there is change in the product data Our Index is de-normalized as per the ES best practices where the same document will contain both product info and sale or other facts and we create one index for each month where each index has 2 replicas and 6 shards each and each monthly index is of 30GB in size and holds 10million documents each and we store 3 years worth of data like this and at service layer we have optimizations like , if someone is querying for 3 months worth of data, then the query which we fire to ES refer to only those 3 monthly indices swap is set to off on all the Nodes, open files limit has been set to 131070 on all nodes and we run force merge on all newly created indices to reduce the number of segments to 6 on a week basis. And now coming to the main problem, if you see the image attached, JVM memory usage on all ES Nodes is always 40% plus and the attached screenshot is taken when there aren't any loads or queries running on top of ES, not sure how to debug this problem ? So any pointers on how can I drill down to the root cause of the issue ? One other important point is, we run a lot of scripted queries on ES and this is how those scripted queries looks like [https://www.dropbox.com/s/qlfem4qs04zchpg/scripted\_queries\_es.json?dl=0](https://www.dropbox.com/s/qlfem4qs04zchpg/scripted_queries_es.json?dl=0) I can't make these inline scripts parameter driven because their definition isn't constant and will change depending on the filters being passed in the front end by the user, some requests may have 1 filter, some 2 and some "N" number of filters Does these kind of scripted queries will have any negative impact on system performance ? > **[scripted\_queries\_es.json](https://www.dropbox.com/s/qlfem4qs04zchpg/scripted_queries_es.json?dl=0)** > > Shared with Dropbox ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [February 25, 2018, 11:01pm UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/2 "2018-02-25T23:01:51Z")
Does this show any fields or parent/child relationships taking up lots of memory? ```auto GET /_nodes/stats/indices/fielddata?level=indices&fields=*&human&pretty ``` Does this show a large percentage of the 16GB heap in the old gen? ```auto GET /_nodes/stats/jvm ``` ---
**Author:** ![rajivchodisetti](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rajivchodisetti](https://discuss.elastic.co/u/rajivchodisetti)\ **Post date:** [February 26, 2018, 2:21am UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/3 "2018-02-26T02:21:58Z")
@loren GET /\_nodes/stats/indices/fielddata?level=indices&fields=\*&human&pretty , this (field data) is only about 132MB per node But this one GET /\_nodes/stats/jvm, shows most of the large percentage of heap in the old gen, have pasted the output of one node for reference | jvm | | | --- | --- | | timestamp | 1519611476408 | | uptime\_in\_millis | 13696410800 | | mem | | | heap\_used\_in\_bytes | 10296816952 | | heap\_used\_percent | 60 | | heap\_committed\_in\_bytes | 17110138880 | | heap\_max\_in\_bytes | 17110138880 | | non\_heap\_used\_in\_bytes | 176787312 | | non\_heap\_committed\_in\_bytes | 194248704 | | pools | | | young | | | used\_in\_bytes | 455208768 | | max\_in\_bytes | 558432256 | | peak\_used\_in\_bytes | 558432256 | | peak\_max\_in\_bytes | 558432256 | | survivor | | | used\_in\_bytes | 4998528 | | max\_in\_bytes | 69730304 | | peak\_used\_in\_bytes | 69730304 | | peak\_max\_in\_bytes | 69730304 | | old | | | used\_in\_bytes | 9836609656 | | max\_in\_bytes | 16481976320 | | peak\_used\_in\_bytes | 14988439840 | | peak\_max\_in\_bytes | 16481976320 | ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [February 26, 2018, 7:26pm UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/4 "2018-02-26T19:26:25Z")
Since 10G of your 16G heap is allocated to objects that are long lived (i.e. aren't going to get garbage collected), it makes sense that you don't see the JVM heap percentage drop below 60% even when the node is idle. The next question is what is in the old gen. If you can't identify any obvious areas in your application (e.g., a plugin that creates an on-heap cache), then you could dump the JVM heap and inspect it manually (jmap/jconsole/jvisualvm) to get a better sense of the allocations. I assume you are using the default `jvm.options` file with the only change being ms/mx=16GB? ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [February 26, 2018, 11:59pm UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/5 "2018-02-26T23:59:17Z")
I totally missed that you have 155 shards per data node. That's 1000 shards for the cluster. Reduce that by one or two orders of magnitude and I think your JVM heap usage will be more sane. ---
**Author:** ![rajivchodisetti](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rajivchodisetti](https://discuss.elastic.co/u/rajivchodisetti)\ **Post date:** [February 27, 2018, 3:12am UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/6 "2018-02-27T03:12:49Z")
Oh, I was planning to add more indices 🙂 and BTW is there any relationship between number of Nodes & shards, because for each index, I have configured 6 shards & 2 replicas as I have 6 machines in place thinking that all machines will participate in query execution Also on the heap size, I have gone through this below blog earlier, where they were saying for each GB heap allocated, we can have 20 shards i.e for 16GB, we can have around 320 shards per Node > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... "TIP: The number of shards you can hold on a node will be proportional to the amount of heap you have available, but there is no fixed limit enforced by Elasticsearch. A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600-750 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health." ---
**Author:** ![rajivchodisetti](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rajivchodisetti](https://discuss.elastic.co/u/rajivchodisetti)\ **Post date:** [February 27, 2018, 8:26am UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/7 "2018-02-27T08:26:33Z")
![58%20PM](https://us1.discourse-cdn.com/elastic/original/3X/5/c/5cba018bdb88484e94ff7ce40ce8eaf181a8b169.png) I have cut down the number of shards by 50% by dropping all the UN-used indexes , but still the heap is at 40% even if nothing is running. Is it a common thing ? ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [February 27, 2018, 5:31pm UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/8 "2018-02-27T17:31:37Z")
I think so. As that article says, "the further below this limit you can keep it the better". I would shoot for fewer shards per node and see what effect it has on your heap usage. Use 1 shard for each index unless the index won't fit on a single node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 27, 2018, 5:31pm UTC](https://discuss.elastic.co/t/jvm-memory-always-at-50-and-above/121365/9 "2018-03-27T17:31:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.