# High ram.percent, low heap.percent, low system ram usage. What is ram.percent? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 6, 2017, 11:38am UTC](https://discuss.elastic.co/t/high-ram-percent-low-heap-percent-low-system-ram-usage-what-is-ram-percent/106511 "2017-11-06T11:38:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\ **Post date:** [November 6, 2017, 11:38am UTC](https://discuss.elastic.co/t/high-ram-percent-low-heap-percent-low-system-ram-usage-what-is-ram-percent/106511/1 "2017-11-06T11:38:32Z")
I had a 5 node cluster that went TU over the weekend (Tits Up). Version: 2.3.1 AWS instance type: m4.2xlarge `vm.swappiness` is set correctly (according to advice given here: [https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html)) `ES_MAX_HEAP` is set to 50% the available RAM (so around 16GB on this instance type). Take a look at the node: `es-live-5`: ![Pasted image at 2017_11_03 09_14 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/4/14271f30d9fa722029a23d56cdc7f6d8a21dc7c2.png) The compare with the Used RAM gauge on my netdata dash taken at the same time: ![Pasted image at 2017_11_03 09_14 AM (1)](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a3c8f581d6baaa86841c8529fb2f7f0e968686d.png) So the system RAM usage is low, jvm heap usage is low, but `ram.percent` is set at 99! What is `ram.percent`? Thanks for any help offered! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 11:41am UTC](https://discuss.elastic.co/t/high-ram-percent-low-heap-percent-low-system-ram-usage-what-is-ram-percent/106511/2 "2017-11-06T11:41:40Z")
What exactly happened? What is in the logs? ---
**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\ **Post date:** [November 6, 2017, 11:56am UTC](https://discuss.elastic.co/t/high-ram-percent-low-heap-percent-low-system-ram-usage-what-is-ram-percent/106511/3 "2017-11-06T11:56:27Z")
From what I can see a similar thing happened across all nodes: 1. An agg from kibana attempted to load fielddata for a field that was just over the allowed limit (9.2GB over 8.9GB), so a circuit breaker kicked in. 2. Within a minute all nodes entered a stop-the-world round of GC. Whilst GC is happening, inter-node communications break down. 3. At least 3 of the nodes ran out of JVM heap space. 4. As inter-node comms breaks down, data nodes lost track of the assigned master. At this point they start suppressing rest. All of this happened before the above scenarios. When I finally got the cluster back together, this is what it looked like. I started up a failover cluster, because the live cluster remains in a hobbled state (it won't assign shards from a snapshot restore to any node despite the fact that they all have capacity). When restoring a snapshot to the failover cluster I got the same sort of stats as above: 1. low heap.percent 2. 99% ram.percent 3. Low used RAM via netdata So my question remains: What _is_ `ram.percent`? It's not an indication of system ram. It's not JVM heap usage. So what resource is it? And how can I increase it if I need to? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2017, 11:57am UTC](https://discuss.elastic.co/t/high-ram-percent-low-heap-percent-low-system-ram-usage-what-is-ram-percent/106511/4 "2017-12-04T11:57:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.