# Elasticsearch uses more memory than JVM heap settings, reaches container memory limit and crash **URL:** https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873 **Category:** Elasticsearch **Tags:** docker **Created:** [February 11, 2020, 11:26pm UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873 "2020-02-11T23:26:25Z") **Posts on this page:** 12 **Page:** 1
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 11, 2020, 11:26pm UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/1 "2020-02-11T23:26:26Z")
Elasticsearch uses more memory than JVM heap settings, which is currently -Xms512m, -Xmx512m. I tried setting those values to 1g and reverted because the container crashed immediately after relaunching containers, because of OOM. I run Elasticsearch 7.3.0 on ECK, and memory usage is reported by Prometheus node exporter. Memory limlt is set to 1.5GiB to allocate some memory for EC2 instances, which are 3x t3.small instance with 2GB of RAM for each. Is it bad idea to set memory limit for Elasticsearch containers? I'm not sure whether it includes virtual memory or not. If it does, that might cause the container to crash even when enough amount of memory is available. > [@ElasticSearch use too much Memory then cause sys cpu too high](https://discuss.elastic.co/t/elasticsearch-use-too-much-memory-then-cause-sys-cpu-too-high/63147): > > I use ElasticSearch to store some datetime based data. Data of each day is stored as a index. The total storage of 75 is about 600GB. when ElasticSearch startup, the proccess's VIRT is up to 145GB. Then when I do lots of aggregation, the proccess's RES and SHR continue growing up quickly. After RES reach 40GB, it will cause the sy CPU growing upto 90% and stay at this level. At that time, the server become very slow and need a long time to respone any command. All I can do is to kill the procc… ---
### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner) #### Post date: [February 12, 2020, 3:14am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/2 "2020-02-12T03:14:08Z")
Quoting the [docs on setting the heap size](https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html) (emphasis mine): > Set `Xmx` and `Xms` to **no more than 50% of your physical RAM**. Elasticsearch requires memory for purposes other than the JVM heap and it is important to leave space for this. For instance, Elasticsearch uses off-heap buffers for efficient network communication, relies on the operating system’s filesystem cache for efficient access to files, and the JVM itself requires some memory too. **It is normal to observe the Elasticsearch process using more memory than the limit configured with the `Xmx` setting.** In a container, "physical RAM" means the memory limit of the container. If you have a 1.5GiB memory limit on your container then you must set the Elasticsearch heap size to no more than 0.75GiB. ---
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 12, 2020, 3:59am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/3 "2020-02-12T03:59:22Z")
Then what is my problem?? JVM heap is set to 512MB currently, which is less than 50% of container memory limit. In my case memory usage sometimes exceeds 1.5GB and crash. Container memory limit is 3 times large as heap size. ---
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 12, 2020, 6:03am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/4 "2020-02-12T06:03:22Z")
This guy on stackoverflow reports similar error as me, and unsetting memory limit fixes the problem. > He also says that linux kernel 4.15 fixes the memory issue. ---
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 12, 2020, 6:30am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/5 "2020-02-12T06:30:20Z")
The issue is still not fixed...? > > > This is a bug report > I searched existing issues before opening this one > Expected behavior > RHEL kernel OOM kills a docker container when... > > > when we upgrade the k8s from 1.6.4 to 1.9.0, after a few days, the product environment report the machine is hang... ---
### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner) #### Post date: [February 12, 2020, 8:28am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/6 "2020-02-12T08:28:47Z")
> [@vroad](#): > > In my case memory usage sometimes exceeds 1.5GB and crash. Sorry, this wasn't clear. You said: > [@vroad](#): > > I tried setting those values to 1g and reverted because the container crashed immediately after relaunching containers, because of OOM [...] Memory limlt is set to 1.5GiB A 1GiB heap is definitely too large for a 1.5GiB container. > [@vroad](#): > > He also says that linux kernel 4.15 fixes the memory issue. Yes there are [known bugs in some kernels](https://github.com/opencontainers/runc/issues/1725) that inappropriately trigger the OOM killer in a container. That still doesn't mean it's a bad idea to set the memory limit on an Elasticsearch container, it just means it's a bad idea to use a buggy kernel. If you think it's not that, please share the full `dmesg` output from such a crash; it could be thousands of lines long, so use [https://gist.github.com/](https://gist.github.com/) if it doesn't fit here. ---
### Author: ![jcastelc](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jcastelc](https://discuss.elastic.co/u/jcastelc) #### Post date: [February 13, 2020, 8:07am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/7 "2020-02-13T08:07:32Z")
Maybe is related to JVM "metaspace" usage, not heap. Check java _MetaspaceSize_ and _MaxMetaspaceSize_ settings (Xmx and Xms too, of course) Try several settings for heap/metaspace, and monitor JVM heap/meta usage with "jstat" command before setting container memory limits. [https://docs.oracle.com/javase/8/docs/technotes/tools/windows/jstat.html](https://docs.oracle.com/javase/8/docs/technotes/tools/windows/jstat.html) > **[Permgen vs Metaspace in Java | Baeldung](https://www.baeldung.com/java-permgen-metaspace)** > > Learn about the differences between the PermGen and Metaspace memory regions in Java ---
### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner) #### Post date: [February 13, 2020, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/8 "2020-02-13T08:25:34Z")
@jcastelc if you are seeing evidence of ongoing metaspace allocation in your cluster then I'd like to see more detail. It is rare to see metaspace memory pressure with Elasticsearch. I think its metaspace usage should be pretty much constant since I'm not aware of any dynamic loading happening after startup, and we account for this in the 2x limit described in the documentation. ---
### Author: ![jcastelc](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jcastelc](https://discuss.elastic.co/u/jcastelc) #### Post date: [February 13, 2020, 9:15am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/9 "2020-02-13T09:15:27Z")
Good to know that. No evidence, it only was a suggestion. Thanks for the details! 🙂 ---
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 17, 2020, 8:24am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/10 "2020-02-17T08:24:29Z")
My ES cluster crashed again, but memory usage of the ES container stayed within the 1.5GiB limit. I've found that EBS burst credit for root volume was running out before the crash. This might be a problem other than memory usage... This time I was able to get logs from failed node. It logged lots of warnings by JvmGcMonitorService. But this might be caused by loss of EBS burst credit for root volume. ---
### Author: ![vroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vroad/32/36242_2.png) [@vroad](https://discuss.elastic.co/u/vroad) #### Post date: [February 17, 2020, 8:35am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/11 "2020-02-17T08:35:35Z")
EBS burst credit decreased slowly before the crush. Once it reaches to 0, the pod gets evicted because of slow I/O caused by that. I should solve this problem first... Thank you for the answers anyway. ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [March 16, 2020, 8:35am UTC](https://discuss.elastic.co/t/elasticsearch-uses-more-memory-than-jvm-heap-settings-reaches-container-memory-limit-and-crash/218873/12 "2020-03-16T08:35:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.