# Elasticsearch Docker container keeps crashing with exit status of 137 **URL:** \ **Category:** Elasticsearch\ **Created:** [March 23, 2018, 8:18pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410 "2018-03-23T20:18:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 23, 2018, 8:18pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/1 "2018-03-23T20:18:16Z")
Hi Everyone, I have a docker-xpack docker image that I deploy on marathon/mesos. The image works great and is uber-stable when xplack security is disabled. When I enable xpack security and use encrypted comms, the containers crash randomly every 2-3 days with no message other than the docker is exiting with status 137. Status 137 usually means mesos killed the container because it's RAM exceeded the configured max. I have upped both the JVM memory heap and the docker RAM via Mesos configuration from 2GB/4GB to 4GB/8GB and I am still getting exit with 137. In addition, I instrumented one of the containers by calling docker stats every 5 minutes and the overall RAM keeps creeping up and eventually reaches the Mesos limit for the container and Mesos kills it off. Anyone else have trouble with an elasticsearch-xpack docker? Any ideas/suggestions are welcome. Thanks --John ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [March 26, 2018, 3:06pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/2 "2018-03-26T15:06:18Z")
What Elasticsearch version are you on? ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 26, 2018, 3:19pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/3 "2018-03-26T15:19:33Z")
Argh, sorry, I should have specified that--ES 5.6.4 ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 26, 2018, 4:26pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/4 "2018-03-26T16:26:51Z")
More information. This is an Elasticsearch container with default jvm.options (and therefore 2GB memory heap) and 4 GB dedicated to the container. The memory heap is staying steady in the 1GB range while the container is up to 3.97 GB. So, there appears to be off-heap memory that is getting total memory usage \> 4 GB at which point Mesos kills the container. --John ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 28, 2018, 1:43pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/5 "2018-03-28T13:43:21Z")
Interesting update. I have a cluster with xpack security enabled, same docker image but with 31 GB memory heap and 62 GB docker memory and this one has been stable for several weeks. Question: is anyone aware of a minimum memory heap size for elasticsearch with xpack enabled? --John ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 29, 2018, 3:01pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/6 "2018-03-29T15:01:21Z")
More interesting information. I set MALLOC\_ARENA\_MAX=4 and that slowed down the growth of off-heap memory, but did not stabilize it. A colleague of mine who knows Elasticsearch way more than I do updated the refresh\_rate from 10s to 300s. The combo of MALLOC\_ARENA\_MAX=4 and refresh\_rate=300s appears to have stabilized things for now. Will monitor and report an update in a few hours. ---
**Author:** ![hokiegeek2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hokiegeek2/32/11076_2.png) [@hokiegeek2](https://discuss.elastic.co/u/hokiegeek2)\ **Post date:** [March 30, 2018, 1:24pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/7 "2018-03-30T13:24:32Z")
On one of the clusters the memory usage for a 2GB/4GB combo is 2.38 GB, so the combination of MALLOC\_ARENA\_MAX=4 and refresh\_rate=300s appears to have stabilized things for now. On the other cluster (with more data), the memory consumed by the ES Docker continues to increase, albeit more slowly. Again, I can confirm that xpack is not causing this as I've seen this in both configs where xpack is enabled and also when it is disabled. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2018, 1:24pm UTC](https://discuss.elastic.co/t/elasticsearch-docker-container-keeps-crashing-with-exit-status-of-137/125410/8 "2018-04-27T13:24:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.