# Outofmemory exception on ES 1.7.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [August 5, 2015, 4:45pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908 "2015-08-05T16:45:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![kkomv](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@kkomv](https://discuss.elastic.co/u/kkomv)\ **Post date:** [August 5, 2015, 4:45pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/1 "2015-08-05T16:45:11Z")
Hi, We are seeing an out of memory exception on the Elasticsearech JVM , as it is running out of heap space after a while. We are using elastic search version 1.7.0, with 1.7.0\_79 version of java, when we looked at the heap dump we could see that around 80% of the retained heap size was with around 1.7 million clusterState objects in the heap. Any pointers on why the heap is being retained by this objects and not being released? How often are the clusterState objects created? We are running a 4 node ES cluster and each ES JVM is being run with a heap space of 8g ``` ES_HEAP_SIZE=8g ``` Any help is much appreciated. Thanks. ``` Class Name | Shallow Heap | Retained Heap | Percentage -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor @ 0x750acc598 | 88 | 3,281,330,168 | 81.40% |- java.util.concurrent.PriorityBlockingQueue @ 0x750b96760 | 40 | 3,281,329,448 | 81.40% | |- java.lang.Object[1707954] @ 0x7f670bc00 | 6,831,832 | 3,281,329,304 | 81.40% | | |- org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable @ 0x74fc721b8| 40 | 532,728 | 0.01% | | |- org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable @ 0x74fc6fc38| 40 | 529,784 | 0.01% | | |- org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable @ 0x74fc70eb0| 40 | 521,008 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x708454380 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7e6174ff8 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7bd933908 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7cacb4780 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x73c460d50 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7292082e8 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x73b610838 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x79fadbb98 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7200bf040 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7306d3430 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x7146e6fb0 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x72b5c88d0 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x724eac8f8 | 56 | 400,104 | 0.01% | | |- org.elasticsearch.cluster.ClusterState @ 0x71d312540 | 56 | 400,104 | 0.01% | | '- Total: 25 of 1,435,266 entries; 1,435,241 more | | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 6, 2015, 2:34am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/2 "2015-08-06T02:34:25Z")
You don't happen to have a lot of mappings do you? ---
**Author:** ![kkomv](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@kkomv](https://discuss.elastic.co/u/kkomv)\ **Post date:** [August 6, 2015, 3:48am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/3 "2015-08-06T03:48:14Z")
No, just 2 , the default and a custom mapping. I do however see a `update_mapping [logs] (dynamic)` Info log whenever a new index is created, index creation is being done on an hourly basis. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 6, 2015, 4:01am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/4 "2015-08-06T04:01:04Z")
That's a lot of indices, why so many? ---
**Author:** ![mosiddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mosiddi/32/577_2.png) [@mosiddi](https://discuss.elastic.co/u/mosiddi)\ **Post date:** [August 6, 2015, 4:21am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/5 "2015-08-06T04:21:53Z")
If you have a lot of index, your cluster state will be huge and will consume space in memory, actually a lot of space. Remember in ES, a lot cluster state sync happens across nodes and all nodes maintain the state, keeping it light is important. If you have marvel enabled, the large # of indices will impact that as well. Marvel client will keep sending index stats for all those indices and consume a lot of memory. ---
**Author:** ![kkomv](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@kkomv](https://discuss.elastic.co/u/kkomv)\ **Post date:** [August 6, 2015, 3:12pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/6 "2015-08-06T15:12:23Z")
We are indexing on an hourly basis because of the volume of the logs, our hourly index sizes frequently go beyond 5gb. We have been running with this setup on ES 1.2 successfully, but now seeing an issue on upgrading to 1.7. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 6, 2015, 11:05pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/7 "2015-08-06T23:05:45Z")
5GB for an index is not big, I'd just stick with a daily one with (at least) 4 shards. To add to that, if you have the default of 5 shards and 1 replica, you have 240 shards _per day_, which is a lot. And a shard is a Lucene instance that requires resources to be maintained. Over sharding, which is what you are doing, is going to be playing a part in this OOM. ---
**Author:** ![kkomv](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@kkomv](https://discuss.elastic.co/u/kkomv)\ **Post date:** [August 6, 2015, 11:57pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/8 "2015-08-06T23:57:36Z")
Thanks for the response @warkolm. I will try out by indexing on a daily basis, also, when I said 5gb as an index size, that's the size for an hour, so if we index on a daily basis, in the worst case the index size could go upto 120GB, would that cause any other problem? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 7, 2015, 12:13am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/9 "2015-08-07T00:13:23Z")
Nope, that's fine. ---
**Author:** ![mosiddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mosiddi/32/577_2.png) [@mosiddi](https://discuss.elastic.co/u/mosiddi)\ **Post date:** [August 11, 2015, 3:20pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/10 "2015-08-11T15:20:13Z")
Seems reasonable to me as well ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:56pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-es-1-7-0/26908/11 "2017-07-05T23:56:28Z")