# Heapdump - out of memory java heap space **URL:** \ **Category:** Elasticsearch\ **Created:** [May 3, 2018, 9:46am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416 "2018-05-03T09:46:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 3, 2018, 9:46am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/1 "2018-05-03T09:46:52Z")
Greetings We are running a 3 node cluster(version-1.4.4) (all masters) with 30G heap with JRE 1.7.80 and replication as 1. Data: 2.5TB(live) Total: 25TB shard per index is 3 daily data is around 70-80GB indexes are created everyday We are having frequent GCs running into hours sometimes making the cluster unstable every week output of jmap histo of one of the servers - [https://pastebin.com/8gcQySx2](https://pastebin.com/8gcQySx2) Can someone point me in the right direction Any help is appreciated Thanks in advance ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 3, 2018, 10:05am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/2 "2018-05-03T10:05:35Z")
How many shards in total do you have? You definitely need to upgrade anyway. ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 3, 2018, 11:01am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/3 "2018-05-03T11:01:43Z")
No of primary shards is around 120 Sorry to say but upgrading is not a option in the current situation ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 4, 2018, 5:32am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/4 "2018-05-04T05:32:34Z")
Can some one explain the top 5-6 items using large chunk of memory mean in terms of elasticsearch given in the jmap -histo Link - [https://pastebin.com/8gcQySx2](https://pastebin.com/8gcQySx2) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2018, 5:49am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/5 "2018-05-04T05:49:42Z")
[Doc values](https://www.elastic.co/blog/disk-based-field-data-a-k-a-doc-values) were introduced early in the Elasticsearch 1.x series, but were initially quite slow. This improved over time and performance improved so much by version 1.7 that we could make them enabled by default in Elasticsearch 2.0. If you have fields that you need to aggregate on but not perform free-text search on, you can save a lot of heap space by mapping these using doc\_values. What does your data and mappings look like? Are you using doc values? ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 4, 2018, 6:03am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/6 "2018-05-04T06:03:06Z")
No we are not using Doc values, A daily index typically looks like: [https://pastebin.com/EJUFpxqK](https://pastebin.com/EJUFpxqK) There is only one type in daily indexes and we don't aggregate on analyzed fields Data we capture are the logs generated from the devices like firewall, windows, linux etc ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2018, 6:05am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/7 "2018-05-04T06:05:49Z")
Not analysed fields are still kept on the heap unless you use doc values, so I suspect that is one thing that is driving your heap usage. I recall heap pressure dropping significantly when users switched to doc\_values, allowing much large data volumes to be stored per node. If I remember correctly though (this was a long time ago as this is a very old version), doc values requires the use of aggregations rather than facets, so Kibana3 may not work with doc values. ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 4, 2018, 6:17am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/8 "2018-05-04T06:17:10Z")
Currently on one of the nodes the heap is 65 percent and the fileddata is 1.4 gb and this remains true to all of the nodes. We have even added scripts to clear cache once it reaches 80-85 percent ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2018, 6:24am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/9 "2018-05-04T06:24:15Z")
I would like David recommend upgrading. I do not really remember enough to have any other suggestions at this point. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 4, 2018, 6:35am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/10 "2018-05-04T06:35:26Z")
If you don't want to upgrade another short term solution is to start new nodes. ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 4, 2018, 7:21am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/11 "2018-05-04T07:21:07Z")
Ok , I will keep that in mind. Meanwhile is there anything we can do to make the occurrence of the heap issue less frequent other than adding nodes ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 4, 2018, 7:57am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/12 "2018-05-04T07:57:53Z")
Some ideas: - Don't run aggregations - Don't sort - Remove old indices ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 6, 2018, 5:42am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/13 "2018-05-06T05:42:06Z")
Ok, one last question do you recommend to enable doc values for ES 1.4.4 to resolve or to delay the heap issue, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 6, 2018, 6:20am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/14 "2018-05-06T06:20:33Z")
How are you querying your data? Kibana3? Kibana4? Aggregations? Facets? Just searches? ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 6, 2018, 6:27am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/15 "2018-05-06T06:27:01Z")
aggregation and searches with sorts using python ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 6, 2018, 6:28am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/16 "2018-05-06T06:28:33Z")
Automated queries don't execute for more than 24 hrs of the old data, so maximum 2 indexes will be queried ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 6, 2018, 6:39am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/17 "2018-05-06T06:39:37Z")
If you are not using facets you should (as far as I can recall) be able to switch to using doc values for not analysed fields and save on heap space. If this causes performance problems, it could be worthwhile upgrading to version 1.7.6 as there were a number of performance improvements introduced throughout the series. ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 6, 2018, 6:45am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/18 "2018-05-06T06:45:01Z")
Ok thank you for the thumps up and we are not using facets, I will enable the doc values and keep monitoring. If it only takes more time to give back the result its not an issue for me. Going by yours and David's suggestion we are planning it to upgrade to 2.x series ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 6, 2018, 6:47am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/19 "2018-05-06T06:47:49Z")
I would recommend testing it properly first though, as my memory stretching this far back is a bit hazy... ---
**Author:** ![TaskDead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taskdead/32/30736_2.png) [@TaskDead](https://discuss.elastic.co/u/TaskDead)\ **Post date:** [May 6, 2018, 6:58am UTC](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416/20 "2018-05-06T06:58:04Z")
Ok, I will test it and only then will apply it on the production [Next page](https://discuss.elastic.co/t/heapdump-out-of-memory-java-heap-space/130416.md?page=2)