# HPROF Created on ES 6.2.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2018, 3:55pm UTC](https://discuss.elastic.co/t/hprof-created-on-es-6-2-2/126967 "2018-04-05T15:55:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Post date:** [April 5, 2018, 3:55pm UTC](https://discuss.elastic.co/t/hprof-created-on-es-6-2-2/126967/1 "2018-04-05T15:55:28Z")
Hi, I installed ELK 6.2 version and allocated -Xms1g as heap space and 370 GB space left out in the drive. Not sure what queries had been passed thru Kibana, Elastic Search stopped abruptly and created a HPROF file, I tried analyzing the HPROF with the Memory Analyzer and below are the findings: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/9/993771ab407f7ab296ec711875119d93fd9f70ce.jpg) ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f5fc863fd5dd9d770a41735afc27350c4e658b88.png) Here are the problem stacktrace for problem suspect 1: ```auto elasticsearch[node-1][search][T#2] at java.lang.OutOfMemoryError.()V (Unknown Source) at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(I)[B (PageCacheRecycler.java:99) at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(I)Ljava/lang/Object; (PageCacheRecycler.java:96) at org.elasticsearch.common.recycler.DequeRecycler.obtain(I)Lorg/elasticsearch/common/recycler/Recycler$V; (DequeRecycler.java:53) at org.elasticsearch.common.recycler.AbstractRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (AbstractRecycler.java:33) at org.elasticsearch.common.recycler.DequeRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (DequeRecycler.java:28) at org.elasticsearch.common.recycler.FilterRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (FilterRecycler.java:39) at org.elasticsearch.common.recycler.Recyclers$3.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (Recyclers.java:119) at org.elasticsearch.common.recycler.FilterRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (FilterRecycler.java:39) at org.elasticsearch.common.util.PageCacheRecycler.bytePage(Z)Lorg/elasticsearch/common/recycler/Recycler$V; (PageCacheRecycler.java:147) at org.elasticsearch.common.util.AbstractBigArray.newBytePage(I)[B (AbstractBigArray.java:117) at org.elasticsearch.common.util.BigByteArray.resize(J)V (BigByteArray.java:143) at org.elasticsearch.common.util.BigArrays.resizeInPlace(Lorg/elasticsearch/common/util/AbstractBigArray;J)Lorg/elasticsearch/common/util/AbstractBigArray; (BigArrays.java:449) at org.elasticsearch.common.util.BigArrays.resize(Lorg/elasticsearch/common/util/ByteArray;J)Lorg/elasticsearch/common/util/ByteArray; (BigArrays.java:496) at org.elasticsearch.common.util.BigArrays.grow(Lorg/elasticsearch/common/util/ByteArray;J)Lorg/elasticsearch/common/util/ByteArray; (BigArrays.java:513) at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(J)V (BytesStreamOutput.java:157) at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.ensureCapacity(J)V (ReleasableBytesStreamOutput.java:69) at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes([BII)V (BytesStreamOutput.java:89) at org.elasticsearch.common.io.Streams$FlushOnCloseOutputStream.writeBytes([BII)V (Streams.java:266) at org.elasticsearch.common.io.stream.StreamOutput.write([BII)V (StreamOutput.java:408) at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer()V (UTF8JsonGenerator.java:2039) at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeRaw([CII)V (UTF8JsonGenerator.java:638) at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeRaw(Ljava/lang/String;)V (UTF8JsonGenerator.java:555) at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(D)V (UTF8JsonGenerator.java:922) at com.fasterxml.jackson.core.JsonGenerator.writeNumberField(Ljava/lang/String;D)V (JsonGenerator.java:1571) at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeNumberField(Ljava/lang/String;D)V (JsonXContentGenerator.java:208) at org.elasticsearch.common.xcontent.XContentBuilder.field(Ljava/lang/String;D)Lorg/elasticsearch/common/xcontent/XContentBuilder; (XContentBuilder.java:331) at org.elasticsearch.search.aggregations.metrics.sum.InternalSum.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalSum.java:85) at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205) at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129) at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram$Bucket.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalDateHistogram.java:152) at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalDateHistogram.java:479) at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205) at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129) at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalSingleBucketAggregation.java:135) at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205) at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129) at org.elasticsearch.search.aggregations.Aggregations.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:120) ``` Here is the stacktrace for problem suspect 2: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9f5406e7b6e9a87e21fb20d9543314f42310e8ad.png) Here is the stacktrace for problem suspect 3: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/0/905351ad34e93ea72531309864f7e0d2f2d26ca0.png) Just trying to understand what made the elasticsearch server down ? Please let me know. Thanks ! ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 6, 2018, 10:12am UTC](https://discuss.elastic.co/t/hprof-created-on-es-6-2-2/126967/2 "2018-04-06T10:12:41Z")
We have some best-effort checks to fail requests before out-of-memory errors known as circuit breakers, but it looks like it didn't work in your case. The heap and stack trace of the exception suggests that the issue might be due to the fact that you requested a very large histogram. I'm not sure how much data you have in your cluster, but it's common to need more than 1GB of heap when working with large amounts of data. You might want to try going with 2GB or 4GB. ---
**Author:** ![paul1243](https://avatars.discourse-cdn.com/v4/letter/p/e99b99/32.png) [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Post date:** [April 6, 2018, 10:29am UTC](https://discuss.elastic.co/t/hprof-created-on-es-6-2-2/126967/3 "2018-04-06T10:29:32Z")
Thanks @jpountz. I'll increase the heapspace and will get back if I hit the same issue again. Thanks ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2018, 10:29am UTC](https://discuss.elastic.co/t/hprof-created-on-es-6-2-2/126967/4 "2018-05-04T10:29:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.