HPROF Created on ES 6.2.2

paul1243 · April 5, 2018, 3:55pm

Hi,
I installed ELK 6.2 version and allocated -Xms1g as heap space and 370 GB space left out in the drive. Not sure what queries had been passed thru Kibana, Elastic Search stopped abruptly and created a HPROF file, I tried analyzing the HPROF with the Memory Analyzer and below are the findings:

Here are the problem stacktrace for problem suspect 1:


elasticsearch[node-1][search][T#2]
  at java.lang.OutOfMemoryError.<init>()V (Unknown Source)
  at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(I)[B (PageCacheRecycler.java:99)
  at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(I)Ljava/lang/Object; (PageCacheRecycler.java:96)
  at org.elasticsearch.common.recycler.DequeRecycler.obtain(I)Lorg/elasticsearch/common/recycler/Recycler$V; (DequeRecycler.java:53)
  at org.elasticsearch.common.recycler.AbstractRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (AbstractRecycler.java:33)
  at org.elasticsearch.common.recycler.DequeRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (DequeRecycler.java:28)
  at org.elasticsearch.common.recycler.FilterRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (FilterRecycler.java:39)
  at org.elasticsearch.common.recycler.Recyclers$3.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (Recyclers.java:119)
  at org.elasticsearch.common.recycler.FilterRecycler.obtain()Lorg/elasticsearch/common/recycler/Recycler$V; (FilterRecycler.java:39)
  at org.elasticsearch.common.util.PageCacheRecycler.bytePage(Z)Lorg/elasticsearch/common/recycler/Recycler$V; (PageCacheRecycler.java:147)
  at org.elasticsearch.common.util.AbstractBigArray.newBytePage(I)[B (AbstractBigArray.java:117)
  at org.elasticsearch.common.util.BigByteArray.resize(J)V (BigByteArray.java:143)
  at org.elasticsearch.common.util.BigArrays.resizeInPlace(Lorg/elasticsearch/common/util/AbstractBigArray;J)Lorg/elasticsearch/common/util/AbstractBigArray; (BigArrays.java:449)
  at org.elasticsearch.common.util.BigArrays.resize(Lorg/elasticsearch/common/util/ByteArray;J)Lorg/elasticsearch/common/util/ByteArray; (BigArrays.java:496)
  at org.elasticsearch.common.util.BigArrays.grow(Lorg/elasticsearch/common/util/ByteArray;J)Lorg/elasticsearch/common/util/ByteArray; (BigArrays.java:513)
  at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(J)V (BytesStreamOutput.java:157)
  at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.ensureCapacity(J)V (ReleasableBytesStreamOutput.java:69)
  at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes([BII)V (BytesStreamOutput.java:89)
  at org.elasticsearch.common.io.Streams$FlushOnCloseOutputStream.writeBytes([BII)V (Streams.java:266)
  at org.elasticsearch.common.io.stream.StreamOutput.write([BII)V (StreamOutput.java:408)
  at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer()V (UTF8JsonGenerator.java:2039)
  at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeRaw([CII)V (UTF8JsonGenerator.java:638)
  at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeRaw(Ljava/lang/String;)V (UTF8JsonGenerator.java:555)
  at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(D)V (UTF8JsonGenerator.java:922)
  at com.fasterxml.jackson.core.JsonGenerator.writeNumberField(Ljava/lang/String;D)V (JsonGenerator.java:1571)
  at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeNumberField(Ljava/lang/String;D)V (JsonXContentGenerator.java:208)
  at org.elasticsearch.common.xcontent.XContentBuilder.field(Ljava/lang/String;D)Lorg/elasticsearch/common/xcontent/XContentBuilder; (XContentBuilder.java:331)
  at org.elasticsearch.search.aggregations.metrics.sum.InternalSum.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalSum.java:85)
  at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205)
  at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129)
  at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram$Bucket.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalDateHistogram.java:152)
  at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalDateHistogram.java:479)
  at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205)
  at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129)
  at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.doXContentBody(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalSingleBucketAggregation.java:135)
  at org.elasticsearch.search.aggregations.InternalAggregation.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (InternalAggregation.java:205)
  at org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:129)
  at org.elasticsearch.search.aggregations.Aggregations.toXContent(Lorg/elasticsearch/common/xcontent/XContentBuilder;Lorg/elasticsearch/common/xcontent/ToXContent$Params;)Lorg/elasticsearch/common/xcontent/XContentBuilder; (Aggregations.java:120)

Here is the stacktrace for problem suspect 2:

Here is the stacktrace for problem suspect 3:

Just trying to understand what made the elasticsearch server down ? Please let me know. Thanks !

jpountz · April 6, 2018, 10:12am

We have some best-effort checks to fail requests before out-of-memory errors known as circuit breakers, but it looks like it didn't work in your case. The heap and stack trace of the exception suggests that the issue might be due to the fact that you requested a very large histogram.

I'm not sure how much data you have in your cluster, but it's common to need more than 1GB of heap when working with large amounts of data. You might want to try going with 2GB or 4GB.

paul1243 · April 6, 2018, 10:29am

Thanks @jpountz.
I'll increase the heapspace and will get back if I hit the same issue again. Thanks !

system · May 4, 2018, 10:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Excess caching issue [Elasticsearch] Elasticsearch	1	428	July 5, 2017
Reasons Behind HPROF Files - What is the Cause? Elasticsearch	12	6187	April 22, 2020
ES 5.6.2 OutOfMemory heap space Elasticsearch	3	2247	December 15, 2017
Large size of .hprof files on data nodes Elasticsearch	4	19469	July 5, 2017
Kibana crashes Elasticsearch Kibana	8	1920	July 6, 2017

HPROF Created on ES 6.2.2

Related topics