Hi,
I have noticed that when using DateHistograms I'm often running out of memory and my Elasticsearch is crashing. Especially with lower configured intervals. I have tried to tweak Elasticsearch circuit breaker settings but they don't seem to help much in this case.
Is there anything I can do to prevent Elasticsearch from breaking? I'm ok with an error returned from the query.
Kind regards,
Which version are you running?
I'm currently running version 7.9.1.
Any chance you have min_doc_count set to 0, have a large range, and have a small interval? I believe we can run out of memory in that case. I merged https://github.com/elastic/elasticsearch/pull/72081 recently to stop it from running out.
If you have something else I'd love to hear about it! Running out of memory is always a bug. Might be one we've fixed, but yeah.
The best way to file an issue is to reproduce it on an empty install with a bash script running curl. Or the kibana dev console. If you can do that the stack trace on the oom would give us a hint. But sometimes it takes detective work to from there.
Yes, I have used min_doc_count of 0 with a small interval. This means it will probably be fixed with your merged changes.
The stack trace of the exception was:
[2021-05-03T09:42:43,756][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [*] fatal error in thread [elasticsearch[*][search][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(ArrayList.java:154) ~[?:?]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:335) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:330) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForSingleBucket(BucketsAggregator.java:294) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregations(NestedAggregator.java:127) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:193) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:193) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]
That one is different! It looks like it happened on the data node rather than the coordinating node. So it'll be a little trickier to reproduce. I've filed https://github.com/elastic/elasticsearch/issues/72619 to tackle it.
I have reduced the example to a smaller case. However, I now get a slightly different exception.
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index' -d '{}'
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index/_mapping' -d '{
"properties": {
"nested": {
"type": "nested"
}
}
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_doc' -d '{
"value": "A",
"dateValue": 100,
"nested": {
"date": 0
}
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_doc' -d '{
"value": "B",
"dateValue": 100000000000,
"nested": {
"date": 10000000000000
}
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_search' -d '{
"size": 0,
"query": {
"bool": {
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"date1": {
"date_histogram": {
"field": "dateValue",
"calendar_interval": "1s",
"offset": 0,
"order": {
"_key": "asc"
},
"keyed": false,
"min_doc_count": 0
},
"aggregations": {
"activities": {
"nested": {
"path": "nested"
},
"aggregations": {
"date2": {
"date_histogram": {
"field": "activities.date",
"calendar_interval": "1s",
"offset": 0,
"order": {
"_key": "asc"
},
"keyed": false,
"min_doc_count": 0
}
}
}
}
}
}
}
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_search' -d '{
"size": 0,
"query": {
"bool": {
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"activities": {
"nested": {
"path": "nested"
},
"aggregations": {
"dateHistogram": {
"date_histogram": {
"field": "nested.date",
"calendar_interval": "1s",
"offset": 0,
"order": {
"_key": "asc"
},
"keyed": false,
"min_doc_count": 0
}
}
}
}
}
}'
The last two queries will result in the following exception:
[2021-05-04T12:51:39,411][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [*] fatal error in thread [elasticsearch[*][search][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3689) ~[?:?]
at java.util.ArrayList.grow(ArrayList.java:238) ~[?:?]
at java.util.ArrayList.grow(ArrayList.java:243) ~[?:?]
at java.util.ArrayList.add(ArrayList.java:518) ~[?:?]
at java.util.ArrayList$ListItr.add(ArrayList.java:1098) ~[?:?]
at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.addEmptyBuckets(InternalDateHistogram.java:417) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.reduce(InternalDateHistogram.java:441) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:258) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:268) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:112) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:258) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:202) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:543) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:519) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:423) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.SearchPhaseController$2.reduce(SearchPhaseController.java:807) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:116) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
I was able to reproduce the initial issue with a reduced example:
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index2' -d '{}'
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index2/_mapping' -d '{
"properties": {
"nested": {
"type": "nested"
}
}
}'
for i in {0..1000}
do
dateValue=$(($i * 100))
date_end=$(($i * 100000))
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index2/_doc' -d "{
\"value\": \"A\",
\"textValue\": $dateValue,
\"nested\": [{
\"date\": 0
}, {
\"date\": $date_end
}]
}"
done
sleep 5s
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index2/_search' -d '{
"size": 0,
"query": {
"bool": {
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"agg1": {
"terms": {
"field": "textValue",
"size": 2147483647,
"min_doc_count": 0
},
"aggregations": {
"agg2": {
"terms": {
"field": "textValue",
"size": 2147483647,
"min_doc_count": 0
},
"aggregations": {
"activities": {
"nested": {
"path": "nested"
},
"aggregations": {
"dateHistogram": {
"date_histogram": {
"field": "nested.date",
"calendar_interval": "1M",
"offset": 0,
"order": {
"_key": "asc"
},
"keyed": false,
"min_doc_count": 0
}
}
}
}
}
}
}
}
}
}'
The returned exception is:
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.ArrayList.<init>(ArrayList.java:154)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:335)
at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForSingleBucket(BucketsAggregator.java:294)
at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregations(NestedAggregator.java:127)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:58)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:288)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:268)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:188)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:151)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:128)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:58)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:288)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:268)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:188)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:151)
at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:128)
at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:160)
at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:128)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$22(IndicesService.java:1393)
at org.elasticsearch.indices.IndicesService$$Lambda$3337/0x0000000100b46040.accept(Unknown Source)
at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$23(IndicesService.java:1445)
at org.elasticsearch.indices.IndicesService$$Lambda$3338/0x0000000100b46440.get(Unknown Source)
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)
at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.