[ES v7.9.1] DateHistogram - OutOfMemory

sag-tobias-frey · April 30, 2021, 9:43pm

Hi,

I have noticed that when using DateHistograms I'm often running out of memory and my Elasticsearch is crashing. Especially with lower configured intervals. I have tried to tweak Elasticsearch circuit breaker settings but they don't seem to help much in this case.

Is there anything I can do to prevent Elasticsearch from breaking? I'm ok with an error returned from the query.

ElasticSearch v7.9.1

Kind regards,

dadoonet · May 1, 2021, 1:07am

Which version are you running?

sag-tobias-frey · May 1, 2021, 11:03am

I'm currently running version 7.9.1.

nik9000 · May 2, 2021, 2:48pm

Any chance you have min_doc_count set to 0, have a large range, and have a small interval? I believe we can run out of memory in that case. I merged https://github.com/elastic/elasticsearch/pull/72081 recently to stop it from running out.

If you have something else I'd love to hear about it! Running out of memory is always a bug. Might be one we've fixed, but yeah.

The best way to file an issue is to reproduce it on an empty install with a bash script running curl. Or the kibana dev console. If you can do that the stack trace on the oom would give us a hint. But sometimes it takes detective work to from there.

sag-tobias-frey · May 3, 2021, 7:47am

Yes, I have used min_doc_count of 0 with a small interval. This means it will probably be fixed with your merged changes.

The stack trace of the exception was:

[2021-05-03T09:42:43,756][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [*] fatal error in thread [elasticsearch[*][search][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
        at java.util.ArrayList.<init>(ArrayList.java:154) ~[?:?]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:335) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:330) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForSingleBucket(BucketsAggregator.java:294) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregations(NestedAggregator.java:127) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:193) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:193) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$700(GlobalOrdinalsStringTermsAggregator.java:65) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:720) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:670) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:586) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:535) ~[elasticsearch-7.9.1.jar:7.9.1]

nik9000 · May 3, 2021, 12:40pm

That one is different! It looks like it happened on the data node rather than the coordinating node. So it'll be a little trickier to reproduce. I've filed https://github.com/elastic/elasticsearch/issues/72619 to tackle it.

sag-tobias-frey · May 4, 2021, 10:52am

I have reduced the example to a smaller case. However, I now get a slightly different exception.

curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index' -d '{}'
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index/_mapping' -d '{
 "properties": {
  "nested": {
   "type": "nested"
  }
 }
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_doc' -d '{
 "value": "A",
 "dateValue": 100,
 "nested": {
  "date": 0
 }
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_doc' -d '{
 "value": "B",
 "dateValue": 100000000000,
 "nested": {
  "date": 10000000000000
 }
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_search' -d '{
 "size": 0,
 "query": {
  "bool": {
   "adjust_pure_negative": true,
   "boost": 1
  }
 },
 "aggregations": {
  "date1": {
   "date_histogram": {
    "field": "dateValue",
    "calendar_interval": "1s",
    "offset": 0,
    "order": {
     "_key": "asc"
    },
    "keyed": false,
    "min_doc_count": 0
   },
   "aggregations": {
    "activities": {
     "nested": {
      "path": "nested"
     },
     "aggregations": {
      "date2": {
       "date_histogram": {
        "field": "activities.date",
        "calendar_interval": "1s",
        "offset": 0,
        "order": {
         "_key": "asc"
        },
        "keyed": false,
        "min_doc_count": 0
       }
      }
     }
    }
   }
  }
 }
}'
curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index/_search' -d '{
 "size": 0,
 "query": {
  "bool": {
   "adjust_pure_negative": true,
   "boost": 1
  }
 },
 "aggregations": {
  "activities": {
   "nested": {
    "path": "nested"
   },
   "aggregations": {
    "dateHistogram": {
     "date_histogram": {
      "field": "nested.date",
      "calendar_interval": "1s",
      "offset": 0,
      "order": {
       "_key": "asc"
      },
      "keyed": false,
      "min_doc_count": 0
     }
    }
   }
  }
 }
}'

The last two queries will result in the following exception:

[2021-05-04T12:51:39,411][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [*] fatal error in thread [elasticsearch[*][search][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3689) ~[?:?]
        at java.util.ArrayList.grow(ArrayList.java:238) ~[?:?]
        at java.util.ArrayList.grow(ArrayList.java:243) ~[?:?]
        at java.util.ArrayList.add(ArrayList.java:518) ~[?:?]
        at java.util.ArrayList$ListItr.add(ArrayList.java:1098) ~[?:?]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.addEmptyBuckets(InternalDateHistogram.java:417) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.reduce(InternalDateHistogram.java:441) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:258) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:268) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:112) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:258) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:202) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:543) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:519) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:423) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.SearchPhaseController$2.reduce(SearchPhaseController.java:807) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:116) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

sag-tobias-frey · May 4, 2021, 12:11pm

I was able to reproduce the initial issue with a reduced example:

curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index2' -d '{}'
curl -H 'Content-type: application/json' -XPUT 'http://localhost:9200/test_index2/_mapping' -d '{
 "properties": {
  "nested": {
   "type": "nested"
  }
 }
}'

for i in {0..1000}
do
  dateValue=$(($i * 100))
  date_end=$(($i * 100000))
  curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index2/_doc' -d "{
	\"value\": \"A\",
	\"textValue\": $dateValue,
	\"nested\": [{
		\"date\": 0
	}, {
		\"date\": $date_end
	}]
  }"
done

sleep 5s

curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/test_index2/_search' -d '{
 "size": 0,
 "query": {
  "bool": {
   "adjust_pure_negative": true,
   "boost": 1
  }
 },
 "aggregations": {
  "agg1": {
   "terms": {
    "field": "textValue",
    "size": 2147483647,
    "min_doc_count": 0
   },
   "aggregations": {
    "agg2": {
     "terms": {
      "field": "textValue",
      "size": 2147483647,
      "min_doc_count": 0
     },
     "aggregations": {
      "activities": {
       "nested": {
        "path": "nested"
       },
       "aggregations": {
        "dateHistogram": {
         "date_histogram": {
          "field": "nested.date",
          "calendar_interval": "1M",
          "offset": 0,
          "order": {
           "_key": "asc"
          },
          "keyed": false,
          "min_doc_count": 0
         }
        }
       }
      }
     }
    }
   }
  }
 }
}'

The returned exception is:

java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.ArrayList.<init>(ArrayList.java:154)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForVariableBuckets(BucketsAggregator.java:335)
        at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.buildAggregations(DateHistogramAggregator.java:145)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForSingleBucket(BucketsAggregator.java:294)
        at org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregations(NestedAggregator.java:127)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:58)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:288)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:268)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:188)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:151)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:128)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:173)
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:236)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:58)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:288)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:268)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:188)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:151)
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:128)
        at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:160)
        at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:128)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:156)
        at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$22(IndicesService.java:1393)
        at org.elasticsearch.indices.IndicesService$$Lambda$3337/0x0000000100b46040.accept(Unknown Source)
        at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$23(IndicesService.java:1445)
        at org.elasticsearch.indices.IndicesService$$Lambda$3338/0x0000000100b46440.get(Unknown Source)
        at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)
        at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)
        at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)

system · June 1, 2021, 12:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.