Problems with aggregations: Data too large

Elastic Stack Elasticsearch
1

Hi,

I have problems when trying to get aggregations. I have a cluster with one node, and I'm trying to interact with an index that contains 534,737,088 documents (around 32 GB). Some aggregations work, and Elasticsearch returns an answer (for example a cardinality aggregation over a field). However, I'm getting an error when running other aggregations such as the following:

curl -X GET 'http://localhost:9200/<my_index>/_search?pretty' -H 'Content-type:application/json' -d'
{
  "size": 0,
  "aggs": {
    "agg_A": {
      "terms": {
        "field": "field_A",
        "size": 10
      },
      "aggs": {
        "agg_B": {
          "terms": {
            "field": "field_B",
            "size": 10
          }
        }
      }
    }
  }
}
'

The error that I'm getting is the following (partial)

{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[parent] Data too large, data for [allocated_buckets] would be [997902256/951.6mb], which is larger than the limit of [996147200/950mb], real usage: [997902256/951.6mb], new bytes reserved: [0/0b], usages [model_inference=0/0b, eql_sequence=0/0b, fielddata=453457236/432.4mb, request=200254400/190.9mb, inflight_requests=10806/10.5kb]",
        "bytes_wanted" : 997902256,
        "bytes_limit" : 996147200,
        "durability" : "PERMANENT"
      },
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[parent] Data too large, data for [allocated_buckets] would be [1000273024/953.9mb], which is larger than the limit of [996147200/950mb], real usage: [1000273024/953.9mb], new bytes reserved: [0/0b], usages [model_inference=0/0b, eql_sequence=0/0b, fielddata=453457236/432.4mb, request=50063600/47.7mb, inflight_requests=526/526b]",
        "bytes_wanted" : 1000273024,
        "bytes_limit" : 996147200,
        "durability" : "PERMANENT"
      },
...
...
...
  "status" : 429
}

I'd appreciate if you could give me some insights about this issue. Is it a configuration problem? Is there any way to solve the problem and get these type of aggregations? Is Elasticsearch able to return these aggregation or it exceeds its capabilities?

Best regards,

Edit: I get this error when running the aggregation in this way, using Kibana, and also using the Python API

2

My understanding is that you're hitting the memory limit of your nodes. The way to solve it would be to increase the memory available to your nodes, though you might also look at ways to make the query smaller by using pagination on the client for example.
It depends on what the query needs to accomplish , of course!

3

Thanks for you answer. What do you mean by memory? If it is disk, I'm using just around the 50% of the available capacity. Could you please elaborate on the 'pagination on the client' option?

Ideally I want to obtain all the different terms (using terms aggregation) and the sub aggregation for these terms

4

Sorry for being unclear! By memory I mean RAM, which is needed for the node to hold the data it reads from disk in order to do the aggregation.

I understand your goal, though I highlight that you might also run into another limit which is the maximum number of buckets allowed per request. This is also a way for Elasticsearch to constrain memory usage and you can increase this limit but it will cause greater memory consumption. So if it's possible to split the work into multiple requests then that might help.

If you're doing aggregations then pagination won't help as much but you may consider things like partitions for the terms aggregation.

5

I'll check the partitions for terms aggregation section, it seems promising for my use case, specially because I'm mainly using the Python API to get the results and present them.

However, I think that RAM might be part of the problem. I'm running my ES cluster and the queries on an external server. In terms of disk we have around 300GB which leaves room to operate the cluster; however I think that RAM may be too limited. When I run the free -h -t command in the server I get the following values, which I think can be part of the problem. What do you think?

              total        used        free      shared  buff/cache   available
Mem:           2.0G        1.7G         78M        3.2M        165M         93M
Swap:          1.0G        289M        734M
Total:         3.0G        2.0G        813M
6

I'm not an expert on sizing Elasticsearch clusters but indeed that looks like an issue given that your data/queries seem large. If this is the "resting" state of the system, it means you only have less than 80 megabytes for Elasticsearch to work with when trying to answer questions.

7

Yes, I think memory is almost certainly the problem (or one of them). I was recently running some other operations and Elasticsearch stopped because of 'out of memory' issue. I'll have to check my resources.

I'll mark your previous comments as the answer to this topic since it provides the key insights.

Thanks.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.