I'm getting this error when running the aggregation below. The index being queried is only 8.44mb. I was able to run these types of queries on a much smaller cluster, in terms of RAM, with ES v6 and never see these types of aggregation errors. On ES 8.3.3 my logs are full of aggregation errors, always on [request]. Sometimes the aggregation will work, but most of the time they will not.
Obviously, I'm missing some key concept here with aggregations, but I just need to know how to reconfigure my Elastic Cloud cluster so aggregations work again without any of these types of exceptions.
In the screenshots below, you can see I'm not having JVM pressure issues and the index being queried is super tiny.
I've tried running POST _cache/clear?fielddata=true
to no avail and I've read through the docs, but I frankly don't get what has changed from ES v6.3 and ES v8.3.3 for aggregations. Running the same aggregations with half the cluster memory in ES v6.3 I never had ES crash even once due to JVM heap issues. I'm sure this is some type of configuration issue, but I've reading other circuit breaker exception posts and I cannot yet sort this riddle for my use case.
GET _cat/nodes?v=true&h=name,node*,heap*
name id node.role heap.current heap.percent heap.max
instance-0000000001 J9uc himrst 3.1gb 42 7.5gb
instance-0000000000 1bLn himrst 2.6gb 35 7.5gb
tiebreaker-0000000002 XjWb mv 322.2mb 62 512mb
{
"error": {
"root_cause": [],
"type": "search_phase_execution_exception",
"reason": "",
"phase": "fetch",
"grouped": true,
"failed_shards": [],
"caused_by": {
"type": "circuit_breaking_exception",
"reason": "[request] Data too large, data for [<reduce_aggs>] would be [11892235471/11gb], which is larger than the limit of [4831838208/4.5gb]",
"bytes_wanted": 11892235471,
"bytes_limit": 4831838208,
"durability": "TRANSIENT"
}
},
"status": 429
}
GET vanguard_products/_search
{
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"tags": "West"
}
}
],
"filter": {
"terms": {
"sku.exact": [
"2153700",
"2153702",
"7150100",
"7150500",
"7150200",
"7150300"
]
}
}
}
},
"aggs": {
"fulfillment_skus": {
"cardinality": {
"field": "sku.exact"
}
}
}
}