# Computation expensive queries

**URL:** <https://discuss.elastic.co/t/computation-expensive-queries/89880>\
**Category:** Elasticsearch\
**Created:** [June 19, 2017, 5:18am UTC](https://discuss.elastic.co/t/computation-expensive-queries/89880 "2017-06-19T05:18:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Allie\_Yang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/allie_yang/32/16538_2.png) [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\
**Post date:** [June 19, 2017, 5:18am UTC](https://discuss.elastic.co/t/computation-expensive-queries/89880/1 "2017-06-19T05:18:42Z")

</div>

I need to run expensive queries on couple of days' of index, then ES server gives me below error:

### Data too large, data for [reused\_arrays] would be larger than limit of [2566520832/2.3gb]

### "bytes\_wanted": 2608572536,

### "bytes\_limit": 2566520832

## 

Any general suggestion for overcoming this? Shall I increase the shards or use instances with more computing power? thanks!

### my query:

("cardinality" is very expensive):  
POST /index\_name/\_search?size=0  
{  
"aggs": {  
"search\_terms": {  
"terms": {  
"field": "search\_term.keyword",  
"size": 10,  
"order": {  
"unique\_session": "desc"  
}  
},  
"aggs": {  
"unique\_session": {  
**"cardinality": {**  
"field": "session\_id.keyword"  
}  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 19, 2017, 5:30am UTC](https://discuss.elastic.co/t/computation-expensive-queries/89880/2 "2017-06-19T05:30:39Z")

</div>

Add more nodes/resources to alleviate this.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [June 19, 2017, 7:54am UTC](https://discuss.elastic.co/t/computation-expensive-queries/89880/3 "2017-06-19T07:54:15Z")

</div>

A couple of other approaches to consider:

A. Reduce accuracy or  
B. Break into multiple requests

For A consider lowering the `precision_threshold` in the cardinality agg [1]. The default value is 3,000 meaning each unique search term will count up to 3,000 session IDs each which is largely at the root of your memory problems given the number of unique search terms there are likely to be.

For option B we have a couple of ways of doing this. The first is to simply issue a query for the top 1,000 `search_term.keyword` values. This should give you an approximation for the search terms used in sessions. There will be false positives (search terms caused by outlier sessions repeating the same search) but no false negatives. Take this list of search terms and use them as a `terms` query in your existing example agg request.  
Another option in 5.4 is to run multiple requests like your existing one but focusing each request on a subset of the search terms in your index. This can be done using the partitioning function [2] on the `include` clause in your `terms` query.

[1] [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html#\_precision\_control](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html#_precision_control)  
[2] [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#\_filtering\_values\_with\_partitions](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values_with_partitions)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2017, 7:54am UTC](https://discuss.elastic.co/t/computation-expensive-queries/89880/4 "2017-07-17T07:54:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.