# How to make elastic querying faster

**URL:** <https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464>\
**Category:** Elasticsearch\
**Created:** [August 2, 2019, 7:11am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464 "2019-08-02T07:11:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![divyang](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@divyang](https://discuss.elastic.co/u/divyang)\
**Post date:** [August 2, 2019, 7:11am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/1 "2019-08-02T07:11:45Z")

</div>

Hi , I need to query a lot of data for a given elastic index . I cannot use fielddata true on the required index fields as it will increase the size of cached memory. Currently , it is taking approx 10 min to run the query for a particular application using partitions as querying all in one query gives outofMemory error .I want to reduce the time taken for querying indexes . Any suggestions ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 2, 2019, 7:30am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/2 "2019-08-02T07:30:37Z")

</div>

What is the query? What does the data look like? How many indices and shards are you querying? How much data do these hold?

---

<div class="post-metadata">

**Author:** ![divyang](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@divyang](https://discuss.elastic.co/u/divyang)\
**Post date:** [August 2, 2019, 7:38am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/3 "2019-08-02T07:38:23Z")

</div>

Query is :{  
"from": 0,  
"size": 0,  
"query": {  
"bool": {  
"filter": [  
{  
"bool": {  
"must": [  
{  
"match\_phrase": {  
"app\_id": {  
"query": "APPID"  
}  
}  
},  
{  
"range": {  
"collector\_tstamp": {  
"from": "FROMDATE",  
"to": "TODATE"  
}  
}  
}  
]  
}  
}  
]  
}  
},  
"aggregations": {  
"page\_urlpath": {  
"terms": {  
"field": "page\_urlpath.keyword",  
"size": 2147483647,  
"include": {  
"partition": "PARTITION\_NUMBER",  
"num\_partitions": "TOTAL\_PARTITIONS"  
}  
},  
"aggregations": {  
"visitors": {  
"cardinality": {  
"field": "domain\_userid.keyword",  
"precision\_threshold": 40000  
}  
},  
"visits": {  
"cardinality": {  
"field": "domain\_sessionid.keyword",  
"precision\_threshold": 40000  
}  
},  
"number\_of\_events": {  
"value\_count": {  
"field": "\_index"  
}  
}  
}  
}  
}  
}

it queries for the entire month together . The month has a doc count of 6392551 records . It has 3800 different buckets for field page\_urlpath . 2 nodes per node 1013 shards .

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 2, 2019, 7:57am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/4 "2019-08-02T07:57:11Z")

</div>

> [@divyang](#):
>
> "size": 2147483647,

You should never set the size parameter to unnecessarily large values as it will use a lot of heap. See [this old blog](https://www.elastic.co/blog/found-crash-elasticsearch) post for a discussion on this.

---

<div class="post-metadata">

**Author:** ![divyang](https://avatars.discourse-cdn.com/v4/letter/d/7ea924/32.png) [@divyang](https://discuss.elastic.co/u/divyang)\
**Post date:** [August 2, 2019, 10:30am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/5 "2019-08-02T10:30:45Z")

</div>

Well i changed that , definitely a good link to read , but perfomance still remains the same , will creating number of threads affect the heap ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 3, 2019, 5:30am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/6 "2019-08-03T05:30:54Z")

</div>

> [@divyang](#):
>
> The month has a doc count of 6392551 records . It has 3800 different buckets for field page\_urlpath . 2 nodes per node 1013 shards .

It sounds like you have far, far to many shards given the amount of data you have. Please read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and then try to dramatically reduce the number of shards in the cluster. I would expect having to query only a few shards to give much better performance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 31, 2019, 5:30am UTC](https://discuss.elastic.co/t/how-to-make-elastic-querying-faster/193464/7 "2019-08-31T05:30:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.