# Concerned about high query times

**URL:** https://discuss.elastic.co/t/concerned-about-high-query-times/54526
**Category:** Elasticsearch
**Created:** [July 1, 2016, 9:38am UTC](https://discuss.elastic.co/t/concerned-about-high-query-times/54526 "2016-07-01T09:38:02Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Emrah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emrah/32/17657_2.png) [@Emrah](https://discuss.elastic.co/u/Emrah)
#### Post date: [July 1, 2016, 9:38am UTC](https://discuss.elastic.co/t/concerned-about-high-query-times/54526/1 "2016-07-01T09:38:02Z")

</div>

Hi there,

We recently decided use Elasticsearch for our website. Rewrote the website, including mobile apps in a short time. So the time we publish the website and apps , something went wrong in our elastic nodes. We have nearly 250ms query time and we are concerned about it. By the way, could you help us to find the root cause and optimize queries and any suggestion to use elasticsearch better. We have currently 4000 concurrent users, and about 2700 pageviews.

**So here are configuration to give you more info about what we have ;**  
4 nodes  
Each node have 32 GB RAM  
Xeon 2.10 ghz proccessors

**Query times ;**

 ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0d660f1c82f7cf9596867384c9e450ea6e44c63f.png)

**Last snaphsot of hot threads ;**  
{node-2}  
Hot threads at 2016-07-01T09:26:08.643Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

31.3% (156.2ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][search][T#11]'  
2/10 snapshots sharing following 21 elements  
org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)  
org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:72)  
org.elasticsearch.search.aggregations.bucket.global.GlobalAggregator$1.collect(GlobalAggregator.java:54)  
org.elasticsearch.search.aggregations.LeafBucketCollector$3.collect(LeafBucketCollector.java:73)  
org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:88)

**131.3% (656.2ms out of 500ms) cpu usage by thread 'elasticsearch[node-4][search][T#10]'**  
2/10 snapshots sharing following 19 elements  
org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(InternalTerms.java:212)  
org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:153)  
org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:170)  
org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.reduce(InternalTerms.java:110)  
org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(InternalTerms.java:220)  
org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:153)  
org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:170)

**125.0% (625ms out of 500ms) cpu usage by thread 'elasticsearch[node-4][search][T#19]'**  
5/10 snapshots sharing following 26 elements  
org.elasticsearch.search.aggregations.AggregatorFactory$1$1.collect(AggregatorFactory.java:208) org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectExistingBucket(BucketsAggregator.java:80)  
org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:72)  
org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator$1.collect(NestedAggregator.java:112)  
org.elasticsearch.search.aggregations.AggregatorFactory$1$1.collect(AggregatorFactory.java:208)

**37.5% (187.5ms out of 500ms) cpu usage by thread 'elasticsearch[node-3][search][T#6]'**  
2/10 snapshots sharing following 42 elements  
java.lang.Thread.isAlive(Native Method)  
org.apache.lucene.util.CloseableThreadLocal.purge(CloseableThreadLocal.java:115)  
org.apache.lucene.util.CloseableThreadLocal.maybePurge(CloseableThreadLocal.java:105)  
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:88)  
org.apache.lucene.index.CodecReader.getSortedSetDocValues(CodecReader.java:243)  
org.apache.lucene.index.FilterLeafReader.getSortedSetDocValues(FilterLeafReader.java:454)  
org.apache.lucene.index.DocValues.getSortedSet(DocValues.java:302)

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [July 4, 2016, 7:58am UTC](https://discuss.elastic.co/t/concerned-about-high-query-times/54526/2 "2016-07-04T07:58:58Z")

</div>

Hey,

the hot threads output shows a lot of CPU spent in aggs (or operations needed for aggregations). If you dont run any aggregations, is your query fast? You can then try to add the aggregations (each by each) back and see which one is a potential performance culprit. Maybe there is a single aggregation that creates a lot of buckets or needs to parse all of your data in your dataset (and you can change your data model execute that aggregation in a more efficient manner). Maybe it is all of your aggregations together taking a lot of time. Dissecting this a first step might help.

--Alex

---

<div class="post-metadata">

### Author: ![Emrah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emrah/32/17657_2.png) [@Emrah](https://discuss.elastic.co/u/Emrah)
#### Post date: [August 24, 2016, 2:47pm UTC](https://discuss.elastic.co/t/concerned-about-high-query-times/54526/3 "2016-08-24T14:47:01Z")

</div>

Yes, it was exactly what you said. We worked on aggregations which has some sub aggregation, also tried to reduce aggregation count.

First we get these query times.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/5/537dad6e6bbec60422858a8af9b7e5300f89d98a.jpg)

And then we decided to remove **global aggregations** which really affect query times, however we experienced that. Here now we have

 ![](https://us1.discourse-cdn.com/elastic/original/2X/6/65bb991fde1e1459f6a34192988b42363b599f1d.png)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:25pm UTC](https://discuss.elastic.co/t/concerned-about-high-query-times/54526/4 "2017-07-05T22:25:31Z")

</div>


