# Increase in response time for aggregations in 7.10.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [June 17, 2021, 5:09pm UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295 "2021-06-17T17:09:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rgopalan](https://avatars.discourse-cdn.com/v4/letter/r/dfb087/32.png) [@rgopalan](https://discuss.elastic.co/u/rgopalan)\ **Post date:** [June 17, 2021, 5:09pm UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/1 "2021-06-17T17:09:20Z")
Hi, We recently upgraded our ElasticSearch cluster from 7.7 to 7.10.1 and we are seeing an increase in the response time. One query that uses aggregations has seen the response time increase by 1-1.5s. **Setup** \* 3 Master nodes + 3 data nodes \* 2 primaries + 1 replica shards \* Master nodes: 16G \* Data nodes: 31G **Index** - It is an index with 962994 documents - Each document contains around 200 fields. - No stored fields in the mapping **Expected query output** - We are trying to group objects of two different types; courses and classes which have a 1:n relation. - There are 20k courses and 334k classes. **Changes** - There are no changes in data or the query. - Confirmed that the only variable is ElasticSearch version. **Observations** - Queries using aggregations are the ones that have seen an increase in response time. - We have seen this increase in response time with ES 7.10.1 and ES 7.10.2 - ES 7.9.3 is showing times comparable to ES 7.7. - Seems to be related to the Lucene upgrade as mentioned in [Query performance regression after upgrade to 7.10.x with huge values for "size" field · Issue #67574 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/67574). Tried the resolution mentioned in the issue, but it had no effect. **Query tuning** - Global Ordinals - Use eager\_global\_ordinals - Use execution hint "map" - Stored fields stored\_fields : _none_ - index.max\_docvalue\_fields decrease to 10 None of these seem to be having any effect. **Query** It is a bit long. Hence shared via Google Drive. > **[query.json](https://drive.google.com/file/d/1NbUoyMJA5VeZQTCttmY8mXdxZSm9-_Qm/view?usp=sharing)** > > Google Drive file. **Profile output** It is a bit long. Hence shared via Google Drive. > **[profile-query-output-7.10.1.json](https://drive.google.com/file/d/1pPtV5CIgOWRsJs3aAzF1UtGdn2yOfsNb/view?usp=sharing)** > > Google Drive file. I can see most of the time is taken in TopHitsAggregator. Not sure why the time has increased from ES 7.7 to ES 7.10.1. **References** - [Improving the performance of high-cardinality terms aggregations in Elasticsearch | Elastic Blog](https://www.elastic.co/blog/improving-the-performance-of-high-cardinality-terms-aggregations-in-elasticsearch) - [eager\_global\_ordinals | Elasticsearch Guide [master] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/eager-global-ordinals.html) What am I missing here ? Is there any setting that needs to be changed ? Thanks in advance. -Ravi ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 21, 2021, 9:08am UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/2 "2021-06-21T09:08:28Z")
Hey, indeed there have been some changes in aggregations. Are you able to upgrade to the latest 7.13. and check if your aggregation speed has been restored? If the problem still persists, that sounds like a book to investigate. Also, taking a look at the [search profiler](https://www.elastic.co/guide/en/kibana/7.13/xpack-profiler.html) might help. --Alex ---
**Author:** ![rgopalan](https://avatars.discourse-cdn.com/v4/letter/r/dfb087/32.png) [@rgopalan](https://discuss.elastic.co/u/rgopalan)\ **Post date:** [June 23, 2021, 2:27pm UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/3 "2021-06-23T14:27:12Z")
Hi Alex, I tried the upgrade to 7.13.0 and still see the same issue with increased response time. I narrowed the degradation down to a commit in which the codecs were changed from Lucene86Code to Lucene87Codec. If I revert the codec (build ES with codec reverted), I see performance on par with ES 7.7.1/ES 7.9.3. Any suggestions on why changes in compression would lead to a degradation ? Thanks. -RaviShekhar G ---
**Author:** ![Jockj](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@Jockj](https://discuss.elastic.co/u/Jockj)\ **Post date:** [June 30, 2021, 4:31am UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/4 "2021-06-30T04:31:40Z")
I experience the same issue with 7.10.2, the response time is 10x compare with 7.7.1 on the same hardware. ---
**Author:** ![Jockj](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@Jockj](https://discuss.elastic.co/u/Jockj)\ **Post date:** [July 14, 2021, 9:42pm UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/5 "2021-07-14T21:42:15Z")
@rgopalan , would you mind share what you changed to make it work? I have the same issue, and when I try it, and I got the following error "failed because of NotSerializableExceptionWrapper[unsupported\_operation\_exception: Old codecs may only be used for reading]" ---
**Author:** ![rgopalan](https://avatars.discourse-cdn.com/v4/letter/r/dfb087/32.png) [@rgopalan](https://discuss.elastic.co/u/rgopalan)\ **Post date:** [July 15, 2021, 4:47pm UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/6 "2021-07-15T16:47:15Z")
Hi Jock, We built a custom ES version reverting the codec from Lucene87 to Lucene86. Ultimately, we had to pause the upgrade because we did not want to maintain a custom version of ES. We are investigating 7.13 where Lucene has been upgraded again and the compression is made configurable as per [https://issues.apache.org/jira/browse/LUCENE-9378](https://issues.apache.org/jira/browse/LUCENE-9378) **Changes to revert the codec** /server/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java - commented out checkLucene() /server/src/main/java/org/elasticsearch/common/lucene/Lucene.java /server/src/main/java/org/elasticsearch/index/codec/CodecService.java /server/src/main/java/org/elasticsearch/index/codec/PerFieldMappingPostingFormatCodec.java /server/src/main/java/org/elasticsearch/Version.java ---
**Author:** ![Jockj](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@Jockj](https://discuss.elastic.co/u/Jockj)\ **Post date:** [July 19, 2021, 12:54am UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/7 "2021-07-19T00:54:56Z")
Thanks a lot @rgopalan for sharing the detail information. It seems the issue is there since elasticsearch 7.7(Lucene 85), and may fix in elasticsearch 7.13(Lucene 88). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2021, 12:55am UTC](https://discuss.elastic.co/t/increase-in-response-time-for-aggregations-in-7-10-1/276295/8 "2021-08-16T00:55:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.