# Elasticsearch performance degrades after upgrading from 6.7 to 7.10 **URL:** \ **Category:** Elasticsearch\ **Tags:** migration\ **Created:** [July 18, 2023, 9:20am UTC](https://discuss.elastic.co/t/elasticsearch-performance-degrades-after-upgrading-from-6-7-to-7-10/338640 "2023-07-18T09:20:21Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![teanoon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/teanoon/32/123577_2.png) [@teanoon](https://discuss.elastic.co/u/teanoon)\ **Post date:** [July 18, 2023, 9:20am UTC](https://discuss.elastic.co/t/elasticsearch-performance-degrades-after-upgrading-from-6-7-to-7-10/338640/1 "2023-07-18T09:20:21Z")
We are trying to migrate the elasticsearch from 6.7 to 7.10. The indices are reindexed in the new elasticsearch cluster. And did some comparisons. ### simple load test is slow A simple load test indicates that the same set of queries uses 91ms per request in 6 while it takes 112ms in 7. An example query looks like this: [request.json ยท GitHub](https://gist.github.com/teanoon/be17fc76a99460c6ec669222c76dbe60) ### profile shows slower parts The profile detail shows a lot of queries take 100% longer in 7 than it in 6. ```auto { "type": "PointInSetQuery", "description": "brandIds:{59 (...omitted 100 brands) 31389}", "time_6_baseline": "1.2531 ms", "time_7_baseline": "4.4807 ms" } ``` Breakdowns with a same query have huge differences. ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3b5a526e5f59b243108d10f22f3fa4ec53815fae.png) The left one comes from ES6 and the right one is ES7. ### page cache is used We know that the elasticsearch 7 has a off-heap changes and [it moves terms index out of the heap](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/breaking-changes-7.0.html#_off_heap_terms_index). so we checked the page cache: `sudo lsof +D /mnt/elasticsearch/data/nodes/0/indices/wu47nPk0TEuVBzLo3WEOsQ` and it looks like this: ```auto COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 30779 elasticsearch mem REG 259,0 108409303 15728663 /mnt/elasticsearch/data/nodes/0/indices/Dznr2xCpSS6VDJV0qLDkqQ/7/index/_5.cfs java.........................................................................cfs java.........................................................................doc java.........................................................................dvd java.........................................................................kdd java.........................................................................kdi java.........................................................................nvd java.........................................................................tim java.........................................................................tip(these files are loaded in mem) java 30779 elasticsearch mem REG 259,0 71445 15729187 /mnt/elasticsearch/data/nodes/0/indices/Dznr2xCpSS6VDJV0qLDkqQ/4/index/_9_Lucene84_0.tip java 30779 elasticsearch 373r REG 259,0 33969522 15728698 /mnt/elasticsearch/data/nodes/0/indices/Dznr2xCpSS6VDJV0qLDkqQ/1/index/_d_Lucene84_0.pos java.........................................................................ckp java.........................................................................fdt java.........................................................................fdx java.........................................................................lock java.........................................................................pos java.........................................................................tlog(these files are not loaded in mem) java 30779 elasticsearch 731w REG 259,0 88 15728658 /mnt/elasticsearch/data/nodes/0/indices/Dznr2xCpSS6VDJV0qLDkqQ/7/translog/translog.ckp ``` we tried with different index store settings([change the store type to mmapfs](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-modules-store.html), [preload custom files](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/preload-data-to-file-system-cache.html)) and they load more files into the mem like elasticsearch 6 but the overall performance is still not as good as 6. ### query optimization Query optimization should improve the performance but it doesn't cover the gaps between 6 and 7. We tried using `filter` to replace `must` and have some improvement but it doesn't fill the gap between 6 and 7. ### index and cluster brief The testing index and cluster are: - 1 index with 2.6 million products and 9 shards 1 replica. 5.8gb(11.6 gb including replica) - we tried different Elasticsearch heap size: 8G and 15G - The cluster has 2 nodes. Each node is a i3.xlarge(4cpu 30.5GB SSD) ### why these 2 versions Because we use `amazon/opendistro-for-elasticsearch` which are 0.9.0 and 1.3.2 and we plan to move to 8 without using that image. ### question So the question is what causes the performance issue? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2023, 9:20am UTC](https://discuss.elastic.co/t/elasticsearch-performance-degrades-after-upgrading-from-6-7-to-7-10/338640/2 "2023-07-18T09:20:22Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [July 18, 2023, 9:28am UTC](https://discuss.elastic.co/t/elasticsearch-performance-degrades-after-upgrading-from-6-7-to-7-10/338640/3 "2023-07-18T09:28:59Z")
Hi @teanoon, It sounds like you are using very old versions that are end of life, so I would recommend upgrading to a supported version. To confirm are you using [amazon/opendistro-for-elasticsearch](https://hub.docker.com/r/amazon/opendistro-for-elasticsearch) or the [official elasticsearch images](https://hub.docker.com/_/elasticsearch)? OpenDistro is an AWS product and from the link I provided out of support. But you might be able to get some help with it on the [OpenSearch forum](https://forum.opensearch.org/) which is the replacement. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2023, 9:29am UTC](https://discuss.elastic.co/t/elasticsearch-performance-degrades-after-upgrading-from-6-7-to-7-10/338640/4 "2023-08-15T09:29:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.