Slowness after upgrading ElasticSearch 6.5 to 7.10

[Thomas_Kang]

Hi folks,

I'm experiencing slowness after upgrading Elasticsearch from 6.5.4 to 7.10.2.

I can reproduce it in my local (using the official docker images) and also in AWS' managed clusters.

Here are the index mappings in v6 and v7. Due to the removal of the mapping type in v7, they look a bit different. However, the list of fields and types are the same.

V6 mapping:

http://jsonblob.com/1120601989569593344

V7 mapping:

http://jsonblob.com/1120602233610977280

Here is the sample query (sorry for the long query). Pls note that there is a nested field, and there are a couple of bucket aggregations in the query.

Search request:

http://jsonblob.com/1120602638478753792

Finally, here are the search results in v6 and v7. Only the top of the response body is different as highlighted.

V7 is about twice slower than V6.

V6 result:

http://jsonblob.com/1120602886773161984

V7 result:

http://jsonblob.com/1120603043161980928

I have checked the v7 release note and this forum desperately. There were some posts about slowness after an upgrade, but I couldn't find any relevant posts to me. Thank you so much for having a look. If you need any other stuff, pls let me know. Cheers!

Regards,
Thomas

[system]

7.10.2 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[dadoonet (David Pilato)]

Welcome!

I'd recommend to use 7.17.10 which is the latest of the 7.x branch so you will get all the security fixes and the improvements.

If you still see the same problem after some runs, please ask again here with your new findings.

Also have a look at Aggregations | Elasticsearch Guide [8.8] | Elastic

[Thomas_Kang]

@Thomas_Kang

http://jsonblob.com/1120602886773161984

Thanks @dadoonet. Yes, I can reproduce it even with 7.17.10. Here's the result after I ran the attached search request with k6 & 5 iterations per version.

ElasticSearch 6.5.4....: avg=2356.2  min=2100 med=2128    max=3201   p(90)=2820.6  p(95)=3010.8
ElasticSearch 7.10.2...: avg=4303.6  min=4245 med=4292    max=4365   p(90)=4362.2  p(95)=4363.6

I also used the below request params to disable the request-level cache.

POST workflow_4829719_node_7/_search?request_cache=false&preference=_local

[dadoonet (David Pilato)]

Elasticsearch 7.10.2...: avg=4303.6 min=4245 med=4292 max=4365 p(90)=4362.2 p(95)=4363.6

This is not 7.17.

Could you share the results with 7.17?
Could you run it something like 10/20 times?

I also used the below request params to disable the request-level cache.

Why? Are you disabling the cache for your end users?

[Thomas_Kang]

It was my mistake on k6 metric labelling. The test was run with 7.17. This is the response when I send a GET request to the root path of my local ES server.

{
  "name" : "local_dev_v7",
  "cluster_name" : "local_cluster",
  "cluster_uuid" : "1mGfFFUoSgyoYw77A4X-sA",
  "version" : {
    "number" : "7.17.10",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "fecd68e3150eda0c307ab9a9d7557f5d5fd71349",
    "build_date" : "2023-04-23T05:33:18.138275597Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I disabled the cache because we're less likely to run the same aggregation query in production. Our ES queries are dynamically created with different fields and values based on the user's filter selection. Regardless, I'm attaching the results w and w/o cache with 20 iterations.

Without cache (20 iterations),

Local - ElasticSearch 6.5.4....: avg=2104.75  min=2070  med=2097    max=2168   p(90)=2142.2  p(95)=2162.3
Local - ElasticSearch 7.17.10...: avg=4305.3   min=4227  med=4317.5  max=4375   p(90)=4362.8  p(95)=4370.25

With cache (20 iterations)

Local - ElasticSearch 6.5.4....: avg=142.6    min=1      med=1      max=2830   p(90)=2       p(95)=143.4
Local - ElasticSearch 7.17.10...: avg=231.35   min=2      med=3      max=4545   p(90)=10.3     p(95)=248.15