Queries latency increased after upgrade to es7

Elastic Stack Elasticsearch
1

Hi!

We upgraded our ES cluster from version 6.8 to 7.17.3, afterwards the query latency was
increased (10 - 30 ms) for all different kind of queries.
We expected for reducing the query latency after upgrade, because of version improvement and indexes shrinking, but the opposite had happened.

comparing the old 6.8 and new 7.17 clusters:

  • Both cluster run on AWS EC2 instances, with machine type "m5d.4xlarge".
  • machine is with 62GB RAM, 25 GB ram used by JVM.
  • All indexes, queries are the same between the versions (slightly changes in query syntax, nothing significant).
    *Both running on openJDK 11 (JAVA version 11).
  • One of my suspect is setting in the jvm : -XX:InitiatingHeapOccupancyPercent=20 which seems kind low (default is 30). It looks that gc time increased after upgrade (at image - blue is version 7.17 and orange is 6.8). Can it cause such increase in query latencty?
    image
    image1610×544 33.9 KB

    G1 gc is being used.
  • No special errors in es logs.
  • New es7 cluster is running on ubuntu 18.04.

Any ideas why the query latency was increased ??

Love for help,
Eylon

2

In addition, I compare both elasticsearch settings, the only changes are the following fields:
added to settings:

  • indices.breaker.total.use_real_memory
  • searchguard.ssl.transport.enabled
  • xpack.monitoring.collection.enabled
    removed from settings:
    *http.enabled

I searched them and they not seems to have impact on the search latency.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.