Queries latency increased after upgrade to es7

[Eylon_Koren1 (Eylon Koren)]

Hi!

We upgraded our ES cluster from version 6.8 to 7.17.3, afterwards the query latency was
increased (10 - 30 ms) for all different kind of queries.
We expected for reducing the query latency after upgrade, because of version improvement and indexes shrinking, but the opposite had happened.

comparing the old 6.8 and new 7.17 clusters:

• Both cluster run on AWS EC2 instances, with machine type "m5d.4xlarge".
• machine is with 62GB RAM, 25 GB ram used by JVM.
• All indexes, queries are the same between the versions (slightly changes in query syntax, nothing significant).
  *Both running on openJDK 11 (JAVA version 11).
• One of my suspect is setting in the jvm : -XX:InitiatingHeapOccupancyPercent=20 which seems kind low (default is 30). It looks that gc time increased after upgrade (at image - blue is version 7.17 and orange is 6.8). Can it cause such increase in query latencty?
  
  
  
  G1 gc is being used.
• No special errors in es logs.
• New es7 cluster is running on ubuntu 18.04.

Any ideas why the query latency was increased ??

Love for help,
Eylon

[Eylon_Koren1 (Eylon Koren)]

In addition, I compare both elasticsearch settings, the only changes are the following fields:
added to settings:

• indices.breaker.total.use_real_memory
• searchguard.ssl.transport.enabled
• xpack.monitoring.collection.enabled
  removed from settings:
  *http.enabled

I searched them and they not seems to have impact on the search latency.