Experiencing high latency post upgrading to elastic 8.10 and 8.12 from 8.7

8.7 Cluster (32 CPU 64 GB RM) 20 data nodes part of this cluster

here the we see p99 latency is around 15 to 20 ms.

after upgrading to 8.10 and then to 8.12, we are seeing high query cache evictions and latency started increasing to 80 to 90 ms now.

sample search query : _search?typed_keys=true&request_cache=false {"_source":{"excludes":,"includes":},"from":0,"query":{"bool":{"filter":[{"terms":{"vendorPoNos.keyword":["XXXXXXXXXX"]}}]}},"size":1,"sort":[{"created":{"order":"desc"}}],"track_total_hits":false}

8.12 graph

are we missing out any parameters that needs to be set to optimise the performance ?