ES Java Client 9.2.4 -> 9.4.4 performance regression. HttpComponents 5.4.4 -> 5.6 to blame?

Elastic Stack Elasticsearch
1

Hey friends :waving_hand:

We just upgraded all our ES clusters from 9.2.4 -> 9.4.4. Server upgrade was buttery smooth, as always! :flexed_biceps:

However, upgrading the ES Java client from 9.2.4 to 9.4.4 we noticed a performance regression - up to 50ms added to an msearch request.

This is our typeahead service - here you can see our p95 jumping by around 30ms, our p95 jumped by around 50ms until we revert:

image
image1264×321 32 KB

We saw same issue in our main search ES cluster with upgraded client too...

We though at first this might be related to the change to GZIP handling with the upgrade to Apache HTTP Client 5.6, but after putting on TRACE logging, that doesn't appear to be the case - gzip headers / compression was working in both directions as expected in both 9.2.4 and 9.4.4

We were able to remove this performance regression by keeping ES Client at 9.4.4 but pinning the transitive dependency on HTTPClient 5 in Gradle with force("org.apache.httpcomponents.client5:httpclient5:5.4.4")

For comparison, here's the 9.4.4 client deployed with Apache HTTP Client 5 pinned to 5.4.4 - no real difference in performance :upside_down_face: :

image
image1264×321 26.6 KB

Here's a couple of OTEL traces showing the performance regression is defo in the msearch client call:

9.2.4 client - ~3ms in the span over msearch:

image
image2455×377 78.4 KB

9.4.4 client - ~62ms in the span over msearch:

image
image2455×377 78.6 KB

We think that there may be some further changes required in the ES Java client to handle the dependency upgrade on HTTP Client 5.4.4 -> 5.6.1 other than just the change to GZIP handling?

I'll open an issue on the client GitHub too.
EDIT: Github issue

Thanks! :person_bowing:

2

Thanks for creating the GitHub issue :folded_hands: we'll keep the discussion there and we'll update this thread when the solution/patch is available.