RestHighLevelClient slow performance

[eugene_p (Eugene)]

Hi everyone.
I noticed that requests executed through RestHighLevelClient slower 6 times than the same requests executed through curl or Postman.

So I configure the RestHighLevelClient:

@Bean
fun restHighLevelClient(): RestHighLevelClient {
    val trustStrategy = TrustSelfSignedStrategy()

    val sslContext = SSLContexts.custom()
            .loadTrustMaterial(File(trustStore), trustStorePassword.toCharArray(), trustStrategy)
            .loadKeyMaterial(File(keyStore), keyStorePassword.toCharArray(), keyStorePassword.toCharArray())
            .build()

    val credentialsProvider = BasicCredentialsProvider()
            .apply { setCredentials(AuthScope.ANY, UsernamePasswordCredentials(username, password)) }

    return RestHighLevelClient(RestClient.builder(
            *getHosts(hosts)
    ).setHttpClientConfigCallback {
        it
                .setMaxConnPerRoute(maxConnPerRoute)
                .setMaxConnTotal(maxConnTotal)
                .setDefaultCredentialsProvider(credentialsProvider)
                .setSSLContext(sslContext)
    }
    )
}

In VisualVM I can see that search request takes 401ms.

But if I copy request from logs and call it in Postman it takes 65ms.

It's a big difference. How can I improve performance in my code?

[eugene_p (Eugene)]

Wrote my own implementation for search:

private fun searchRequest(index: Index,
                          indexType: IndexType,
                          source: SearchSourceBuilder): SearchResponse {
    val searchRequest = SearchRequest(index.index)
            .types(indexType.type)
            .source(source)

    val request = Converter.createSearchRequest(searchRequest).entity.content
    val writer = StringWriter()
    IOUtils.copy(request, StringWriter(), StandardCharsets.UTF_8.name())
    val httpEntity = HttpEntity(writer.toString(), getHeders())

    val postForEntity = restTemplateElastic.postForEntity(getHost(), httpEntity, String::class.java)

    val httpEntityResponse = BasicHttpEntity()
    httpEntityResponse.content = IOUtils.toInputStream(postForEntity.body, StandardCharsets.UTF_8.name())
    return Converter.createSearchResponse(httpEntityResponse, restHighLevelClient)
}

It's quicker than the standard implementation

My implementation:

Lifting the server siege...
Transactions: 25226 hits
Availability: 100.00 %
Elapsed time: 61.06 secs
Data transferred: 314.29 MB
Response time: 0.24 secs
Transaction rate: 413.13 trans/sec
Throughput: 5.15 MB/sec
Concurrency: 99.83
Successful transactions: 25226
Failed transactions: 0
Longest transaction: 8.00
Shortest transaction: 0.06

RestHighLevelClient.search

Lifting the server siege...
Transactions: 1886 hits
Availability: 100.00 %
Elapsed time: 60.78 secs
Data transferred: 23.49 MB
Response time: 3.15 secs
Transaction rate: 31.03 trans/sec
Throughput: 0.39 MB/sec
Concurrency: 97.60
Successful transactions: 1886
Failed transactions: 0
Longest transaction: 6.77
Shortest transaction: 0.80

[dadoonet (David Pilato)]

That's great. Could you share your findings in an elasticsearch issue so the client team might have a chance to look at this.

Cc @javanna aware of anything like this?

[dadoonet (David Pilato)]

Ha. Thanks for opening the issue in github:

Issue: RestHighLevelClient low performance for search

I didn't know initially that you were using an "old" elasticsearch version of the rest client. Any chance you could run the same tests with 6.8.0?

[eugene_p (Eugene)]

Reproduced on 6.8.0

[dadoonet (David Pilato)]

Great. Thanks!