RestHighLevelClient SocketTimeoutException

[zharenkov (Danila)]

I've got ES cluster on Azure, and communicate with it from Spring Boot app using elasticsearch RestHighLevelClient.

The bean in spring is the follow

@Bean
public RestHighLevelClient elasticsearchRestClient() {
    CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
    return new RestHighLevelClient(RestClient.builder(new HttpHost(ipAddress, port))
            .setRequestConfigCallback(builder -> builder
                    .setConnectTimeout(10000)
                    .setSocketTimeout(90000)
                    .setConnectionRequestTimeout(0)
            )
            .setHttpClientConfigCallback(httpClientBuilder ->
                    httpClientBuilder
                            .setDefaultCredentialsProvider(credentialsProvider)));
}

Here ipAddress is loadbalancer's ip on Azure ES cluster, port is 9200. The problem is - if ES idle for few minutes - next request to ES fails with SocketTimeoutException.

2019-09-26T07:20:50.228101050Z: [INFO] java.net.SocketTimeoutException: 90,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE] 2019-09-26T07:20:50.228104650Z: [INFO] at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:773) ~[elasticsearch-rest-client-7.3.2.jar!/:7.3.2] 2019-09-26T07:20:50.228108350Z: [INFO] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[elasticsearch-rest-client-7.3.2.jar!/:7.3.2] 2019-09-26T07:20:50.228111750Z: [INFO] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205) ~[elasticsearch-rest-client-7.3.2.jar!/:7.3.2] 2019-09-26T07:20:50.228115350Z: [INFO] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.3.2.jar!/:7.3.2] 2019-09-26T07:20:50.228118850Z: [INFO] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1439) ~[elasticsearch-rest-high-level-client-7.3.2.jar!/:7.3.2] 2019-09-26T07:20:50.228122150Z: [INFO] at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:784) ~[elasticsearch-rest-high-level-client-7.3.2.jar!/:7.3.2]

After this fail, retry attemp finishes succesfully.

I've tried to add sniffer on LowLevelRestClient - but no effect.

Also, i tried to add some kind of polling ES every minute like this

    @Scheduled(cron = "0 0/1 * * * *")
public void checkHealth() throws IOException {
    LOGGER.info(restHighLevelClient.info(RequestOptions.DEFAULT));
}

And in this case - every request works fine. But I believe it's not the best option.

What can be the possible trouble? Thanks!