We are dealing in querying large amount of data from our elastic search server. However, in random scenarios the connection created by the High Level Rest client is not getting released and it's blocking our code flow.
Rest client opens connection to get the data but doesn't return. Although we have specified the timeouts as well as shown below, but the rest client doesn't timeout as well.
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(httpHost).setRequestConfigCallback( requestConfigBuilder -> requestConfigBuilder .setConnectTimeout(30000) .setConnectionRequestTimeout(90000) .setSocketTimeout(90000)).setMaxRetryTimeoutMillis(90000));
We have checked on Elastic search server as well, the query gets finished but client still doesn't release the connection. Also, the load on elastic search and server is not high at all when we trigger these queries.
Queries are also not that complex. It consists of only filtering and single level of aggregation.
The thread, in which this query is triggered, has been observed to be blocked for over 10 hours. Expected time is just a few minutes.
However when we retry the query later, it goes through fine. So it's happening randomly.
Elasticsearch version: 6.5.4
Rest client version: 6.5.4
allow_partial_search_results : false (both on client side and server side cluster level)
Can you please suggest what are we missing ?
Related question: Rest High Level Client : Request timeout is not working