Elasticsearch random ConnectionnTimeout error

Hi all,
I'm using Amazon Elasticsearch with RestHighLevelClient for our team application and have noticed occasional small dips in our success rate from canary traffic calling our API due to the error below.

    Caused by: java.net.ConnectException: Timeout connecting to [search-mydomain-htb7yrdp7xvt2k3apw5phcz7ku.us-east-1.es.amazonaws.com/52.72.66.108:443]
        	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:628) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:894) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:184) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:214) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
        	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194) ~[httpasyncclient-4.1.x.jar:?]
        	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.x.jar:?]
        	... 1 more

This is my configuration for the RestHighLevelClient.

       @Provides
       @Singleton
       public RestHighLevelClient providesHttpClient(@Named("CurrentRegion") String currentRegion,
                                                                                               @Named("DomainEndpoint") String endpoint) {
                    AWSCredentialsProvider awsCredentialsProvider = new DefaultAWSCredentialsProviderChain();

                    AWS4Signer awsSigV4Signer = new AWS4Signer();
                    awsSigV4Signer.setRegionName(currentRegion);
                    awsSigV4Signer.setServiceName(AES_SERVICE_NAME);

                    HttpRequestInterceptor requestInterceptor = new AWSRequestSigningApacheInterceptor(AES_SERVICE_NAME,
                            awsSigV4Signer, awsCredentialsProvider);

                    return new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint))
                        .setRequestConfigCallback(rcb -> rcb.setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(requestInterceptor)));
                }

I have tried setting the RequestConfigCallback below but the problem persisted

    return new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint))
                .setRequestConfigCallback(rcb -> rcb.setSocketTimeout(60000).setConnectTimeout(30000).setConnectionRequestTimeout(0))
                .setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(requestInterceptor)));

I have also tried adding a retry strategy and that seemed to get rid of the issue but with a trade off of the RestHighLevelClient created and close per call instead of being a singleton. I believe this is not recommended per documentation.

Regardless, I still do not understand what is the root cause here causing the ConnectionTimeout error and I have only seen this with out canary traffic which is running once per minute.

Thank you in advance!

Welcome!

We don't support this service here but could you check elasticsearch logs in case it gives a clue?

BTW did you look at Cloud by Elastic, also available if needed from AWS Marketplace ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next :slight_smile: ...