HighLevel REST Client : Retry requests on failure

[Gopala_Krishna]

Hello!

Background: We use an Elasticsearch cluster with a single node for indexing text documents. The High Level REST Client is used for interacting with the cluster. The cluster, the client all run in their own docker containers on the same docker network.

Problem: We notice that often the requests to Elasticsearch (index exists, index requests etc) fail due to connection problems.

1. Is there a built in mechanism for retrying requests when the client cant connect to the cluster? I searched a lot but couldn't find any relevant documentation.

2. What could be the reason for this frequent connection resets?

Please find the relevant stacktrace below

  2019-01-08 15:37:48.938  INFO 1 --- [nio-8080-exec-8] a.d.elasticsearch.documents.DocumentDao  : documentsdata is already exist
java.io.IOException: Connection reset by peer
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:954)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:229)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1593)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1563)
        at org.elasticsearch.client.IndicesClient.existsAlias(IndicesClient.java:475)
        at analyzelaw.dao.elasticsearch.documents.DocumentDao.insertDocument(DocumentDao.java:284)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:204)
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        ... 1 more
        Suppressed: java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
                at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
                at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
                at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
                ... 1 more

[xerrad (김석윤)]

Hi, I got the same problem with you. and recently I solved it.
In my case, the reason is calling disableAuthCaching() method while creating HighLevelRestClient instance.

too many trying authentication to elasticsearch cause connection refusal.
so do not call disableAuthCaching() method.