RestHighLevelClient error while performing request

Elastic Stack Elasticsearch
1

I recently upgraded to es 6.2.4 and migrated the java client to use the RestHighLevelClient. I'm getting this error regularly performing a query on elasticsearch v6.2.4:

java.lang.RuntimeException: error while performing request at 
org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:686) at
 org.elasticsearch.client.RestClient.performRequest(RestClient.java:223) at
 org.elasticsearch.client.RestClient.performRequest(RestClient.java:195) at
 org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:488) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:474) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:391)
...
at org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:654) at
 org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at
 java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException at
 org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:364) at
org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(AbstractNIOConnPool.java:344) at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:318) at 
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(PoolingNHttpClientConnectionManager.java:303) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnection(AbstractClientExchangeHandler.java:239) at 
org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:387) at  org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:168) at 
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at 
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at
 org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at 
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at
 org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at
 org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)

It looks to be the same as this: RestClient java.lang.RuntimeException: error while performing request but I'm not 100% sure what the solution was.

Also similar to this: RestHighLevelClient timeout exception (no responses)

By RestClient is built with the following:

            builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
                    .setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(30000)
                    .setSocketTimeout(120000))
                    .setMaxRetryTimeoutMillis(120000);

What settings need to be changed to prevent this from happening? The timeouts are already huge. Is there a server setting I need to change on the es install end? Is increasing the timeouts a bit naive and I need to solve it a different way?

Network connectivity is not the problem - it works lots of the time, but I get this error regularly. It might be that the ES node is under too much strain...and maybe I need more search threads in the thread_pool settings? Looking at netdata, the stress on the servers is minimal. They could definitely take a lot more traffic.

Thanks for any help offered.

Update:

I have now also tried this:

builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
  .setMaxConnTotal(100)
  .setMaxConnPerRoute(100)
  .setDefaultIOReactorConfig(
    IOReactorConfig.custom()
            .setIoThreadCount(100)
            .build()
  )
)
.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
  .setConnectTimeout(0) //Timeout until connection is established
  .setSocketTimeout(0) //Timeout when waiting for data
  .setConnectionRequestTimeout(0) //Timeout when requesting a connection from the connection manager
)
.setMaxRetryTimeoutMillis(120000);

can you sense my desperation yet?!

2

Hi,
I believe this was fixed with https://github.com/elastic/elasticsearch/pull/30384 starting from 6.3.1.

If you use an older version, I'd suggest setting the connection request time out to -1 .in the request config callback, through the setConnectionRequestTimeout method.

Sorry for the trouble.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.