I recently upgraded to es 6.2.4 and migrated the java client to use the RestHighLevelClient. I'm getting this error regularly performing a query on elasticsearch v6.2.4:
java.lang.RuntimeException: error while performing request at
org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:686) at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:223) at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:195) at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:488) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:474) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:391)
...
at org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:654) at
org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at
java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException at
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:364) at
org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(AbstractNIOConnPool.java:344) at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:318) at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(PoolingNHttpClientConnectionManager.java:303) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnection(AbstractClientExchangeHandler.java:239) at
org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:387) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:168) at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
It looks to be the same as this: RestClient java.lang.RuntimeException: error while performing request but I'm not 100% sure what the solution was.
Also similar to this: RestHighLevelClient timeout exception (no responses)
By RestClient is built with the following:
builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(30000)
.setSocketTimeout(120000))
.setMaxRetryTimeoutMillis(120000);
What settings need to be changed to prevent this from happening? The timeouts are already huge. Is there a server setting I need to change on the es install end? Is increasing the timeouts a bit naive and I need to solve it a different way?
Network connectivity is not the problem - it works lots of the time, but I get this error regularly. It might be that the ES node is under too much strain...and maybe I need more search threads in the thread_pool settings? Looking at netdata, the stress on the servers is minimal. They could definitely take a lot more traffic.
Thanks for any help offered.
Update:
I have now also tried this:
builder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
.setMaxConnTotal(100)
.setMaxConnPerRoute(100)
.setDefaultIOReactorConfig(
IOReactorConfig.custom()
.setIoThreadCount(100)
.build()
)
)
.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
.setConnectTimeout(0) //Timeout until connection is established
.setSocketTimeout(0) //Timeout when waiting for data
.setConnectionRequestTimeout(0) //Timeout when requesting a connection from the connection manager
)
.setMaxRetryTimeoutMillis(120000);
can you sense my desperation yet?!