BulkProcessor: Cannot send more than 5 concurrent requests

Elastic Stack Elasticsearch
1 
new BulkProcessor.Builder(esClient.getHighLevelClient()::bulkAsync, listener, threadPool)
                .setBulkActions(3000)
                .setBulkSize(7)
                .setConcurrentRequests(6)
                .setFlushInterval(5000)
                .setBackoffPolicy(
                    BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(50),
                        8)).build();

Multiple threads are calling the bulkProcessor.add(indexRequest) method. So a lot of data is being put.
But this works great only if ConcurrentRequest=5. As soon as i make it 6 it starts giving below error:

java.util.concurrent.TimeoutException: null
	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:364) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(AbstractNIOConnPool.java:344) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:318) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseConnection(PoolingNHttpClientConnectionManager.java:303) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnection(AbstractClientExchangeHandler.java:239) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientExec.java:387) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:168) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_73]

Please help !

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

3

I realize that this is late, but I recently discovered the cause of this issue. It is due to the fact that we configure the Low Level REST Client's internal connection manager to have a timeout of 500ms. If your bulk payloads take longer than that, then the Apache threadpool's threads can be overwhelmed (defaults to the number of cores on the machine, which I suspect this case had 4 or possibly 5 from a Docker container).

The underlying Apache HTTP client's timeout is -1, which represents an unlimited time that a request can sit in the connection pool even attempting to be sent. As noted in the line above, the current default for the Low Level REST client (and thus also the High Level REST client) is 500 milliseconds. So if a request gets stuck behind N other requests -- where N depends on the connection pool's settings -- then it will end up being ignored when it comes time to even try to process it.

To get rid of this timeout, you can tell the connection manager to not have a timeout when configuring the RestClient. This can be done like the other timeout configurations by setting

requestConfigBuilder.setConnectionRequestTimeout(-1)

The implication of this change is that you may end up waiting an unpredictable amount of time before any of your requests actually get fired from your process, which is probably fine for a bulk request, but it may be less useful (and thus predictable) for any other types of requests unless your application is fine with the asynchronous nature of things.

Hope that helps someone,
Chris

2 Likes