I am running ES through a docker image where I am trying to read a few documents from ES and then prepare a new document and then update them. While reading the documents I face this error.
Here is the full stack trace:
java.io.IOException: entity content is too long [110846684] for the configured buffer limit [104857600]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:937) ~[elasticsearch-rest-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) ~[elasticsearch-rest-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137) ~[elasticsearch-rest-high-level-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105) ~[elasticsearch-rest-high-level-client-7.17.14.jar!/:7.17.14]
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1367) ~[elasticsearch-rest-high-level-client-7.17.14.jar!/:7.17.14]
at fareye.elastic.connector.modules.elasticsearch.UpgradedElasticSearchService.getRecordsById(UpgradedElasticSearchService.java:154) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.elasticsearch.UpgradedElasticSearchService.getRecordsById(UpgradedElasticSearchService.java:167) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.elasticsearch.UpgradedElasticSearchService.getRecordsById(UpgradedElasticSearchService.java:167) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.processor.UpgradedPrepareDocumentService.getRecordsFromElasticSearchAndUpdate(UpgradedPrepareDocumentService.java:172) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.processor.UpgradedPrepareDocumentService.readAndSendProcessToElasticSearch(UpgradedPrepareDocumentService.java:130) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.processor.ProcessorConsumerService.extractDataFromKafka(ProcessorConsumerService.java:112) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.processor.ProcessorConsumerService.start(ProcessorConsumerService.java:91) [classes!/:v24.618.0]
at fareye.elastic.connector.modules.admin.StartApp.lambda$startApps$0(StartApp.java:47) [classes!/:v24.618.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_342]
Caused by: org.apache.http.ContentTooLongException: entity content is too long [110846684] for the configured buffer limit [104857600]
at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76) ~[elasticsearch-rest-client-7.17.14.jar!/:7.17.14]
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
I am using Elasticsearch RestHighLevelClient to call search API.
Here is the code of my rest client builder.
private RestClientBuilder newRestClient() {
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials("******", "*******"));
RestClientBuilder builder = RestClient.builder(new HttpHost("localhost", 9201,"http"));
//add credentials to the rest client
builder.setHttpClientConfigCallback(httpClientBuilder -> {
DefaultConnectingIOReactor ioreactor = null;
try {
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setIoThreadCount(Runtime.getRuntime().availableProcessors())
.setTcpNoDelay(true)
.setSoReuseAddress(true)
.setSoKeepAlive(true)
.build();
ioreactor = new DefaultConnectingIOReactor(ioReactorConfig);
PoolingNHttpClientConnectionManager connectionManager = new PoolingNHttpClientConnectionManager(ioreactor);
httpClientBuilder.setConnectionManager(connectionManager);
} catch (IOReactorException e) {
throw new RuntimeException(e);
}
return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
});
builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
.setConnectionRequestTimeout(0));
return builder;
}
Kindly suggest how can I increase the configured buffer limit. I also tried to change the http.max_content_length in elasticsearch.yml but that didn't solve the issue.