Hi,
version: 7.9.1
Context
We are using the Java high-level REST client.
When executing a search request, and the response's size exceeds the buffered heap, then a ContentTooLongException and IOReactorException are thrown. By default, the heap is initialized to 100Mb.
For all subsequent requests, the REST client will throw a RuntimeException, even if the heap is larger than the size of the response entity. There seems to be no way to fix the situation without restarting the application. I'd expect that subsequent requests would not fail assuming they were small enough to fit into the max heap buffered response.
Stacktrace
ContentTooLongException
org.apache.http.ContentTooLongException: entity content is too long [147830] for the configured buffer limit [1024]
at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137)
at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:834)
IOReactorException
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Incorrect number of labels.
at io.prometheus.client.SimpleCollector.labels(SimpleCollector.java:64)
at com.blablacar.trip.common.config.CustomHttpAsyncRequestExecutor.exception(CustomHttpAsyncRequestExecutor.java:37)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 common frames omitted
RuntimeException
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:857)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
….
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:255)
... 28 common frames omitted
How to reproduce
We can reproduce the problem by playing with the heap size. In the snippet of code below, the very first search will use a 1Kb heap. All others will use a 100Mb heap size.
The first search will throw both the ContentTooLongException and IOReactorException.
Allo others will throw the RuntimeException even if the heap is larger than the size of the response entity.
Snippet of code:
class DataSource {
// Very small heap size
private static final int MAX_HEAP_BUFFER_LIMIT_IN_BYTES_1K = 1024;
// Default heap size 100Mb
private static final int MAX_HEAP_BUFFER_LIMIT_IN_BYTES_100M = 100 * 1024 * 1024;
private final RestHighLevelClient client;
/**
* Switch the value of the buffered heap.
* For the very first call, the heap equals MAX_HEAP_BUFFER_LIMIT_IN_BYTES_1K,
* and MAX_HEAP_BUFFER_LIMIT_IN_BYTES_100M for all subsequent calls.
*/
private final AtomicBoolean heapSwitch;
public DataSource(RestHighLevelClient client) {
this.client = client;
this.heapSwitch = new AtomicBoolean(false);
}
public List<String> search() {
SearchRequest searchRequest = new SearchRequest("index-1")
.requestCache(true)
.source(new SearchSourceBuilder().query(boolQuery()));
// Choose the heap value.
int heap;
if (!heapSwitch.get()) {
heap = MAX_HEAP_BUFFER_LIMIT_IN_BYTES_1K;
heapSwitch.getAndSet(true);
} else {
heap = MAX_HEAP_BUFFER_LIMIT_IN_BYTES_100M;
}
Builder customOptions = RequestOptions.DEFAULT.toBuilder();
customOptions.setHttpAsyncResponseConsumerFactory(
new HeapBufferedResponseConsumerFactory(heap)
);
try {
SearchResponse response = client.search(searchRequest, customOptions.build());
return ...;
} catch (IOException e) {
return ...;
}
}
}