The issue:
After the upgrade of our Elasticsearch server from version 8.6 to version 8.11 one of our services stopped working.
The following errors we are receiving on this
BulkRequest: POST /_bulk [{"update":{"_id":"xxx","_index":"test1_xxx"}}]
The error:
org.apache.http.ConnectionClosedException: Connection is closed
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:920) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91) ~[elasticsearch-java-8.10.4.jar:na]
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:137) ~[elasticsearch-java-8.10.4.jar:na]
at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:337) ~[elasticsearch-java-8.10.4.jar:na]
at com.xxx.xxx.xxx.service.storage.impl.ElasticsearchStorageService.bulk(ElasticsearchStorageService.java:89) ~[classes/:na]
at com.xxx.xxx.xxx.service.storage.impl.ElasticsearchStorageService.upsert(ElasticsearchStorageService.java:78) ~[classes/:na]
at com.xxx.xxx.xxx.service.util.StorageUpsertHelperCallable.call(StorageUpsertHelperCallable.java:48) ~[classes/:na]
at com.xxx.xxx.xxx.service.util.StorageUpsertHelperCallable.call(StorageUpsertHelperCallable.java:17) ~[classes/:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.16.jar:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.16.jar:4.4.16]
Some additional information in order to be the big picture full:
-
Just before this request is executed, index delete and index create operations for the particular index and they complete well.
-
The application that experienced the error uses Spring Boot 3.2.0 and <elasticsearch-client.version>8.10.4</elasticsearch-client.version>.
-
How we instantiate the ElasticsearchClient variable:
@Bean
public ElasticsearchClient getElasticsearchClient() {
ElasticsearchTransport transport = this.getElasticsearchTransport();
return new ElasticsearchClient(transport);
}
private ElasticsearchTransport getElasticsearchTransport() {
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(configProperties.getEsUser(), configProperties.getEsPassword()));
RestClientBuilder builder = RestClient
.builder(new HttpHost(configProperties.getEsHost(), configProperties.getEsPort(), configProperties.getEsProtocol()))
.setHttpClientConfigCallback(httpClientBuilder -> {
httpClientBuilder.disableAuthCaching();
if (configProperties.isEnableProxy()) {
httpClientBuilder
.setProxy(new HttpHost(configProperties.getProxyHost(), configProperties.getProxyPort(), configProperties.getProxyProtocol()));
}
return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
}).setRequestConfigCallback(
arg0 -> arg0.setConnectTimeout(configProperties.getEsConnectionTimeout()).setSocketTimeout(configProperties.getEsSocketTimeout()));
return new RestClientTransport(builder.build(), new JacksonJsonpMapper(getObjectMapper()));
}