co.elastic.clients.elasticsearch.core.BulkRequest does not work against the Elasticsearch 8.11

The issue:
After the upgrade of our Elasticsearch server from version 8.6 to version 8.11 one of our services stopped working.

The following errors we are receiving on this

BulkRequest: POST /_bulk [{"update":{"_id":"xxx","_index":"test1_xxx"}}]

The error:

org.apache.http.ConnectionClosedException: Connection is closed
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:920) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-8.10.4.jar:8.10.4]
	at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91) ~[elasticsearch-java-8.10.4.jar:na]
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:137) ~[elasticsearch-java-8.10.4.jar:na]
	at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:337) ~[elasticsearch-java-8.10.4.jar:na]
	at com.xxx.xxx.xxx.service.storage.impl.ElasticsearchStorageService.bulk(ElasticsearchStorageService.java:89) ~[classes/:na]
	at com.xxx.xxx.xxx.service.storage.impl.ElasticsearchStorageService.upsert(ElasticsearchStorageService.java:78) ~[classes/:na]
	at com.xxx.xxx.xxx.service.util.StorageUpsertHelperCallable.call(StorageUpsertHelperCallable.java:48) ~[classes/:na]
	at com.xxx.xxx.xxx.service.util.StorageUpsertHelperCallable.call(StorageUpsertHelperCallable.java:17) ~[classes/:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.16.jar:4.4.16]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.16.jar:4.4.16]

Some additional information in order to be the big picture full:

  • Just before this request is executed, index delete and index create operations for the particular index and they complete well.

  • The application that experienced the error uses Spring Boot 3.2.0 and <elasticsearch-client.version>8.10.4</elasticsearch-client.version>.

  • How we instantiate the ElasticsearchClient variable:

    @Bean
    public ElasticsearchClient getElasticsearchClient() {

        ElasticsearchTransport transport = this.getElasticsearchTransport();

        return new ElasticsearchClient(transport);
    }
	
	private ElasticsearchTransport getElasticsearchTransport() {

        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(configProperties.getEsUser(), configProperties.getEsPassword()));

        RestClientBuilder builder = RestClient
                .builder(new HttpHost(configProperties.getEsHost(), configProperties.getEsPort(), configProperties.getEsProtocol()))
                .setHttpClientConfigCallback(httpClientBuilder -> {
                    httpClientBuilder.disableAuthCaching();
                    if (configProperties.isEnableProxy()) {
                        httpClientBuilder
                                .setProxy(new HttpHost(configProperties.getProxyHost(), configProperties.getProxyPort(), configProperties.getProxyProtocol()));
                    }
                    return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                }).setRequestConfigCallback(
                        arg0 -> arg0.setConnectTimeout(configProperties.getEsConnectionTimeout()).setSocketTimeout(configProperties.getEsSocketTimeout()));

        return new RestClientTransport(builder.build(), new JacksonJsonpMapper(getObjectMapper()));
    }

Is it the same discussion as Upgrade of ES to 8.11.3 and Java app using Spring Boot 3.2.0 - socket TImeout?

Yep, they are the same, we have created two of them by mistake. We will delete the other one, 10x.

You mentioned a move to Elasticsearch 8.11 but you are using:

<elasticsearch-client.version>8.10.4</elasticsearch-client.version>.

Why this?

Not sure it's related though, but I'm curious.

We also tried with 8.11.2 version but the same result.

I don't know exactly what is happening. May be you are not using https instead of http?
I see that you are mentioning a proxy in the code. Are you using a proxy here?

Could you log/debug this code:

new HttpHost(configProperties.getEsHost(), configProperties.getEsPort(), configProperties.getEsProtocol())

Also, could you share the full logs of your Elasticsearch service from the start?

1 Like

Hello,
thank you very much for your effort.
Yes, we communicate with Elasticsearch service via proxy.
The problem has been fixed, I have no information about what exactly was the reason, but I will share it when I have the info.
But it seems is in the network, most likely not in the Elasticsearch service itself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.