Out of memorry Error When Using Java Client API

Elastic Stack Elasticsearch
1

I'm migrating my code from Java Rest High Level API to Java Client API 7.16 version. Everything goes all right in my local enviroment but got the OutOfMemoryError: Direct buffer memory error when I pull it to the Test Enviroment.
Here is the error logs:

Exception in thread "I/O dispatcher 79" java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
        at sun.nio.ch.IOUtil.read(IOUtil.java:195)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at ......

Once it happenes, all of my operations will be failed with error logs bellow before I restart my project.

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46) ~[httpcore-4.4.11.jar!/:4.4.11]
        at ......

and

 org.apache.http.ConnectionClosedException: Connection closed unexpectedly
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:778) ~[elasticsearch-rest-client-7.4.2.jar!/:7.4.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[elasticsearch-rest-client-7.4.2.jar!/:7.4.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205) ~[elasticsearch-rest-client-7.4.2.jar!/:7.4.2]
        at ......

I've increased the memory and the direct buffer memory of jvm, but it still doesn't work.
And it never happenned by using Java Rest High Level API.
Is it a bug or an enhancement of the new client API?
Can anyone suggest me how to troubleshoot this issue?

2

In the Java client, the only persistent memory allocations happen in the http library that maintains a pool of connections and buffers.

Are you creating new transport objects (RestClientTransport or the underlying RestClient) repeatedly? Could it be that they're not properly closed once used, leading their internal buffer pools to stay in memory? Once used, make sure to call RestClientTransport.close() or RestClient.close() to release their resources. Or even better, only create one transport object that is used throughout the entire application lifetime.

3

Thanks for your replay. And here is how I create connections which is the same as official documents.

    public static ElasticsearchClient initEsConnection() {
        RestClient restClient = RestClient.builder(new HttpHost(HOST_NAME, PORT, SCHEME)).build();
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
        return new ElasticsearchClient(transport);
    }

There is only one connection throughout my whole function. So you mean before the function is finished, I should close the connection manually by using RestClientTransport.close()?
By the way, I cannot find methods like client.close() to close the searchRequest connections in Java Client API. Is it replaced by RestClientTransport.close() and RestClient.close()?
Thanks again.

4

It seems like I solved the problem.
Finishing my search requests, a close client method will be called:

     public static void destroyConnection(ElasticsearchClient client) throws IOException {
        ElasticsearchTransport elasticsearchTransport = client._transport();
        elasticsearchTransport.close();
    }

I used to think it will be automatically closed. But it doesn`t. And the way to close a connection is hard to find. So, could it be possible that Java Client API offers a simpler method to close a ElasticsearchClient just like Java Rest High Level Client API.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.