ElasticsearchClient didn't close socket connect and file open

Adam_Lin · February 4, 2023, 9:32am

Hi all,

I tried to query logs from the ELK server through java RestClient.

This is how I set up search client

try {
	restClient = RestClient
				.builder(new HttpHost(host, port))
				.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
					@Override
					public Builder customizeRequestConfig(Builder requestConfigBuilder) {
						return requestConfigBuilder
								.setConnectTimeout(60000)
								.setSocketTimeout(60000);
						}
				})
				.build();
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
	ElasticsearchClient client = new ElasticsearchClient(transport);
        // Do search 
        ....
        // close connection
        client._transport().close();
} catch(Exception e) {
}

the socket connection should close after every query.

I run the command on the client:

watch "ss -an | grep -an 9200 | wc -l"

and

watch "ls -lrt /proc/1382/fd | wc -l"

to trace the socket connect and file open, it increases in every query from the client and didn't close.

the version I use is 8.1.

Could anyone help?

Thanks.

dadoonet · February 4, 2023, 11:58am

I'm not sure about what is happening and I'm not going to answer exactly your question but here are some thoughts:

If you have an application running, it would be better IMO to have a singleton which you create when the application starts and only close when the application stops.
try the 8.6 version of the client.
It's normally not needed, but may be close as well the restClient?

Adam_Lin · February 6, 2023, 1:21am

Hi @dadoonet

Thanks for your reply.

l mean there will be a few connections to port 9200 and some files open in every request.
but these connections and files open didn't release after the request was finished.
the file open will keep growing until reaches the upper limit of 65535.

in the beginning, there have 1455 file handles, and I post 5 queries, it increases to 1642, and decrease to 1638, then stay at 1638.

[root]# ls -lrt /proc/1382/fd | wc -l
1450
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1642
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638

same in the connection

[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87

I have closed the transport connection after every request, so I am guessing whether there is some reason that the connection is not closed.

I'll try using the singleton to see if it could duplicate.

Adam_Lin · February 6, 2023, 5:50am

Hi @dadoonet

The socket connection didn't increase after I change to using a singleton.

but the file open is still the same.

[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
560
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]#

I send the search query three times and it grows up to 547.

I saw there have many pipes in the pid.

dadoonet · February 6, 2023, 8:47am

What about:

try the 8.6 version of the client.
It's normally not needed, but may be close as well the restClient?

And last question:

Is that a problem? Do yo stop the JVM entirely?

Adam_Lin · February 8, 2023, 1:46am

Hi @dadoonet

Thanks for your reply.

I've solved the file handle issue. some pipe leaking on my test environment.
I think this might not a problem.

And I found that when I create a restClient, there are actually multiple connections more than two being created, I wonder why and why not requests through two connections (client to server and server to client)?

Christian_Dahlqvist · February 8, 2023, 6:14am

The official clients are generally designed to be set up as singletons and reused by multiple threads. When connecting to Elasticsearch they set up a connection pool of long running connections. Each client request will use one of these connections. This improves performance and latency as new connections do not need to be established repeatedly, which can be slow, especially if SSL is used.

system · March 8, 2023, 6:15am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Java client 7.6.2 socket timeout seems has problem Elasticsearch language-clients	4	140	July 15, 2024
How to properly close a connection in a Java API Client? Elasticsearch language-clients	6	4086	November 9, 2022
Java Elasticsearch REST Client reuse or pooling Elasticsearch	4	10907	May 9, 2017
How to manage high level client connection to Elasticsearch Elasticsearch language-clients	4	1067	October 4, 2021
High level rest client connection Elasticsearch language-clients	5	479	December 21, 2021

ElasticsearchClient didn't close socket connect and file open

Related Topics