ElasticsearchClient didn't close socket connect and file open

[Adam_Lin]

Hi all,

I tried to query logs from the ELK server through java RestClient.

This is how I set up search client

try {
	restClient = RestClient
				.builder(new HttpHost(host, port))
				.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
					@Override
					public Builder customizeRequestConfig(Builder requestConfigBuilder) {
						return requestConfigBuilder
								.setConnectTimeout(60000)
								.setSocketTimeout(60000);
						}
				})
				.build();
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
	ElasticsearchClient client = new ElasticsearchClient(transport);
        // Do search 
        ....
        // close connection
        client._transport().close();
} catch(Exception e) {
}

the socket connection should close after every query.

I run the command on the client:

watch "ss -an | grep -an 9200 | wc -l"

and

watch "ls -lrt /proc/1382/fd | wc -l"

to trace the socket connect and file open, it increases in every query from the client and didn't close.

the version I use is 8.1.

Could anyone help?

Thanks.

[dadoonet (David Pilato)]

I'm not sure about what is happening and I'm not going to answer exactly your question but here are some thoughts:

• If you have an application running, it would be better IMO to have a singleton which you create when the application starts and only close when the application stops.
• try the 8.6 version of the client.
• It's normally not needed, but may be close as well the restClient?

[Adam_Lin]

Hi @dadoonet

Thanks for your reply.

l mean there will be a few connections to port 9200 and some files open in every request.
but these connections and files open didn't release after the request was finished.
the file open will keep growing until reaches the upper limit of 65535.

in the beginning, there have 1455 file handles, and I post 5 queries, it increases to 1642, and decrease to 1638, then stay at 1638.

[root]# ls -lrt /proc/1382/fd | wc -l
1450
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1455
[root]# ls -lrt /proc/1382/fd | wc -l
1642
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638
[root]# ls -lrt /proc/1382/fd | wc -l
1638

same in the connection

[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
73
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87
[root]# ss -an | grep -an 9200 | wc -l
87

I have closed the transport connection after every request, so I am guessing whether there is some reason that the connection is not closed.

I'll try using the singleton to see if it could duplicate.

[Adam_Lin]

Hi @dadoonet

The socket connection didn't increase after I change to using a singleton.

but the file open is still the same.

[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
529
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
536
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
547
[root /]# ls -lrt /proc/1602780/fd | wc -l
560
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]# ls -lrt /proc/1602780/fd | wc -l
562
[root /]#

I send the search query three times and it grows up to 547.

I saw there have many pipes in the pid.

[dadoonet (David Pilato)]

What about:

• try the 8.6 version of the client.
• It's normally not needed, but may be close as well the restClient?

And last question:

Is that a problem? Do yo stop the JVM entirely?

[Adam_Lin]

Hi @dadoonet

Thanks for your reply.

I've solved the file handle issue. some pipe leaking on my test environment.
I think this might not a problem.

And I found that when I create a restClient, there are actually multiple connections more than two being created, I wonder why and why not requests through two connections (client to server and server to client)?

[Christian_Dahlqvist]

The official clients are generally designed to be set up as singletons and reused by multiple threads. When connecting to Elasticsearch they set up a connection pool of long running connections. Each client request will use one of these connections. This improves performance and latency as new connections do not need to be established repeatedly, which can be slow, especially if SSL is used.