Bulk api consuming new tcp port for every write which happens periodically every 1m

Elastic Stack Elasticsearch
1

Hi, I'm using elastic java highclientrestclient and bulk processor, when writing data periodically every 1m, noticed new tcp port is used and old port remains in established state, after a while noticed lot of tcp connections are established with elastic search and eventually process die with not able to create a tcp port/file descriptor.

Below are settings i'm using in java rest client and bulk processor

   public BulkProcessor createBulkProcessor()  {
   ....
    BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
            (request, bulkListener) ->
                    client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
    BulkProcessor.Builder builder =
            BulkProcessor.builder(bulkConsumer, listener)
                    .setFlushInterval(TimeValue.timeValueSeconds(10L))
                    .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueSeconds(1L),
                            5));
                    .setBulkActions();
                    .setBulkSize(new ByteSizeValue(5, ByteSizeUnit.MB)).setConcurrentRequests(2)

}

    RestClientBuilder restClientBuilder;
        restClientBuilder = RestClient.builder(new HttpHost(config.elasticHostName, config.elasticPort, "https"));
        try {
            restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                @Override
                public HttpAsyncClientBuilder customizeHttpClient(
                        HttpAsyncClientBuilder httpClientBuilder) {
                    return httpClientBuilder.setSSLContext(sslContext);
                }
            });
        }
        catch (Exception ex) {
            ex.printStackTrace();
        }

    this.client = new RestHighLevelClient(restClientBuilder);
    this.bulkProcessor = createBulkProcessor();
}

# nsenter -t 10962 -n netstat -tanp  | grep 9200  ; date                                                   
tcp6       0      0 30.0.220.59:38648       100.80.111.216:9200     ESTABLISHED 10960/java         
tcp6       0      0 30.0.220.59:38314       100.80.111.216:9200     ESTABLISHED 10960/java         
Thu Oct  3 14:19:38 UTC 2019

# nsenter -t 10962 -n netstat -tanp  | grep 9200  ; date
tcp6       0      0 30.0.220.59:39058       100.80.111.216:9200     ESTABLISHED 10960/java <<< New connection

tcp6       0      0 30.0.220.59:38648       100.80.111.216:9200     ESTABLISHED 10960/java . <<< old conection is still open

tcp6       0      0 30.0.220.59:39458       100.80.111.216:9200     ESTABLISHED 10960/java
tcp6       0      0 30.0.220.59:38314       100.80.111.216:9200     ESTABLISHED 10960/java
Thu Oct  3 14:21:16 UTC 2019
2

Do you have a load balancer in front of the cluster or network settings that could prevent long-running connections?

3

Yes there is load balancer in front of the ES cluster, but instead of reusing the existing TCP connections, its creating new TCP connection over a period TCP connection going above 1000, all of them remaining in ESTABLISHED state.

4

Then I would recommend checking the load balancer configuration or see if connecting without load balancer resolves the problem.

5

With python to same Elastic load balancer this issue is not happening, i'm seeing only in Java i guess some configuration issue either in my restClientBuilder, or in java high level rest client handler.

I tried setting maxconnection total, none of them is taking effect, suspect for each bulk write ending up with new restclient instead of using the existing one.

I'm using ES version 6.2.4

7

That was my guess, and would explain it. You must use the same client across all the requests.

8

Using same Client in my requests, bulk async write seems to be consuming new TCP session, instead of reusing the existing ones.

2019-10-07 06:23:09 Executing bulk [6] with 6748 requests. >>> After executing bulk request, i see 6 TCP sessions are present.

# nsenter -t 23587 -n netstat -tanp  | grep 9200;  date
tcp6       0      0 30.0.220.108:46530      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:47376      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:48272      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:47822      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:46454      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:46904      100.80.111.216:9200     ESTABLISHED 23587/java
Mon Oct  7 06:23:19 UTC 2019

2019-10-07 06:24:11 Bulk [7] completed in 1439 milliseconds >> after this send tcp session count become 7, it matches with number of bulk writes.

# nsenter -t 23587 -n netstat -tanp  | grep 9200;  date
tcp6       0      0 30.0.220.108:46530      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:47376      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:48272      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:47822      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:48796      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:46454      100.80.111.216:9200     ESTABLISHED 23587/java
tcp6       0      0 30.0.220.108:46904      100.80.111.216:9200     ESTABLISHED 23587/java
Mon Oct  7 06:24:24 UTC 2019
9

This issue is not happening after changing version to 7.4 on client, connection count is remained with in the configured limit, but its not reusing the existing connections, instead of creating new and old connections are going in TIME WAIT and eventually getting closed.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.