# Bulk api consuming new tcp port for every write which happens periodically every 1m **URL:** \ **Category:** Elasticsearch\ **Created:** [October 6, 2019, 5:31am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436 "2019-10-06T05:31:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![singamsundeep](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@singamsundeep](https://discuss.elastic.co/u/singamsundeep)\ **Post date:** [October 6, 2019, 5:31am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/1 "2019-10-06T05:31:43Z")
Hi, I'm using elastic java highclientrestclient and bulk processor, when writing data periodically every 1m, noticed new tcp port is used and old port remains in established state, after a while noticed lot of tcp connections are established with elastic search and eventually process die with not able to create a tcp port/file descriptor. Below are settings i'm using in java rest client and bulk processor ``` public BulkProcessor createBulkProcessor() { .... BiConsumer> bulkConsumer = (request, bulkListener) -> client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener); BulkProcessor.Builder builder = BulkProcessor.builder(bulkConsumer, listener) .setFlushInterval(TimeValue.timeValueSeconds(10L)) .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueSeconds(1L), 5)); .setBulkActions(); .setBulkSize(new ByteSizeValue(5, ByteSizeUnit.MB)).setConcurrentRequests(2) ``` } ``` RestClientBuilder restClientBuilder; restClientBuilder = RestClient.builder(new HttpHost(config.elasticHostName, config.elasticPort, "https")); try { restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() { @Override public HttpAsyncClientBuilder customizeHttpClient( HttpAsyncClientBuilder httpClientBuilder) { return httpClientBuilder.setSSLContext(sslContext); } }); } catch (Exception ex) { ex.printStackTrace(); } this.client = new RestHighLevelClient(restClientBuilder); this.bulkProcessor = createBulkProcessor(); } ``` ```auto # nsenter -t 10962 -n netstat -tanp | grep 9200 ; date tcp6 0 0 30.0.220.59:38648 100.80.111.216:9200 ESTABLISHED 10960/java tcp6 0 0 30.0.220.59:38314 100.80.111.216:9200 ESTABLISHED 10960/java Thu Oct 3 14:19:38 UTC 2019 # nsenter -t 10962 -n netstat -tanp | grep 9200 ; date tcp6 0 0 30.0.220.59:39058 100.80.111.216:9200 ESTABLISHED 10960/java <<< New connection tcp6 0 0 30.0.220.59:38648 100.80.111.216:9200 ESTABLISHED 10960/java . <<< old conection is still open tcp6 0 0 30.0.220.59:39458 100.80.111.216:9200 ESTABLISHED 10960/java tcp6 0 0 30.0.220.59:38314 100.80.111.216:9200 ESTABLISHED 10960/java Thu Oct 3 14:21:16 UTC 2019 ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2019, 9:15am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/2 "2019-10-06T09:15:22Z")
Do you have a load balancer in front of the cluster or network settings that could prevent long-running connections? ---
**Author:** ![singamsundeep](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@singamsundeep](https://discuss.elastic.co/u/singamsundeep)\ **Post date:** [October 6, 2019, 3:55pm UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/3 "2019-10-06T15:55:36Z")
Yes there is load balancer in front of the ES cluster, but instead of reusing the existing TCP connections, its creating new TCP connection over a period TCP connection going above 1000, all of them remaining in ESTABLISHED state. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 6, 2019, 3:59pm UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/4 "2019-10-06T15:59:35Z")
Then I would recommend checking the load balancer configuration or see if connecting without load balancer resolves the problem. ---
**Author:** ![singamsundeep](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@singamsundeep](https://discuss.elastic.co/u/singamsundeep)\ **Post date:** [October 7, 2019, 2:16am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/5 "2019-10-07T02:16:00Z")
With python to same Elastic load balancer this issue is not happening, i'm seeing only in Java i guess some configuration issue either in my restClientBuilder, or in java high level rest client handler. I tried setting maxconnection total, none of them is taking effect, suspect for each bulk write ending up with new restclient instead of using the existing one. I'm using ES version 6.2.4 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 7, 2019, 5:15am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/7 "2019-10-07T05:15:54Z")
> [@singamsundeep](#): > > suspect for each bulk write ending up with new restclient instead of using the existing one. That was my guess, and would explain it. You must use the same client across all the requests. ---
**Author:** ![singamsundeep](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@singamsundeep](https://discuss.elastic.co/u/singamsundeep)\ **Post date:** [October 7, 2019, 6:32am UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/8 "2019-10-07T06:32:50Z")
Using same Client in my requests, bulk async write seems to be consuming new TCP session, instead of reusing the existing ones. 2019-10-07 06:23:09 Executing bulk [6] with 6748 requests. \>\>\> After executing bulk request, i see 6 TCP sessions are present. ``` # nsenter -t 23587 -n netstat -tanp | grep 9200; date tcp6 0 0 30.0.220.108:46530 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:47376 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:48272 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:47822 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:46454 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:46904 100.80.111.216:9200 ESTABLISHED 23587/java Mon Oct 7 06:23:19 UTC 2019 ``` 2019-10-07 06:24:11 Bulk [7] completed in 1439 milliseconds \>\> after this send tcp session count become 7, it matches with number of bulk writes. ``` # nsenter -t 23587 -n netstat -tanp | grep 9200; date tcp6 0 0 30.0.220.108:46530 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:47376 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:48272 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:47822 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:48796 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:46454 100.80.111.216:9200 ESTABLISHED 23587/java tcp6 0 0 30.0.220.108:46904 100.80.111.216:9200 ESTABLISHED 23587/java Mon Oct 7 06:24:24 UTC 2019 ``` ---
**Author:** ![singamsundeep](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@singamsundeep](https://discuss.elastic.co/u/singamsundeep)\ **Post date:** [October 7, 2019, 9:08pm UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/9 "2019-10-07T21:08:06Z")
This issue is not happening after changing version to 7.4 on client, connection count is remained with in the configured limit, but its not reusing the existing connections, instead of creating new and old connections are going in TIME WAIT and eventually getting closed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2019, 9:08pm UTC](https://discuss.elastic.co/t/bulk-api-consuming-new-tcp-port-for-every-write-which-happens-periodically-every-1m/202436/10 "2019-11-04T21:08:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.