# Higher number of active threads with elasticsearch-java dependency

**URL:** <https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385>\
**Category:** Elasticsearch\
**Tags:** language-clients\
**Created:** [August 5, 2024, 1:10pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385 "2024-08-05T13:10:18Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 5, 2024, 1:10pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/1 "2024-08-05T13:10:18Z")

</div>

Hi  
we are currently migrating to Elasticsearch v8.12.0 and we have below code to create the Elasticsearch client to make the connections to Elasticsearch.  
we are using `elasticsearch-java: v8.12.0` dependency

```auto
@Provides
@Singleton
public ElasticsearchClient provideESRestClient() {
        RestClientBuilder restClientBuilder = RestClient.builder(
                new HttpHost("host", port, "https"));

        ElasticsearchTransport transport = new RestClientTransport(restClientBuilder.build(), new JacksonJsonpMapper());

        return new ElasticsearchClient(transport);
    }

```

we are seeing high number of active thread counts when we use this for a longer time with load compare to the older versions.

is there any recommendation on handling thread uses and have a better thread performance.

Any help will be really very helpful

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 5, 2024, 1:38pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/2 "2024-08-05T13:38:50Z")

</div>

Can you reproduce this in 8.13.0 or later? If so, could you share a list of all the names of the threads you're seeing?

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 6, 2024, 5:26am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/3 "2024-08-06T05:26:20Z")

</div>

@DavidTurner Thank you for the suggestion. sure we are trying with `8.14.2`. will let you know once we are done with the testing

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 7, 2024, 10:53am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/4 "2024-08-07T10:53:42Z")

</div>

@DavidTurner same issue with `8.13.0` as well. one thing we noticed is most of those threads are in waiting state with below stack trace.

```auto
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1907)

```

as a pattern we are seeing every ~12 to15 minutes interval its spikes and mostly its not able to kill the idle threads(suspecting).

Is there any configurations we can add with above code to close any idle or blocked connection like a time out setting or something ?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 7, 2024, 11:14am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/5 "2024-08-07T11:14:56Z")

</div>

As per my previous message:

> [@DavidTurner](#):
>
> could you share a list of all the names of the threads you're seeing?

The next steps depend on the outcome of this (in ≥8.13.0)

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 7:21am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/6 "2024-08-08T07:21:17Z")

</div>

@DavidTurner Below is the thread details we are noticing when we run the tests

- `elasticsearch-rest-client-0-thread-1` to `elasticsearch-rest-client-0-thread-9` with status something similar to below

`elasticsearch-rest-client-0-thread-1: running , holding [0x00000200736ccb68 0x00000200736ccb58 0x00000200736cca00]`

- there are maximum number threads which have name pattern as `-<number>`, for example `-12388 awaiting notification`. these threads points to below stack trace

```auto
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1923)

```

- there are some threads with format `pool-8-thread-10667` and these threads also is pointing to above stack trace

Please let me know if this information will help to debug the issue or any other data is required.

if this data is not helpful, can you please share the steps to collect the data which will be helpful for the debugging.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 8, 2024, 7:27am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/7 "2024-08-08T07:27:30Z")

</div>

To repeat, again:

> [@DavidTurner](#):
>
> could you share a list of all the names of the threads you're seeing?

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 7:34am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/8 "2024-08-08T07:34:28Z")

</div>

`elasticsearch-rest-client-0-thread-1` these are the thread names what we are seeing. is it not the correct thread names?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 8, 2024, 7:36am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/9 "2024-08-08T07:36:36Z")

</div>

That's not a list of all the thread names (unless you only have one thread, but then that doesn't seem like a high number of threads). Sorry if I'm missing something obvious here.

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 9:47am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/10 "2024-08-08T09:47:05Z")

</div>

Sorry for the misunderstanding, below is the list of thread name.

Please let me know if i am doing any mistake capturing the thread names.  
we are using this command `sudo /jdk8/bin/jstack <pid> > "thread_dump.log"` to capture the thread dump

if i am doing any mistake on capturing the information please share the steps to capture the thread names

```auto
AbstractQueuedSynchronizer$ConditionObject
main-EventThread
AbstractQueuedSynchronizer$ConditionObject
AbstractQueuedSynchronizer$ConditionObject
PersistentEphemeralNodeThread-0
AbstractQueuedSynchronizer$ConditionObject
PersistentEphemeralNodeThread-1
AbstractQueuedSynchronizer$ConditionObject
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-1
AbstractQueuedSynchronizer$ConditionObject
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-0
AbstractQueuedSynchronizer$ConditionObject
AbstractQueuedSynchronizer$ConditionObject
pollingConfigurationSource
-9004
dw-admin-115
-12359
-8934
-12380
-2268
-1821
pool-17-thread-773
 -7362 
 -12389
 -54 - 
 -3480 
 pool-17-thread-734
 -2269 
 -2466 
 -12384
 -3482 
 pool-17-thread-652
 pool-17-thread-720
 -12409
 pool-17-thread-738
 -3339
 pool-17-thread-767
 -1489
 -101 
 -2731
 -8931
 -12385
 -56 
-12368
 -9042
 -3757
 -58 
 -3330
 -8898
 -1843
 -2468
 -1589
 pool-17-thread-719
 -3697
 -1605
 -8936
 -2250
 -3498
 -8990
 -1824
 -8705
 pool-17-thread-740
 -12332
 -2440 
 pool-17-thread-754
 pool-17-thread-733
pool-17-thread-725
-8724 
 -8737
 -8719
 -12386
 -3751 
 -2427 
 -12418
 -12342
 -8994 
 pool-17-thread-771
 -9041 
 -12366
 pool-17-thread-644
 pool-17-thread-747
 -9038 
 -1596 
 -3343 
 -12408
 pool-17-thread-774
 pool-17-thread-743
 pool-17-thread-750
 -8709 
 -3346 
 -8703 
 -9028 
 -1827 
 pool-17-thread-770
 -8901 
 -12420
 -84 
 -3493
 -2469
 -3341
 -7364
 -3694
 pool-17-thread-751
 -2431
 pool-17-thread-567
 pool-17-thread-764
 -12411
 -66 
 -1840
 -3740
 -8888
 pool-17-thread-436
 -12381
 pool-17-thread-745
 pool-17-thread-759
 pool-17-thread-768
 -3499 
 -107 
 -2249
 -8997
 pool-17-thread-737
 -9001
 -12404
 -3338 
 -3488 
 pool-17-thread-713
 pool-17-thread-594
 -9003 
 pool-17-thread-526
 -8903 
 pool-17-thread-706
 -8716 
 -3767 
 -6311 
 -12415
 pool-17-thread-646
 -9039 
 pool-17-thread-666
 -8937 
 -1595 
 -3481 
 -71 
 -6305 
 -12356
 -1614 
 -12370
 -12416
 -8894 
 -12352
 pool-17-thread-655
 -12350
 pool-17-thread-362
 -8893 
 -8991 
 pool-17-thread-571
 -8922 
 -2252 
 -9035 
 -12422
 -2737 
 pool-17-thread-656
 pool-17-thread-746
 pool-17-thread-727
 -9033 
 -12392
 -3467 
 pool-17-thread-739
 -8745
 -105 
 -3494
 pool-17-thread-732
 -3501 
 -12401
 -6308 
 pool-17-thread-700
 -3327 
 -1590 
 -60 
 -6296 
 -2330 
 pool-17-thread-766
 -2738 
 -12412
 pool-17-thread-633
 -12424
 -7368 
 -47 
 -1602 
 pool-17-thread-769
 -8730
 -7283
 pool-17-thread-762
 -7356
 -79 
 -9029
 -3340
 -12358
 -1842 
 -1820 
 -3334 
 pool-17-thread-604
 -3505 
 -12351
 -12423
 pool-17-thread-654
 -7282 
 -6307 
 pool-17-thread-555
 -8721 
 pool-17-thread-695
 -8727 
 pool-17-thread-724
 -3475 
 pool-17-thread-741
 -3491 
 pool-17-thread-722
 -12400
 pool-17-thread-686
 -3328
 pool-17-thread-775
 -3479
 -8897
 -12396
 -3477 
 -1584 
 pool-17-thread-730
 pool-17-thread-726
 -2740 
 -8725 
 -3758 
 -8933 
 pool-17-thread-676
 -12375
 pool-17-thread-711
 -8925 
 pool-17-thread-772
 -3342 
 pool-17-thread-758
 -8718 
 pool-17-thread-718
 pool-17-thread-755
 -12407
 -8999 
 -2439 
 pool-17-thread-651
 -8919 
 pool-17-thread-714
 -3753
 -3750
 -3507
 -8708
 -2736
 -3326
 -3487
 -12374
 pool-17-thread-659
 -1848 
 -1811 
 -12394
 pool-17-thread-763
 -12333
 pool-17-thread-683
 pool-17-thread-699
 -12340
 -8729 
 pool-17-thread-765
 -3500 
 pool-17-thread-697
 pool-17-thread-749
 pool-17-thread-694
 pool-17-thread-678
 -46
 -12426
 pool-17-thread-731
 -2727
 pool-17-thread-542
 pool-17-thread-760
 pool-17-thread-693
 pool-17-thread-708
 -82 
 pool-17-thread-684
 -1838
 -12399 
 pool-17-thread-761
 pool-17-thread-748
 pool-17-thread-736
 -1484 
 -3698 
 pool-17-thread-710
 pool-17-thread-757
 -2695 
 pool-17-thread-648
 pool-17-thread-735
 -8739 
 pool-17-thread-612
 -3755 
 -12364
 -3474 
 -7358 
 -12354
 -12334
 -8891 
 -12382
 -8904 
 -1825 
 -2424 
 -2255 
 pool-17-thread-685
 -3348 
 -2470 
 -9031 
 -12357
 pool-17-thread-756
 pool-17-thread-721
 pool-17-thread-546
 -9007 
 -1594 
 -3695 
 -12335
 -8929 
 pool-17-thread-559
 -12421
 -8899 
 -51 - 
 pool-17-thread-742
 -2437 
 -6301 
 -9027 
 Okio Watchdog
 -12397
 -6294 
 dw-admin-137
 dw-admin-359
 dw-admin-146
 dw-admin-139
 main-SendThread
 elasticsearch-rest-client-0-thread-2
 elasticsearch-rest-client-0-thread-3
 elasticsearch-rest-client-0-thread-4
 elasticsearch-rest-client-0-thread-5
 elasticsearch-rest-client-0-thread-6
 elasticsearch-rest-client-0-thread-7
 elasticsearch-rest-client-0-thread-8
 elasticsearch-rest-client-0-thread-9
 -2248
 -8988
 elasticsearch-rest-client-0-thread-1
 OkHttp ConnectionPool
 OkHttp ConnectionPool
 OkHttp ConnectionPool
 -12397
 -6294
 dw-admin-137
 dw-admin-359
 dw-admin-146
 dw-admin-139
 main-SendThread
 dw-admin-114-acceptor-0@495e7171-admin@41bb447d{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
 main-EventThread
PersistentEphemeralNodeThread-0
PersistentEphemeralNodeThread-1
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-1
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-0
pollingConfigurationSource
-9004 
dw-admin-115
-12359
-8934 
-12380
-2268 
-1821 
pool-17-thread-773

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 8, 2024, 10:12am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/11 "2024-08-08T10:12:19Z")

</div>

Thanks, that's helpful. I think the only ones created by the Elasticsearch client are the ones that start `elasticsearch-rest-client-*-thread-*`, and they look ok to me. Can you work out (e.g. with a debugger) what is creating the other threads?

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 11:29am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/12 "2024-08-08T11:29:43Z")

</div>

Thank you for the reply, but we are noticing all the other threads which are in `awaiting notification` is stuck with below stack tarce

```auto
waiting on VM lock 'org.apache.http.concurrent.BasicFuture', polling bits: safep ]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
	- locked <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
	at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
	at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.mget(ElasticsearchClient.java:1279)
	at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:88)
	at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:74)
	at abc.cde.client.ESClientService.execute(ESClientService.java:66)
	at abc.cde.execution.MultiGetExecutorImpl.execute(MultiGetExecutorImpl.java:104)
	at abc.cde.impl.DefaultProductInfoService.getBatchExpandedProducts(DefaultProductInfoService.java:263)
	at abc.cde.guice.modules.ProductInfoServiceModule.loadProducts(ProductInfoServiceModule.java:154)
	at abc.cde.guice.modules.ProductInfoServiceModule.access$200(ProductInfoServiceModule.java:38)
	at abc.cde.guice.modules.ProductInfoServiceModule$3.loadAll(ProductInfoServiceModule.java:103)
	at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4073)
	at com.google.common.cache.LocalCache.getAll(LocalCache.java:4036)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4964)
	at abc.cde.request.env.Stats.getStatsBlock(StatsBackend.java:156)
	at abc.cde.request.env.Stats.getStatsTaskResult(Stats.java:326)
	at abc.cde.request.env.Stats.lambda$execMultiThreadedStatsTasks$1(Stats.java:279)
	at abc.cde.request.env.Stats$$Lambda$lambda$execMultiThreadedStatsTasks$1$81078421/0x000000000000437b.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:277)
	at abc.cde.request.env.Stats.execMultiThreadedStatsTasks(Stats.java:285)
	at abc.cde.request.env.Stats.calculateStats(Stats.java:168)
	at abc.cde.request.env.SingleResourceEnvRequestHandler.handle(SingleResourceEnvRequestHandler.java:96)
	at abc.cde.request.env.EnvRequestHandler.handle(EnvRequestHandler.java:44)
	at abc.cde.resource.Resource.doHandleApiQueryRequest(Resource.java:163)
	at abc.cde.resource.Resource.handleApiQueryRequest(Resource.java:116)
	at abc.cde.resource.Resource.query(Resource.java:99)
	at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$lambda$static$0$3446075016/0x0000000000003632.invoke(Unknown Source)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at abc.cde.jersey.filters.MaxSimultaneousRequestsFilter.doFilter(MaxSimultaneousRequestsFilter.java:222)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:501)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel$$Lambda$lambda$handle$1$2621999213/0x00000000000037ad.dispatch(Unknown Source)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:807)

```

Any suggestion on any customisation which might be helpful to debug the issue

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 8, 2024, 11:45am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/13 "2024-08-08T11:45:49Z")

</div>

That's not the whole stack trace, but this is what you'd expect if your own application thread makes a blocking request to the Elasticsearch client. What's the rest of that stack trace?

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 12:08pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/14 "2024-08-08T12:08:57Z")

</div>

Complete stack trace

```auto
waiting on VM lock 'org.apache.http.concurrent.BasicFuture', polling bits: safep ]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
	- locked <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
	at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
	at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
	at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.mget(ElasticsearchClient.java:1279)
	at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:88)
	at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:74)
	at abc.cde.client.ESClientService.execute(ESClientService.java:66)
	at abc.cde.execution.MultiGetExecutorImpl.execute(MultiGetExecutorImpl.java:104)
	at abc.cde.impl.DefaultProductInfoService.getBatchExpandedProducts(DefaultProductInfoService.java:263)
	at abc.cde.guice.modules.ProductInfoServiceModule.loadProducts(ProductInfoServiceModule.java:154)
	at abc.cde.guice.modules.ProductInfoServiceModule.access$200(ProductInfoServiceModule.java:38)
	at abc.cde.guice.modules.ProductInfoServiceModule$3.loadAll(ProductInfoServiceModule.java:103)
	at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4073)
	at com.google.common.cache.LocalCache.getAll(LocalCache.java:4036)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4964)
	at abc.cde.request.env.Stats.getStatsBlock(StatsBackend.java:156)
	at abc.cde.request.env.Stats.getStatsTaskResult(Stats.java:326)
	at abc.cde.request.env.Stats.lambda$execMultiThreadedStatsTasks$1(Stats.java:279)
	at abc.cde.request.env.Stats$$Lambda$lambda$execMultiThreadedStatsTasks$1$81078421/0x000000000000437b.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:277)
	at abc.cde.request.env.Stats.execMultiThreadedStatsTasks(Stats.java:285)
	at abc.cde.request.env.Stats.calculateStats(Stats.java:168)
	at abc.cde.request.env.SingleResourceEnvRequestHandler.handle(SingleResourceEnvRequestHandler.java:96)
	at abc.cde.request.env.EnvRequestHandler.handle(EnvRequestHandler.java:44)
	at abc.cde.resource.Resource.doHandleApiQueryRequest(Resource.java:163)
	at abc.cde.resource.Resource.handleApiQueryRequest(Resource.java:116)
	at abc.cde.resource.Resource.query(Resource.java:99)
	at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$lambda$static$0$3446075016/0x0000000000003632.invoke(Unknown Source)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at abc.cde.jersey.filters.MaxSimultaneousRequestsFilter.doFilter(MaxSimultaneousRequestsFilter.java:222)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:501)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel$$Lambda$lambda$handle$1$2621999213/0x00000000000037ad.dispatch(Unknown Source)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:807)

```

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 8, 2024, 12:17pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/15 "2024-08-08T12:17:57Z")

</div>

replied in a new thread but the system is flagging that as a spam as it having more number of threads in the conversations, updated the stack trace in the previous thread as well

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 8, 2024, 12:39pm UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/16 "2024-08-08T12:39:09Z")

</div>

> [@sibasish.palo](#):
>
> ```auto
> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> 
> ```

Right so this is a thread in a Jetty pool, nothing to do with Elasticsearch

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 13, 2024, 4:40am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/17 "2024-08-13T04:40:13Z")

</div>

Thank you for the help, we are able to figure out the issue.

we have an NetworkLoadBalancer(NLB) in front of the cluster, when we try to create a connection to the NLB, the client always connects to one IP of the NLB. That is why its not able to spin more threads(after a limit) and everything else is blocked.

Is there any way we can configure the client to pick all the IPs of the NLB.

The reason we are adding a NLB in front of our clusters is because we wanted to connect to multiple clusters at a time.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 13, 2024, 7:02am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/18 "2024-08-13T07:02:10Z")

</div>

> [@sibasish.palo](#):
>
> the client always connects to one IP of the NLB

If you configure the client with multiple IP addresses then it will use all of them.

---

<div class="post-metadata">

**Author:** ![sibasish.palo](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@sibasish.palo](https://discuss.elastic.co/u/sibasish.palo)\
**Post date:** [August 13, 2024, 10:41am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/19 "2024-08-13T10:41:55Z")

</div>

is there any recommended way to use the client, such as use a load balancer in front of the cluster or configure all the node IPs which might cause problem when a node gets replaced(could this be resolved by sniffer)?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [August 13, 2024, 10:43am UTC](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385/20 "2024-08-13T10:43:23Z")

</div>

> [@sibasish.palo](#):
>
> is there any recommended way to use the client

All of your suggestions seem reasonable.

[Next page](https://discuss.elastic.co/t/higher-number-of-active-threads-with-elasticsearch-java-dependency/364385.md?page=2)