Hi
we are currently migrating to Elasticsearch v8.12.0 and we have below code to create the Elasticsearch client to make the connections to Elasticsearch.
we are using elasticsearch-java: v8.12.0 dependency
@Provides
@Singleton
public ElasticsearchClient provideESRestClient() {
RestClientBuilder restClientBuilder = RestClient.builder(
new HttpHost("host", port, "https"));
ElasticsearchTransport transport = new RestClientTransport(restClientBuilder.build(), new JacksonJsonpMapper());
return new ElasticsearchClient(transport);
}
we are seeing high number of active thread counts when we use this for a longer time with load compare to the older versions.
is there any recommendation on handling thread uses and have a better thread performance.
Any help will be really very helpful
Can you reproduce this in 8.13.0 or later? If so, could you share a list of all the names of the threads you're seeing?
@DavidTurner Thank you for the suggestion. sure we are trying with 8.14.2. will let you know once we are done with the testing
@DavidTurner same issue with 8.13.0 as well. one thing we noticed is most of those threads are in waiting state with below stack trace.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1907)
as a pattern we are seeing every ~12 to15 minutes interval its spikes and mostly its not able to kill the idle threads(suspecting).
Is there any configurations we can add with above code to close any idle or blocked connection like a time out setting or something ?
As per my previous message:
The next steps depend on the outcome of this (in ≥8.13.0)
@DavidTurner Below is the thread details we are noticing when we run the tests
elasticsearch-rest-client-0-thread-1 to elasticsearch-rest-client-0-thread-9 with status something similar to below
elasticsearch-rest-client-0-thread-1: running , holding [ 0x00000200736ccb68 0x00000200736ccb58 0x00000200736cca00 ]
- there are maximum number threads which have name pattern as
-<number>, for example -12388 awaiting notification. these threads points to below stack trace
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1923)
- there are some threads with format
pool-8-thread-10667 and these threads also is pointing to above stack trace
Please let me know if this information will help to debug the issue or any other data is required.
if this data is not helpful, can you please share the steps to collect the data which will be helpful for the debugging.
elasticsearch-rest-client-0-thread-1 these are the thread names what we are seeing. is it not the correct thread names?
That's not a list of all the thread names (unless you only have one thread, but then that doesn't seem like a high number of threads). Sorry if I'm missing something obvious here.
Sorry for the misunderstanding, below is the list of thread name.
Please let me know if i am doing any mistake capturing the thread names.
we are using this command sudo /jdk8/bin/jstack <pid> > "thread_dump.log" to capture the thread dump
if i am doing any mistake on capturing the information please share the steps to capture the thread names
AbstractQueuedSynchronizer$ConditionObject
main-EventThread
AbstractQueuedSynchronizer$ConditionObject
AbstractQueuedSynchronizer$ConditionObject
PersistentEphemeralNodeThread-0
AbstractQueuedSynchronizer$ConditionObject
PersistentEphemeralNodeThread-1
AbstractQueuedSynchronizer$ConditionObject
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-1
AbstractQueuedSynchronizer$ConditionObject
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-0
AbstractQueuedSynchronizer$ConditionObject
AbstractQueuedSynchronizer$ConditionObject
pollingConfigurationSource
-9004
dw-admin-115
-12359
-8934
-12380
-2268
-1821
pool-17-thread-773
-7362
-12389
-54 -
-3480
pool-17-thread-734
-2269
-2466
-12384
-3482
pool-17-thread-652
pool-17-thread-720
-12409
pool-17-thread-738
-3339
pool-17-thread-767
-1489
-101
-2731
-8931
-12385
-56
-12368
-9042
-3757
-58
-3330
-8898
-1843
-2468
-1589
pool-17-thread-719
-3697
-1605
-8936
-2250
-3498
-8990
-1824
-8705
pool-17-thread-740
-12332
-2440
pool-17-thread-754
pool-17-thread-733
pool-17-thread-725
-8724
-8737
-8719
-12386
-3751
-2427
-12418
-12342
-8994
pool-17-thread-771
-9041
-12366
pool-17-thread-644
pool-17-thread-747
-9038
-1596
-3343
-12408
pool-17-thread-774
pool-17-thread-743
pool-17-thread-750
-8709
-3346
-8703
-9028
-1827
pool-17-thread-770
-8901
-12420
-84
-3493
-2469
-3341
-7364
-3694
pool-17-thread-751
-2431
pool-17-thread-567
pool-17-thread-764
-12411
-66
-1840
-3740
-8888
pool-17-thread-436
-12381
pool-17-thread-745
pool-17-thread-759
pool-17-thread-768
-3499
-107
-2249
-8997
pool-17-thread-737
-9001
-12404
-3338
-3488
pool-17-thread-713
pool-17-thread-594
-9003
pool-17-thread-526
-8903
pool-17-thread-706
-8716
-3767
-6311
-12415
pool-17-thread-646
-9039
pool-17-thread-666
-8937
-1595
-3481
-71
-6305
-12356
-1614
-12370
-12416
-8894
-12352
pool-17-thread-655
-12350
pool-17-thread-362
-8893
-8991
pool-17-thread-571
-8922
-2252
-9035
-12422
-2737
pool-17-thread-656
pool-17-thread-746
pool-17-thread-727
-9033
-12392
-3467
pool-17-thread-739
-8745
-105
-3494
pool-17-thread-732
-3501
-12401
-6308
pool-17-thread-700
-3327
-1590
-60
-6296
-2330
pool-17-thread-766
-2738
-12412
pool-17-thread-633
-12424
-7368
-47
-1602
pool-17-thread-769
-8730
-7283
pool-17-thread-762
-7356
-79
-9029
-3340
-12358
-1842
-1820
-3334
pool-17-thread-604
-3505
-12351
-12423
pool-17-thread-654
-7282
-6307
pool-17-thread-555
-8721
pool-17-thread-695
-8727
pool-17-thread-724
-3475
pool-17-thread-741
-3491
pool-17-thread-722
-12400
pool-17-thread-686
-3328
pool-17-thread-775
-3479
-8897
-12396
-3477
-1584
pool-17-thread-730
pool-17-thread-726
-2740
-8725
-3758
-8933
pool-17-thread-676
-12375
pool-17-thread-711
-8925
pool-17-thread-772
-3342
pool-17-thread-758
-8718
pool-17-thread-718
pool-17-thread-755
-12407
-8999
-2439
pool-17-thread-651
-8919
pool-17-thread-714
-3753
-3750
-3507
-8708
-2736
-3326
-3487
-12374
pool-17-thread-659
-1848
-1811
-12394
pool-17-thread-763
-12333
pool-17-thread-683
pool-17-thread-699
-12340
-8729
pool-17-thread-765
-3500
pool-17-thread-697
pool-17-thread-749
pool-17-thread-694
pool-17-thread-678
-46
-12426
pool-17-thread-731
-2727
pool-17-thread-542
pool-17-thread-760
pool-17-thread-693
pool-17-thread-708
-82
pool-17-thread-684
-1838
-12399
pool-17-thread-761
pool-17-thread-748
pool-17-thread-736
-1484
-3698
pool-17-thread-710
pool-17-thread-757
-2695
pool-17-thread-648
pool-17-thread-735
-8739
pool-17-thread-612
-3755
-12364
-3474
-7358
-12354
-12334
-8891
-12382
-8904
-1825
-2424
-2255
pool-17-thread-685
-3348
-2470
-9031
-12357
pool-17-thread-756
pool-17-thread-721
pool-17-thread-546
-9007
-1594
-3695
-12335
-8929
pool-17-thread-559
-12421
-8899
-51 -
pool-17-thread-742
-2437
-6301
-9027
Okio Watchdog
-12397
-6294
dw-admin-137
dw-admin-359
dw-admin-146
dw-admin-139
main-SendThread
elasticsearch-rest-client-0-thread-2
elasticsearch-rest-client-0-thread-3
elasticsearch-rest-client-0-thread-4
elasticsearch-rest-client-0-thread-5
elasticsearch-rest-client-0-thread-6
elasticsearch-rest-client-0-thread-7
elasticsearch-rest-client-0-thread-8
elasticsearch-rest-client-0-thread-9
-2248
-8988
elasticsearch-rest-client-0-thread-1
OkHttp ConnectionPool
OkHttp ConnectionPool
OkHttp ConnectionPool
-12397
-6294
dw-admin-137
dw-admin-359
dw-admin-146
dw-admin-139
main-SendThread
dw-admin-114-acceptor-0@495e7171-admin@41bb447d{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
main-EventThread
PersistentEphemeralNodeThread-0
PersistentEphemeralNodeThread-1
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-1
CuratorFramework[zookeeper1:2181,zookeeper2:2181,zookeeper3:2181]-0
pollingConfigurationSource
-9004
dw-admin-115
-12359
-8934
-12380
-2268
-1821
pool-17-thread-773
Thanks, that's helpful. I think the only ones created by the Elasticsearch client are the ones that start elasticsearch-rest-client-*-thread-*, and they look ok to me. Can you work out (e.g. with a debugger) what is creating the other threads?
Thank you for the reply, but we are noticing all the other threads which are in awaiting notification is stuck with below stack tarce
waiting on VM lock 'org.apache.http.concurrent.BasicFuture', polling bits: safep ]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
- locked <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.mget(ElasticsearchClient.java:1279)
at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:88)
at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:74)
at abc.cde.client.ESClientService.execute(ESClientService.java:66)
at abc.cde.execution.MultiGetExecutorImpl.execute(MultiGetExecutorImpl.java:104)
at abc.cde.impl.DefaultProductInfoService.getBatchExpandedProducts(DefaultProductInfoService.java:263)
at abc.cde.guice.modules.ProductInfoServiceModule.loadProducts(ProductInfoServiceModule.java:154)
at abc.cde.guice.modules.ProductInfoServiceModule.access$200(ProductInfoServiceModule.java:38)
at abc.cde.guice.modules.ProductInfoServiceModule$3.loadAll(ProductInfoServiceModule.java:103)
at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4073)
at com.google.common.cache.LocalCache.getAll(LocalCache.java:4036)
at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4964)
at abc.cde.request.env.Stats.getStatsBlock(StatsBackend.java:156)
at abc.cde.request.env.Stats.getStatsTaskResult(Stats.java:326)
at abc.cde.request.env.Stats.lambda$execMultiThreadedStatsTasks$1(Stats.java:279)
at abc.cde.request.env.Stats$$Lambda$lambda$execMultiThreadedStatsTasks$1$81078421/0x000000000000437b.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:277)
at abc.cde.request.env.Stats.execMultiThreadedStatsTasks(Stats.java:285)
at abc.cde.request.env.Stats.calculateStats(Stats.java:168)
at abc.cde.request.env.SingleResourceEnvRequestHandler.handle(SingleResourceEnvRequestHandler.java:96)
at abc.cde.request.env.EnvRequestHandler.handle(EnvRequestHandler.java:44)
at abc.cde.resource.Resource.doHandleApiQueryRequest(Resource.java:163)
at abc.cde.resource.Resource.handleApiQueryRequest(Resource.java:116)
at abc.cde.resource.Resource.query(Resource.java:99)
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$lambda$static$0$3446075016/0x0000000000003632.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at abc.cde.jersey.filters.MaxSimultaneousRequestsFilter.doFilter(MaxSimultaneousRequestsFilter.java:222)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel$$Lambda$lambda$handle$1$2621999213/0x00000000000037ad.dispatch(Unknown Source)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:807)
Any suggestion on any customisation which might be helpful to debug the issue
That's not the whole stack trace, but this is what you'd expect if your own application thread makes a blocking request to the Elasticsearch client. What's the rest of that stack trace?
Complete stack trace
waiting on VM lock 'org.apache.http.concurrent.BasicFuture', polling bits: safep ]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
at java.lang.Object.wait(Object.java:502)
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:82)
- locked <0x0000200396e94910> (a org.apache.http.concurrent.BasicFuture)
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:91)
at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:144)
at co.elastic.clients.elasticsearch.ElasticsearchClient.mget(ElasticsearchClient.java:1279)
at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:88)
at abc.cde.execution.MultiGetExecutorImpl$1.call(MultiGetExecutorImpl.java:74)
at abc.cde.client.ESClientService.execute(ESClientService.java:66)
at abc.cde.execution.MultiGetExecutorImpl.execute(MultiGetExecutorImpl.java:104)
at abc.cde.impl.DefaultProductInfoService.getBatchExpandedProducts(DefaultProductInfoService.java:263)
at abc.cde.guice.modules.ProductInfoServiceModule.loadProducts(ProductInfoServiceModule.java:154)
at abc.cde.guice.modules.ProductInfoServiceModule.access$200(ProductInfoServiceModule.java:38)
at abc.cde.guice.modules.ProductInfoServiceModule$3.loadAll(ProductInfoServiceModule.java:103)
at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4073)
at com.google.common.cache.LocalCache.getAll(LocalCache.java:4036)
at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4964)
at abc.cde.request.env.Stats.getStatsBlock(StatsBackend.java:156)
at abc.cde.request.env.Stats.getStatsTaskResult(Stats.java:326)
at abc.cde.request.env.Stats.lambda$execMultiThreadedStatsTasks$1(Stats.java:279)
at abc.cde.request.env.Stats$$Lambda$lambda$execMultiThreadedStatsTasks$1$81078421/0x000000000000437b.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2038)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:277)
at abc.cde.request.env.Stats.execMultiThreadedStatsTasks(Stats.java:285)
at abc.cde.request.env.Stats.calculateStats(Stats.java:168)
at abc.cde.request.env.SingleResourceEnvRequestHandler.handle(SingleResourceEnvRequestHandler.java:96)
at abc.cde.request.env.EnvRequestHandler.handle(EnvRequestHandler.java:44)
at abc.cde.resource.Resource.doHandleApiQueryRequest(Resource.java:163)
at abc.cde.resource.Resource.handleApiQueryRequest(Resource.java:116)
at abc.cde.resource.Resource.query(Resource.java:99)
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$lambda$static$0$3446075016/0x0000000000003632.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at abc.cde.jersey.filters.MaxSimultaneousRequestsFilter.doFilter(MaxSimultaneousRequestsFilter.java:222)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel$$Lambda$lambda$handle$1$2621999213/0x00000000000037ad.dispatch(Unknown Source)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:807)
replied in a new thread but the system is flagging that as a spam as it having more number of threads in the conversations, updated the stack trace in the previous thread as well
Right so this is a thread in a Jetty pool, nothing to do with Elasticsearch
1 Like
Thank you for the help, we are able to figure out the issue.
we have an NetworkLoadBalancer(NLB) in front of the cluster, when we try to create a connection to the NLB, the client always connects to one IP of the NLB. That is why its not able to spin more threads(after a limit) and everything else is blocked.
Is there any way we can configure the client to pick all the IPs of the NLB.
The reason we are adding a NLB in front of our clusters is because we wanted to connect to multiple clusters at a time.
If you configure the client with multiple IP addresses then it will use all of them.
is there any recommended way to use the client, such as use a load balancer in front of the cluster or configure all the node IPs which might cause problem when a node gets replaced(could this be resolved by sniffer)?
All of your suggestions seem reasonable.