TransportClient - how to control number of threads - not answered anywhere

Elastic Stack Elasticsearch
1

Hi,
Is there a way to specify the number of threads that TransportClient can use in its netty connection pool? We see hundreds and hundreds of threads for 10 clients, and would like to decrease it.

I found that this question was asked multiple times , in different forums, but was never answered:

ES discussion - no answer on the controlling of the pool size:

StackOverflow Posts - no answers, or no working answers:

thanks,
Marina

2

Thread pools are well documented, see https://www.elastic.co/guide/en/elasticsearch/reference/1.6/modules-threadpool.html

If your problem is that you are having 'hundreds and hundreds of threads' in a single TransportClient, it can not be solved by simply decreasing thread pool. You should create thread dumps of the JVM and examine what threads are in an active or blocked state, and find out the reason of the trouble if the number of blocked threads is high. In most cases, when threads are blocked, the cluster is not running stable, nodes go up and down quickly, cluster states change rapidly, and clients lose connections and try hard to reconnect.

3

Thanks, Jorg, for the quick response.
From the documentation you have linked to - I'm still not clear which of those pools is used, specifically, by the TransportClient client? For example, when I bulk-index documents into ES, using the following code, is the "index" or "bulk" or other pool used?

BulkRequestBuilder bulkRequestBuilder = esClient().prepareBulk();
...
bulkRequestBuilder.add(
      esClient().prepareIndex(
          indexName, 
          indexType)
      .setSource(transformedMessage)
 );

I also agree that my problem might be elsewhere, not the # of threads, and I do see tons of BLOCKED threads in the jstack dump - I just cannot tell what those threads are blocked on easily, and I was wondering if the shear number of them might be causing GC issues, in which case they could be blocked by GC...

Below is an except from my stack trace - just a small one, as there are many more threads, and I cannot see an option of attaching a file to the post.... If you see anything obviously wrong - would appreciate the help!

Thanks,
Marina

Thread 28457: (state = BLOCKED)

  • java.util.Arrays.copyOfRange(char[], int,
    int) @bci=40, line=3664 (Compiled frame)
  • java.lang.String.(char[], int,
    int) @bci=55, line=201 (Compiled frame)
  • java.lang.StringBuilder.toString() @bci=13,
    line=407 (Compiled frame)
  • java.lang.Throwable.toString() @bci=37,
    line=481 (Interpreted frame)

java.lang.Throwable.(java.lang.Throwable) @bci=38, line=311
(Interpreted frame)

java.lang.Exception.(java.lang.Throwable) @bci=2, line=102
(Interpreted frame)

java.util.concurrent.ExecutionException.(java.lang.Throwable)
@bci=2, line=90 (Interpreted frame)

org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue() @bci=47,
line=293 (Compiled frame)

org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=6, line=280
(Compiled frame)

org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=9, line=118
(Compiled frame)

org.elasticsearch.transport.PlainTransportFuture.txGet() @bci=1, line=45
(Compiled frame)

org.elasticsearch.transport.PlainTransportFuture.txGet() @bci=1, line=34
(Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample()
@bci=199, line=350 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample()
@bci=24, line=289 (Compiled frame)

Thread 28416: (state = BLOCKED)

  • java.util.HashMap$KeySet.iterator() @bci=0,
    line=912 (Compiled frame)
  • java.util.HashSet.iterator() @bci=7,
    line=172 (Compiled frame)

org.elasticsearch.client.support.Headers.applyTo(org.elasticsearch.transport.TransportMessage)
@bci=9, line=50 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample()
@bci=163, line=350 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample()
@bci=24, line=289 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run()
@bci=7, line=322 (Compiled frame)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)

java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled
frame)

  • java.lang.Thread.run() @bci=11, line=745
    (Compiled frame)

Thread 28260: (state = BLOCKED)

  • java.lang.Object.wait(long) @bci=0 (Compiled
    frame; information may be imprecise)
  • java.lang.Object.wait(long, int) @bci=58, line=460 (Interpreted frame)

org.elasticsearch.common.netty.channel.DefaultChannelFuture.await0(long,
boolean) @bci=108, line=289 (Compiled frame)

org.elasticsearch.common.netty.channel.DefaultChannelFuture.awaitUninterruptibly(long)
@bci=9, line=263 (Compiled frame)

org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(org.elasticsearch.cluster.node.DiscoveryNode)
@bci=34, line=760 (Interpreted frame)

org.elasticsearch.transport.netty.NettyTransport.connectToNode(org.elasticsearch.cluster.node.DiscoveryNode,
boolean) @bci=133, line=727 (Interpreted frame)

org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(org.elasticsearch.cluster.node.DiscoveryNode)
@bci=3, line=699 (Compiled frame)

org.elasticsearch.transport.TransportService.connectToNodeLight(org.elasticsearch.cluster.node.DiscoveryNode)
@bci=5, line=220 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample()
@bci=94, line=343 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample()
@bci=24, line=289 (Compiled frame)

org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run()
@bci=7, line=322 (Compiled frame)