Huge number of 'elasticsearch[cached]-pool-1-thread-NNN' threads

Elastic Stack Elasticsearch
1

While helping someone debug a logstash issue tonight, we noticed that for an
elasticsearch native java client had 1700 elasticsearch threads. After
further digging, I concluded that the high thread count for elasticsearch,
while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort|
uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of
them?

Are they pending queries? Something else?

-Jordan

2

Which version of elasticsearch are you using? The thread pool names changed between versions.
On Sunday, May 8, 2011 at 9:53 AM, Jordan Sissel wrote:

While helping someone debug a logstash issue tonight, we noticed that for an elasticsearch native java client had 1700 elasticsearch threads. After further digging, I concluded that the high thread count for elasticsearch, while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort| uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of them?

Are they pending queries? Something else?

-Jordan

3

On Sat, May 7, 2011 at 11:56 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Which version of elasticsearch are you using? The thread pool names
changed between versions.

The client I shipped should be version 0.16.0

-Jordan

On Sunday, May 8, 2011 at 9:53 AM, Jordan Sissel wrote:

While helping someone debug a logstash issue tonight, we noticed that for
an elasticsearch native java client had 1700 elasticsearch threads. After
further digging, I concluded that the high thread count for elasticsearch,
while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort|
uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of
them?

Are they pending queries? Something else?

-Jordan

4

Then thats strange, I would expect to see other elasticsearch threads in this case. Which client are you using in this case, the NodeClient or the TransportClient?
On Monday, May 9, 2011 at 3:29 AM, Jordan Sissel wrote:

On Sat, May 7, 2011 at 11:56 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Which version of elasticsearch are you using? The thread pool names changed between versions.

The client I shipped should be version 0.16.0

-Jordan

On Sunday, May 8, 2011 at 9:53 AM, Jordan Sissel wrote:

While helping someone debug a logstash issue tonight, we noticed that for an elasticsearch native java client had 1700 elasticsearch threads. After further digging, I concluded that the high thread count for elasticsearch, while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort| uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of them?

Are they pending queries? Something else?

-Jordan

5

On Mon, May 9, 2011 at 1:48 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Then thats strange, I would expect to see other elasticsearch threads in
this case. Which client are you using in this case, the NodeClient or the
TransportClient?

I believe the 'node' client? Though I don't know for sure, here's the code
that creates it:

org.elasticsearch.node.NodeBuilder.nodeBuilder.client(true).node.client

-Jordan

On Monday, May 9, 2011 at 3:29 AM, Jordan Sissel wrote:

On Sat, May 7, 2011 at 11:56 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Which version of elasticsearch are you using? The thread pool names
changed between versions.

The client I shipped should be version 0.16.0

-Jordan

On Sunday, May 8, 2011 at 9:53 AM, Jordan Sissel wrote:

While helping someone debug a logstash issue tonight, we noticed that for
an elasticsearch native java client had 1700 elasticsearch threads. After
further digging, I concluded that the high thread count for elasticsearch,
while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort|
uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of
them?

Are they pending queries? Something else?

-Jordan

6

Yea, thats the node client. strange... . I will try and run some tests to see if I can repro, if you can do the same, it would be great...
On Monday, May 9, 2011 at 11:11 PM, Jordan Sissel wrote:

On Mon, May 9, 2011 at 1:48 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Then thats strange, I would expect to see other elasticsearch threads in this case. Which client are you using in this case, the NodeClient or the TransportClient?

I believe the 'node' client? Though I don't know for sure, here's the code that creates it:

org.elasticsearch.node.NodeBuilder.nodeBuilder.client(true).node.client

-Jordan

On Monday, May 9, 2011 at 3:29 AM, Jordan Sissel wrote:

On Sat, May 7, 2011 at 11:56 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Which version of elasticsearch are you using? The thread pool names changed between versions.

The client I shipped should be version 0.16.0

-Jordan

On Sunday, May 8, 2011 at 9:53 AM, Jordan Sissel wrote:

While helping someone debug a logstash issue tonight, we noticed that for an elasticsearch native java client had 1700 elasticsearch threads. After further digging, I concluded that the high thread count for elasticsearch, while abnormal, was not related to the problem I was debugging.

Emailing the list anyway to get some details about what is going on -

Here's a breakdown of elasticsearch threads according to jstack:

% grep '^"' logstash.jstack | fex '"1' | sed -re 's/[0-9]+/XXX/g' |sort| uniq -c | sort -n | grep elastic
1 elasticsearch[Artemis]clusterService#updateTask-pool-XXX-thread-XXX
1 elasticsearch[Artemis][scheduler]-pool-XXX-thread-XXX
1 elasticsearch[Artemis][timer]
1 elasticsearch[Artemis]transport_client_boss-pool-XXX-thread-XXX
6 elasticsearch[Artemis]transport_client_worker-pool-XXX-thread-XXX
1694 elasticsearch[cached]-pool-XXX-thread-XXX

What are these 'cached' pool threads? Why would there be almost 1700 of them?

Are they pending queries? Something else?

-Jordan