Limit large number of threads

Elastic Stack Elasticsearch
1

Hi all,

I know I'm not the first one wondering about the number of threads but I
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely
(i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain hussain@novacom.mygbiz.com
wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but I
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely
(i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbbvJBv%3DVnE1cOTRuGQFBEBJ9rYBETwO99NzJn3kKN%2BmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for clarification.

Still I wonder why such a huge amount of thread is created and if this can
lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:

Hi,

please note that Bigdesk does not show all internal thread pools in ES now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <hus...@novacom.mygbiz.com
<javascript:>> wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but I
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation
safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

If thread counts go out of bounds, it may be a lockup somewhere. What
version of ES do you use?

Jörg

On Fri, Mar 20, 2015 at 2:08 PM, Abid Hussain hussain@novacom.mygbiz.com
wrote:

Thanks for clarification.

Still I wonder why such a huge amount of thread is created and if this can
lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:

Hi,

please note that Bigdesk does not show all internal thread pools in ES
now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <hus...@novacom.mygbiz.com

wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but I
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation
safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF_OoTYTK7WT1gWhcMkmvy3mqpWtVHZrp3-F8Voqhkahw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Each segment, each network connection and a whole bunch of other things
will add to this.
280 is a very low number and I wouldn't even worry about it.

On 20 March 2015 at 06:08, Abid Hussain hussain@novacom.mygbiz.com wrote:

Thanks for clarification.

Still I wonder why such a huge amount of thread is created and if this can
lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:

Hi,

please note that Bigdesk does not show all internal thread pools in ES
now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <hus...@novacom.mygbiz.com

wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but I
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation
safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uV%3DWf3VAzGuZL6rC_7iQA6-%3DpXrn%2Bn5bT%2BFFuggrCJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

We're using 1.4.2 currently. Are there any issues known with this version?

If there is nothing to worry about 400 threads, it's ok for me. I just
wonder why they never seem to be released.

Am Freitag, 20. März 2015 16:20:03 UTC+1 schrieb Jörg Prante:

If thread counts go out of bounds, it may be a lockup somewhere. What
version of ES do you use?

Jörg

On Fri, Mar 20, 2015 at 2:08 PM, Abid Hussain <hus...@novacom.mygbiz.com
<javascript:>> wrote:

Thanks for clarification.

Still I wonder why such a huge amount of thread is created and if this
can lead to issues - especially as it seems to me that they are never
released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:

Hi,

please note that Bigdesk does not show all internal thread pools in ES
now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <
hus...@novacom.mygbiz.com> wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but
I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation
safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e13d3eb5-f90d-45e3-ab24-901e1b0c284a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Hm, I doubt it is ok if a 1.4.0 node has 195 threads in state BLOCKED:

Thread 19374: (state = BLOCKED)

  • sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
    may be imprecise)
  • java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
    line=175 (Compiled frame)

java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node,
java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean,
long) @bci=184, line=737 (Compiled frame)

  • java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean,
    int, long) @bci=286, line=647 (Compiled frame)
  • java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265
    (Compiled frame)
  • org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take()
    @bci=4, line=162 (Compiled frame)
  • java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
    (Compiled frame)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)

  • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
    (Compiled frame)
  • java.lang.Thread.run() @bci=11, line=745 (Compiled frame)

Jörg

On Fri, Mar 20, 2015 at 4:30 PM, Mark Walkom markwalkom@gmail.com wrote:

Each segment, each network connection and a whole bunch of other things
will add to this.
280 is a very low number and I wouldn't even worry about it.

On 20 March 2015 at 06:08, Abid Hussain hussain@novacom.mygbiz.com
wrote:

Thanks for clarification.

Still I wonder why such a huge amount of thread is created and if this
can lead to issues - especially as it seems to me that they are never
released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:

Hi,

please note that Bigdesk does not show all internal thread pools in ES
now.

Regards,
Lukas

On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <
hus...@novacom.mygbiz.com> wrote:

Hi all,

I know I'm not the first one wondering about the number of threads but
I didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is
actually (according to what bigdesk says):

  • Search 72
  • Index 24
  • Bulk 24
  • Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation
safely (i.e. without compromising cluster functionality)?

Regards,

Abid

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uV%3DWf3VAzGuZL6rC_7iQA6-%3DpXrn%2Bn5bT%2BFFuggrCJw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-uV%3DWf3VAzGuZL6rC_7iQA6-%3DpXrn%2Bn5bT%2BFFuggrCJw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE4t%3DfgaE%3D9SW3dX08U5nUzpXLPi9T0iZsvbEwqfEKdHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

I think you should check a thread dump created by tools like jstack if you
have a high JVM thread count in state BLOCKED. This might be a pointer that
something unusual is going on, but I'm not sure.

Jörg

On Fri, Mar 20, 2015 at 4:41 PM, Abid Hussain hussain@novacom.mygbiz.com
wrote:

We're using 1.4.2 currently. Are there any issues known with this version?

If there is nothing to worry about 400 threads, it's ok for me. I just
wonder why they never seem to be released.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZaDaBTRw_hY9%3DaJmrXCPqfO4i%2B4sOU1Svy1%2BJbHCJNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

9

Thanks Jörg, I did a thread dump: 60 % of ~400 threads are in state
WAITING, 35 % are in state RUNNABLE, the rest is in state TIMED_WAITING,
none is in state BLOCKED.

So I assume everything is OK - still wondering whats the point of creating
hundreds of threads as there are "only" 24 cores available on our machine.

Best regards,

Abid

Am Freitag, 20. März 2015 16:47:28 UTC+1 schrieb Jörg Prante:

I think you should check a thread dump created by tools like jstack if you
have a high JVM thread count in state BLOCKED. This might be a pointer that
something unusual is going on, but I'm not sure.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

ES uses several threadpools. Some are fixed sized, some are scalable, and
the reference is the JVM available core count, i.e.
Runtime.getRuntime().availableProcessors(), which can be overridden by a
"processors" directive:

I have 32 cores per machine so I'm also observing hundreds of idle threads.
I do not use percolators/warmers/snapshot/suggest in my use case. Since
they are not active, they steal a lot of my stack memory. Right now I don't
care about idle threads but this can be a problem for small memory sized
machines.

If you feel safe about reducing thread pool resources, you can decrease
them in your config, e.g. -Des.processors=4. But you should monitor your
performance, if you see bad numbers, go back to default.

Jörg

On Mon, Mar 23, 2015 at 8:48 AM, Abid Hussain hussain@novacom.mygbiz.com
wrote:

Thanks Jörg, I did a thread dump: 60 % of ~400 threads are in state
WAITING, 35 % are in state RUNNABLE, the rest is in state TIMED_WAITING,
none is in state BLOCKED.

So I assume everything is OK - still wondering whats the point of creating
hundreds of threads as there are "only" 24 cores available on our machine.

Best regards,

Abid

Am Freitag, 20. März 2015 16:47:28 UTC+1 schrieb Jörg Prante:

I think you should check a thread dump created by tools like jstack if
you have a high JVM thread count in state BLOCKED. This might be a pointer
that something unusual is going on, but I'm not sure.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF%2B2jrzPH6w%2BHa9v6ixJXOYS8Rdu_%3DMdRc5PVDG6%3DK8-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.