Node spikes to 1000 threads and hangs, once or twice a day. Help?

Elastic Stack Elasticsearch
1

Odd behavior - our 5-node cluster hums along happily but then, once or
twice a day, one node pops to all 1000 threads (the default limit) being
hit and the node becomes unresponsive, causing our whole cluster to become
extremely slow.

Has anyone experienced this? Any good way to diagnose this?

Memory and CPU appear normal, if that helps... I'm not even sure where to
start here.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a92150b-ea74-410a-a5e1-8c9de24c461b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Check the query logs. Maybe your site is crawled and you do not use
robots.txt

There is no default limit of 1000 threads, what are you talking about?

Jörg

On Sat, Nov 22, 2014 at 6:51 AM, Christopher Ambler <
const.dogberry@gmail.com> wrote:

Odd behavior - our 5-node cluster hums along happily but then, once or
twice a day, one node pops to all 1000 threads (the default limit) being
hit and the node becomes unresponsive, causing our whole cluster to become
extremely slow.

Has anyone experienced this? Any good way to diagnose this?

Memory and CPU appear normal, if that helps... I'm not even sure where to
start here.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a92150b-ea74-410a-a5e1-8c9de24c461b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a92150b-ea74-410a-a5e1-8c9de24c461b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGH0JQAFp2YpTN-hHYmP3GV%3DqrXnts%3DNkfmOeDdnJcv2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi,

Look at query rates and see if they correlate. I'm guessing they jumped,
too. SPM http://sematext.com/spm will help with that. Once you confirm
you can trace the source of queries further upstream.

Otis

Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

On Saturday, November 22, 2014 12:51:31 AM UTC-5, Christopher Ambler wrote:

Odd behavior - our 5-node cluster hums along happily but then, once or
twice a day, one node pops to all 1000 threads (the default limit) being
hit and the node becomes unresponsive, causing our whole cluster to become
extremely slow.

Has anyone experienced this? Any good way to diagnose this?

Memory and CPU appear normal, if that helps... I'm not even sure where to
start here.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e9816ce-63da-4143-a1fa-7f939d4fc657%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.