# What do these metrics mean? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 11, 2014, 8:19pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613 "2014-07-11T20:19:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [July 11, 2014, 8:19pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/1 "2014-07-11T20:19:17Z")
I should probably preface everything with I'm running a 5 node cluster with version 0.19.3 and should be up to version 1.1.2 by the middle of August, but I have some confusion around metrics I'm seeing, what they mean and what are good values. In thread\_pools I see threads, active and queued. Queued + active != threads. I assume this really is a work pool and you have active threads, a thread count in the work pool and queued work. So some explanation around this would be nice. I've correlated some spikes in search threads with heap mem utilization explosions. current searches sort of also correlate, but I have more current searches than search threads and there is not search threadpool queueing. I'm not sure how current searches correlate (or if they should/do) with search threads. I've observed the following: Devestating: 10,000 current searches on worst index sustained over hours with not much change ending at the same time as spikes of \> 1000 search threads (where we generally average \< 50) and a heap explosion. Oddly OK: current searches averaging 15 on worst index spiking to 105 with search threads averaging 50 with maxes of 300 spiking to averages of 120 and maxes of \> 1000 So... I guess, what are good ranges for search threads and current searches? --Shannon Monasco -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 11, 2014, 8:52pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/2 "2014-07-11T20:52:55Z")
Your second paragraph is correct. The threads are the total number of search threads at your disposal, active is the number of ongoing threads and queue are the number of threads that cannot be run since your thread pool is exhausted, which should be when active == threads, but not always the case. The default number of search threads is based upon the number of processors (3x # of available processors). There is no good metric for determining a balance since searches can be either lightweight (milliseconds) or heavyweight (minutes), but I would argue that the key metric to monitor is your queue. Is it normally empty? Spiky behavior? Requests constantly queued? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Cheers, Ivan On Fri, Jul 11, 2014 at 1:19 PM, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > I should probably preface everything with I'm running a 5 node cluster > with version 0.19.3 and should be up to version 1.1.2 by the middle of > August, but I have some confusion around metrics I'm seeing, what they mean > and what are good values. > > In thread\_pools I see threads, active and queued. Queued + active != > threads. I assume this really is a work pool and you have active threads, > a thread count in the work pool and queued work. So some explanation > around this would be nice. > > I've correlated some spikes in search threads with heap mem utilization > explosions. current searches sort of also correlate, but I have more > current searches than search threads and there is not search threadpool > queueing. > > I'm not sure how current searches correlate (or if they should/do) with > search threads. > > I've observed the following: > > Devestating: 10,000 current searches on worst index sustained over hours > with not much change ending at the same time as spikes of \> 1000 search > threads (where we generally average \< 50) and a heap explosion. > > Oddly OK: current searches averaging 15 on worst index spiking to 105 with > search threads averaging 50 with maxes of 300 spiking to averages of 120 > and maxes of \> 1000 > > So... I guess, what are good ranges for search threads and current > searches? > > --Shannon Monasco > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [July 11, 2014, 10:09pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/3 "2014-07-11T22:09:49Z")
I have never seen any queuing on search threads. Not sure if in .19 it defaults to cache or not but that's the behavior I see. What about current queries from indices stats? On Jul 11, 2014 2:53 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Your second paragraph is correct. The threads are the total number of > search threads at your disposal, active is the number of ongoing threads > and queue are the number of threads that cannot be run since your thread > pool is exhausted, which should be when active == threads, but not always > the case. > > The default number of search threads is based upon the number of > processors (3x # of available processors). There is no good metric for > determining a balance since searches can be either lightweight > (milliseconds) or heavyweight (minutes), but I would argue that the key > metric to monitor is your queue. Is it normally empty? Spiky behavior? > Requests constantly queued? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html) > > Cheers, > > Ivan > > On Fri, Jul 11, 2014 at 1:19 PM, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > > > I should probably preface everything with I'm running a 5 node cluster > > with version 0.19.3 and should be up to version 1.1.2 by the middle of > > August, but I have some confusion around metrics I'm seeing, what they mean > > and what are good values. > > > > In thread\_pools I see threads, active and queued. Queued + active != > > threads. I assume this really is a work pool and you have active threads, > > a thread count in the work pool and queued work. So some explanation > > around this would be nice. > > > > I've correlated some spikes in search threads with heap mem utilization > > explosions. current searches sort of also correlate, but I have more > > current searches than search threads and there is not search threadpool > > queueing. > > > > I'm not sure how current searches correlate (or if they should/do) with > > search threads. > > > > I've observed the following: > > > > Devestating: 10,000 current searches on worst index sustained over hours > > with not much change ending at the same time as spikes of \> 1000 search > > threads (where we generally average \< 50) and a heap explosion. > > > > Oddly OK: current searches averaging 15 on worst index spiking to 105 > > with search threads averaging 50 with maxes of 300 spiking to averages of > > 120 and maxes of \> 1000 > > > > So... I guess, what are good ranges for search threads and current > > searches? > > > > --Shannon Monasco > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 11, 2014, 11:56pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/4 "2014-07-11T23:56:36Z")
The default in 0.90 (not sure about 0.19.) should still be a fixed search thread pool: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. I find that the current queries and active threads tends to be the same number. When I look at my graphs, they have the same movements. If you do not have any queued requests you might want to set a lower cap if your cluster is experiencing slowness. BTW, the best course of action that you can take is to simply upgrade and not worry about thread settings. The Lucene 4 improvements (Elasticsearch 0.90 I believe) were monumental in the space savings. New versions will offer tons of other improvements, but the 0.90 release was especially great! Cheers, Ivan On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > I have never seen any queuing on search threads. Not sure if in .19 it > defaults to cache or not but that's the behavior I see. > > What about current queries from indices stats? > On Jul 11, 2014 2:53 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > Your second paragraph is correct. The threads are the total number of > > search threads at your disposal, active is the number of ongoing threads > > and queue are the number of threads that cannot be run since your thread > > pool is exhausted, which should be when active == threads, but not always > > the case. > > > > The default number of search threads is based upon the number of > > processors (3x # of available processors). There is no good metric for > > determining a balance since searches can be either lightweight > > (milliseconds) or heavyweight (minutes), but I would argue that the key > > metric to monitor is your queue. Is it normally empty? Spiky behavior? > > Requests constantly queued? > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html) > > > > Cheers, > > > > Ivan > > > > On Fri, Jul 11, 2014 at 1:19 PM, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > > > > > I should probably preface everything with I'm running a 5 node cluster > > > with version 0.19.3 and should be up to version 1.1.2 by the middle of > > > August, but I have some confusion around metrics I'm seeing, what they mean > > > and what are good values. > > > > > > In thread\_pools I see threads, active and queued. Queued + active != > > > threads. I assume this really is a work pool and you have active threads, > > > a thread count in the work pool and queued work. So some explanation > > > around this would be nice. > > > > > > I've correlated some spikes in search threads with heap mem utilization > > > explosions. current searches sort of also correlate, but I have more > > > current searches than search threads and there is not search threadpool > > > queueing. > > > > > > I'm not sure how current searches correlate (or if they should/do) with > > > search threads. > > > > > > I've observed the following: > > > > > > Devestating: 10,000 current searches on worst index sustained over hours > > > with not much change ending at the same time as spikes of \> 1000 search > > > threads (where we generally average \< 50) and a heap explosion. > > > > > > Oddly OK: current searches averaging 15 on worst index spiking to 105 > > > with search threads averaging 50 with maxes of 300 spiking to averages of > > > 120 and maxes of \> 1000 > > > > > > So... I guess, what are good ranges for search threads and current > > > searches? > > > > > > --Shannon Monasco > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [July 12, 2014, 3:57am UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/5 "2014-07-12T03:57:52Z")
Thanks Ivan! On Jul 11, 2014 5:56 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > The default in 0.90 (not sure about 0.19.) should still be a fixed search > thread pool: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-threadpool.html) > > I find that the current queries and active threads tends to be the same > number. When I look at my graphs, they have the same movements. > > If you do not have any queued requests you might want to set a lower cap > if your cluster is experiencing slowness. > > BTW, the best course of action that you can take is to simply upgrade and > not worry about thread settings. The Lucene 4 improvements (Elasticsearch > 0.90 I believe) were monumental in the space savings. New versions will > offer tons of other improvements, but the 0.90 release was especially great! > > Cheers, > > Ivan > > On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco [smonasco@gmail.com](mailto:smonasco@gmail.com) > wrote: > > > I have never seen any queuing on search threads. Not sure if in .19 it > > defaults to cache or not but that's the behavior I see. > > > > What about current queries from indices stats? > > On Jul 11, 2014 2:53 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > > > Your second paragraph is correct. The threads are the total number of > > > search threads at your disposal, active is the number of ongoing threads > > > and queue are the number of threads that cannot be run since your thread > > > pool is exhausted, which should be when active == threads, but not always > > > the case. > > > > > > The default number of search threads is based upon the number of > > > processors (3x # of available processors). There is no good metric for > > > determining a balance since searches can be either lightweight > > > (milliseconds) or heavyweight (minutes), but I would argue that the key > > > metric to monitor is your queue. Is it normally empty? Spiky behavior? > > > Requests constantly queued? > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html) > > > > > > Cheers, > > > > > > Ivan > > > > > > On Fri, Jul 11, 2014 at 1:19 PM, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > > > > > > > I should probably preface everything with I'm running a 5 node cluster > > > > with version 0.19.3 and should be up to version 1.1.2 by the middle of > > > > August, but I have some confusion around metrics I'm seeing, what they mean > > > > and what are good values. > > > > > > > > In thread\_pools I see threads, active and queued. Queued + active != > > > > threads. I assume this really is a work pool and you have active threads, > > > > a thread count in the work pool and queued work. So some explanation > > > > around this would be nice. > > > > > > > > I've correlated some spikes in search threads with heap mem utilization > > > > explosions. current searches sort of also correlate, but I have more > > > > current searches than search threads and there is not search threadpool > > > > queueing. > > > > > > > > I'm not sure how current searches correlate (or if they should/do) with > > > > search threads. > > > > > > > > I've observed the following: > > > > > > > > Devestating: 10,000 current searches on worst index sustained over > > > > hours with not much change ending at the same time as spikes of \> 1000 > > > > search threads (where we generally average \< 50) and a heap explosion. > > > > > > > > Oddly OK: current searches averaging 15 on worst index spiking to 105 > > > > with search threads averaging 50 with maxes of 300 spiking to averages of > > > > 120 and maxes of \> 1000 > > > > > > > > So... I guess, what are good ranges for search threads and current > > > > searches? > > > > > > > > --Shannon Monasco > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAFDU5WKEgCs1s7wn2h0oh4R9T13ZpZQYgCu5-BZTiRX36WE0RA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFDU5WKEgCs1s7wn2h0oh4R9T13ZpZQYgCu5-BZTiRX36WE0RA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [July 14, 2014, 4:44pm UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/6 "2014-07-14T16:44:05Z")
So it doesn't appear that the sum of currentqueries = K \* active search threads . In other words they are not proportional. Currentqueries was flat, and search.active jumped 2 orders of magnitude. We have highly varied indexes and searches. Some indexes have only one shard with one replica and some have 40 shards with 1 replica. Some queries are nothing more than a search string with no boosting ordered by a date and with no facets. Others have 6 facets, boosts, interesting analyzers and ordered by score. I'm sure some searches use more resources than others, but do they expand to multiple threads? I would imagine one could easily expand this out at the shard (or maybe segment) level. I could also imagine that facets could get their own threads once the reduced document set was determined. --Shannon Monasco P.S. If you don't know about version 0.19.3, but know about .90+ or even 1.x that's fine. I'll take answers I can get. On Friday, July 11, 2014 9:58:03 PM UTC-6, smonasco wrote: > Thanks Ivan! > On Jul 11, 2014 5:56 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > The default in 0.90 (not sure about 0.19.) should still be a fixed search > > thread pool: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-threadpool.html) > > > > I find that the current queries and active threads tends to be the same > > number. When I look at my graphs, they have the same movements. > > > > If you do not have any queued requests you might want to set a lower cap > > if your cluster is experiencing slowness. > > > > BTW, the best course of action that you can take is to simply upgrade and > > not worry about thread settings. The Lucene 4 improvements (Elasticsearch > > 0.90 I believe) were monumental in the space savings. New versions will > > offer tons of other improvements, but the 0.90 release was especially great! > > > > Cheers, > > > > Ivan > > > > On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco [smonasco@gmail.com](mailto:smonasco@gmail.com) > > wrote: > > > > > I have never seen any queuing on search threads. Not sure if in .19 it > > > defaults to cache or not but that's the behavior I see. > > > > > > What about current queries from indices stats? > > > On Jul 11, 2014 2:53 PM, "Ivan Brusic" [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > > > > > Your second paragraph is correct. The threads are the total number of > > > > search threads at your disposal, active is the number of ongoing threads > > > > and queue are the number of threads that cannot be run since your thread > > > > pool is exhausted, which should be when active == threads, but not always > > > > the case. > > > > > > > > The default number of search threads is based upon the number of > > > > processors (3x # of available processors). There is no good metric for > > > > determining a balance since searches can be either lightweight > > > > (milliseconds) or heavyweight (minutes), but I would argue that the key > > > > metric to monitor is your queue. Is it normally empty? Spiky behavior? > > > > Requests constantly queued? > > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html) > > > > > > > > Cheers, > > > > > > > > Ivan > > > > > > > > On Fri, Jul 11, 2014 at 1:19 PM, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote: > > > > > > > > > I should probably preface everything with I'm running a 5 node cluster > > > > > with version 0.19.3 and should be up to version 1.1.2 by the middle of > > > > > August, but I have some confusion around metrics I'm seeing, what they mean > > > > > and what are good values. > > > > > > > > > > In thread\_pools I see threads, active and queued. Queued + active != > > > > > threads. I assume this really is a work pool and you have active threads, > > > > > a thread count in the work pool and queued work. So some explanation > > > > > around this would be nice. > > > > > > > > > > I've correlated some spikes in search threads with heap mem > > > > > utilization explosions. current searches sort of also correlate, but I > > > > > have more current searches than search threads and there is not search > > > > > threadpool queueing. > > > > > > > > > > I'm not sure how current searches correlate (or if they should/do) > > > > > with search threads. > > > > > > > > > > I've observed the following: > > > > > > > > > > Devestating: 10,000 current searches on worst index sustained over > > > > > hours with not much change ending at the same time as spikes of \> 1000 > > > > > search threads (where we generally average \< 50) and a heap explosion. > > > > > > > > > > Oddly OK: current searches averaging 15 on worst index spiking to 105 > > > > > with search threads averaging 50 with maxes of 300 spiking to averages of > > > > > 120 and maxes of \> 1000 > > > > > > > > > > So... I guess, what are good ranges for search threads and current > > > > > searches? > > > > > > > > > > --Shannon Monasco > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8723911c-f764-4286-9f52-7750273a7610%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w\_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCMhHOKnX4LFSF5KsQk24CJj6u23w_VyXw%3D2NbOAhjEow%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAFDU5W%2BNEi9W5FmjuLsxAQzgQNU2p-LMh%2BV5i3x%2B%2B%3D8cgDHdPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/d9V58pThwWY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAgVoyQJo%3D-LRhpWq3bqLP1-MC5dOa1JYPETbTa3eNFLQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e4d4bf22-e57c-4520-9418-23d2c0e7b56e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e4d4bf22-e57c-4520-9418-23d2c0e7b56e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/what-do-these-metrics-mean/18613/7 "2017-07-06T01:16:00Z")