I should probably preface everything with I'm running a 5 node cluster with
version 0.19.3 and should be up to version 1.1.2 by the middle of August,
but I have some confusion around metrics I'm seeing, what they mean and
what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem utilization
explosions. current searches sort of also correlate, but I have more
current searches than search threads and there is not search threadpool
queueing.
I'm not sure how current searches correlate (or if they should/do) with
search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over hours
with not much change ending at the same time as spikes of > 1000 search
threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105 with
search threads averaging 50 with maxes of 300 spiking to averages of 120
and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
Your second paragraph is correct. The threads are the total number of
search threads at your disposal, active is the number of ongoing threads
and queue are the number of threads that cannot be run since your thread
pool is exhausted, which should be when active == threads, but not always
the case.
The default number of search threads is based upon the number of processors
(3x # of available processors). There is no good metric for determining a
balance since searches can be either lightweight (milliseconds) or
heavyweight (minutes), but I would argue that the key metric to monitor is
your queue. Is it normally empty? Spiky behavior? Requests constantly
queued?
I should probably preface everything with I'm running a 5 node cluster
with version 0.19.3 and should be up to version 1.1.2 by the middle of
August, but I have some confusion around metrics I'm seeing, what they mean
and what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem utilization
explosions. current searches sort of also correlate, but I have more
current searches than search threads and there is not search threadpool
queueing.
I'm not sure how current searches correlate (or if they should/do) with
search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over hours
with not much change ending at the same time as spikes of > 1000 search
threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105 with
search threads averaging 50 with maxes of 300 spiking to averages of 120
and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
I have never seen any queuing on search threads. Not sure if in .19 it
defaults to cache or not but that's the behavior I see.
What about current queries from indices stats?
On Jul 11, 2014 2:53 PM, "Ivan Brusic" ivan@brusic.com wrote:
Your second paragraph is correct. The threads are the total number of
search threads at your disposal, active is the number of ongoing threads
and queue are the number of threads that cannot be run since your thread
pool is exhausted, which should be when active == threads, but not always
the case.
The default number of search threads is based upon the number of
processors (3x # of available processors). There is no good metric for
determining a balance since searches can be either lightweight
(milliseconds) or heavyweight (minutes), but I would argue that the key
metric to monitor is your queue. Is it normally empty? Spiky behavior?
Requests constantly queued?
I should probably preface everything with I'm running a 5 node cluster
with version 0.19.3 and should be up to version 1.1.2 by the middle of
August, but I have some confusion around metrics I'm seeing, what they mean
and what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem utilization
explosions. current searches sort of also correlate, but I have more
current searches than search threads and there is not search threadpool
queueing.
I'm not sure how current searches correlate (or if they should/do) with
search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over hours
with not much change ending at the same time as spikes of > 1000 search
threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105
with search threads averaging 50 with maxes of 300 spiking to averages of
120 and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
The default in 0.90 (not sure about 0.19.) should still be a fixed search
thread pool:
I find that the current queries and active threads tends to be the same
number. When I look at my graphs, they have the same movements.
If you do not have any queued requests you might want to set a lower cap if
your cluster is experiencing slowness.
BTW, the best course of action that you can take is to simply upgrade and
not worry about thread settings. The Lucene 4 improvements (Elasticsearch
0.90 I believe) were monumental in the space savings. New versions will
offer tons of other improvements, but the 0.90 release was especially great!
Cheers,
Ivan
On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco smonasco@gmail.com wrote:
I have never seen any queuing on search threads. Not sure if in .19 it
defaults to cache or not but that's the behavior I see.
What about current queries from indices stats?
On Jul 11, 2014 2:53 PM, "Ivan Brusic" ivan@brusic.com wrote:
Your second paragraph is correct. The threads are the total number of
search threads at your disposal, active is the number of ongoing threads
and queue are the number of threads that cannot be run since your thread
pool is exhausted, which should be when active == threads, but not always
the case.
The default number of search threads is based upon the number of
processors (3x # of available processors). There is no good metric for
determining a balance since searches can be either lightweight
(milliseconds) or heavyweight (minutes), but I would argue that the key
metric to monitor is your queue. Is it normally empty? Spiky behavior?
Requests constantly queued?
I should probably preface everything with I'm running a 5 node cluster
with version 0.19.3 and should be up to version 1.1.2 by the middle of
August, but I have some confusion around metrics I'm seeing, what they mean
and what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem utilization
explosions. current searches sort of also correlate, but I have more
current searches than search threads and there is not search threadpool
queueing.
I'm not sure how current searches correlate (or if they should/do) with
search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over hours
with not much change ending at the same time as spikes of > 1000 search
threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105
with search threads averaging 50 with maxes of 300 spiking to averages of
120 and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
I find that the current queries and active threads tends to be the same
number. When I look at my graphs, they have the same movements.
If you do not have any queued requests you might want to set a lower cap
if your cluster is experiencing slowness.
BTW, the best course of action that you can take is to simply upgrade and
not worry about thread settings. The Lucene 4 improvements (Elasticsearch
0.90 I believe) were monumental in the space savings. New versions will
offer tons of other improvements, but the 0.90 release was especially great!
Cheers,
Ivan
On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco smonasco@gmail.com
wrote:
I have never seen any queuing on search threads. Not sure if in .19 it
defaults to cache or not but that's the behavior I see.
What about current queries from indices stats?
On Jul 11, 2014 2:53 PM, "Ivan Brusic" ivan@brusic.com wrote:
Your second paragraph is correct. The threads are the total number of
search threads at your disposal, active is the number of ongoing threads
and queue are the number of threads that cannot be run since your thread
pool is exhausted, which should be when active == threads, but not always
the case.
The default number of search threads is based upon the number of
processors (3x # of available processors). There is no good metric for
determining a balance since searches can be either lightweight
(milliseconds) or heavyweight (minutes), but I would argue that the key
metric to monitor is your queue. Is it normally empty? Spiky behavior?
Requests constantly queued?
I should probably preface everything with I'm running a 5 node cluster
with version 0.19.3 and should be up to version 1.1.2 by the middle of
August, but I have some confusion around metrics I'm seeing, what they mean
and what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem utilization
explosions. current searches sort of also correlate, but I have more
current searches than search threads and there is not search threadpool
queueing.
I'm not sure how current searches correlate (or if they should/do) with
search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over
hours with not much change ending at the same time as spikes of > 1000
search threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105
with search threads averaging 50 with maxes of 300 spiking to averages of
120 and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
So it doesn't appear that the sum of currentqueries = K * active search
threads . In other words they are not proportional.
Currentqueries was flat, and search.active jumped 2 orders of magnitude.
We have highly varied indexes and searches. Some indexes have only one
shard with one replica and some have 40 shards with 1 replica. Some
queries are nothing more than a search string with no boosting ordered by a
date and with no facets. Others have 6 facets, boosts, interesting
analyzers and ordered by score.
I'm sure some searches use more resources than others, but do they expand
to multiple threads? I would imagine one could easily expand this out at
the shard (or maybe segment) level. I could also imagine that facets could
get their own threads once the reduced document set was determined.
--Shannon Monasco
P.S. If you don't know about version 0.19.3, but know about .90+ or even
1.x that's fine. I'll take answers I can get.
On Friday, July 11, 2014 9:58:03 PM UTC-6, smonasco wrote:
I find that the current queries and active threads tends to be the same
number. When I look at my graphs, they have the same movements.
If you do not have any queued requests you might want to set a lower cap
if your cluster is experiencing slowness.
BTW, the best course of action that you can take is to simply upgrade and
not worry about thread settings. The Lucene 4 improvements (Elasticsearch
0.90 I believe) were monumental in the space savings. New versions will
offer tons of other improvements, but the 0.90 release was especially great!
Cheers,
Ivan
On Fri, Jul 11, 2014 at 3:09 PM, Shannon Monasco smonasco@gmail.com
wrote:
I have never seen any queuing on search threads. Not sure if in .19 it
defaults to cache or not but that's the behavior I see.
What about current queries from indices stats?
On Jul 11, 2014 2:53 PM, "Ivan Brusic" ivan@brusic.com wrote:
Your second paragraph is correct. The threads are the total number of
search threads at your disposal, active is the number of ongoing threads
and queue are the number of threads that cannot be run since your thread
pool is exhausted, which should be when active == threads, but not always
the case.
The default number of search threads is based upon the number of
processors (3x # of available processors). There is no good metric for
determining a balance since searches can be either lightweight
(milliseconds) or heavyweight (minutes), but I would argue that the key
metric to monitor is your queue. Is it normally empty? Spiky behavior?
Requests constantly queued?
I should probably preface everything with I'm running a 5 node cluster
with version 0.19.3 and should be up to version 1.1.2 by the middle of
August, but I have some confusion around metrics I'm seeing, what they mean
and what are good values.
In thread_pools I see threads, active and queued. Queued + active !=
threads. I assume this really is a work pool and you have active threads,
a thread count in the work pool and queued work. So some explanation
around this would be nice.
I've correlated some spikes in search threads with heap mem
utilization explosions. current searches sort of also correlate, but I
have more current searches than search threads and there is not search
threadpool queueing.
I'm not sure how current searches correlate (or if they should/do)
with search threads.
I've observed the following:
Devestating: 10,000 current searches on worst index sustained over
hours with not much change ending at the same time as spikes of > 1000
search threads (where we generally average < 50) and a heap explosion.
Oddly OK: current searches averaging 15 on worst index spiking to 105
with search threads averaging 50 with maxes of 300 spiking to averages of
120 and maxes of > 1000
So... I guess, what are good ranges for search threads and current
searches?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.