# Low priority queries or query throttling?

**URL:** <https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366>\
**Category:** Elasticsearch\
**Created:** [March 13, 2014, 9:15pm UTC](https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366 "2014-03-13T21:15:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Peter\_Wright](https://avatars.discourse-cdn.com/v4/letter/p/8c91f0/32.png) [@Peter\_Wright](https://discuss.elastic.co/u/Peter_Wright)\
**Post date:** [March 13, 2014, 9:15pm UTC](https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366/1 "2014-03-13T21:15:58Z")

</div>

Hi,  
I am currently having trouble with fairly slow and intensive queries  
causing excessive load on my elasticsearch cluster and I would like to know  
people's opinions on ways to mitigate or prevent that excessive load.

We attempt about 50 of these slow queries per second, and they take an  
average of 300ms which adds up to more than we can process,, causing  
excessive load and sometimes causing elasticsearch to become non-responsive.

The slow queries are all low priority, and we have other, high priority  
queries running on the index. Slow queries could take 10 seconds to process  
for all we care, and we'd rather have them fail than cause excessive load  
on the cluster.

Is there a way to give these queries a lower priority and to force them to  
use no more than a certain percentage of the cluster's resources? Or is it  
possible to refuse certain types of queries if elasticsearch is under  
excessive load?

I am also curious if people have thoughts on what could improve the  
throughput of these queries based on the information given below. I can  
give more details about the structure of the queries themselves if  
necessary.

The cluster is made of two machines (both have 16 CPU cores and let  
elasticsearch use 15G of memory) running ES 0.90.12 with the G1 garbage  
collector. I began experiencing higher load with this setup than before  
when I upgraded to 0.90.12 from 0.90.0 beta (which was using the CMS  
collector with the default settings), however, several other changes were  
made at the same time and it isn't yet fully clear whether either the  
change of version or the change of GC is responsible, or whether it was  
simply a coincidence. Some thoughts on that would be appreciated.

The index has 5 shards and one replica (both machines each have a version  
of all shards), is a couple gigs in size and contains a couple million  
documents.

Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/38e8c947-7dff-4eab-9cc0-b492e20a769e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/38e8c947-7dff-4eab-9cc0-b492e20a769e%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [March 14, 2014, 12:29am UTC](https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366/2 "2014-03-14T00:29:20Z")

</div>

What's the nature of the queries? There may be some optimizations that can  
be made.

How much memory is on the box total?

I would not recommend G1 GC. It is promising but we still see bug reports  
where G1 just straight up crashes. For now, the official ES recommendation  
is still CMS. FWIW, G1 will use more CPU than CMS by definition, because  
of the way G1 operates (e.g. shorter pauses at the cost of more CPU). That  
could partially explain your increased load.

There is currently no way to give a priority to queries, although I agree  
that would be very nice. There are some tricks you can do to control where  
queries go using search preferences (see  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-preference.html)).  
For example, you could send all "slow" queries to a single node, and send  
all other queries to the rest of your cluster. That would effectively  
bottleneck the slow queries, assuming the "slow node" has all the shards  
required to execute the query.

Similarly, you can use Allocation Awareness and Forced Zones to control  
which indices end up on which shards, etc.

On Thursday, March 13, 2014 5:15:58 PM UTC-4, Peter Wright wrote:

> Hi,  
> I am currently having trouble with fairly slow and intensive queries  
> causing excessive load on my elasticsearch cluster and I would like to  
> know people's opinions on ways to mitigate or prevent that excessive load.
> 
> We attempt about 50 of these slow queries per second, and they take an  
> average of 300ms which adds up to more than we can process,, causing  
> excessive load and sometimes causing elasticsearch to become  
> non-responsive.
> 
> The slow queries are all low priority, and we have other, high priority  
> queries running on the index. Slow queries could take 10 seconds to process  
> for all we care, and we'd rather have them fail than cause excessive load  
> on the cluster.
> 
> Is there a way to give these queries a lower priority and to force them to  
> use no more than a certain percentage of the cluster's resources? Or is it  
> possible to refuse certain types of queries if elasticsearch is under  
> excessive load?
> 
> I am also curious if people have thoughts on what could improve the  
> throughput of these queries based on the information given below. I can  
> give more details about the structure of the queries themselves if  
> necessary.
> 
> The cluster is made of two machines (both have 16 CPU cores and let  
> elasticsearch use 15G of memory) running ES 0.90.12 with the G1 garbage  
> collector. I began experiencing higher load with this setup than before  
> when I upgraded to 0.90.12 from 0.90.0 beta (which was using the CMS  
> collector with the default settings), however, several other changes were  
> made at the same time and it isn't yet fully clear whether either the  
> change of version or the change of GC is responsible, or whether it was  
> simply a coincidence. Some thoughts on that would be appreciated.
> 
> The index has 5 shards and one replica (both machines each have a version  
> of all shards), is a couple gigs in size and contains a couple million  
> documents.
> 
> Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [March 14, 2014, 10:34am UTC](https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366/3 "2014-03-14T10:34:39Z")

</div>

Adding to what Zach said, I'd also be interested in looking at what causes  
these queries to be so slow. Potentially their performance could be greatly  
improved.

clint

On 14 March 2014 01:29, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) wrote:

> What's the nature of the queries? There may be some optimizations that  
> can be made.
> 
> How much memory is on the box total?
> 
> I would not recommend G1 GC. It is promising but we still see bug reports  
> where G1 just straight up crashes. For now, the official ES recommendation  
> is still CMS. FWIW, G1 will use more CPU than CMS by definition, because  
> of the way G1 operates (e.g. shorter pauses at the cost of more CPU). That  
> could partially explain your increased load.
> 
> There is currently no way to give a priority to queries, although I agree  
> that would be very nice. There are some tricks you can do to control where  
> queries go using search preferences (see  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-preference.html)).  
> For example, you could send all "slow" queries to a single node, and send  
> all other queries to the rest of your cluster. That would effectively  
> bottleneck the slow queries, assuming the "slow node" has all the shards  
> required to execute the query.
> 
> Similarly, you can use Allocation Awareness and Forced Zones to control  
> which indices end up on which shards, etc.
> 
> On Thursday, March 13, 2014 5:15:58 PM UTC-4, Peter Wright wrote:
> 
> > Hi,  
> > I am currently having trouble with fairly slow and intensive queries  
> > causing excessive load on my elasticsearch cluster and I would like to  
> > know people's opinions on ways to mitigate or prevent that excessive load.
> > 
> > We attempt about 50 of these slow queries per second, and they take an  
> > average of 300ms which adds up to more than we can process,, causing  
> > excessive load and sometimes causing elasticsearch to become  
> > non-responsive.
> > 
> > The slow queries are all low priority, and we have other, high priority  
> > queries running on the index. Slow queries could take 10 seconds to process  
> > for all we care, and we'd rather have them fail than cause excessive load  
> > on the cluster.
> > 
> > Is there a way to give these queries a lower priority and to force them  
> > to use no more than a certain percentage of the cluster's resources? Or is  
> > it possible to refuse certain types of queries if elasticsearch is under  
> > excessive load?
> > 
> > I am also curious if people have thoughts on what could improve the  
> > throughput of these queries based on the information given below. I can  
> > give more details about the structure of the queries themselves if  
> > necessary.
> > 
> > The cluster is made of two machines (both have 16 CPU cores and let  
> > elasticsearch use 15G of memory) running ES 0.90.12 with the G1 garbage  
> > collector. I began experiencing higher load with this setup than before  
> > when I upgraded to 0.90.12 from 0.90.0 beta (which was using the CMS  
> > collector with the default settings), however, several other changes were  
> > made at the same time and it isn't yet fully clear whether either the  
> > change of version or the change of GC is responsible, or whether it was  
> > simply a coincidence. Some thoughts on that would be appreciated.
> > 
> > The index has 5 shards and one replica (both machines each have a  
> > version of all shards), is a couple gigs in size and contains a couple  
> > million documents.
> > 
> > Thanks!
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a13d004a-e180-43ad-90c1-132bd05bfdfa%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQTwVo-aRE2SA6xy5fzJ6tkOHOAckq4zCMPAr4D08t4rw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQTwVo-aRE2SA6xy5fzJ6tkOHOAckq4zCMPAr4D08t4rw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:43am UTC](https://discuss.elastic.co/t/low-priority-queries-or-query-throttling/16366/4 "2017-07-06T01:43:10Z")

</div>