# Shard query cache size
**URL:** https://discuss.elastic.co/t/shard-query-cache-size/23663
**Category:** Elasticsearch
**Created:** [May 21, 2015, 1:41pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663 "2015-05-21T13:41:45Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@msukmanowsky](https://discuss.elastic.co/u/msukmanowsky)
#### Post date: [May 21, 2015, 1:41pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/1 "2015-05-21T13:41:45Z")
Hi all,
We store Marvel-style timeseries data in Elasticsearch and make very heavy
use of aggregations (all queries are effectively aggregations).
We've been playing around with the shard query cache and have a question.
Is there a reason the shard query cache is set to such a low level of JVM
heap by default? 1% seems awfully low unless ES assumes most people aren't
making heavy use of aggregations? Any harm in us significantly boosting
this from 1% to say 15% of heap? Most of our machines have 30GB of RAM and
heap at 50% of that (15GB) so the query cache is 150MB by default. Think
we'd like to experiment growing that to at least 10% of heap to have 1GB in
use for this cache.
Mike
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [May 21, 2015, 2:28pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/2 "2015-05-21T14:28:28Z")
It depends how likely it is for you to run the same aggregation again. Note
that this cache is fully invalidated at every refresh (meaning either every
second by default, or every time that you update/add/remove documents if
you perform less than 1 operation per second). So this cache will only be
used if you are likely to run the exact same request twice or more in a
short period of time.
We assumed that this situation is not common, so a small cache would be
enough for the maybe 4 or 5 requests that would be run again and again. You
can increase it if you think it will be helpful in your case although I
would advise to be careful, maybe memory would be better spent on eg. the
filesystem cache.
On Thu, May 21, 2015 at 3:41 PM, Mike Sukmanowsky \<
[mike.sukmanowsky@gmail.com](mailto:mike.sukmanowsky@gmail.com)\> wrote:
> Hi all,
>
> We store Marvel-style timeseries data in Elasticsearch and make very heavy
> use of aggregations (all queries are effectively aggregations).
>
> We've been playing around with the shard query cache and have a question.
>
> Is there a reason the shard query cache is set to such a low level of JVM
> heap by default? 1% seems awfully low unless ES assumes most people aren't
> making heavy use of aggregations? Any harm in us significantly boosting
> this from 1% to say 15% of heap? Most of our machines have 30GB of RAM and
> heap at 50% of that (15GB) so the query cache is 150MB by default. Think
> we'd like to experiment growing that to at least 10% of heap to have 1GB in
> use for this cache.
>
> Mike
>
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
>
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
Adrien
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@James\_Macdonald](https://discuss.elastic.co/u/James_Macdonald)
#### Post date: [May 21, 2015, 9:49pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/3 "2015-05-21T21:49:39Z")
Hi, I am a little confused by your response. Are you saying that
query/filter caches are invalidated across all data in a shard every time
the refresh interval ticks over?
I was under the impression that all field data and caching related
operations were performed on a Lucene index segment level and that the
caches would only be invalidated for a given segment if that segment had
changed since the last refresh. Since most data is stored in large segments
that don't take fresh writes and seldom merge this would mean that most
caches are good for long periods of time; even if the shard is under
constant indexing load. Am I mistaken?
Thanks,
James
On Thu, May 21, 2015 at 9:28 AM, Adrien Grand [adrien@elastic.co](mailto:adrien@elastic.co) wrote:
> It depends how likely it is for you to run the same aggregation again.
> Note that this cache is fully invalidated at every refresh (meaning either
> every second by default, or every time that you update/add/remove documents
> if you perform less than 1 operation per second). So this cache will only
> be used if you are likely to run the exact same request twice or more in a
> short period of time.
>
> We assumed that this situation is not common, so a small cache would be
> enough for the maybe 4 or 5 requests that would be run again and again. You
> can increase it if you think it will be helpful in your case although I
> would advise to be careful, maybe memory would be better spent on eg. the
> filesystem cache.
>
> On Thu, May 21, 2015 at 3:41 PM, Mike Sukmanowsky \<
> [mike.sukmanowsky@gmail.com](mailto:mike.sukmanowsky@gmail.com)\> wrote:
>
> > Hi all,
> >
> > We store Marvel-style timeseries data in Elasticsearch and make very
> > heavy use of aggregations (all queries are effectively aggregations).
> >
> > We've been playing around with the shard query cache and have a question.
> >
> > Is there a reason the shard query cache is set to such a low level of JVM
> > heap by default? 1% seems awfully low unless ES assumes most people aren't
> > making heavy use of aggregations? Any harm in us significantly boosting
> > this from 1% to say 15% of heap? Most of our machines have 30GB of RAM and
> > heap at 50% of that (15GB) so the query cache is 150MB by default. Think
> > we'd like to experiment growing that to at least 10% of heap to have 1GB in
> > use for this cache.
> >
> > Mike
> >
> > ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
> >
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3a0fa7b-49f8-4d78-a520-6eeb16d53de3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> Adrien
>
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
>
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhjZW%2B7BoLqXOk8nEfOvaKrgUroZhbG41BVUDh2bHhEnQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAABsnTYY85YeKRUBFLoMcAfnPj%3DBP6GqM020N%2BrZ9yp32%2BQY0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAABsnTYY85YeKRUBFLoMcAfnPj%3DBP6GqM020N%2BrZ9yp32%2BQY0A%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [May 21, 2015, 11:12pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/4 "2015-05-21T23:12:37Z")
On Thu, May 21, 2015 at 11:49 PM, James Macdonald \<
[james.macdonald@geofeedia.com](mailto:james.macdonald@geofeedia.com)\> wrote:
> Hi, I am a little confused by your response. Are you saying that
> query/filter caches are invalidated across all data in a shard every time
> the refresh interval ticks over?
Sorry for the confusion:
- the query cache caches entire requests per index, and is competely
invalidated across all data every time the refresh interval ticks over AND
there have been changes since the last refresh
- the filter cache caches matching documents per segment, it is
invalidated per segment only when a segment goes away (typically because
it's been merged to a larger segment), which is unfrequent for large
segments
- the fielddata cache caches the document-\>value mapping per segment and
has the same invalidation rules as the filter cache
> I was under the impression that all field data and caching related
> operations were performed on a Lucene index segment level and that the
> caches would only be invalidated for a given segment if that segment had
> changed since the last refresh. Since most data is stored in large segments
> that don't take fresh writes and seldom merge this would mean that most
> caches are good for long periods of time; even if the shard is under
> constant indexing load. Am I mistaken?
This is right for the fielddata and filter caches, but not for the query
cache.
--
Adrien
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1\_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@msukmanowsky](https://discuss.elastic.co/u/msukmanowsky)
#### Post date: [May 22, 2015, 3:00pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/5 "2015-05-22T15:00:35Z")
Thanks for the responses guys. We have an ES setup so that we have: hot,
warm and cold ES nodes. Hot nodes are the only ones receiving realtime
updates and have refresh intervals fairly low for indices there thus making
a query cache pretty useless for data there.
Indices on warm nodes on the other hand are only updated every night and
indices cold nodes are similar. Assuming we do have repetitive aggregation
queries, sounds like bumping up query cache on warm/cold tier could have
some significant speed ups for our more expensive aggregations.
On Thursday, 21 May 2015 19:12:44 UTC-4, Adrien Grand wrote:
> On Thu, May 21, 2015 at 11:49 PM, James Macdonald \<
> [james.m...@geofeedia.com](mailto:james.m...@geofeedia.com) \\> wrote:
>
> > Hi, I am a little confused by your response. Are you saying that
> > query/filter caches are invalidated across all data in a shard every time
> > the refresh interval ticks over?
>
> Sorry for the confusion:
>
> - the query cache caches entire requests per index, and is competely
> invalidated across all data every time the refresh interval ticks over AND
> there have been changes since the last refresh
> - the filter cache caches matching documents per segment, it is
> invalidated per segment only when a segment goes away (typically because
> it's been merged to a larger segment), which is unfrequent for large
> segments
> - the fielddata cache caches the document-\>value mapping per segment and
> has the same invalidation rules as the filter cache
>
> > I was under the impression that all field data and caching related
> > operations were performed on a Lucene index segment level and that the
> > caches would only be invalidated for a given segment if that segment had
> > changed since the last refresh. Since most data is stored in large segments
> > that don't take fresh writes and seldom merge this would mean that most
> > caches are good for long periods of time; even if the shard is under
> > constant indexing load. Am I mistaken?
>
> This is right for the fielddata and filter caches, but not for the query
> cache.
>
> --
> Adrien
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c9ca4e07-42c4-41d4-a919-50cb9c56f8eb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c9ca4e07-42c4-41d4-a919-50cb9c56f8eb%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@James\_Macdonald](https://discuss.elastic.co/u/James_Macdonald)
#### Post date: [May 23, 2015, 4:12pm UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/6 "2015-05-23T16:12:59Z")
Hi Adrien, thanks very much for this clarification. I am always trying to
learn more about how Elasticsearch works, and that clarification was very
helpful.
James
On Thu, May 21, 2015 at 6:12 PM, Adrien Grand [adrien@elastic.co](mailto:adrien@elastic.co) wrote:
> On Thu, May 21, 2015 at 11:49 PM, James Macdonald \<
> [james.macdonald@geofeedia.com](mailto:james.macdonald@geofeedia.com)\> wrote:
>
> > Hi, I am a little confused by your response. Are you saying that
> > query/filter caches are invalidated across all data in a shard every time
> > the refresh interval ticks over?
>
> Sorry for the confusion:
>
> - the query cache caches entire requests per index, and is competely
> invalidated across all data every time the refresh interval ticks over AND
> there have been changes since the last refresh
> - the filter cache caches matching documents per segment, it is
> invalidated per segment only when a segment goes away (typically because
> it's been merged to a larger segment), which is unfrequent for large
> segments
> - the fielddata cache caches the document-\>value mapping per segment and
> has the same invalidation rules as the filter cache
>
> > I was under the impression that all field data and caching related
> > operations were performed on a Lucene index segment level and that the
> > caches would only be invalidated for a given segment if that segment had
> > changed since the last refresh. Since most data is stored in large segments
> > that don't take fresh writes and seldom merge this would mean that most
> > caches are good for long periods of time; even if the shard is under
> > constant indexing load. Am I mistaken?
>
> This is right for the fielddata and filter caches, but not for the query
> cache.
>
> --
> Adrien
>
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
>
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1\_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1\_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAjZBXouCRxNtvVmtsx%2BZK1_iymRC-7YSojULF7pSK%2Bebg%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAABsnTZtOWEF%3DRjBj\_Xmt9x%2BHQdREn4vT2dFnbjRiF88oPnofA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAABsnTZtOWEF%3DRjBj_Xmt9x%2BHQdREn4vT2dFnbjRiF88oPnofA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:12am UTC](https://discuss.elastic.co/t/shard-query-cache-size/23663/7 "2017-07-06T00:12:14Z")