# Heap size/filter cache **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2014, 8:58am UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976 "2014-11-28T08:58:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Audrius\_Bugas](https://avatars.discourse-cdn.com/v4/letter/a/b9e5f3/32.png) [@Audrius\_Bugas](https://discuss.elastic.co/u/Audrius_Bugas)\ **Post date:** [November 28, 2014, 8:58am UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976/1 "2014-11-28T08:58:44Z")
Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters cache. When clear filter cache from api it clears ~4GB memory from heap (way more then filters cache). Is it normal, or I have some problems with cached filters? We are using warmers, but this should not be a reason... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [November 29, 2014, 11:00am UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976/2 "2014-11-29T11:00:41Z")
When you cache filters, there are two things that can take memory: the cached bitsets and the filter itself (which is used as a key). And while we have APIs to estimate the memory usage for bitsets, we do not have anything for filters unfortunately. So if you happen to have lots of small segments on your machine or if you use filters that can have a high memory footprint (like terms filters with lots of terms) then the memory usage of your filter cache might be underestimated because of the filters which are not taken into account. By the way, 20% is a bit high for a filter cache size. Recent versions of elasticsearch have a default value of 10% ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-cache.html)) which should already allow for caching lots of caches. On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas [audrius.bugas@hometogo.com](mailto:audrius.bugas@hometogo.com) wrote: > Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters > cache. When clear filter cache from api it clears ~4GB memory from heap > (way more then filters cache). Is it normal, or I have some problems with > cached filters? We are using warmers, but this should not be a reason... > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6L7OB5Fxc364z6%3DMt-ecSmRpgsxu%3D0OR-vZ9ODjqRR0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6L7OB5Fxc364z6%3DMt-ecSmRpgsxu%3D0OR-vZ9ODjqRR0w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Audrius\_Bugas\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/audrius_bugas_2/32/1072_2.png) [@Audrius\_Bugas\_2](https://discuss.elastic.co/u/Audrius_Bugas_2)\ **Post date:** [December 1, 2014, 12:05pm UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976/3 "2014-12-01T12:05:24Z")
Thank you for answer. We had lot's of cache evictions so we increased cache size. Should I use custom cache keys and try to make it smaller? What cache keys are generated by default? Is it full filter query dsl string? 2014 m. lapkritis 29 d., šeštadienis 13:01:59 UTC+2, Adrien Grand rašė: > When you cache filters, there are two things that can take memory: the > cached bitsets and the filter itself (which is used as a key). And while we > have APIs to estimate the memory usage for bitsets, we do not have anything > for filters unfortunately. So if you happen to have lots of small segments > on your machine or if you use filters that can have a high memory footprint > (like terms filters with lots of terms) then the memory usage of your > filter cache might be underestimated because of the filters which are not > taken into account. > > By the way, 20% is a bit high for a filter cache size. Recent versions of > elasticsearch have a default value of 10% ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-cache.html)) > which should already allow for caching lots of caches. > > On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas \<[audriu...@hometogo.com](mailto:audriu...@hometogo.com) > \\> wrote: > > > Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters > > cache. When clear filter cache from api it clears ~4GB memory from heap > > (way more then filters cache). Is it normal, or I have some problems with > > cached filters? We are using warmers, but this should not be a reason... > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 1, 2014, 12:19pm UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976/4 "2014-12-01T12:19:21Z")
Cache keys are pretty much the input filter by default. Specifying your own keys can certainly help. On Dec 1, 2014 7:05 AM, "Audrius Bugas" [bugas.audrius@gmail.com](mailto:bugas.audrius@gmail.com) wrote: > Thank you for answer. We had lot's of cache evictions so we increased > cache size. Should I use custom cache keys and try to make it smaller? > What cache keys are generated by default? Is it full filter query dsl > string? > > 2014 m. lapkritis 29 d., šeštadienis 13:01:59 UTC+2, Adrien Grand rašė: > > > When you cache filters, there are two things that can take memory: the > > cached bitsets and the filter itself (which is used as a key). And while we > > have APIs to estimate the memory usage for bitsets, we do not have anything > > for filters unfortunately. So if you happen to have lots of small segments > > on your machine or if you use filters that can have a high memory footprint > > (like terms filters with lots of terms) then the memory usage of your > > filter cache might be underestimated because of the filters which are not > > taken into account. > > > > By the way, 20% is a bit high for a filter cache size. Recent versions of > > elasticsearch have a default value of 10% ([http://www.elasticsearch.org/](http://www.elasticsearch.org/) > > guide/en/elasticsearch/reference/current/index-modules-cache.html) which > > should already allow for caching lots of caches. > > > > On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas [audriu...@hometogo.com](mailto:audriu...@hometogo.com) > > wrote: > > > > > Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters > > > cache. When clear filter cache from api it clears ~4GB memory from heap > > > (way more then filters cache). Is it normal, or I have some problems with > > > cached filters? We are using warmers, but this should not be a reason... > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0s%3DS5kZbw\_cug2LB1Lt%2BekOGGGzmGF\_uAJRkW6yN5s%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0s%3DS5kZbw_cug2LB1Lt%2BekOGGGzmGF_uAJRkW6yN5s%3Dg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:46am UTC](https://discuss.elastic.co/t/heap-size-filter-cache/20976/5 "2017-07-06T00:46:51Z")