# Clear single field cache

**URL:** <https://discuss.elastic.co/t/clear-single-field-cache/4138>\
**Category:** Elasticsearch\
**Created:** [March 21, 2011, 10:04pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138 "2011-03-21T22:04:24Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\
**Post date:** [March 21, 2011, 10:04pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/1 "2011-03-21T22:04:24Z")

</div>

Hi all,

I have a use case where a user can search by date recency, in fixed  
intervals, eg: {today, yesterday, past week, past month}

I was planning to use field cache, as the filter is going to be used  
repeatedly, but I would need to call cache clear for that specific  
date filter alone, one every new day (at 24:00).

The problem is that the clear cache API, as I understood from the docs  
and Java code of ClearIndicesCacheRequest, clears all the caches of a  
certain type and index as a minimum granularity. I would like to clear  
just one field cache, for example "publish\_date". I could of course  
clear all the caches, but I would loose a lot of other important  
fields that won't have changed just for one field that did change.

Is clearing the cache of a single field (or list of fields) possible  
somehow today? Can I add a feature request for it? Any other ideas?

Thanks,  
Sebastian.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 21, 2011, 10:11pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/2 "2011-03-21T22:11:54Z")

</div>

I think there is a confusion here... . There is the filter cache, which is used when using certain filters in different places when searching, and field cache, which is used for things like range faceting, date histogram and the like. I did not understand which one are you going to use in order to implement it (not enough info to guess).

There is no API to clear specific field/fields from the field level cache, but can be added. Though, in your case (if you are going to use it), it does not make sense to clear it, since you will still use it next time around. Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly. They do get cleared when memory becomes scarce.

-shay.banon  
On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:

> Hi all,
> 
> I have a use case where a user can search by date recency, in fixed  
> intervals, eg: {today, yesterday, past week, past month}
> 
> I was planning to use field cache, as the filter is going to be used  
> repeatedly, but I would need to call cache clear for that specific  
> date filter alone, one every new day (at 24:00).
> 
> The problem is that the clear cache API, as I understood from the docs  
> and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> certain type and index as a minimum granularity. I would like to clear  
> just one field cache, for example "publish\_date". I could of course  
> clear all the caches, but I would loose a lot of other important  
> fields that won't have changed just for one field that did change.
> 
> Is clearing the cache of a single field (or list of fields) possible  
> somehow today? Can I add a feature request for it? Any other ideas?
> 
> Thanks,  
> Sebastian.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\
**Post date:** [March 21, 2011, 10:24pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/3 "2011-03-21T22:24:01Z")

</div>

Hi Shay,

Sorry for the confusion, I'll try to clarify it.

In my case I was planning to use "rangeFilter", with the epoch milliseconds  
of the filter the user selected, eg: if the user selects "yesterday", I  
would calculate, based on the date today, the milliseconds that corresponds  
to "yesterday". I think a cached filter would be appropriate because until  
the day changes at 24:00, I could keep using the cached filters.

I understand that there would be many filters associated with my  
field "publish\_date", one for each possible value {today, yesterday, past  
week, past month}.

I would like to clean all the range filters associated with the  
field "publish\_date".

> Specific filters make less sense, since they are so discrete you would  
> want to have some sort of way to group them possibly

Is this covered with my explanation of publish\_date and the possible values  
as a way to group them, or why do you think this won't be useful?

Thanks,  
Sebastian.

On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> I think there is a confusion here... . There is the filter cache, which  
> is used when using certain filters in different places when searching, and  
> field cache, which is used for things like range faceting, date histogram  
> and the like. I did not understand which one are you going to use in order  
> to implement it (not enough info to guess).
> 
> There is no API to clear specific field/fields from the field level cache,  
> but can be added. Though, in your case (if you are going to use it), it does  
> not make sense to clear it, since you will still use it next time around.  
> Specific filters make less sense, since they are so discrete you would want  
> to have some sort of way to group them possibly. They do get cleared when  
> memory becomes scarce.
> 
> -shay.banon
> 
> On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> 
> Hi all,
> 
> I have a use case where a user can search by date recency, in fixed  
> intervals, eg: {today, yesterday, past week, past month}
> 
> I was planning to use field cache, as the filter is going to be used  
> repeatedly, but I would need to call cache clear for that specific  
> date filter alone, one every new day (at 24:00).
> 
> The problem is that the clear cache API, as I understood from the docs  
> and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> certain type and index as a minimum granularity. I would like to clear  
> just one field cache, for example "publish\_date". I could of course  
> clear all the caches, but I would loose a lot of other important  
> fields that won't have changed just for one field that did change.
> 
> Is clearing the cache of a single field (or list of fields) possible  
> somehow today? Can I add a feature request for it? Any other ideas?
> 
> Thanks,  
> Sebastian.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 21, 2011, 10:30pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/4 "2011-03-21T22:30:28Z")

</div>

If it is filters, then they will get cleared out when memory get scarce. There are other options (not documented yet, I should really document it), to control the maximum number of filters that are allowed to be cached: [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721).

The reason why its harder to say something clear all filters that match X is because filters are very different from one another, so its more difficult to know what group them and then expose an API to do that. Even if you specify the field that they are used on.

I don't think you will need this option, but, if you see that you do, we can think harder on how to do that 🙂  
On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:

> Hi Shay,
> 
> Sorry for the confusion, I'll try to clarify it.
> 
> In my case I was planning to use "rangeFilter", with the epoch milliseconds of the filter the user selected, eg: if the user selects "yesterday", I would calculate, based on the date today, the milliseconds that corresponds to "yesterday". I think a cached filter would be appropriate because until the day changes at 24:00, I could keep using the cached filters.
> 
> I understand that there would be many filters associated with my field "publish\_date", one for each possible value {today, yesterday, past week, past month}.
> 
> I would like to clean all the range filters associated with the field "publish\_date".
> 
> > Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly
> 
> Is this covered with my explanation of publish\_date and the possible values as a way to group them, or why do you think this won't be useful?
> 
> Thanks,  
> Sebastian.
> 
> On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > I think there is a confusion here... . There is the filter cache, which is used when using certain filters in different places when searching, and field cache, which is used for things like range faceting, date histogram and the like. I did not understand which one are you going to use in order to implement it (not enough info to guess).
> > 
> > There is no API to clear specific field/fields from the field level cache, but can be added. Though, in your case (if you are going to use it), it does not make sense to clear it, since you will still use it next time around. Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly. They do get cleared when memory becomes scarce.
> > 
> > -shay.banon  
> > On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> > 
> > > Hi all,
> > > 
> > > I have a use case where a user can search by date recency, in fixed  
> > > intervals, eg: {today, yesterday, past week, past month}
> > > 
> > > I was planning to use field cache, as the filter is going to be used  
> > > repeatedly, but I would need to call cache clear for that specific  
> > > date filter alone, one every new day (at 24:00).
> > > 
> > > The problem is that the clear cache API, as I understood from the docs  
> > > and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> > > certain type and index as a minimum granularity. I would like to clear  
> > > just one field cache, for example "publish\_date". I could of course  
> > > clear all the caches, but I would loose a lot of other important  
> > > fields that won't have changed just for one field that did change.
> > > 
> > > Is clearing the cache of a single field (or list of fields) possible  
> > > somehow today? Can I add a feature request for it? Any other ideas?
> > > 
> > > Thanks,  
> > > Sebastian.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\
**Post date:** [March 21, 2011, 11:25pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/5 "2011-03-21T23:25:42Z")

</div>

Agreed about difficulty to clear a filter in an API friendly way.

I think selecting the size of the filter cache would be fine.

Just one more thing, how do you know (in advance for sizing and in runtime  
to monitor) the number of filters used? I know the ones I request from my  
searches, but are some others created implicitly inside ES? is there an API  
call to find out the size of that cache (in items quantity, not bytes)?

On Mon, Mar 21, 2011 at 7:30 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> If it is filters, then they will get cleared out when memory get scarce.  
> There are other options (not documented yet, I should really document it),  
> to control the maximum number of filters that are allowed to be cached:  
> [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721)  
> .
> 
> The reason why its harder to say something clear all filters that match X  
> is because filters are very different from one another, so its more  
> difficult to know what group them and then expose an API to do that. Even if  
> you specify the field that they are used on.
> 
> I don't think you will need this option, but, if you see that you do, we  
> can think harder on how to do that 🙂
> 
> On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:
> 
> Hi Shay,
> 
> Sorry for the confusion, I'll try to clarify it.
> 
> In my case I was planning to use "rangeFilter", with the epoch milliseconds  
> of the filter the user selected, eg: if the user selects "yesterday", I  
> would calculate, based on the date today, the milliseconds that corresponds  
> to "yesterday". I think a cached filter would be appropriate because until  
> the day changes at 24:00, I could keep using the cached filters.
> 
> I understand that there would be many filters associated with my  
> field "publish\_date", one for each possible value {today, yesterday, past  
> week, past month}.
> 
> I would like to clean all the range filters associated with the  
> field "publish\_date".
> 
> > Specific filters make less sense, since they are so discrete you would  
> > want to have some sort of way to group them possibly
> 
> Is this covered with my explanation of publish\_date and the possible values  
> as a way to group them, or why do you think this won't be useful?
> 
> Thanks,  
> Sebastian.
> 
> On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> I think there is a confusion here... . There is the filter cache, which  
> is used when using certain filters in different places when searching, and  
> field cache, which is used for things like range faceting, date histogram  
> and the like. I did not understand which one are you going to use in order  
> to implement it (not enough info to guess).
> 
> There is no API to clear specific field/fields from the field level cache,  
> but can be added. Though, in your case (if you are going to use it), it does  
> not make sense to clear it, since you will still use it next time around.  
> Specific filters make less sense, since they are so discrete you would want  
> to have some sort of way to group them possibly. They do get cleared when  
> memory becomes scarce.
> 
> -shay.banon
> 
> On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> 
> Hi all,
> 
> I have a use case where a user can search by date recency, in fixed  
> intervals, eg: {today, yesterday, past week, past month}
> 
> I was planning to use field cache, as the filter is going to be used  
> repeatedly, but I would need to call cache clear for that specific  
> date filter alone, one every new day (at 24:00).
> 
> The problem is that the clear cache API, as I understood from the docs  
> and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> certain type and index as a minimum granularity. I would like to clear  
> just one field cache, for example "publish\_date". I could of course  
> clear all the caches, but I would loose a lot of other important  
> fields that won't have changed just for one field that did change.
> 
> Is clearing the cache of a single field (or list of fields) possible  
> somehow today? Can I add a feature request for it? Any other ideas?
> 
> Thanks,  
> Sebastian.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 21, 2011, 11:28pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/6 "2011-03-21T23:28:11Z")

</div>

On Tuesday, March 22, 2011 at 1:25 AM, Sebastian Gavarini wrote:  
Agreed about difficulty to clear a filter in an API friendly way.

> I think selecting the size of the filter cache would be fine.
> 
> Just one more thing, how do you know (in advance for sizing and in runtime to monitor) the number of filters used? I know the ones I request from my searches, but are some others created implicitly inside ES? is there an API call to find out the size of that cache (in items quantity, not bytes)?  
> The most common other place where elasticsearch uses filters is when using type level searches (the type is used as a term filter). But, I agree, we can enhance the stats API to include counts as well, care to open a feature request for that?
> 
> On Mon, Mar 21, 2011 at 7:30 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > If it is filters, then they will get cleared out when memory get scarce. There are other options (not documented yet, I should really document it), to control the maximum number of filters that are allowed to be cached: [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721).
> > 
> > The reason why its harder to say something clear all filters that match X is because filters are very different from one another, so its more difficult to know what group them and then expose an API to do that. Even if you specify the field that they are used on.
> > 
> > I don't think you will need this option, but, if you see that you do, we can think harder on how to do that 🙂  
> > On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:
> > 
> > > Hi Shay,
> > > 
> > > Sorry for the confusion, I'll try to clarify it.
> > > 
> > > In my case I was planning to use "rangeFilter", with the epoch milliseconds of the filter the user selected, eg: if the user selects "yesterday", I would calculate, based on the date today, the milliseconds that corresponds to "yesterday". I think a cached filter would be appropriate because until the day changes at 24:00, I could keep using the cached filters.
> > > 
> > > I understand that there would be many filters associated with my field "publish\_date", one for each possible value {today, yesterday, past week, past month}.
> > > 
> > > I would like to clean all the range filters associated with the field "publish\_date".
> > > 
> > > > Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly
> > > 
> > > Is this covered with my explanation of publish\_date and the possible values as a way to group them, or why do you think this won't be useful?
> > > 
> > > Thanks,  
> > > Sebastian.
> > > 
> > > On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > 
> > > > I think there is a confusion here... . There is the filter cache, which is used when using certain filters in different places when searching, and field cache, which is used for things like range faceting, date histogram and the like. I did not understand which one are you going to use in order to implement it (not enough info to guess).
> > > > 
> > > > There is no API to clear specific field/fields from the field level cache, but can be added. Though, in your case (if you are going to use it), it does not make sense to clear it, since you will still use it next time around. Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly. They do get cleared when memory becomes scarce.
> > > > 
> > > > -shay.banon  
> > > > On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> > > > 
> > > > > Hi all,
> > > > > 
> > > > > I have a use case where a user can search by date recency, in fixed  
> > > > > intervals, eg: {today, yesterday, past week, past month}
> > > > > 
> > > > > I was planning to use field cache, as the filter is going to be used  
> > > > > repeatedly, but I would need to call cache clear for that specific  
> > > > > date filter alone, one every new day (at 24:00).
> > > > > 
> > > > > The problem is that the clear cache API, as I understood from the docs  
> > > > > and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> > > > > certain type and index as a minimum granularity. I would like to clear  
> > > > > just one field cache, for example "publish\_date". I could of course  
> > > > > clear all the caches, but I would loose a lot of other important  
> > > > > fields that won't have changed just for one field that did change.
> > > > > 
> > > > > Is clearing the cache of a single field (or list of fields) possible  
> > > > > somehow today? Can I add a feature request for it? Any other ideas?
> > > > > 
> > > > > Thanks,  
> > > > > Sebastian.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\
**Post date:** [March 21, 2011, 11:52pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/7 "2011-03-21T23:52:58Z")

</div>

As I was typing the feature request, I had a thought and I think it won't  
do, but not sure though.

Here's the thing, the filter caches are either LRU, Soft or Weak (these last  
two use some sort of eviction based on LRU as stated in the docs, but not  
sure exactly the algorithm), but none of them evict on idle time, so the  
count API would grow up to the maximum and stay there, bringing little use  
as a monitor tool. Maybe if the GC decides to end some Soft/Weak references  
it would decrement, but no guarantee.

I think the whole thing won't add much information, maybe the best way to  
find the right running footprint would be an LRU with active eviction based  
on idle access time, but it's probably more complex to implement and not  
sure if it adds real value.

Maybe adding to the "stats" API not only "count" (returns the current cache  
size) but also "evictedCount" (or something like that, which counts the  
number of evicted filters). So with a couple of calls to the API, separated  
in time, you could understand the rate of creation/eviction of filters,  
according to your use case (in my case once a day I would create 4 new  
filters based on publish\_date).

Is this correct? Do I add a feature request for those two stats?

On Mon, Mar 21, 2011 at 8:28 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> On Tuesday, March 22, 2011 at 1:25 AM, Sebastian Gavarini wrote:
> 
> Agreed about difficulty to clear a filter in an API friendly way.
> 
> I think selecting the size of the filter cache would be fine.
> 
> Just one more thing, how do you know (in advance for sizing and in runtime  
> to monitor) the number of filters used? I know the ones I request from my  
> searches, but are some others created implicitly inside ES? is there an API  
> call to find out the size of that cache (in items quantity, not bytes)?
> 
> The most common other place where elasticsearch uses filters is when using  
> type level searches (the type is used as a term filter). But, I agree, we  
> can enhance the stats API to include counts as well, care to open a feature  
> request for that?
> 
> On Mon, Mar 21, 2011 at 7:30 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> If it is filters, then they will get cleared out when memory get scarce.  
> There are other options (not documented yet, I should really document it),  
> to control the maximum number of filters that are allowed to be cached:  
> [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721)  
> .
> 
> The reason why its harder to say something clear all filters that match X  
> is because filters are very different from one another, so its more  
> difficult to know what group them and then expose an API to do that. Even if  
> you specify the field that they are used on.
> 
> I don't think you will need this option, but, if you see that you do, we  
> can think harder on how to do that 🙂
> 
> On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:
> 
> Hi Shay,
> 
> Sorry for the confusion, I'll try to clarify it.
> 
> In my case I was planning to use "rangeFilter", with the epoch milliseconds  
> of the filter the user selected, eg: if the user selects "yesterday", I  
> would calculate, based on the date today, the milliseconds that corresponds  
> to "yesterday". I think a cached filter would be appropriate because until  
> the day changes at 24:00, I could keep using the cached filters.
> 
> I understand that there would be many filters associated with my  
> field "publish\_date", one for each possible value {today, yesterday, past  
> week, past month}.
> 
> I would like to clean all the range filters associated with the  
> field "publish\_date".
> 
> > Specific filters make less sense, since they are so discrete you would  
> > want to have some sort of way to group them possibly
> 
> Is this covered with my explanation of publish\_date and the possible values  
> as a way to group them, or why do you think this won't be useful?
> 
> Thanks,  
> Sebastian.
> 
> On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> I think there is a confusion here... . There is the filter cache, which  
> is used when using certain filters in different places when searching, and  
> field cache, which is used for things like range faceting, date histogram  
> and the like. I did not understand which one are you going to use in order  
> to implement it (not enough info to guess).
> 
> There is no API to clear specific field/fields from the field level cache,  
> but can be added. Though, in your case (if you are going to use it), it does  
> not make sense to clear it, since you will still use it next time around.  
> Specific filters make less sense, since they are so discrete you would want  
> to have some sort of way to group them possibly. They do get cleared when  
> memory becomes scarce.
> 
> -shay.banon
> 
> On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> 
> Hi all,
> 
> I have a use case where a user can search by date recency, in fixed  
> intervals, eg: {today, yesterday, past week, past month}
> 
> I was planning to use field cache, as the filter is going to be used  
> repeatedly, but I would need to call cache clear for that specific  
> date filter alone, one every new day (at 24:00).
> 
> The problem is that the clear cache API, as I understood from the docs  
> and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> certain type and index as a minimum granularity. I would like to clear  
> just one field cache, for example "publish\_date". I could of course  
> clear all the caches, but I would loose a lot of other important  
> fields that won't have changed just for one field that did change.
> 
> Is clearing the cache of a single field (or list of fields) possible  
> somehow today? Can I add a feature request for it? Any other ideas?
> 
> Thanks,  
> Sebastian.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 21, 2011, 11:56pm UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/8 "2011-03-21T23:56:41Z")

</div>

Need to check the eviction listener thingy, but, adding an option to expire (after time, last access based) is certainly something that can be added. Open a feature request for it.  
On Tuesday, March 22, 2011 at 1:52 AM, Sebastian Gavarini wrote:

> As I was typing the feature request, I had a thought and I think it won't do, but not sure though.
> 
> Here's the thing, the filter caches are either LRU, Soft or Weak (these last two use some sort of eviction based on LRU as stated in the docs, but not sure exactly the algorithm), but none of them evict on idle time, so the count API would grow up to the maximum and stay there, bringing little use as a monitor tool. Maybe if the GC decides to end some Soft/Weak references it would decrement, but no guarantee.
> 
> I think the whole thing won't add much information, maybe the best way to find the right running footprint would be an LRU with active eviction based on idle access time, but it's probably more complex to implement and not sure if it adds real value.
> 
> Maybe adding to the "stats" API not only "count" (returns the current cache size) but also "evictedCount" (or something like that, which counts the number of evicted filters). So with a couple of calls to the API, separated in time, you could understand the rate of creation/eviction of filters, according to your use case (in my case once a day I would create 4 new filters based on publish\_date).
> 
> Is this correct? Do I add a feature request for those two stats?
> 
> On Mon, Mar 21, 2011 at 8:28 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > On Tuesday, March 22, 2011 at 1:25 AM, Sebastian Gavarini wrote:
> > 
> > > Agreed about difficulty to clear a filter in an API friendly way.
> > > 
> > > I think selecting the size of the filter cache would be fine.
> > > 
> > > Just one more thing, how do you know (in advance for sizing and in runtime to monitor) the number of filters used? I know the ones I request from my searches, but are some others created implicitly inside ES? is there an API call to find out the size of that cache (in items quantity, not bytes)?  
> > > The most common other place where elasticsearch uses filters is when using type level searches (the type is used as a term filter). But, I agree, we can enhance the stats API to include counts as well, care to open a feature request for that?
> > 
> > > On Mon, Mar 21, 2011 at 7:30 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > 
> > > > If it is filters, then they will get cleared out when memory get scarce. There are other options (not documented yet, I should really document it), to control the maximum number of filters that are allowed to be cached: [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721).
> > > > 
> > > > The reason why its harder to say something clear all filters that match X is because filters are very different from one another, so its more difficult to know what group them and then expose an API to do that. Even if you specify the field that they are used on.
> > > > 
> > > > I don't think you will need this option, but, if you see that you do, we can think harder on how to do that 🙂  
> > > > On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:
> > > > 
> > > > > Hi Shay,
> > > > > 
> > > > > Sorry for the confusion, I'll try to clarify it.
> > > > > 
> > > > > In my case I was planning to use "rangeFilter", with the epoch milliseconds of the filter the user selected, eg: if the user selects "yesterday", I would calculate, based on the date today, the milliseconds that corresponds to "yesterday". I think a cached filter would be appropriate because until the day changes at 24:00, I could keep using the cached filters.
> > > > > 
> > > > > I understand that there would be many filters associated with my field "publish\_date", one for each possible value {today, yesterday, past week, past month}.
> > > > > 
> > > > > I would like to clean all the range filters associated with the field "publish\_date".
> > > > > 
> > > > > > Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly
> > > > > 
> > > > > Is this covered with my explanation of publish\_date and the possible values as a way to group them, or why do you think this won't be useful?
> > > > > 
> > > > > Thanks,  
> > > > > Sebastian.
> > > > > 
> > > > > On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > > > 
> > > > > > I think there is a confusion here... . There is the filter cache, which is used when using certain filters in different places when searching, and field cache, which is used for things like range faceting, date histogram and the like. I did not understand which one are you going to use in order to implement it (not enough info to guess).
> > > > > > 
> > > > > > There is no API to clear specific field/fields from the field level cache, but can be added. Though, in your case (if you are going to use it), it does not make sense to clear it, since you will still use it next time around. Specific filters make less sense, since they are so discrete you would want to have some sort of way to group them possibly. They do get cleared when memory becomes scarce.
> > > > > > 
> > > > > > -shay.banon  
> > > > > > On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> > > > > > 
> > > > > > > Hi all,
> > > > > > > 
> > > > > > > I have a use case where a user can search by date recency, in fixed  
> > > > > > > intervals, eg: {today, yesterday, past week, past month}
> > > > > > > 
> > > > > > > I was planning to use field cache, as the filter is going to be used  
> > > > > > > repeatedly, but I would need to call cache clear for that specific  
> > > > > > > date filter alone, one every new day (at 24:00).
> > > > > > > 
> > > > > > > The problem is that the clear cache API, as I understood from the docs  
> > > > > > > and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> > > > > > > certain type and index as a minimum granularity. I would like to clear  
> > > > > > > just one field cache, for example "publish\_date". I could of course  
> > > > > > > clear all the caches, but I would loose a lot of other important  
> > > > > > > fields that won't have changed just for one field that did change.
> > > > > > > 
> > > > > > > Is clearing the cache of a single field (or list of fields) possible  
> > > > > > > somehow today? Can I add a feature request for it? Any other ideas?
> > > > > > > 
> > > > > > > Thanks,  
> > > > > > > Sebastian.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\
**Post date:** [March 22, 2011, 12:08am UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/9 "2011-03-22T00:08:26Z")

</div>

Done,

Stats extension:

> **[Issues · elastic/elasticsearch](https://github.com/elastic/elasticsearch/issues#issue/802)**
>
> Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub.

Last access time based expiry caches:

> **[Issues · elastic/elasticsearch](https://github.com/elastic/elasticsearch/issues#issue/803)**
>
> Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub.

Thanks,  
Sebastian.

On Mon, Mar 21, 2011 at 8:56 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Need to check the eviction listener thingy, but, adding an option to  
> expire (after time, last access based) is certainly something that can be  
> added. Open a feature request for it.
> 
> On Tuesday, March 22, 2011 at 1:52 AM, Sebastian Gavarini wrote:
> 
> As I was typing the feature request, I had a thought and I think it won't  
> do, but not sure though.
> 
> Here's the thing, the filter caches are either LRU, Soft or Weak (these  
> last two use some sort of eviction based on LRU as stated in the docs, but  
> not sure exactly the algorithm), but none of them evict on idle time, so the  
> count API would grow up to the maximum and stay there, bringing little use  
> as a monitor tool. Maybe if the GC decides to end some Soft/Weak references  
> it would decrement, but no guarantee.
> 
> I think the whole thing won't add much information, maybe the best way to  
> find the right running footprint would be an LRU with active eviction based  
> on idle access time, but it's probably more complex to implement and not  
> sure if it adds real value.
> 
> Maybe adding to the "stats" API not only "count" (returns the current cache  
> size) but also "evictedCount" (or something like that, which counts the  
> number of evicted filters). So with a couple of calls to the API, separated  
> in time, you could understand the rate of creation/eviction of filters,  
> according to your use case (in my case once a day I would create 4 new  
> filters based on publish\_date).
> 
> Is this correct? Do I add a feature request for those two stats?
> 
> On Mon, Mar 21, 2011 at 8:28 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > On Tuesday, March 22, 2011 at 1:25 AM, Sebastian Gavarini wrote:
> > 
> > Agreed about difficulty to clear a filter in an API friendly way.
> > 
> > I think selecting the size of the filter cache would be fine.
> > 
> > Just one more thing, how do you know (in advance for sizing and in runtime  
> > to monitor) the number of filters used? I know the ones I request from my  
> > searches, but are some others created implicitly inside ES? is there an API  
> > call to find out the size of that cache (in items quantity, not bytes)?
> > 
> > The most common other place where elasticsearch uses filters is when using  
> > type level searches (the type is used as a term filter). But, I agree, we  
> > can enhance the stats API to include counts as well, care to open a feature  
> > request for that?
> > 
> > On Mon, Mar 21, 2011 at 7:30 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > wrote:
> > 
> > If it is filters, then they will get cleared out when memory get scarce.  
> > There are other options (not documented yet, I should really document it),  
> > to control the maximum number of filters that are allowed to be cached:  
> > [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed?page=2#issue/721)  
> > .
> > 
> > The reason why its harder to say something clear all filters that match X  
> > is because filters are very different from one another, so its more  
> > difficult to know what group them and then expose an API to do that. Even if  
> > you specify the field that they are used on.
> > 
> > I don't think you will need this option, but, if you see that you do, we  
> > can think harder on how to do that 🙂
> > 
> > On Tuesday, March 22, 2011 at 12:24 AM, Sebastian Gavarini wrote:
> > 
> > Hi Shay,
> > 
> > Sorry for the confusion, I'll try to clarify it.
> > 
> > In my case I was planning to use "rangeFilter", with the epoch  
> > milliseconds of the filter the user selected, eg: if the user selects  
> > "yesterday", I would calculate, based on the date today, the milliseconds  
> > that corresponds to "yesterday". I think a cached filter would be  
> > appropriate because until the day changes at 24:00, I could keep using the  
> > cached filters.
> > 
> > I understand that there would be many filters associated with my  
> > field "publish\_date", one for each possible value {today, yesterday, past  
> > week, past month}.
> > 
> > I would like to clean all the range filters associated with the  
> > field "publish\_date".
> > 
> > > Specific filters make less sense, since they are so discrete you would  
> > > want to have some sort of way to group them possibly
> > 
> > Is this covered with my explanation of publish\_date and the possible  
> > values as a way to group them, or why do you think this won't be useful?
> > 
> > Thanks,  
> > Sebastian.
> > 
> > On Mon, Mar 21, 2011 at 7:11 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > wrote:
> > 
> > I think there is a confusion here... . There is the filter cache,  
> > which is used when using certain filters in different places when searching,  
> > and field cache, which is used for things like range faceting, date  
> > histogram and the like. I did not understand which one are you going to use  
> > in order to implement it (not enough info to guess).
> > 
> > There is no API to clear specific field/fields from the field level cache,  
> > but can be added. Though, in your case (if you are going to use it), it does  
> > not make sense to clear it, since you will still use it next time around.  
> > Specific filters make less sense, since they are so discrete you would want  
> > to have some sort of way to group them possibly. They do get cleared when  
> > memory becomes scarce.
> > 
> > -shay.banon
> > 
> > On Tuesday, March 22, 2011 at 12:04 AM, Sebastian wrote:
> > 
> > Hi all,
> > 
> > I have a use case where a user can search by date recency, in fixed  
> > intervals, eg: {today, yesterday, past week, past month}
> > 
> > I was planning to use field cache, as the filter is going to be used  
> > repeatedly, but I would need to call cache clear for that specific  
> > date filter alone, one every new day (at 24:00).
> > 
> > The problem is that the clear cache API, as I understood from the docs  
> > and Java code of ClearIndicesCacheRequest, clears all the caches of a  
> > certain type and index as a minimum granularity. I would like to clear  
> > just one field cache, for example "publish\_date". I could of course  
> > clear all the caches, but I would loose a lot of other important  
> > fields that won't have changed just for one field that did change.
> > 
> > Is clearing the cache of a single field (or list of fields) possible  
> > somehow today? Can I add a feature request for it? Any other ideas?
> > 
> > Thanks,  
> > Sebastian.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:09am UTC](https://discuss.elastic.co/t/clear-single-field-cache/4138/10 "2017-07-06T04:09:32Z")

</div>