# Indices.cache.filter.size limit not enforce? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2013, 9:13am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068 "2013-10-23T09:13:57Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Benoit](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@Benoit](https://discuss.elastic.co/u/Benoit)\ **Post date:** [October 23, 2013, 9:13am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/1 "2013-10-23T09:13:57Z")
Hello, Yesterday I had the same problem as last time ( [https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8\_KmhJ0wcQJ](https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8_KmhJ0wcQJ)). But this time I have a suspicion about indices.cache.filter.size limit that might not be enforced. The story is that filter\_cache has grown beyond its limit up to 80% of the total JVM heap instead of the 30% configured. At one time a GC longer that 3x30s timeout make the node leave the cluster [2013-10-22 07:16:40,459][INFO][discovery.zen] [sissor1] master\_left [[sissor2][sBQ1oCTbRsGexVcQpu466Q][inet[/192.168.110.90:9300]]], reason [failed to ping, tried [3] times, each with maximum [30s] timeout] The settings : $ curl -XGET xxxxxx/\_cluster/settings?pretty=true { "persistent" : { }, "transient" : { "indices.cache.filter.size" : "30%" } } This is a two nodes cluster with - Elasticsearch version 0.90.3, - 256 Go RAM - 127.8 Go heap - index total size: 956gb - 150 000 000 docs - 14 indices, 28 shards + 1 replica I know this is an unusual heap size, but until now I had not so much problems. Could it be a bug or did i miss somethings ? Benoît -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Benoit](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@Benoit](https://discuss.elastic.co/u/Benoit)\ **Post date:** [October 24, 2013, 7:33am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/2 "2013-10-24T07:33:41Z")
Nobody has an opinion? On Wednesday, October 23, 2013 11:13:57 AM UTC+2, Benoît wrote: > Hello, > > Yesterday I had the same problem as last time ( > [https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8\_KmhJ0wcQJ](https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8_KmhJ0wcQJ)). > > But this time I have a suspicion about indices.cache.filter.size limit > that might not be enforced. > > The story is that filter\_cache has grown beyond its limit up to 80% of > the total JVM heap instead of the 30% configured. > > At one time a GC longer that 3x30s timeout make the node leave the cluster > > [2013-10-22 07:16:40,459][INFO][discovery.zen] [sissor1] > master\_left > [[sissor2][sBQ1oCTbRsGexVcQpu466Q][inet[/192.168.110.90:9300]]], reason > [failed to ping, tried [3] times, each with maximum [30s] timeout] > > The settings : > > $ curl -XGET xxxxxx/\_cluster/settings?pretty=true > { > "persistent" : { }, > "transient" : { > "indices.cache.filter.size" : "30%" > } > } > > This is a two nodes cluster with > > - Elasticsearch version 0.90.3, > - 256 Go RAM > - 127.8 Go heap > - index total size: 956gb > - 150 000 000 docs > - 14 indices, 28 shards + 1 replica > > I know this is an unusual heap size, but until now I had not so much > problems. > > Could it be a bug or did i miss somethings ? > > Benoît -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [October 24, 2013, 6:57pm UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/3 "2013-10-24T18:57:21Z")
I've had no issues with changing the settings. Is 256 Go/127.8 Go a typo for GB? You might be better off running two instances on a single node, since giving Java more than 32GB of RAM is detrimental. Ivan On Thu, Oct 24, 2013 at 12:33 AM, Benoît [benoit.intrw@gmail.com](mailto:benoit.intrw@gmail.com) wrote: > Nobody has an opinion? > > On Wednesday, October 23, 2013 11:13:57 AM UTC+2, Benoît wrote: > > > Hello, > > > > Yesterday I had the same problem as last time ( > > [https://groups.google.com/d/\*\*msg/elasticsearch/PQhJezsaQrg/](https://groups.google.com/d/**msg/elasticsearch/PQhJezsaQrg/)\*\* > > w8\_KmhJ0wcQJ[https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8\_KmhJ0wcQJ](https://groups.google.com/d/msg/elasticsearch/PQhJezsaQrg/w8_KmhJ0wcQJ) > > ). > > > > But this time I have a suspicion about indices.cache.filter.size limit > > that might not be enforced. > > > > The story is that filter\_cache has grown beyond its limit up to 80% of > > the total JVM heap instead of the 30% configured. > > > > At one time a GC longer that 3x30s timeout make the node leave the cluster > > > > [2013-10-22 07:16:40,459][INFO][discovery.zen] [sissor1] > > master\_left [[sissor2][**sBQ1oCTbRsGexVcQpu466Q][inet[/** > > 192.168.110.90:9300]]], reason [failed to ping, tried [3] times, each > > with maximum [30s] timeout] > > > > The settings : > > > > $ curl -XGET xxxxxx/\_cluster/settings?\*\*pretty=true > > { > > "persistent" : { }, > > "transient" : { > > "indices.cache.filter.size" : "30%" > > } > > } > > > > This is a two nodes cluster with > > > > - Elasticsearch version 0.90.3, > > - 256 Go RAM > > - 127.8 Go heap > > - index total size: 956gb > > - 150 000 000 docs > > - 14 indices, 28 shards + 1 replica > > > > I know this is an unusual heap size, but until now I had not so much > > problems. > > > > Could it be a bug or did i miss somethings ? > > > > Benoît > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Benoit](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@Benoit](https://discuss.elastic.co/u/Benoit)\ **Post date:** [October 25, 2013, 8:05am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/4 "2013-10-25T08:05:52Z")
Hello, Thank you for looking at this, comments below. On Thursday, October 24, 2013 8:57:21 PM UTC+2, Ivan Brusic wrote: > I've had no issues with changing the settings. > > Is 256 Go/127.8 Go a typo for GB? Oh yes sorry, o is for "Octet" which is the french of Byte 😉 > You might be better off running two instances on a single node, I've never read support about running two instance of ES on one box. > since giving Java more than 32GB of RAM is detrimental. I know that, but when we started this project, there was little room in the rack and so we chose two large machines rather than many small. The question is is it a bug or is it normal that indices.cache.filter.size can go over the limit. Benoît -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [October 25, 2013, 12:06pm UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/5 "2013-10-25T12:06:58Z")
Hi I've never seen the filter cache limit not being enforced. If you can provide supporting data, ie the filter cache size from the nodes\_stats plus the settings you had in place at the time, would be helpful. I support Ivan's comment about heap size: the bigger the heap, the longer GC takes. And using a heap above 32GB means the JVM can't use compressed pointers. So better to run multiple nodes on one machine, using "shard awareness" to ensure that you don't have copies of the same data on the same machine. Clint On 25 October 2013 10:05, Benoît [benoit.intrw@gmail.com](mailto:benoit.intrw@gmail.com) wrote: > Hello, > > Thank you for looking at this, comments below. > > On Thursday, October 24, 2013 8:57:21 PM UTC+2, Ivan Brusic wrote: > > > I've had no issues with changing the settings. > > > > Is 256 Go/127.8 Go a typo for GB? > > Oh yes sorry, o is for "Octet" which is the french of Byte 😉 > > > You might be better off running two instances on a single node, > > I've never read support about running two instance of ES on one box. > > > since giving Java more than 32GB of RAM is detrimental. > > I know that, but when we started this project, there was little room in > the rack and so we chose two large machines rather than many small. > > The question is is it a bug or is it normal that indices.cache.filter.size > can go over the limit. > > Benoît > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Benoit](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@Benoit](https://discuss.elastic.co/u/Benoit)\ **Post date:** [October 25, 2013, 12:55pm UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/6 "2013-10-25T12:55:37Z")
Hi ! On Friday, October 25, 2013 2:06:58 PM UTC+2, Clinton Gormley wrote: > I've never seen the filter cache limit not being enforced. If you can > provide supporting data, ie the filter cache size from the nodes\_stats plus > the settings you had in place at the time, would be helpful. Output of \_cluster/settings and \_nodes/stats?all=true in the following gist : [nodes stats and cluster setting · GitHub](https://gist.github.com/benoit-intrw/7154161) The value is not really high right now but 44.5gb is over 30% of commited heap (127.8gb) "filter\_cache": { "memory\_size": "44.5gb", "memory\_size\_in\_bytes": 47819287444, "evictions": 0 }, > I support Ivan's comment about heap size: the bigger the heap, the longer > GC takes. And using a heap above 32GB means the JVM can't use compressed > pointers. So better to run multiple nodes on one machine, using "shard > awareness" to ensure that you don't have copies of the same data on the > same machine. ok i will think about it but the machine are in production ... Regards Benoît -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Low](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_low/32/1549_2.png) [@Daniel\_Low](https://discuss.elastic.co/u/Daniel_Low)\ **Post date:** [May 21, 2014, 6:03pm UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/7 "2014-05-21T18:03:50Z")
Hello, Has there been any updates to this? We are using nodes with 256GB of ram and heap sizes of 96GB and also seeing this exact same issue where filter cache sizes grow above the limit. What I also discovered was that when I set the filter cache size to 31.9GB or lower the limit worked fine, but anything above and it did not. Thanks, Daniel On Friday, October 25, 2013 5:55:37 AM UTC-7, Benoît wrote: > Hi ! > > On Friday, October 25, 2013 2:06:58 PM UTC+2, Clinton Gormley wrote: > > > I've never seen the filter cache limit not being enforced. If you can > > provide supporting data, ie the filter cache size from the nodes\_stats plus > > the settings you had in place at the time, would be helpful. > > Output of \_cluster/settings and \_nodes/stats?all=true in the following > gist : [nodes stats and cluster setting · GitHub](https://gist.github.com/benoit-intrw/7154161) > > The value is not really high right now but 44.5gb is over 30% of commited > heap (127.8gb) > > "filter\_cache": { > "memory\_size": "44.5gb", > "memory\_size\_in\_bytes": 47819287444, > "evictions": 0 > }, > > > I support Ivan's comment about heap size: the bigger the heap, the longer > > GC takes. And using a heap above 32GB means the JVM can't use compressed > > pointers. So better to run multiple nodes on one machine, using "shard > > awareness" to ensure that you don't have copies of the same data on the > > same machine. > > ok i will think about it but the machine are in production ... > > Regards > > Benoît -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 23, 2014, 12:12am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/8 "2014-05-23T00:12:37Z")
For those who would come to this thread through a search engine, Dan found the root cause of this issue > > > Hi, > > We are running an elasticsearch 1.1.1 6 node cluster with 256GB of ram, and… using 96GB JVM heap sizes. I've noticed that when I set the filter cache size to 32GB or over with this command: > > \`\`\` > curl -XPUT "http://localhost:9200/\_cluster/settings" -d' > { > "transient" : { > "indices.cache.filter.size" : "50%" > } > }' > \`\`\` > > The field cache size keeps growing above and beyond the indicated limit. The relevant node stats show that the filter cache size is about 69GB in size, which is over the configured limit of 48GB > > \`\`\` > "filter\_cache" : { > "memory\_size\_in\_bytes" : 74550217274, > "evictions" : 8665179 > }, > \`\`\` > > I've enable debug logging on the node itself and it looks like the cache itself is getting created with the correct values: > > \`\`\` > \[2014-05-21 00:31:57,215\]\[DEBUG\]\[indices.cache.filter \] \[ess02-006\] using \[node\] weighted filter cache with size \[50%\], actual\_size \[47.9gb\], expire \[null\], clean\_interval \[1m\] > \`\`\` > > Whats strange is that when I set the limit to 31.9GB, the limit is enforced, which leads me to believe there is some sort of overflow going on. > > Thanks, > Daniel On Wed, May 21, 2014 at 8:03 PM, Daniel Low [danglow@gmail.com](mailto:danglow@gmail.com) wrote: > Hello, > > Has there been any updates to this? We are using nodes with 256GB of ram > and heap sizes of 96GB and also seeing this exact same issue where filter > cache sizes grow above the limit. What I also discovered was that when I > set the filter cache size to 31.9GB or lower the limit worked fine, but > anything above and it did not. > > Thanks, > Daniel > > On Friday, October 25, 2013 5:55:37 AM UTC-7, Benoît wrote: > > > Hi ! > > > > On Friday, October 25, 2013 2:06:58 PM UTC+2, Clinton Gormley wrote: > > > > > I've never seen the filter cache limit not being enforced. If you can > > > provide supporting data, ie the filter cache size from the nodes\_stats plus > > > the settings you had in place at the time, would be helpful. > > > > Output of \_cluster/settings and \_nodes/stats?all=true in the following > > gist : [nodes stats and cluster setting · GitHub](https://gist.github.com/benoit-intrw/7154161) > > > > The value is not really high right now but 44.5gb is over 30% of commited > > heap (127.8gb) > > > > "filter\_cache": { > > "memory\_size": "44.5gb", > > "memory\_size\_in\_bytes": 47819287444, > > "evictions": 0 > > }, > > > > > I support Ivan's comment about heap size: the bigger the heap, the > > > longer GC takes. And using a heap above 32GB means the JVM can't use > > > compressed pointers. So better to run multiple nodes on one machine, using > > > "shard awareness" to ensure that you don't have copies of the same data on > > > the same machine. > > > > ok i will think about it but the machine are in production ... > > > > Regards > > > > Benoît > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f5158eda-7e43-4b05-9f32-52bd3984c766%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4h9EcB6DoW0zSU2bpWCxoWm1\_kkMt7vDYY0vqXLZif6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4h9EcB6DoW0zSU2bpWCxoWm1_kkMt7vDYY0vqXLZif6g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:27am UTC](https://discuss.elastic.co/t/indices-cache-filter-size-limit-not-enforce/14068/9 "2017-07-06T01:27:32Z")