# Field Data Cache Size and Eviction **URL:** \ **Category:** Elasticsearch\ **Created:** [September 12, 2014, 1:21pm UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757 "2014-09-12T13:21:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Philippe\_Laflamme](https://avatars.discourse-cdn.com/v4/letter/p/7ab992/32.png) [@Philippe\_Laflamme](https://discuss.elastic.co/u/Philippe_Laflamme)\ **Post date:** [September 12, 2014, 1:21pm UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757/1 "2014-09-12T13:21:23Z")
Hi, I have a cluster with nodes configured with a 18G heap. We've noticed a degradation in performance recently after increasing the volume of data we're indexing. I think the issue is due to the field data cache doing eviction. Some nodes are doing lots of them, some aren't doing any. This is explained by our routing strategy which results in non-uniform document distribution. Maybe we can improve this eventually, but in the meantime, I'm trying to understand why the nodes are evicting cached data. The metrics show that the field data cache is only ~1.5GB in size, yet we have this in our elasticsearch.yml: indices.fielddata.cache.size: 10gb Why would a node evict cache entries when it should still have plenty of room to store more? Are we missing another setting? Is there a way to tell what the actual fielddata cache size is at runtime (maybe it did not pickup the configuration setting for some reason)? Thanks, Philippe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e619f974-1632-4694-a0f9-40c32100c504%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e619f974-1632-4694-a0f9-40c32100c504%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Philippe\_Laflamme](https://avatars.discourse-cdn.com/v4/letter/p/7ab992/32.png) [@Philippe\_Laflamme](https://discuss.elastic.co/u/Philippe_Laflamme)\ **Post date:** [September 12, 2014, 1:33pm UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757/2 "2014-09-12T13:33:16Z")
Forgot to mention that we're using ES 1.1.1 On Friday, September 12, 2014 9:21:23 AM UTC-4, Philippe Laflamme wrote: > Hi, > > I have a cluster with nodes configured with a 18G heap. We've noticed a > degradation in performance recently after increasing the volume of data > we're indexing. > > I think the issue is due to the field data cache doing eviction. Some > nodes are doing lots of them, some aren't doing any. This is explained by > our routing strategy which results in non-uniform document distribution. > Maybe we can improve this eventually, but in the meantime, I'm trying to > understand why the nodes are evicting cached data. > > The metrics show that the field data cache is only ~1.5GB in size, yet we > have this in our elasticsearch.yml: > > indices.fielddata.cache.size: 10gb > > Why would a node evict cache entries when it should still have plenty of > room to store more? Are we missing another setting? Is there a way to tell > what the actual fielddata cache size is at runtime (maybe it did not pickup > the configuration setting for some reason)? > > Thanks, > Philippe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/512be87e-561a-4031-a465-d256ad400bbb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/512be87e-561a-4031-a465-d256ad400bbb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Philippe\_Laflamme](https://avatars.discourse-cdn.com/v4/letter/p/7ab992/32.png) [@Philippe\_Laflamme](https://discuss.elastic.co/u/Philippe_Laflamme)\ **Post date:** [September 16, 2014, 2:25pm UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757/3 "2014-09-16T14:25:07Z")
Sorry for bumping this, but I'm a little stumped here. We have some nodes that are evicting fielddata cache entries for seemingly no reason: 1. we've set indices.fielddata.cache.size to 10gb 2. the metrics from the node stats endpoint show that the indices.fielddata.memory\_size\_in\_bytes never exceeded 3.6GB on any node. 3. the rate of eviction is normally 0, but goes up above that eventhough the fielddata cache size is nowhere near 10GB Attached is a plot of the max(indices.fielddata.memory\_size\_in\_bytes) (red line) and sum(indices.fielddata.evictions) (green line) across all nodes in the cluster. Note that we create a fresh new index every day that replaces an older one (that explains the change in profile around midnight). As you can see, the size (on any given node) never exceeds 3.6GB, yet even at a lower value (around 2.2GB), some nodes start evicting entries from the cache. Also, starting around Tue 8AM, the max(field cache size) becomes erratic and jumps up and down. I can't explain this behaviour, especially since we've been operating for a while at this volume and rate of documents. This was not happening before. Though it's possible that we're getting a higher volume of data, it doesn't look substantially different from the past. Under what circumstances will an ES node evict entries from it's field data cache? We're also deleting documents from the index, can this have an impact? What other things should I be looking it to find a correlation (GC time does not seem to be correlated)? Thanks, Philippe On Friday, September 12, 2014 9:33:16 AM UTC-4, Philippe Laflamme wrote: > Forgot to mention that we're using ES 1.1.1 > > On Friday, September 12, 2014 9:21:23 AM UTC-4, Philippe Laflamme wrote: > > > Hi, > > > > I have a cluster with nodes configured with a 18G heap. We've noticed a > > degradation in performance recently after increasing the volume of data > > we're indexing. > > > > I think the issue is due to the field data cache doing eviction. Some > > nodes are doing lots of them, some aren't doing any. This is explained by > > our routing strategy which results in non-uniform document distribution. > > Maybe we can improve this eventually, but in the meantime, I'm trying to > > understand why the nodes are evicting cached data. > > > > The metrics show that the field data cache is only ~1.5GB in size, yet we > > have this in our elasticsearch.yml: > > > > indices.fielddata.cache.size: 10gb > > > > Why would a node evict cache entries when it should still have plenty of > > room to store more? Are we missing another setting? Is there a way to tell > > what the actual fielddata cache size is at runtime (maybe it did not pickup > > the configuration setting for some reason)? > > > > Thanks, > > Philippe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eafa1b0a-dbd6-4127-94d5-3733a3067bc7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eafa1b0a-dbd6-4127-94d5-3733a3067bc7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [April 23, 2015, 8:10am UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757/4 "2015-04-23T08:10:33Z")
A bit late from the OP posted this, not sure if it is still relevant but anyway... > > Under what circumstances will an ES node evict entries from it's field > > data cache? We're also deleting documents from the index, can this have an > > impact? What other things should I be looking it to find a correlation (GC > > time does not seem to be correlated)? The cache implements an LRU eviction policy: when a cache becomes full, the least recently used data is evicted to make way for new data. [http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-cache.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-cache.html) more information here [Monitoring Individual Nodes | Elasticsearch: The Definitive Guide [2.x] | Elastic](http://www.elastic.co/guide/en/elasticsearch/guide/current/_monitoring_individual_nodes.html#_indices_section) It's puzzling in your case when you set to 10GB for cache size but per node usage only 3.6GB . Have you use the other api to check the cache if it is also the case? [http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-shard-query-cache.html#\_monitoring\_cache\_usage](http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-shard-query-cache.html#_monitoring_cache_usage) There are also a few additional links which might give you hints. > **[3 Performance Tuning Tips For ElasticSearch - EverTrue Engineering](http://evertrue.github.io/blog/2014/11/16/3-performance-tuning-tips-for-elasticsearch/)** > > 3 Performance Tuning Tips For ElasticSearch Nov 16th, 2014 Mark Greene Over the last year, we’ve run into three main tuning scenarios where … > > > This is followup for \[my email\](https://groups.google.com/forum/#!searchin/elast…icsearch/CPU$20consumption$20after$20long$20period$20of$20time/elasticsearch/\_ihDrmVExLA/HaOgH3SE2RsJ). > > Here you may see what happens if you fill the whole heap with field cache (big query with faceting for 50gb of data). Indexing happens every 10 minutes and with full cache it becomes very painful because of GC. > > !\[cache full\](http://puu.sh/4kios.png) > > And this is what happens after \`/\_cache/clear\`: > > !\[cache clear\](http://puu.sh/4kiri.png) > > We run elasticsearch 0.90.2 like that: > > \`/usr/lib/jvm/oracle-jdk-bin-1.7/bin/java -Xms5g -Xmx5g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/opt/elasticsearch -cp :/opt/elasticsearch/lib/elasticsearch-0.90.2.jar:/opt/elasticsearch/lib/\*:/opt/elasticsearch/lib/sigar/\* -Des.config=/etc/elasticsearch/elasticsearch.yml org.elasticsearch.bootstrap.ElasticSearch\` > > Settings in yml file only affect data location and network settings. > > Is this expected behaviour? Elasticsearch becomes very slow even for simple "get doc by id" (like 10-20 seconds instead of 10-500ms). > **[Limiting Memory Usage | Elasticsearch: The Definitive Guide \[2.x\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html)** Hope it helps. Jason On Tuesday, September 16, 2014 at 10:25:08 PM UTC+8, Philippe Laflamme wrote: > Sorry for bumping this, but I'm a little stumped here. > > We have some nodes that are evicting fielddata cache entries for seemingly > no reason: > > 1. we've set indices.fielddata.cache.size to 10gb > 2. the metrics from the node stats endpoint show that the > indices.fielddata.memory\_size\_in\_bytes never exceeded 3.6GB on any node. > 3. the rate of eviction is normally 0, but goes up above that eventhough > the fielddata cache size is nowhere near 10GB > > Attached is a plot of the max(indices.fielddata.memory\_size\_in\_bytes) (red > line) and sum(indices.fielddata.evictions) (green line) across all nodes in > the cluster. Note that we create a fresh new index every day that replaces > an older one (that explains the change in profile around midnight). > > As you can see, the size (on any given node) never exceeds 3.6GB, yet even > at a lower value (around 2.2GB), some nodes start evicting entries from the > cache. Also, starting around Tue 8AM, the max(field cache size) becomes > erratic and jumps up and down. > > I can't explain this behaviour, especially since we've been operating for > a while at this volume and rate of documents. This was not happening > before. Though it's possible that we're getting a higher volume of data, it > doesn't look substantially different from the past. > > Under what circumstances will an ES node evict entries from it's field > data cache? We're also deleting documents from the index, can this have an > impact? What other things should I be looking it to find a correlation (GC > time does not seem to be correlated)? > > Thanks, > Philippe > > On Friday, September 12, 2014 9:33:16 AM UTC-4, Philippe Laflamme wrote: > > > Forgot to mention that we're using ES 1.1.1 > > > > On Friday, September 12, 2014 9:21:23 AM UTC-4, Philippe Laflamme wrote: > > > > > Hi, > > > > > > I have a cluster with nodes configured with a 18G heap. We've noticed a > > > degradation in performance recently after increasing the volume of data > > > we're indexing. > > > > > > I think the issue is due to the field data cache doing eviction. Some > > > nodes are doing lots of them, some aren't doing any. This is explained by > > > our routing strategy which results in non-uniform document distribution. > > > Maybe we can improve this eventually, but in the meantime, I'm trying to > > > understand why the nodes are evicting cached data. > > > > > > The metrics show that the field data cache is only ~1.5GB in size, yet > > > we have this in our elasticsearch.yml: > > > > > > indices.fielddata.cache.size: 10gb > > > > > > Why would a node evict cache entries when it should still have plenty of > > > room to store more? Are we missing another setting? Is there a way to tell > > > what the actual fielddata cache size is at runtime (maybe it did not pickup > > > the configuration setting for some reason)? > > > > > > Thanks, > > > Philippe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b12d4d45-94ab-4110-831a-0abd8a651a9b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b12d4d45-94ab-4110-831a-0abd8a651a9b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:18am UTC](https://discuss.elastic.co/t/field-data-cache-size-and-eviction/19757/5 "2017-07-06T00:18:05Z")