# Memory building up while faceting on multi valued fields **URL:** \ **Category:** Elasticsearch\ **Created:** [February 19, 2013, 9:01am UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794 "2013-02-19T09:01:04Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Leonardo\_Menezes\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leonardo_menezes_2/32/2487_2.png) [@Leonardo\_Menezes\_2](https://discuss.elastic.co/u/Leonardo_Menezes_2)\ **Post date:** [February 19, 2013, 9:01am UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/1 "2013-02-19T09:01:04Z")
Hey hello everybody, for the past weeks We have been experiencing some memory problems with elasticsearch, and after running some tests We narrowed it down to faceting on multi valued string fields. We don't see OOM, but We see how overtime memory start to build up and after a few days it's just not able to free up any memory(leading to really big pauses of GC). Attached goes a picture of heap size for the 2 clusters. They are exactly the same(same hardware, settings, data,queries and etc), with the only difference that for the first one we removed faceting on the multi valued string fields. The test has been running since friday the 15h until this morning, when the cluster already start to struggle for memory. We have field data/filter cache limited(they amount to less than 5GB summed up). Every node runs on a 40GB JVM, on servers with 64GB and 24 cores. Each index has 4 shards, and the shard size on disk is about 8GB. Any suggestions on where to look for? Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 19, 2013, 8:20pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/2 "2013-02-19T20:20:05Z")
There is a known issue regarding facets on multi-valued fields: > > > Hi, > > Our ElasticSearch instance contains circa 240 million messages. Each messag…e can have one or more tags id associated with it. These ids are stored as an array of integers and we facet it using the standard terms facet. > > The facet cache size for this field is ~55GB which is highly surprising as only 5 million messages actually do have tags (tagging is manual). Also the total number of tag applications is only ~15 million. > > Yesterday our ES cluster died due to lack of memory. > > Researching it I have pin down the issue to the way the MultiValued\*Field caches work - For every segment it allocates memory space which is proportionate to the max number of values per docs \\\* maxDocs of that segment. > > In our case we had 3 messages with 100 tags which caused ElasticSearch to allocate 100\*24 million integers on 3 of the 10 shards we use (27.5 GB in total ). The rest of the shards each had at least one message with ~50 tags which is less dramatic but has a similar high consumption. > > I understand why the current MultiValueIntFieldData implementation is set as it is right now, but in our case it leads to extreme results. > > We currently worked around it by delete the tags from the top 200 messages which reduced memory considerably but this a short term solution. > > I have started working on a an alternative data structure which will solve things for us. I will submit a pull request as soon as it is ready. > > Cheers, > Boaz Do your documents vary in the amount of terms for the faceted field? People that had high variance tended to have the most issues. Facets have been re-architected in master (0.21). Shay posted something about the release last week. You can try running master to see if it helps. -- Ivan On Tue, Feb 19, 2013 at 1:01 AM, Leonardo Menezes [mail@lmenezes.com](mailto:mail@lmenezes.com) wrote: > Hey hello everybody, > for the past weeks We have been experiencing some memory problems with > elasticsearch, and after running some tests We narrowed it down to faceting > on multi valued string fields. We don't see OOM, but We see how overtime > memory start to build up and after a few days it's just not able to free up > any memory(leading to really big pauses of GC). Attached goes a picture of > heap size for the 2 clusters. They are exactly the same(same hardware, > settings, data,queries and etc), with the only difference that for the > first one we removed faceting on the multi valued string fields. The test > has been running since friday the 15h until this morning, when the cluster > already start to struggle for memory. > We have field data/filter cache limited(they amount to less than 5GB > summed up). Every node runs on a 40GB JVM, on servers with 64GB and 24 > cores. Each index has 4 shards, and the shard size on disk is about 8GB. > Any suggestions on where to look for? Thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Leonardo\_Menezes](https://avatars.discourse-cdn.com/v4/letter/l/b487fb/32.png) [@Leonardo\_Menezes](https://discuss.elastic.co/u/Leonardo_Menezes)\ **Post date:** [February 20, 2013, 9:15am UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/3 "2013-02-20T09:15:44Z")
Hey Ivan, thanks for your reply. I have read that before, and even though I also experience some high memory usage, that isn't really a problem for me(even though it would be nicer having less memory requirements). I'm more worried now about the fact that the memory seems to be building up, and that for each iteration of the CMS GC, less memory is actually freed up, until it runs really tight on memory and eventually collapses due to really long GC pauses. Thanks anyway [http://twitter.com/leonardomenezes](http://twitter.com/leonardomenezes) On Tue, Feb 19, 2013 at 9:20 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > There is a known issue regarding facets on multi-valued fields: > [Unrealistic high memory consumption for faceting of infrequent array fields with many members · Issue #2468 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2468) > > Do your documents vary in the amount of terms for the faceted field? > People that had high variance tended to have the most issues. > > Facets have been re-architected in master (0.21). Shay posted something > about the release last week. You can try running master to see if it helps. > > -- > Ivan > > On Tue, Feb 19, 2013 at 1:01 AM, Leonardo Menezes [mail@lmenezes.com](mailto:mail@lmenezes.com)wrote: > > > Hey hello everybody, > > for the past weeks We have been experiencing some memory problems > > with elasticsearch, and after running some tests We narrowed it down to > > faceting on multi valued string fields. We don't see OOM, but We see how > > overtime memory start to build up and after a few days it's just not able > > to free up any memory(leading to really big pauses of GC). Attached goes a > > picture of heap size for the 2 clusters. They are exactly the same(same > > hardware, settings, data,queries and etc), with the only difference that > > for the first one we removed faceting on the multi valued string fields. > > The test has been running since friday the 15h until this morning, when the > > cluster already start to struggle for memory. > > We have field data/filter cache limited(they amount to less than 5GB > > summed up). Every node runs on a 40GB JVM, on servers with 64GB and 24 > > cores. Each index has 4 shards, and the shard size on disk is about 8GB. > > Any suggestions on where to look for? Thanks > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 21, 2013, 2:25pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/4 "2013-02-21T14:25:15Z")
Hi Leo > thanks for your reply. I have read that before, and even though I > also experience some high memory usage, that isn't really a problem > for me(even though it would be nicer having less memory requirements). > I'm more worried now about the fact that the memory seems to be > building up, and that for each iteration of the CMS GC, less memory is > actually freed up, until it runs really tight on memory and eventually > collapses due to really long GC pauses. Thanks anyway Note that the field data "cache" isn't really a cache. It doesn't get freed because, chances are, you're just going to need that data again the next time you run the query anyway. For this reason, in the next version of ES, field data is no longer referred to as a cache. Also in the next version, memory usage for multi-valued fields is much better than in the current version. All I can suggest for now is to add nodes (or RAM). clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Leonardo\_Menezes](https://avatars.discourse-cdn.com/v4/letter/l/b487fb/32.png) [@Leonardo\_Menezes](https://discuss.elastic.co/u/Leonardo_Menezes)\ **Post date:** [February 21, 2013, 2:32pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/5 "2013-02-21T14:32:21Z")
Hey Clinton, thanks for the reply. Being considered as "cache" or not, that should still be reported by the stats, right? I mean, I get around 3gb reported as being used for field data and 1gb for filter cache, but we have a total of 40gb of heap. Regarding 0.21, we are already preparing our systems to try that, since we are still not live and can just try it. Anyway, if you have any suggestion on where to look for the "rest" of the memory... cheers, leo [http://es.linkedin.com/in/leonardomenezess](http://es.linkedin.com/in/leonardomenezess) [http://twitter.com/leonardomenezes](http://twitter.com/leonardomenezes) On Thu, Feb 21, 2013 at 3:25 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > Hi Leo > > > thanks for your reply. I have read that before, and even though I > > also experience some high memory usage, that isn't really a problem > > for me(even though it would be nicer having less memory requirements). > > I'm more worried now about the fact that the memory seems to be > > building up, and that for each iteration of the CMS GC, less memory is > > actually freed up, until it runs really tight on memory and eventually > > collapses due to really long GC pauses. Thanks anyway > > Note that the field data "cache" isn't really a cache. It doesn't get > freed because, chances are, you're just going to need that data again > the next time you run the query anyway. > > For this reason, in the next version of ES, field data is no longer > referred to as a cache. > > Also in the next version, memory usage for multi-valued fields is much > better than in the current version. All I can suggest for now is to add > nodes (or RAM). > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 21, 2013, 3:03pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/6 "2013-02-21T15:03:03Z")
Heya > ``` > thanks for the reply. Being considered as "cache" or not, that > > ``` > > should still be reported by the stats, right? I mean, I get around 3gb > reported as being used for field data and 1gb for filter cache, but we > have a total of 40gb of heap. OK - if it is reporting it as 3GB then that should be all there is that is being used for field values (you haven't got soft refs turned on, have you). Btw, you really don't want to use 40GB of heap. Below 32GB Java can use compressed pointers. Above that and you're wasting space and making GC more difficult. Are you using mmapfs? If not, consider doing that and reducing your heap size. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Leonardo\_Menezes](https://avatars.discourse-cdn.com/v4/letter/l/b487fb/32.png) [@Leonardo\_Menezes](https://discuss.elastic.co/u/Leonardo_Menezes)\ **Post date:** [February 21, 2013, 3:08pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/7 "2013-02-21T15:08:44Z")
We tried 30GB but we ran into the same problem, that's why we actually incremented to 40gb. About the mmapfs, isnt nio the recommended one? Any reason to believe nio might have a problem? [http://twitter.com/leonardomenezes](http://twitter.com/leonardomenezes) On Thu, Feb 21, 2013 at 4:03 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > Heya > > > ``` > > thanks for the reply. Being considered as "cache" or not, that > > > > ``` > > > > should still be reported by the stats, right? I mean, I get around 3gb > > reported as being used for field data and 1gb for filter cache, but we > > have a total of 40gb of heap. > > OK - if it is reporting it as 3GB then that should be all there is that > is being used for field values (you haven't got soft refs turned on, > have you). > > Btw, you really don't want to use 40GB of heap. Below 32GB Java can use > compressed pointers. Above that and you're wasting space and making GC > more difficult. > > Are you using mmapfs? If not, consider doing that and reducing your heap > size. > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 21, 2013, 3:17pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/8 "2013-02-21T15:17:50Z")
On Thu, 2013-02-21 at 16:08 +0100, Leonardo Menezes wrote: > We tried 30GB but we ran into the same problem, that's why we actually > incremented to 40gb. About the mmapfs, isnt nio the recommended one? > Any reason to believe nio might have a problem? You're on 64 bit? If so, give this a read: > **[Use Lucene’s MMapDirectory on 64bit platforms, please!](https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html)** > > Don’t be afraid – Some clarification to common misunderstandings Since version 3.1, Apache Lucene and Solr use MMapDirectory by ... but either way, you don't want your heap above 30GB. Rather leave the rest of the RAM for your file system caches. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Leonardo\_Menezes](https://avatars.discourse-cdn.com/v4/letter/l/b487fb/32.png) [@Leonardo\_Menezes](https://discuss.elastic.co/u/Leonardo_Menezes)\ **Post date:** [February 22, 2013, 12:51pm UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/9 "2013-02-22T12:51:47Z")
Clinton, excuse my french, but you are a fucking genius 😉 Changing the store type to mmapfs radically changed the memory buildup. It's still a bit soon to say if it really fixed the problem, but We have a test that would consistently make the problem appear in about 5h, and now it's been running for more than 24h and still ok. I dont know if other people experienced that before, but maybe mmap should be the default, or at least, recommended setting for 64bit systems? thanks again leo [http://twitter.com/leonardomenezes](http://twitter.com/leonardomenezes) On Thu, Feb 21, 2013 at 4:17 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > On Thu, 2013-02-21 at 16:08 +0100, Leonardo Menezes wrote: > > > We tried 30GB but we ran into the same problem, that's why we actually > > incremented to 40gb. About the mmapfs, isnt nio the recommended one? > > Any reason to believe nio might have a problem? > > You're on 64 bit? > > If so, give this a read: > [The Generics Policeman Blog: Use Lucene’s MMapDirectory on 64bit platforms, please!](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) > > but either way, you don't want your heap above 30GB. Rather leave the > rest of the RAM for your file system caches. > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:50am UTC](https://discuss.elastic.co/t/memory-building-up-while-faceting-on-multi-valued-fields/10794/10 "2017-07-06T02:50:05Z")