# Faceting too memory hungry **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2012, 3:32pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061 "2012-06-12T15:32:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [June 12, 2012, 3:32pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/1 "2012-06-12T15:32:26Z")
Hi all, I want to return facet counts on a field that has many possible values. It is an "author" field, in a large database of documents = there are potentially even more distinct values than there are total documents. The field mapping: "author": { "type": "multi\_field", "fields": { "author": { "type" : "string", }, "untouched" : { "index" : "not\_analyzed", "type": "string", } } } (the reason for multifield is that I want to be able to search the field for "Jan Ramon" as well as "ramon jan" = analyzed, but I also want to return facet counts for the whole field = unanalyzed.) Trouble is, ES runs out of heap space as soon I ask for this facet, such as with "facets": { "author": { "terms": { "field": "author.untouched", "size": 10 } } } Is there a way to make faceting less memory-hungry? What is a good way to facet on the author field? Many thanks in advance. ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [June 12, 2012, 3:40pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/2 "2012-06-12T15:40:27Z")
Hello! Faceting on high cardinality fields can be expensive in terms of memory needed. How much heap memory your Elasticsearch nodes have assigned ? -- Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch > Hi all, > I want to return facet counts on a field that has many possible > values. It is an "author" field, in a large database of documents = > there are potentially even more distinct values than there are total > documents. > The field mapping: > "author": { > "type": "multi\_field", > "fields": { > "author": { > "type" : "string", > }, > "untouched" : { > "index" : "not\_analyzed", > "type": "string", > } > } > } > (the reason for multifield is that I want to be able to search the > field for "Jan Ramon" as well as "ramon jan" = analyzed, but I also > want to return facet counts for the whole field = unanalyzed.) > Trouble is, ES runs out of heap space as soon I ask for this facet, > such as with > "facets": { > "author": { > "terms": { > "field": "author.untouched", > "size": 10 > } > } > } > Is there a way to make faceting less memory-hungry? What is a good way > to facet on the author field? > Many thanks in advance. ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [June 12, 2012, 3:49pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/3 "2012-06-12T15:49:50Z")
Hello Rafał, On Jun 12, 5:40 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > Faceting on high cardinality fields can be expensive in terms of > memory needed. How much heap memory your Elasticsearch nodes > have assigned ? right now, it's 4GB, with memlock turned on. But if the memory for faceting grows linearly with the number of distinct values (does it?), it's bound to exceed the heap size sooner or later, no matter the absolute value. Is there a way/workaround/hack to get faceting working on such a nasty "author" field? I imagine faceting on author is a very common scenario? Btw the facet counts don't need to be 100% accurate, but if inaccurate, it must be at least "consistently inaccurate", across queries. Cheers. ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [June 12, 2012, 4:05pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/4 "2012-06-12T16:05:28Z")
Hello! You may try with different field data cache type than the default resident or setting the maximum size of the cache. I hope nobody will get mad if I point you here - [ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/) . You can get some information what to change. However if you will use limited field data cache or soft type field data cache, please be aware that rebuilding field data cache is expensive and your faceting performance will suffer because of that. -- Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch > Hello Rafał, > On Jun 12, 5:40 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > > > Faceting on high cardinality fields can be expensive in terms of > > memory needed. How much heap memory your Elasticsearch nodes > > have assigned ? > right now, it's 4GB, with memlock turned on. > But if the memory for faceting grows linearly with the number of > distinct values (does it?), it's bound to exceed the heap size sooner > or later, no matter the absolute value. > Is there a way/workaround/hack to get faceting working on such a nasty > "author" field? I imagine faceting on author is a very common > scenario? > Btw the facet counts don't need to be 100% accurate, but if > inaccurate, it must be at least "consistently inaccurate", across > queries. > Cheers. ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [June 12, 2012, 4:42pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/5 "2012-06-12T16:42:04Z")
Thanks for the link, Rafał, very useful. Counting frequency of unique strings is such a fundamental operation (+well researched and optimized to death) that I am surprised this is such a big deal in ES. Maybe I'm better off using a separate application to keep track of the string facet counts + return the top-N on request? I'm thinking a simple string-\>count mapping, based on a DB of sorts, like SQLite or Berkeley DB. Then I'd merge the search results from ES with the facet results from this app. Downside is I'd need to index things twice (in two separate places), at least for the author field, which adds a lot of system complexity. What do you think? On Jun 12, 6:05 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > Hello! > > You may try with different field data cache type than the default > resident or setting the maximum size of the cache. I hope nobody will > get mad if I point you here -[ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/). You > can get some information what to change. > > However if you will use limited field data cache or soft type field > data cache, please be aware that rebuilding field data cache is > expensive and your faceting performance will suffer because of that. > > -- > Regards, > Rafał Kuć > Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch - Elasticsearch > > > Hello Rafał, > > On Jun 12, 5:40 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > > > > > Faceting on high cardinality fields can be expensive in terms of > > > memory needed. How much heap memory your Elasticsearch nodes > > > have assigned ? > > > right now, it's 4GB, with memlock turned on. > > > But if the memory for faceting grows linearly with the number of > > > distinct values (does it?), it's bound to exceed the heap size sooner > > > or later, no matter the absolute value. > > > Is there a way/workaround/hack to get faceting working on such a nasty > > > "author" field? I imagine faceting on author is a very common > > > scenario? > > > Btw the facet counts don't need to be 100% accurate, but if > > > inaccurate, it must be at least "consistently inaccurate", across > > > queries. > > > Cheers. ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [June 12, 2012, 4:44pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/6 "2012-06-12T16:44:19Z")
Ah no, scratch that, I forgot the facet counts are query-dependent 🙂 I'll have to give this more thought, and in the mean time, turn off author faceting. On Jun 12, 6:42 pm, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > Thanks for the link, Rafał, very useful. > > Counting frequency of unique strings is such a fundamental operation > (+well researched and optimized to death) that I am surprised this is > such a big deal in ES. > > Maybe I'm better off using a separate application to keep track of the > string facet counts + return the top-N on request? I'm thinking a > simple string-\>count mapping, based on a DB of sorts, like SQLite or > Berkeley DB. Then I'd merge the search results from ES with the facet > results from this app. Downside is I'd need to index things twice (in > two separate places), at least for the author field, which adds a lot > of system complexity. > > What do you think? > > On Jun 12, 6:05 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > > > Hello! > > > You may try with different field data cache type than the default > > resident or setting the maximum size of the cache. I hope nobody will > > get mad if I point you here -[ElasticSearch Cache Usage - Sematext](http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/). You > > can get some information what to change. > > > However if you will use limited field data cache or soft type field > > data cache, please be aware that rebuilding field data cache is > > expensive and your faceting performance will suffer because of that. > > > -- > > Regards, > > Rafał Kuć > > Sematext ::[http://sematext.com/::Solr](http://sematext.com/::Solr) - Lucene - Nutch - Elasticsearch > > > > Hello Rafał, > > > On Jun 12, 5:40 pm, Rafał Kuć [r....@solr.pl](mailto:r....@solr.pl) wrote: > > > > > > > Faceting on high cardinality fields can be expensive in terms of > > > > memory needed. How much heap memory your Elasticsearch nodes > > > > have assigned ? > > > > right now, it's 4GB, with memlock turned on. > > > > But if the memory for faceting grows linearly with the number of > > > > distinct values (does it?), it's bound to exceed the heap size sooner > > > > or later, no matter the absolute value. > > > > Is there a way/workaround/hack to get faceting working on such a nasty > > > > "author" field? I imagine faceting on author is a very common > > > > scenario? > > > > Btw the facet counts don't need to be 100% accurate, but if > > > > inaccurate, it must be at least "consistently inaccurate", across > > > > queries. > > > > Cheers. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 12, 2012, 5:33pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/7 "2012-06-12T17:33:59Z")
Also have a look at [https://groups.google.com/d/msg/elasticsearch/ePJgCtBpyrs/39pzczoRokoJ](https://groups.google.com/d/msg/elasticsearch/ePJgCtBpyrs/39pzczoRokoJ) ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Tuesday, June 12, 2012 11:32:26 AM UTC-4, Crwe wrote: > Hi all, > > I want to return facet counts on a field that has many possible > values. It is an "author" field, in a large database of documents = > there are potentially even more distinct values than there are total > documents. > > The field mapping: > > "author": { > "type": "multi\_field", > "fields": { > "author": { > "type" : "string", > }, > "untouched" : { > "index" : "not\_analyzed", > "type": "string", > } > } > } > > (the reason for multifield is that I want to be able to search the > field for "Jan Ramon" as well as "ramon jan" = analyzed, but I also > want to return facet counts for the whole field = unanalyzed.) > > Trouble is, ES runs out of heap space as soon I ask for this facet, > such as with > > "facets": { > "author": { > "terms": { > "field": "author.untouched", > "size": 10 > } > } > } > > Is there a way to make faceting less memory-hungry? What is a good way > to facet on the author field? > > Many thanks in advance. ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [June 12, 2012, 6:11pm UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/8 "2012-06-12T18:11:30Z")
On Tuesday, June 12, 2012 7:33:59 PM UTC+2, Otis Gospodnetic wrote: > Also have a look at > [https://groups.google.com/d/msg/elasticsearch/ePJgCtBpyrs/39pzczoRokoJ](https://groups.google.com/d/msg/elasticsearch/ePJgCtBpyrs/39pzczoRokoJ) Excellent info, thanks Otis. Solr's FieldCache looks exactly like what we need. I'll keep an eye on the "improved faceting" progress of ES, while keeping an eye on the "improved NRT" progress in Solr -- because I need them both! Whichever engine gets there first 🙂 > > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/faceting-too-memory-hungry/8061/9 "2017-07-06T03:24:44Z")