# Creating new facet types **URL:** \ **Category:** Elasticsearch\ **Created:** [February 15, 2012, 12:01am UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708 "2012-02-15T00:01:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 15, 2012, 12:01am UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/1 "2012-02-15T00:01:09Z")
I am currently attempting to convert a custom Lucene collector to an ElasticSearch facet. The facet is similar to a RangeFacet, but with dynamic ranges. I have tried to use the existing RangeFacet and HistogramFacet, but have not been able to create the a facet with a fixed number of buckets and no predefined ranges. From what I have gathered from viewing the source, these appear to be the main steps (and questions) when creating a new facet: 1. Create a plugin that creates a new FacetProcessor (see below) to add to the FacetModule. 2. Create a new FacetProcessor class. The types() and parse(...) are pretty simple to understand. The reduce method is a bit trickier, but I think I understand its usage. When/why do facets need to be merge? Also, what is the purpose of the various regsiterStreams in the FacetProcessor constuctors? 3. Create a new FacetCollector class. The meat of the faceting process. Contains the logic that creates the actual facets. 4. Create a new Facet class. Primarily a POJO. Am I missing anything else? The process seems daunting, but hopefully it will be alleviated since I should be able to build upon the existing classes (skipping points 3 + 4). What the dynamic range algorithm is actualy doing is executing a range facet on pre-established static ranges (based on historical data) and then combining those ranges evenly among a defined number of buckets. Cheers, Ivan ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 22, 2012, 6:24am UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/2 "2012-02-22T06:24:34Z")
Any insights on creating a new facet type? Especially the purpose of registerStreams(). -- Ivan On Tue, Feb 14, 2012 at 4:01 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > I am currently attempting to convert a custom Lucene collector to an > Elasticsearch facet. The facet is similar to a RangeFacet, but with > dynamic ranges. I have tried to use the existing RangeFacet and > HistogramFacet, but have not been able to create the a facet with a > fixed number of buckets and no predefined ranges. > > From what I have gathered from viewing the source, these appear to be > the main steps (and questions) when creating a new facet: > > 1. Create a plugin that creates a new FacetProcessor (see below) to > add to the FacetModule. > > 2. Create a new FacetProcessor class. The types() and parse(...) are > pretty simple to understand. The reduce method is a bit trickier, but > I think I understand its usage. When/why do facets need to be merge? > Also, what is the purpose of the various regsiterStreams in the > FacetProcessor constuctors? > > 3. Create a new FacetCollector class. The meat of the faceting > process. Contains the logic that creates the actual facets. > > 4. Create a new Facet class. Primarily a POJO. > > Am I missing anything else? The process seems daunting, but hopefully > it will be alleviated since I should be able to build upon the > existing classes (skipping points 3 + 4). What the dynamic range > algorithm is actualy doing is executing a range facet on > pre-established static ranges (based on historical data) and then > combining those ranges evenly among a defined number of buckets. > > Cheers, > > Ivan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 24, 2012, 1:26pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/3 "2012-02-24T13:26:30Z")
registerStreams is aimed at registering how the facets will get serialized between nodes. The reduce part is important, because facet execute on each shard, and then they need to be reduced into a single result. Note, this API is quite low level and expected to change, very very rough currently. On Wednesday, February 22, 2012 at 8:24 AM, Ivan Brusic wrote: > Any insights on creating a new facet type? Especially the purpose of > registerStreams(). > > -- > Ivan > > On Tue, Feb 14, 2012 at 4:01 PM, Ivan Brusic \<[ivan@brusic.com](mailto:ivan@brusic.com) ([mailto:ivan@brusic.com](mailto:ivan@brusic.com))\> wrote: > > > I am currently attempting to convert a custom Lucene collector to an > > Elasticsearch facet. The facet is similar to a RangeFacet, but with > > dynamic ranges. I have tried to use the existing RangeFacet and > > HistogramFacet, but have not been able to create the a facet with a > > fixed number of buckets and no predefined ranges. > > > > From what I have gathered from viewing the source, these appear to be > > the main steps (and questions) when creating a new facet: > > > > 1. Create a plugin that creates a new FacetProcessor (see below) to > > add to the FacetModule. > > > > 2. Create a new FacetProcessor class. The types() and parse(...) are > > pretty simple to understand. The reduce method is a bit trickier, but > > I think I understand its usage. When/why do facets need to be merge? > > Also, what is the purpose of the various regsiterStreams in the > > FacetProcessor constuctors? > > > > 3. Create a new FacetCollector class. The meat of the faceting > > process. Contains the logic that creates the actual facets. > > > > 4. Create a new Facet class. Primarily a POJO. > > > > Am I missing anything else? The process seems daunting, but hopefully > > it will be alleviated since I should be able to build upon the > > existing classes (skipping points 3 + 4). What the dynamic range > > algorithm is actualy doing is executing a range facet on > > pre-established static ranges (based on historical data) and then > > combining those ranges evenly among a defined number of buckets. > > > > Cheers, > > > > Ivan ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [June 26, 2012, 7:21pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/4 "2012-06-26T19:21:08Z")
Out of curiosity, is there any plans for revisiting the facet API in the near future? After months of delaying creating a new facet, the time has finally come for me to write it! -- Ivan On Fri, Feb 24, 2012 at 5:26 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > registerStreams is aimed at registering how the facets will get serialized > between nodes. The reduce part is important, because facet execute on each > shard, and then they need to be reduced into a single result. > > Note, this API is quite low level and expected to change, very very rough > currently. > > On Wednesday, February 22, 2012 at 8:24 AM, Ivan Brusic wrote: > > Any insights on creating a new facet type? Especially the purpose of > registerStreams(). > > -- > Ivan > > On Tue, Feb 14, 2012 at 4:01 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > I am currently attempting to convert a custom Lucene collector to an > Elasticsearch facet. The facet is similar to a RangeFacet, but with > dynamic ranges. I have tried to use the existing RangeFacet and > HistogramFacet, but have not been able to create the a facet with a > fixed number of buckets and no predefined ranges. > > From what I have gathered from viewing the source, these appear to be > the main steps (and questions) when creating a new facet: > > 1. Create a plugin that creates a new FacetProcessor (see below) to > add to the FacetModule. > > 2. Create a new FacetProcessor class. The types() and parse(...) are > pretty simple to understand. The reduce method is a bit trickier, but > I think I understand its usage. When/why do facets need to be merge? > Also, what is the purpose of the various regsiterStreams in the > FacetProcessor constuctors? > > 3. Create a new FacetCollector class. The meat of the faceting > process. Contains the logic that creates the actual facets. > > 4. Create a new Facet class. Primarily a POJO. > > Am I missing anything else? The process seems daunting, but hopefully > it will be alleviated since I should be able to build upon the > existing classes (skipping points 3 + 4). What the dynamic range > algorithm is actualy doing is executing a range facet on > pre-established static ranges (based on historical data) and then > combining those ranges evenly among a defined number of buckets. > > Cheers, > > Ivan ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [June 26, 2012, 11:10pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/5 "2012-06-26T23:10:47Z")
On Tuesday, June 26, 2012 12:21:08 PM UTC-7, Ivan Brusic wrote: > Out of curiosity, is there any plans for revisiting the facet API in > the near future? After months of delaying creating a new facet, the > time has finally come for me to write it! I also wanted to get working on that geo-clustering facet [1], so this would be good to know! [1] [Facet for clustering geo\_points · Issue #1689 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1689) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 27, 2012, 6:09pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/6 "2012-06-27T18:09:54Z")
Yes, there is a plan to revisit the facet execution model, so there will probably be changes. @eric: Really want to get the geo clustering as a facet, want to tackle it after the refactoring. On Wed, Jun 27, 2012 at 1:10 AM, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > On Tuesday, June 26, 2012 12:21:08 PM UTC-7, Ivan Brusic wrote: > > > Out of curiosity, is there any plans for revisiting the facet API in > > the near future? After months of delaying creating a new facet, the > > time has finally come for me to write it! > > I also wanted to get working on that geo-clustering facet [1], so this > would be good to know! > > [1] [Facet for clustering geo\_points · Issue #1689 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1689) ---
**Author:** ![sujoysett](https://avatars.discourse-cdn.com/v4/letter/s/2acd7d/32.png) [@sujoysett](https://discuss.elastic.co/u/sujoysett)\ **Post date:** [July 18, 2012, 1:35pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/7 "2012-07-18T13:35:50Z")
Hi, As far I understand from the structure of custom facet plugin development, the code within the the Collector class is executed as many times as the number of shards, one thread for each shard, whereas the code within the Processor class is executed once for aggregating the result available from each Collector response. Please correct me if I am wrong in this basic structure. Assuming the above, is there any way to access the lucene based file system for a shard directly in its raw format from the Collector class? What I am trying is to bypass the doCollect(docId) or forEachValueInDoc(aggregator) methods, which deals with the tokens as available from the index, and access the raw lucene file system directly for some analysis purpose. Is this easily possible? Thanks, Sujoy. On Wednesday, June 27, 2012 11:39:54 PM UTC+5:30, kimchy wrote: > Yes, there is a plan to revisit the facet execution model, so there will > probably be changes. > @eric: Really want to get the geo clustering as a facet, want to tackle it > after the refactoring. > > On Wed, Jun 27, 2012 at 1:10 AM, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > > > On Tuesday, June 26, 2012 12:21:08 PM UTC-7, Ivan Brusic wrote: > > > > > Out of curiosity, is there any plans for revisiting the facet API in > > > the near future? After months of delaying creating a new facet, the > > > time has finally come for me to write it! > > > > I also wanted to get working on that geo-clustering facet [1], so this > > would be good to know! > > > > [1] [Facet for clustering geo\_points · Issue #1689 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1689) ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [July 19, 2012, 12:23am UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/8 "2012-07-19T00:23:12Z")
I'm note sure this applies in your case, but in a regular Lucene Collector you can access the field cache for each segment within a Lucene index. I believe this is the recommended way to work with a few fields within a doc and not try to load the whole document. See information in the book "Lucene in Action" for how this is done. If you can put just what you need for collection into a nice small field then any one field cache per segment can be small. -Paul On 7/18/2012 6:35 AM, Sujoy Sett wrote: > Assuming the above, is there any way to access the lucene based file > system for a shard directly in its raw format from the Collector > class? What I am trying is to bypass the doCollect(docId) > or forEachValueInDoc(aggregator) methods, which deals with the tokens > as available from the index, and access the raw lucene file system > directly for some analysis purpose. Is this easily possible? ---
**Author:** ![sujoysett](https://avatars.discourse-cdn.com/v4/letter/s/2acd7d/32.png) [@sujoysett](https://discuss.elastic.co/u/sujoysett)\ **Post date:** [July 25, 2012, 2:49pm UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/9 "2012-07-25T14:49:08Z")
Hi Paul, Thanks for your response. My question was more directed towards Elasticsearch facet module structure, though eventually it redirects towards using lucene fields in its raw forms. Let me illustrate my requirement. 1. First of all, search is not my primary target, but faceting is. 2. A custom analyzer parses (text parsing, using nlp libraries) a particular field of the document and indexes it with some associated information available from the parsing. 3. Now developing custom faceting logic, I want to aggregate (using distributed map-reduce from elasticsearch facet architecture) the elements from the said field from all docs (returned by search on some other field) as per another text aggregating logic. 4. While doing the above, I have the elements from said field as a java String (or java basic types) in the faceting module (FacetCollector classes and all). What I am currently doing is parsing the elements again in this phase, and applying the text aggregating logic. (I am partially redoing here what the analyzer has already done.) 5. If I can access the elements in raw form with details (I mean as lucene TermAttribute, etc., instead of java basic types), I will be saving a huge percentage of the time needed here, cutting out the redoing of text parsing. 6. A bit of more explanation, getting the indexed field content as it was is not my requirement. I am not doing search/highlight. I want some information retrieval to run on a field and give me composite result. Elasticsearch facet provides the perfect map-reduce distributed framework for the purpose. I am trying to add to it information retrieval logic. I hope I have explained my requirement well. I can provide more clarification if needed. Ivan's starting post giving a gist towards developing custom facets was a great motivation towards using elasticsearch for this task. However, I am yet not quite familiar with all the internal code details of elasticsearch, so a suggestion / guideline on the requirement as stated above will be really helpful. Thanks, Sujoy. On Thursday, July 19, 2012 5:53:28 AM UTC+5:30, P Hill wrote: > I'm note sure this applies in your case, but in a regular Lucene > Collector you can access the field cache for each segment within a > Lucene index. > I believe this is the recommended way to work with a few fields within a > doc and not try to load the whole document. See information in the book > "Lucene in Action" for how this is done. > If you can put just what you need for collection into a nice small field > then any one field cache per segment can be small. > > -Paul > > On 7/18/2012 6:35 AM, Sujoy Sett wrote: > > > Assuming the above, is there any way to access the lucene based file > > system for a shard directly in its raw format from the Collector > > class? What I am trying is to bypass the doCollect(docId) > > or forEachValueInDoc(aggregator) methods, which deals with the tokens > > as available from the index, and access the raw lucene file system > > directly for some analysis purpose. Is this easily possible? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/creating-new-facet-types/6708/10 "2017-07-06T03:19:01Z")