# Adding a new field data type to field-data-cache service seems to be impossible? (0.90.5) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 21, 2013, 10:36am UTC](https://discuss.elastic.co/t/adding-a-new-field-data-type-to-field-data-cache-service-seems-to-be-impossible-0-90-5/14034 "2013-10-21T10:36:23Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Tikitu\_de\_Jager](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tikitu_de_jager/32/1258_2.png) [@Tikitu\_de\_Jager](https://discuss.elastic.co/u/Tikitu_de_Jager)\ **Post date:** [October 21, 2013, 10:36am UTC](https://discuss.elastic.co/t/adding-a-new-field-data-type-to-field-data-cache-service-seems-to-be-impossible-0-90-5/14034/1 "2013-10-21T10:36:23Z")
Hi folks, I've inherited some 0.20 code which essentially adds a new field data type (analysed strings stored only as hash-values for the low memory signature; that's enough to do some statistical analysis on things like term frequencies). (The code is the experimental part of Boaz Leskes's elasticfacets library, [http://github.com/bleskes/elasticfacets](http://github.com/bleskes/elasticfacets) -- we're probably the only folk using it in production, and we'd rather move to 0.90 now instead of waiting for the 1.0 release, which I presume he will support when it arrives.) I'm concerned that this may not be possible at all in 0.90. I'm looking at index.fielddata.IndexFieldDataService and I don't see a way to get an entry into the cache there unless it's one of the builtin types. (Once the value is in, one can get it back out using getForField() with a custom subclass of FieldDataType for the type argument, but as far as I can see getForField() is also the only way to put values _into_ the cache, and it works with a fixed list of builders.) Can anyone confirm that there is no way to get custom field data types into the cache (in 0.90), or point me in the right direction if there is? Cheers, Tikitu -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 21, 2013, 11:23am UTC](https://discuss.elastic.co/t/adding-a-new-field-data-type-to-field-data-cache-service-seems-to-be-impossible-0-90-5/14034/2 "2013-10-21T11:23:20Z")
Hi Tikitu, You are looking at the right place and there is no way to inject custom fielddata types without writing your own facet which doesn't rely on the standard IndexFieldDataService to load it's data. There are some thoughts on introducing approximation methods for statistical methods (like hyperloglog, which is different than what you need) but these are all at a very early stage and will be post 1.0. That said, v0.90 includes some huge memory savers, one of which is storing strings in utf8 in memory. This should save ~50% of the memory signature for those values. Another feature is the ability to to tweak which values are loaded into the field data cache. See: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html#field-data-filtering) . I would investigate whether these are enough of a saving for you to not need to rely on the hashed strings field data cache offered by the elasticfacet plugin. Cheers, Boaz On Monday, October 21, 2013 12:36:23 PM UTC+2, Tikitu de Jager wrote: > Hi folks, > > I've inherited some 0.20 code which essentially adds a new field data type > (analysed strings stored only as hash-values for the low memory signature; > that's enough to do some statistical analysis on things like term > frequencies). (The code is the experimental part of Boaz Leskes's > elasticfacets library, [GitHub - bleskes/elasticfacets: A set of facets and related tools for ElasticSearch](http://github.com/bleskes/elasticfacets) -- we're > probably the only folk using it in production, and we'd rather move to 0.90 > now instead of waiting for the 1.0 release, which I presume he will support > when it arrives.) > > I'm concerned that this may not be possible at all in 0.90. I'm looking at > index.fielddata.IndexFieldDataService and I don't see a way to get an entry > into the cache there unless it's one of the builtin types. (Once the value > is in, one can get it back out using getForField() with a custom subclass > of FieldDataType for the type argument, but as far as I can see > getForField() is also the only way to put values _into_ the cache, and it > works with a fixed list of builders.) > > Can anyone confirm that there is no way to get custom field data types > into the cache (in 0.90), or point me in the right direction if there is? > > Cheers, > Tikitu -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tikitu\_de\_Jager](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tikitu_de_jager/32/1258_2.png) [@Tikitu\_de\_Jager](https://discuss.elastic.co/u/Tikitu_de_Jager)\ **Post date:** [October 21, 2013, 11:53am UTC](https://discuss.elastic.co/t/adding-a-new-field-data-type-to-field-data-cache-service-seems-to-be-impossible-0-90-5/14034/3 "2013-10-21T11:53:53Z")
Thanks Boaz for the quick reply. It seems like we don't have much choice (not wanting to reimplement the caching layer ourselves...). It would be fantastic if a plugin were able to register a field data type (with Builder) instead of restricting things to the private-final-Immutable builtins. In our case, for instance, just using the .hashCode() of the terms is enough for our use case: no matter what memory improvements come with 0.90, replacing strings with ints has _got_ to give (yet) more gain. Thanks, Tikitu On Monday, 21 October 2013 14:23:20 UTC+3, Boaz Leskes wrote: > Hi Tikitu, > > You are looking at the right place and there is no way to inject custom > fielddata types without writing your own facet which doesn't rely on the > standard IndexFieldDataService to load it's data. > > There are some thoughts on introducing approximation methods for > statistical methods (like hyperloglog, which is different than what you > need) but these are all at a very early stage and will be post 1.0. > > That said, v0.90 includes some huge memory savers, one of which is storing > strings in utf8 in memory. This should save ~50% of the memory signature > for those values. Another feature is the ability to to tweak which values > are loaded into the field data cache. See: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html#field-data-filtering). I would investigate whether these are enough of a saving for you to not > need to rely on the hashed strings field data cache offered by the > elasticfacet plugin. > > Cheers, > Boaz > > On Monday, October 21, 2013 12:36:23 PM UTC+2, Tikitu de Jager wrote: > > > Hi folks, > > > > I've inherited some 0.20 code which essentially adds a new field data > > type (analysed strings stored only as hash-values for the low memory > > signature; that's enough to do some statistical analysis on things like > > term frequencies). (The code is the experimental part of Boaz Leskes's > > elasticfacets library, [GitHub - bleskes/elasticfacets: A set of facets and related tools for ElasticSearch](http://github.com/bleskes/elasticfacets) -- we're > > probably the only folk using it in production, and we'd rather move to 0.90 > > now instead of waiting for the 1.0 release, which I presume he will support > > when it arrives.) > > > > I'm concerned that this may not be possible at all in 0.90. I'm looking > > at index.fielddata.IndexFieldDataService and I don't see a way to get an > > entry into the cache there unless it's one of the builtin types. (Once the > > value is in, one can get it back out using getForField() with a custom > > subclass of FieldDataType for the type argument, but as far as I can see > > getForField() is also the only way to put values _into_ the cache, and > > it works with a fixed list of builders.) > > > > Can anyone confirm that there is no way to get custom field data types > > into the cache (in 0.90), or point me in the right direction if there is? > > > > Cheers, > > Tikitu -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:11am UTC](https://discuss.elastic.co/t/adding-a-new-field-data-type-to-field-data-cache-service-seems-to-be-impossible-0-90-5/14034/4 "2017-07-06T02:11:27Z")