# Accessing array field within Native Plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [October 19, 2013, 3:08pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027 "2013-10-19T15:08:05Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![peter2](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@peter2](https://discuss.elastic.co/u/peter2)\ **Post date:** [October 19, 2013, 3:08pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/1 "2013-10-19T15:08:05Z")
I'm storing some data in array type field which needs to be accessed within Native Script which is used as custom scorer with function\_score query. But when I access the field values within Native Script using docFieldDoubles I do not get the values in order. Does the array data type not maintain ordering? When I do a GET on that doc, it does show the values in that field in order, but not from within the Native script plugin. Is this a bug or is it expected? What I'm really trying to do is this. I need to maintain a Map or a set of key/value pairs where the keys are different for each document. And I need to access the key/value pairs using a known field name (from both the scoring plugin as well as from search clients). Right now, I'm storing two fields, one with keys and other with values and have both them store these in a comma delimited form. Then from within the plugin, I split on comma and based on position I figure out which key maps to which value. This is of course not very performant and I'd prefer to avoid doing that. As a first step, I tried arrays as mentioned above (instead of comma delimited string), but that seems to lose ordering. What's the best way to do this? Thanks, Peter -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 21, 2013, 11:41am UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/2 "2013-10-21T11:41:42Z")
Hi Peter, The docFieldDoubles method gets it's values from the in memory structures of the field data cache. This is done for performance. The field data cache is not loaded from source of the document (because this will be slow) but from the lucene index, where the values are sorted (for lookup speed). The get api does work based on the original document source which is why you see those values in order (note- ES doesn't the parse the source for the get api, it just gives you back what you've put in it). You can access the original document (which will be parsed) using the SourceLookup (available from the source method) but it will be slow as it needs to go to disk for every document. I'm not sure about the exact semantics of what you are trying to achieve, but did you try looking at nested objects? those allow you to store a list of object in a why that keeps values together, like [{ "key": "k1" , "value" : "v1"},...] . Cheers, Boaz On Saturday, October 19, 2013 5:08:05 PM UTC+2, [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > I'm storing some data in array type field which needs to be accessed > within Native Script which is used as custom scorer with function\_score > query. But when I access the field values within Native Script using > docFieldDoubles I do not get the values in order. Does the array data type > not maintain ordering? When I do a GET on that doc, it does show the values > in that field in order, but not from within the Native script plugin. Is > this a bug or is it expected? > > What I'm really trying to do is this. I need to maintain a Map or a set of > key/value pairs where the keys are different for each document. And I need > to access the key/value pairs using a known field name (from both the > scoring plugin as well as from search clients). Right now, I'm storing two > fields, one with keys and other with values and have both them store these > in a comma delimited form. Then from within the plugin, I split on comma > and based on position I figure out which key maps to which value. This is > of course not very performant and I'd prefer to avoid doing that. As a > first step, I tried arrays as mentioned above (instead of comma delimited > string), but that seems to lose ordering. What's the best way to do this? > > Thanks, > Peter -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![peter2](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@peter2](https://discuss.elastic.co/u/peter2)\ **Post date:** [October 21, 2013, 1:10pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/3 "2013-10-21T13:10:16Z")
Thanks, Boaz. That makes sense now. Nested objects seems like a solution, but I'm not quite sure on how I might access nested objects values from within a script scoring plugin. There seems to be two options, 1. doc().get("field") 2. fields().get("field") Both seems to use a some form of cache, but #1 only seems to support Longs, Doubles and Strings. #2 looks like it will support complex objects (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", "value": "v2"}] ). So it looks like #2 is the only option here. What's the difference between the two? #2 seems to be using a SingleFieldsVisitor to access values while #1 uses a IndexFieldDataService. It looks like both have some form of cache but #1 seems to have a proper field cache underneath the top level cache while #2 doesn't. So it looks like #2 is is not going to perform that well. Am I looking at it wrong? Thanks again for your help. Peter On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > Hi Peter, > > The docFieldDoubles method gets it's values from the in memory structures > of the field data cache. This is done for performance. The field data cache > is not loaded from source of the document (because this will be slow) but > from the lucene index, where the values are sorted (for lookup speed). The > get api does work based on the original document source which is why you > see those values in order (note- ES doesn't the parse the source for the > get api, it just gives you back what you've put in it). > > You can access the original document (which will be parsed) using the > SourceLookup (available from the source method) but it will be slow as it > needs to go to disk for every document. > > I'm not sure about the exact semantics of what you are trying to achieve, > but did you try looking at nested objects? those allow you to store a list > of object in a why that keeps values together, like [{ "key": "k1" , > "value" : "v1"},...] . > > Cheers, > Boaz > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > > > I'm storing some data in array type field which needs to be accessed > > within Native Script which is used as custom scorer with function\_score > > query. But when I access the field values within Native Script using > > docFieldDoubles I do not get the values in order. Does the array data type > > not maintain ordering? When I do a GET on that doc, it does show the values > > in that field in order, but not from within the Native script plugin. Is > > this a bug or is it expected? > > > > What I'm really trying to do is this. I need to maintain a Map or a set > > of key/value pairs where the keys are different for each document. And I > > need to access the key/value pairs using a known field name (from both the > > scoring plugin as well as from search clients). Right now, I'm storing two > > fields, one with keys and other with values and have both them store these > > in a comma delimited form. Then from within the plugin, I split on comma > > and based on position I figure out which key maps to which value. This is > > of course not very performant and I'd prefer to avoid doing that. As a > > first step, I tried arrays as mentioned above (instead of comma delimited > > string), but that seems to lose ordering. What's the best way to do this? > > > > Thanks, > > Peter -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 21, 2013, 2:28pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/4 "2013-10-21T14:28:08Z")
Hi Peter, doc().get("field") uses the field data cache discussed before. fields().get("field") uses lucene stored fields which are on disk and thus cached by the file system cache (and are typically too slow for scoring). It will sadly not support nested object as it works on the lucene document level (and nested docsare separate lucene docs). As far as I can tell the only way to get to the nested structures in a script right now is using the sourcelookup which is slow. I have some ideas about how we can potentially extend it but needs some more thinking and time. I was hoping you can do whatever you need with nested queries... If that doesn't work, perhaps you can give some examples of what you need (json + neede score) and I'll try to come up with something else. Cheers, Boaz On Mon, Oct 21, 2013 at 3:10 PM, [peter@vagaband.co](mailto:peter@vagaband.co) [peter@vagaband.co](mailto:peter@vagaband.co)wrote: > Thanks, Boaz. That makes sense now. Nested objects seems like a solution, > but I'm not quite sure on how I might access nested objects values from > within a script scoring plugin. > > There seems to be two options, > > 1. doc().get("field") > 2. fields().get("field") > > Both seems to use a some form of cache, but #1 only seems to support > Longs, Doubles and Strings. #2 looks like it will support complex objects > (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", > "value": "v2"}] ). So it looks like #2 is the only option here. > > What's the difference between the two? #2 seems to be using > a SingleFieldsVisitor to access values while #1 uses > a IndexFieldDataService. It looks like both have some form of cache but #1 > seems to have a proper field cache underneath the top level cache while #2 > doesn't. So it looks like #2 is is not going to perform that well. Am I > looking at it wrong? > > Thanks again for your help. > Peter > > On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > > > Hi Peter, > > > > The docFieldDoubles method gets it's values from the in memory structures > > of the field data cache. This is done for performance. The field data cache > > is not loaded from source of the document (because this will be slow) but > > from the lucene index, where the values are sorted (for lookup speed). The > > get api does work based on the original document source which is why you > > see those values in order (note- ES doesn't the parse the source for the > > get api, it just gives you back what you've put in it). > > > > You can access the original document (which will be parsed) using the > > SourceLookup (available from the source method) but it will be slow as it > > needs to go to disk for every document. > > > > I'm not sure about the exact semantics of what you are trying to achieve, > > but did you try looking at nested objects? those allow you to store a list > > of object in a why that keeps values together, like [{ "key": "k1" , > > "value" : "v1"},...] . > > > > Cheers, > > Boaz > > > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > > > > > I'm storing some data in array type field which needs to be accessed > > > within Native Script which is used as custom scorer with function\_score > > > query. But when I access the field values within Native Script using > > > docFieldDoubles I do not get the values in order. Does the array data type > > > not maintain ordering? When I do a GET on that doc, it does show the values > > > in that field in order, but not from within the Native script plugin. Is > > > this a bug or is it expected? > > > > > > What I'm really trying to do is this. I need to maintain a Map or a set > > > of key/value pairs where the keys are different for each document. And I > > > need to access the key/value pairs using a known field name (from both the > > > scoring plugin as well as from search clients). Right now, I'm storing two > > > fields, one with keys and other with values and have both them store these > > > in a comma delimited form. Then from within the plugin, I split on comma > > > and based on position I figure out which key maps to which value. This is > > > of course not very performant and I'd prefer to avoid doing that. As a > > > first step, I tried arrays as mentioned above (instead of comma delimited > > > string), but that seems to lose ordering. What's the best way to do this? > > > > > > Thanks, > > > Peter > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![peter2](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@peter2](https://discuss.elastic.co/u/peter2)\ **Post date:** [October 24, 2013, 2:53pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/5 "2013-10-24T14:53:42Z")
Hey Boaz, Sorry for the delay in getting back.. was out of town. So right now, I'm storing the keys and values in two separate fields as strings and delimiting them with commas within the string. And within the plugin, splitting them out. But splitting them out for every single doc during scoring is not very performant. Here's a gist with 3 files, current version of plugin, current index mappings, and function score query I'm running on it. > If you can suggest a better (a more performant way of either modeling the data or writing this scoring logic), I'd be a very happy camper. Thank you, Peter On Monday, October 21, 2013 10:28:08 AM UTC-4, Boaz Leskes wrote: > Hi Peter, > > doc().get("field") uses the field data cache discussed before. > fields().get("field") uses lucene stored fields which are on disk and thus > cached by the file system cache (and are typically too slow for scoring). > It will sadly not support nested object as it works on the lucene document > level (and nested docsare separate lucene docs). > > As far as I can tell the only way to get to the nested structures in a > script right now is using the sourcelookup which is slow. I have some ideas > about how we can potentially extend it but needs some more thinking and > time. > > I was hoping you can do whatever you need with nested queries... > > If that doesn't work, perhaps you can give some examples of what you need > (json + neede score) and I'll try to come up with something else. > > Cheers, > Boaz > > On Mon, Oct 21, 2013 at 3:10 PM, [pe...@vagaband.co](mailto:pe...@vagaband.co) \ \< > [pe...@vagaband.co](mailto:pe...@vagaband.co) \\> wrote: > > > Thanks, Boaz. That makes sense now. Nested objects seems like a solution, > > but I'm not quite sure on how I might access nested objects values from > > within a script scoring plugin. > > > > There seems to be two options, > > > > 1. doc().get("field") > > 2. fields().get("field") > > > > Both seems to use a some form of cache, but #1 only seems to support > > Longs, Doubles and Strings. #2 looks like it will support complex objects > > (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", > > "value": "v2"}] ). So it looks like #2 is the only option here. > > > > What's the difference between the two? #2 seems to be using > > a SingleFieldsVisitor to access values while #1 uses > > a IndexFieldDataService. It looks like both have some form of cache but #1 > > seems to have a proper field cache underneath the top level cache while #2 > > doesn't. So it looks like #2 is is not going to perform that well. Am I > > looking at it wrong? > > > > Thanks again for your help. > > Peter > > > > On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > > > > > Hi Peter, > > > > > > The docFieldDoubles method gets it's values from the in memory > > > structures of the field data cache. This is done for performance. The field > > > data cache is not loaded from source of the document (because this will be > > > slow) but from the lucene index, where the values are sorted (for lookup > > > speed). The get api does work based on the original document source which > > > is why you see those values in order (note- ES doesn't the parse the source > > > for the get api, it just gives you back what you've put in it). > > > > > > You can access the original document (which will be parsed) using the > > > SourceLookup (available from the source method) but it will be slow as it > > > needs to go to disk for every document. > > > > > > I'm not sure about the exact semantics of what you are trying to > > > achieve, but did you try looking at nested objects? those allow you to > > > store a list of object in a why that keeps values together, like [{ "key": > > > "k1" , "value" : "v1"},...] . > > > > > > Cheers, > > > Boaz > > > > > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > > > > > > > I'm storing some data in array type field which needs to be accessed > > > > within Native Script which is used as custom scorer with function\_score > > > > query. But when I access the field values within Native Script using > > > > docFieldDoubles I do not get the values in order. Does the array data type > > > > not maintain ordering? When I do a GET on that doc, it does show the values > > > > in that field in order, but not from within the Native script plugin. Is > > > > this a bug or is it expected? > > > > > > > > What I'm really trying to do is this. I need to maintain a Map or a set > > > > of key/value pairs where the keys are different for each document. And I > > > > need to access the key/value pairs using a known field name (from both the > > > > scoring plugin as well as from search clients). Right now, I'm storing two > > > > fields, one with keys and other with values and have both them store these > > > > in a comma delimited form. Then from within the plugin, I split on comma > > > > and based on position I figure out which key maps to which value. This is > > > > of course not very performant and I'd prefer to avoid doing that. As a > > > > first step, I tried arrays as mentioned above (instead of comma delimited > > > > string), but that seems to lose ordering. What's the best way to do this? > > > > > > > > Thanks, > > > > Peter > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 29, 2013, 11:45am UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/6 "2013-10-29T11:45:56Z")
Hi Peter, Nice! I have some ideas on how you could speed things up by using nested documents, loading those value into memory and writing your own custom score function (and a plugin) but that will quite a bit of work. As an alternative you might want to consider the query rescorer ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html#_query_rescorer)). The query rescorer allows you the first quickly get the top N results based on a lighter approximate scoring metric and the only apply the more complex one (your script) to those top N. Out curiosity - how are you planning to use the Jaccard score for? what is the use case? Cheers, Boaz On Thu, Oct 24, 2013 at 4:53 PM, [peter@vagaband.co](mailto:peter@vagaband.co) [peter@vagaband.co](mailto:peter@vagaband.co)wrote: > Hey Boaz, > > Sorry for the delay in getting back.. was out of town. > > So right now, I'm storing the keys and values in two separate fields as > strings and delimiting them with commas within the string. And within the > plugin, splitting them out. But splitting them out for every single doc > during scoring is not very performant. > > Here's a gist with 3 files, current version of plugin, current index > mappings, and function score query I'm running on it. > [JaccardScoreScript · GitHub](https://gist.github.com/ppat/7138638) > > If you can suggest a better (a more performant way of either modeling the > data or writing this scoring logic), I'd be a very happy camper. > > Thank you, > Peter > > On Monday, October 21, 2013 10:28:08 AM UTC-4, Boaz Leskes wrote: > > > Hi Peter, > > > > doc().get("field") uses the field data cache discussed before. > > fields().get("field") uses lucene stored fields which are on disk and thus > > cached by the file system cache (and are typically too slow for scoring). > > It will sadly not support nested object as it works on the lucene document > > level (and nested docsare separate lucene docs). > > > > As far as I can tell the only way to get to the nested structures in a > > script right now is using the sourcelookup which is slow. I have some ideas > > about how we can potentially extend it but needs some more thinking and > > time. > > > > I was hoping you can do whatever you need with nested queries... > > > > If that doesn't work, perhaps you can give some examples of what you need > > (json + neede score) and I'll try to come up with something else. > > > > Cheers, > > Boaz > > > > On Mon, Oct 21, 2013 at 3:10 PM, [pe...@vagaband.co](mailto:pe...@vagaband.co) [pe...@vagaband.co](mailto:pe...@vagaband.co)wrote: > > > > > Thanks, Boaz. That makes sense now. Nested objects seems like a > > > solution, but I'm not quite sure on how I might access nested objects > > > values from within a script scoring plugin. > > > > > > There seems to be two options, > > > > > > 1. doc().get("field") > > > 2. fields().get("field") > > > > > > Both seems to use a some form of cache, but #1 only seems to support > > > Longs, Doubles and Strings. #2 looks like it will support complex objects > > > (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", > > > "value": "v2"}] ). So it looks like #2 is the only option here. > > > > > > What's the difference between the two? #2 seems to be using > > > a SingleFieldsVisitor to access values while #1 uses > > > a IndexFieldDataService. It looks like both have some form of cache but #1 > > > seems to have a proper field cache underneath the top level cache while #2 > > > doesn't. So it looks like #2 is is not going to perform that well. Am I > > > looking at it wrong? > > > > > > Thanks again for your help. > > > Peter > > > > > > On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > > > > > > > Hi Peter, > > > > > > > > The docFieldDoubles method gets it's values from the in memory > > > > structures of the field data cache. This is done for performance. The field > > > > data cache is not loaded from source of the document (because this will be > > > > slow) but from the lucene index, where the values are sorted (for lookup > > > > speed). The get api does work based on the original document source which > > > > is why you see those values in order (note- ES doesn't the parse the source > > > > for the get api, it just gives you back what you've put in it). > > > > > > > > You can access the original document (which will be parsed) using the > > > > SourceLookup (available from the source method) but it will be slow as it > > > > needs to go to disk for every document. > > > > > > > > I'm not sure about the exact semantics of what you are trying to > > > > achieve, but did you try looking at nested objects? those allow you to > > > > store a list of object in a why that keeps values together, like [{ "key": > > > > "k1" , "value" : "v1"},...] . > > > > > > > > Cheers, > > > > Boaz > > > > > > > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, pe...@vagaband.cowrote: > > > > > > > > > I'm storing some data in array type field which needs to be accessed > > > > > within Native Script which is used as custom scorer with function\_score > > > > > query. But when I access the field values within Native Script using > > > > > docFieldDoubles I do not get the values in order. Does the array data type > > > > > not maintain ordering? When I do a GET on that doc, it does show the values > > > > > in that field in order, but not from within the Native script plugin. Is > > > > > this a bug or is it expected? > > > > > > > > > > What I'm really trying to do is this. I need to maintain a Map or a > > > > > set of key/value pairs where the keys are different for each document. And > > > > > I need to access the key/value pairs using a known field name (from both > > > > > the scoring plugin as well as from search clients). Right now, I'm storing > > > > > two fields, one with keys and other with values and have both them store > > > > > these in a comma delimited form. Then from within the plugin, I split on > > > > > comma and based on position I figure out which key maps to which value. > > > > > This is of course not very performant and I'd prefer to avoid doing that. > > > > > As a first step, I tried arrays as mentioned above (instead of comma > > > > > delimited string), but that seems to lose ordering. What's the best way to > > > > > do this? > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > > > > topic/elasticsearch/cI5im\_\*\*EYIDY/unsubscribe[https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![peter2](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@peter2](https://discuss.elastic.co/u/peter2)\ **Post date:** [October 29, 2013, 4:52pm UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/7 "2013-10-29T16:52:56Z")
Hi Boaz, Not quite sure if we can get away with the rescorer alone. Not sure whether it allows multiple scores to be aggregated ala function score style. Also rescoring sounds like it makes using node client tricky as that might require our score plugin to be deployed to client nodes running node client. Let me tell you about our use case from a higher level. We have machine learning processes that generate "matches" for our users or cohorts of users. We put these lists of matches per user/per cohort into ES. We use ES as the part of our infrastructure that serves this matches to our users in real time. As part of serving matches, it will allow additional filtering (maybe based on UI interaction) or sorting (i.e. Scoring by additional measures of relevance with regards to "match"), etc. So in this particular case, we're running an additional scoring algorithm on vectors (fields x,y,z in the index mappings I gave) to personalize the results to a give user. User's particular values for these vectors are given via the query and maybe behavior driven. We are using a Jaccard-esque format for determining the distance of values for these vectors between documents and the user. That might sound a bit confusing, if it does.. I can explain further. Thanks, Peter > On Oct 29, 2013, at 7:45 AM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > Hi Peter, > > Nice! > > I have some ideas on how you could speed things up by using nested documents, loading those value into memory and writing your own custom score function (and a plugin) but that will quite a bit of work. > > As an alternative you might want to consider the query rescorer ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html#_query_rescorer) ). The query rescorer allows you the first quickly get the top N results based on a lighter approximate scoring metric and the only apply the more complex one (your script) to those top N. > > Out curiosity - how are you planning to use the Jaccard score for? what is the use case? > > Cheers, > Boaz > > > On Thu, Oct 24, 2013 at 4:53 PM, [peter@vagaband.co](mailto:peter@vagaband.co) [peter@vagaband.co](mailto:peter@vagaband.co) wrote: > > Hey Boaz, > > > > Sorry for the delay in getting back.. was out of town. > > > > So right now, I'm storing the keys and values in two separate fields as strings and delimiting them with commas within the string. And within the plugin, splitting them out. But splitting them out for every single doc during scoring is not very performant. > > > > Here's a gist with 3 files, current version of plugin, current index mappings, and function score query I'm running on it. > > [JaccardScoreScript · GitHub](https://gist.github.com/ppat/7138638) > > > > If you can suggest a better (a more performant way of either modeling the data or writing this scoring logic), I'd be a very happy camper. > > > > Thank you, > > Peter > > > > > On Monday, October 21, 2013 10:28:08 AM UTC-4, Boaz Leskes wrote: > > > Hi Peter, > > > > > > doc().get("field") uses the field data cache discussed before. fields().get("field") uses lucene stored fields which are on disk and thus cached by the file system cache (and are typically too slow for scoring). It will sadly not support nested object as it works on the lucene document level (and nested docsare separate lucene docs). > > > > > > As far as I can tell the only way to get to the nested structures in a script right now is using the sourcelookup which is slow. I have some ideas about how we can potentially extend it but needs some more thinking and time. > > > > > > I was hoping you can do whatever you need with nested queries... > > > > > > If that doesn't work, perhaps you can give some examples of what you need (json + neede score) and I'll try to come up with something else. > > > > > > Cheers, > > > Boaz > > > > > > > On Mon, Oct 21, 2013 at 3:10 PM, [pe...@vagaband.co](mailto:pe...@vagaband.co) [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > > > > Thanks, Boaz. That makes sense now. Nested objects seems like a solution, but I'm not quite sure on how I might access nested objects values from within a script scoring plugin. > > > > > > > > There seems to be two options, > > > > > > > > 1. doc().get("field") > > > > 2. fields().get("field") > > > > > > > > Both seems to use a some form of cache, but #1 only seems to support Longs, Doubles and Strings. #2 looks like it will support complex objects (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", "value": "v2"}] ). So it looks like #2 is the only option here. > > > > > > > > What's the difference between the two? #2 seems to be using a SingleFieldsVisitor to access values while #1 uses a IndexFieldDataService. It looks like both have some form of cache but #1 seems to have a proper field cache underneath the top level cache while #2 doesn't. So it looks like #2 is is not going to perform that well. Am I looking at it wrong? > > > > > > > > Thanks again for your help. > > > > Peter > > > > > > > > > On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > > > > > Hi Peter, > > > > > > > > > > The docFieldDoubles method gets it's values from the in memory structures of the field data cache. This is done for performance. The field data cache is not loaded from source of the document (because this will be slow) but from the lucene index, where the values are sorted (for lookup speed). The get api does work based on the original document source which is why you see those values in order (note- ES doesn't the parse the source for the get api, it just gives you back what you've put in it). > > > > > > > > > > You can access the original document (which will be parsed) using the SourceLookup (available from the source method) but it will be slow as it needs to go to disk for every document. > > > > > > > > > > I'm not sure about the exact semantics of what you are trying to achieve, but did you try looking at nested objects? those allow you to store a list of object in a why that keeps values together, like [{ "key": "k1" , "value" : "v1"},...] . > > > > > > > > > > Cheers, > > > > > Boaz > > > > > > > > > > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, [pe...@vagaband.co](mailto:pe...@vagaband.co) wrote: > > > > > > I'm storing some data in array type field which needs to be accessed within Native Script which is used as custom scorer with function\_score query. But when I access the field values within Native Script using docFieldDoubles I do not get the values in order. Does the array data type not maintain ordering? When I do a GET on that doc, it does show the values in that field in order, but not from within the Native script plugin. Is this a bug or is it expected? > > > > > > > > > > > > What I'm really trying to do is this. I need to maintain a Map or a set of key/value pairs where the keys are different for each document. And I need to access the key/value pairs using a known field name (from both the scoring plugin as well as from search clients). Right now, I'm storing two fields, one with keys and other with values and have both them store these in a comma delimited form. Then from within the plugin, I split on comma and based on position I figure out which key maps to which value. This is of course not very performant and I'd prefer to avoid doing that. As a first step, I tried arrays as mentioned above (instead of comma delimited string), but that seems to lose ordering. What's the best way to do this? > > > > > > > > > > > > Thanks, > > > > > > Peter > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > > > To unsubscribe from this group and all its topics, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 30, 2013, 11:45am UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/8 "2013-10-30T11:45:56Z")
Hi Peter, The rescorer uses any query so you can use all the machinery out there, including function score. If you put your function score query under the rescorer. It always does a weighted sum of the query score and the recorer score, so you can tweak things to your desire. Set query\_weight to 0 if you want only the rescorer score. The rescorer runs on every shard before the results return so no need to include your scripts on the client nodes. I think I understand where you're going. Here is a trick that may speed things up: Assuming the scores are always between 0 & 1, you can store the feature index and score together - 1.20 would mean that index no 1 has score 0.2 . 20.44 means that index 20 has score 44. This has the upside that you can use the field data to load this into memory and access it via the double values. They sort in the right order. To save memory, you can go further say you only want scores to have 2 byte (or byte) accuracy. Then you can always store 3 bytes numbers where the 1 byte (highest order) is the index and the 2 least significant bytes are the score. Cheers, Boaz On Tue, Oct 29, 2013 at 5:52 PM, Peter Pathirana [peter@vagaband.co](mailto:peter@vagaband.co) wrote: > Hi Boaz, > > Not quite sure if we can get away with the rescorer alone. Not sure > whether it allows multiple scores to be aggregated ala function score > style. Also rescoring sounds like it makes using node client tricky as that > might require our score plugin to be deployed to client nodes running node > client. > > Let me tell you about our use case from a higher level. We have machine > learning processes that generate "matches" for our users or cohorts of > users. We put these lists of matches per user/per cohort into ES. We use ES > as the part of our infrastructure that serves this matches to our users in > real time. As part of serving matches, it will allow additional filtering > (maybe based on UI interaction) or sorting (i.e. Scoring by additional > measures of relevance with regards to "match"), etc. > > So in this particular case, we're running an additional scoring algorithm > on vectors (fields x,y,z in the index mappings I gave) to personalize the > results to a give user. User's particular values for these vectors are > given via the query and maybe behavior driven. > > We are using a Jaccard-esque format for determining the distance of values > for these vectors between documents and the user. > > That might sound a bit confusing, if it does.. I can explain further. > > Thanks, > Peter > > On Oct 29, 2013, at 7:45 AM, Boaz Leskes [b.leskes@gmail.com](mailto:b.leskes@gmail.com) wrote: > > Hi Peter, > > Nice! > > I have some ideas on how you could speed things up by using nested > documents, loading those value into memory and writing your own custom > score function (and a plugin) but that will quite a bit of work. > > As an alternative you might want to consider the query rescorer ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-rescore.html#_query_rescorer)). The query rescorer allows you the first quickly get the top N results > based on a lighter approximate scoring metric and the only apply the more > complex one (your script) to those top N. > > Out curiosity - how are you planning to use the Jaccard score for? what > is the use case? > > Cheers, > Boaz > > On Thu, Oct 24, 2013 at 4:53 PM, [peter@vagaband.co](mailto:peter@vagaband.co) [peter@vagaband.co](mailto:peter@vagaband.co)wrote: > > > Hey Boaz, > > > > Sorry for the delay in getting back.. was out of town. > > > > So right now, I'm storing the keys and values in two separate fields as > > strings and delimiting them with commas within the string. And within the > > plugin, splitting them out. But splitting them out for every single doc > > during scoring is not very performant. > > > > Here's a gist with 3 files, current version of plugin, current index > > mappings, and function score query I'm running on it. > > [JaccardScoreScript · GitHub](https://gist.github.com/ppat/7138638) > > > > If you can suggest a better (a more performant way of either modeling the > > data or writing this scoring logic), I'd be a very happy camper. > > > > Thank you, > > Peter > > > > On Monday, October 21, 2013 10:28:08 AM UTC-4, Boaz Leskes wrote: > > > > > Hi Peter, > > > > > > doc().get("field") uses the field data cache discussed before. > > > fields().get("field") uses lucene stored fields which are on disk and thus > > > cached by the file system cache (and are typically too slow for scoring). > > > It will sadly not support nested object as it works on the lucene document > > > level (and nested docsare separate lucene docs). > > > > > > As far as I can tell the only way to get to the nested structures in a > > > script right now is using the sourcelookup which is slow. I have some ideas > > > about how we can potentially extend it but needs some more thinking and > > > time. > > > > > > I was hoping you can do whatever you need with nested queries... > > > > > > If that doesn't work, perhaps you can give some examples of what you > > > need (json + neede score) and I'll try to come up with something else. > > > > > > Cheers, > > > Boaz > > > > > > On Mon, Oct 21, 2013 at 3:10 PM, [pe...@vagaband.co](mailto:pe...@vagaband.co) [pe...@vagaband.co](mailto:pe...@vagaband.co)wrote: > > > > > > > Thanks, Boaz. That makes sense now. Nested objects seems like a > > > > solution, but I'm not quite sure on how I might access nested objects > > > > values from within a script scoring plugin. > > > > > > > > There seems to be two options, > > > > > > > > 1. doc().get("field") > > > > 2. fields().get("field") > > > > > > > > Both seems to use a some form of cache, but #1 only seems to support > > > > Longs, Doubles and Strings. #2 looks like it will support complex objects > > > > (like the one you mentioned - [{"key": "k1", "value": "v1"},{"key": "k2", > > > > "value": "v2"}] ). So it looks like #2 is the only option here. > > > > > > > > What's the difference between the two? #2 seems to be using > > > > a SingleFieldsVisitor to access values while #1 uses > > > > a IndexFieldDataService. It looks like both have some form of cache but #1 > > > > seems to have a proper field cache underneath the top level cache while #2 > > > > doesn't. So it looks like #2 is is not going to perform that well. Am I > > > > looking at it wrong? > > > > > > > > Thanks again for your help. > > > > Peter > > > > > > > > On Monday, October 21, 2013 7:41:42 AM UTC-4, Boaz Leskes wrote: > > > > > > > > > Hi Peter, > > > > > > > > > > The docFieldDoubles method gets it's values from the in memory > > > > > structures of the field data cache. This is done for performance. The field > > > > > data cache is not loaded from source of the document (because this will be > > > > > slow) but from the lucene index, where the values are sorted (for lookup > > > > > speed). The get api does work based on the original document source which > > > > > is why you see those values in order (note- ES doesn't the parse the source > > > > > for the get api, it just gives you back what you've put in it). > > > > > > > > > > You can access the original document (which will be parsed) using the > > > > > SourceLookup (available from the source method) but it will be slow as it > > > > > needs to go to disk for every document. > > > > > > > > > > I'm not sure about the exact semantics of what you are trying to > > > > > achieve, but did you try looking at nested objects? those allow you to > > > > > store a list of object in a why that keeps values together, like [{ "key": > > > > > "k1" , "value" : "v1"},...] . > > > > > > > > > > Cheers, > > > > > Boaz > > > > > > > > > > On Saturday, October 19, 2013 5:08:05 PM UTC+2, pe...@vagaband.cowrote: > > > > > > > > > > > I'm storing some data in array type field which needs to be accessed > > > > > > within Native Script which is used as custom scorer with function\_score > > > > > > query. But when I access the field values within Native Script using > > > > > > docFieldDoubles I do not get the values in order. Does the array data type > > > > > > not maintain ordering? When I do a GET on that doc, it does show the values > > > > > > in that field in order, but not from within the Native script plugin. Is > > > > > > this a bug or is it expected? > > > > > > > > > > > > What I'm really trying to do is this. I need to maintain a Map or a > > > > > > set of key/value pairs where the keys are different for each document. And > > > > > > I need to access the key/value pairs using a known field name (from both > > > > > > the scoring plugin as well as from search clients). Right now, I'm storing > > > > > > two fields, one with keys and other with values and have both them store > > > > > > these in a comma delimited form. Then from within the plugin, I split on > > > > > > comma and based on position I figure out which key maps to which value. > > > > > > This is of course not very performant and I'd prefer to avoid doing that. > > > > > > As a first step, I tried arrays as mentioned above (instead of comma > > > > > > delimited string), but that seems to lose ordering. What's the best way to > > > > > > do this? > > > > > > > > > > > > Thanks, > > > > > > Peter > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > > > > > topic/elasticsearch/cI5im\_\*\*EYIDY/unsubscribe[https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cI5im\_EYIDY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cI5im_EYIDY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:09am UTC](https://discuss.elastic.co/t/accessing-array-field-within-native-plugin/14027/9 "2017-07-06T02:09:49Z")