# Topics/Entities with relevancy scores and searching **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2014, 1:04pm UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419 "2014-08-23T13:04:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\ **Post date:** [August 23, 2014, 1:04pm UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/1 "2014-08-23T13:04:17Z")
Hey all, a question on possible search paths/structure. If we have a text document, and we have run our magic over it and come away with Topics and Entities (Like, Barack Obama and Apple Inc.) and we have a relevancy score for each one, what would be the best way to store and query against them? we currently are trying a parent/child relationship, where the children are the terms with their relevancy score and the scoring of the parent text document gets done from the relevancy scores of the children. That works. Just worried about speed of parent/child against millions of documents. Another way we could think of was, build our own scorer/analyzer. If we are reading in tokens like BarackObama.93345|AppleInc.0034 where it has the topic and the relevancy score to the document in it, i can build an analyzer to read those sorts of tokens, but is there any way to build a scorer that can use that token match data to score? and third, is there any other way to normalize this data into one document so we can score on it. That seems like it would be the fastest way to query, but my #2 option here is the only way I can think of doing it. Anyone else tagging their documents with relevancy scores to topics, on the document and then letting people search for those topics and pulling back the relevant docs based on the per document relevancy scores? Thanks, Scott -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 23, 2014, 2:49pm UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/2 "2014-08-23T14:49:21Z")
Have a look at: - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. \* > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On 23 August 2014 15:04, Scott Decker [scott@publishthis.com](mailto:scott@publishthis.com) wrote: > Hey all, > a question on possible search paths/structure. If we have a text > document, and we have run our magic over it and come away with Topics and > Entities (Like, Barack Obama and Apple Inc.) and we have a relevancy score > for each one, what would be the best way to store and query against them? > > we currently are trying a parent/child relationship, where the children > are the terms with their relevancy score and the scoring of the parent text > document gets done from the relevancy scores of the children. That works. > Just worried about speed of parent/child against millions of documents. > > Another way we could think of was, build our own scorer/analyzer. If we > are reading in tokens like BarackObama.93345|AppleInc.0034 > where it has the topic and the relevancy score to the document in it, i > can build an analyzer to read those sorts of tokens, but is there any way > to build a scorer that can use that token match data to score? > > and third, is there any other way to normalize this data into one document > so we can score on it. That seems like it would be the fastest way to > query, but my #2 option here is the only way I can think of doing it. > Anyone else tagging their documents with relevancy scores to topics, on the > document and then letting people search for those topics and pulling back > the relevant docs based on the per document relevancy scores? > > Thanks, > Scott > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQmp%3D4Tjft6REtvWZ-2%3DmhqPqnnk-OXtK3oPqw4iNFJmw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQmp%3D4Tjft6REtvWZ-2%3DmhqPqnnk-OXtK3oPqw4iNFJmw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Scott\_Decker](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Scott\_Decker](https://discuss.elastic.co/u/Scott_Decker)\ **Post date:** [August 24, 2014, 5:46pm UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/3 "2014-08-24T17:46:07Z")
Interesting. so, set a payload on the term, in this case the topic/entity, and the payload is the relevancy value. Then, you can do your function score on the query of the main documents themselves, no need for parent/child. Have you done this? any concerns to performance with this sort of scoring, or, it is just as fast if you were doing base lucene scoring if we override the score function and just use our own? -- we will of course try it and run our own performance tests, just looking to see if you all ready have any insights. Super helpful! Scott On Saturday, August 23, 2014 7:50:18 AM UTC-7, Clinton Gormley wrote: > Have a look at: > > - > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-delimited-payload-tokenfilter.html) > \* > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-advanced-scripting.html) > > On 23 August 2014 15:04, Scott Decker \<[sc...@publishthis.com](mailto:sc...@publishthis.com) \ > > > wrote: > > > Hey all, > > a question on possible search paths/structure. If we have a text > > document, and we have run our magic over it and come away with Topics and > > Entities (Like, Barack Obama and Apple Inc.) and we have a relevancy score > > for each one, what would be the best way to store and query against them? > > > > we currently are trying a parent/child relationship, where the children > > are the terms with their relevancy score and the scoring of the parent text > > document gets done from the relevancy scores of the children. That works. > > Just worried about speed of parent/child against millions of documents. > > > > Another way we could think of was, build our own scorer/analyzer. If we > > are reading in tokens like BarackObama.93345|AppleInc.0034 > > where it has the topic and the relevancy score to the document in it, i > > can build an analyzer to read those sorts of tokens, but is there any way > > to build a scorer that can use that token match data to score? > > > > and third, is there any other way to normalize this data into one > > document so we can score on it. That seems like it would be the fastest way > > to query, but my #2 option here is the only way I can think of doing it. > > Anyone else tagging their documents with relevancy scores to topics, on the > > document and then letting people search for those topics and pulling back > > the relevant docs based on the per document relevancy scores? > > > > Thanks, > > Scott > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9434db79-363f-4470-bf91-b960908c2de6%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b3dd847f-99dc-4bad-9a2c-da9b6337ed8c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b3dd847f-99dc-4bad-9a2c-da9b6337ed8c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 25, 2014, 10:48am UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/4 "2014-08-25T10:48:35Z")
On 24 August 2014 19:46, Scott Decker [scott@publishthis.com](mailto:scott@publishthis.com) wrote: > Have you done this? any concerns to performance with this sort of scoring, > or, it is just as fast if you were doing base lucene scoring if we override > the score function and just use our own? > -- we will of course try it and run our own performance tests, just > looking to see if you all ready have any insights. I haven't benchmarked it myself. Obviously accessing payloads is slower than not, and some further work could be done on the scripting side to cache some term statistics lookups, but I don't know how performance will compare to doing this natively. Would be interested in your feedback clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQi%3DLMo83S6w-LZyrGz%3DD3gHPf0B1ZbU-EGkS6p9c9jPA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQi%3DLMo83S6w-LZyrGz%3DD3gHPf0B1ZbU-EGkS6p9c9jPA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![hespoddi](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@hespoddi](https://discuss.elastic.co/u/hespoddi)\ **Post date:** [August 26, 2014, 2:01pm UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/5 "2014-08-26T14:01:54Z")
I'm curious: Using the delimited\_payload\_filter, how do you know which term in delimited list was hit by query in script? From the "text scoring in scripts" documentation, it seems you have to know the term: \_index['FIELD'].get('TERM', \_PAYLOADS) Is the matched term accessible in the script in some way? On Monday, August 25, 2014 6:49:01 AM UTC-4, Clinton Gormley wrote: > On 24 August 2014 19:46, Scott Decker \<[sc...@publishthis.com](mailto:sc...@publishthis.com) \ > > > wrote: > > > Have you done this? any concerns to performance with this sort of > > scoring, or, it is just as fast if you were doing base lucene scoring if we > > override the score function and just use our own? > > -- we will of course try it and run our own performance tests, just > > looking to see if you all ready have any insights. > > I haven't benchmarked it myself. Obviously accessing payloads is slower > than not, and some further work could be done on the scripting side to > cache some term statistics lookups, but I don't know how performance will > compare to doing this natively. > > Would be interested in your feedback > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ba060bc0-a9bc-4cfd-b0fd-29167022f249%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba060bc0-a9bc-4cfd-b0fd-29167022f249%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/topics-entities-with-relevancy-scores-and-searching/19419/6 "2017-07-06T01:06:21Z")