# Sourceless Highlighting **URL:** \ **Category:** Elasticsearch\ **Created:** [February 18, 2013, 1:10pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775 "2013-02-18T13:10:42Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![gelmajjouti](https://avatars.discourse-cdn.com/v4/letter/g/4af34b/32.png) [@gelmajjouti](https://discuss.elastic.co/u/gelmajjouti)\ **Post date:** [February 18, 2013, 1:10pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/1 "2013-02-18T13:10:42Z")
Hello, I'm trying to make search highlighting work without storing sources in ElasticSearch. In other to do this, I intend to get search hits position from Lucene's \*TermFreqVector \*class. I gather from an older thread that I need to write a plugin for ElasticSearch. I have two questions : 1. Is it feasible? Is there a way to extend that part of ElasticSearch? 2. If so, where should I hook my plugin? Cheers, Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [February 18, 2013, 3:45pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/2 "2013-02-18T15:45:12Z")
Before you start and create a plugin / fork ES, you can just store a specific field instead of the whole source and highlight on that instead. Is that sufficient in your case? Martijn On 18 February 2013 14:10, gelmajjouti [gelmajjouti@gmail.com](mailto:gelmajjouti@gmail.com) wrote: > Hello, > > I'm trying to make search highlighting work without storing sources in > Elasticsearch. > > In other to do this, I intend to get search hits position from Lucene's \*TermFreqVector > \*class. I gather from an older thread that I need to write a plugin for > Elasticsearch. > > I have two questions : > > 1. Is it feasible? Is there a way to extend that part of Elasticsearch? > 2. If so, where should I hook my plugin? > > Cheers, > Greg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![gelmajjouti](https://avatars.discourse-cdn.com/v4/letter/g/4af34b/32.png) [@gelmajjouti](https://discuss.elastic.co/u/gelmajjouti)\ **Post date:** [February 18, 2013, 3:55pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/3 "2013-02-18T15:55:39Z")
Thanks for your reply. Most of my data is searchable fulltext. I can't afford to store it in ES. Greg On Monday, February 18, 2013 4:45:12 PM UTC+1, Martijn v Groningen wrote: > Before you start and create a plugin / fork ES, you can just store a > specific field instead of the whole source and > highlight on that instead. Is that sufficient in your case? > > Martijn > > On 18 February 2013 14:10, gelmajjouti \<[gelma...@gmail.com](mailto:gelma...@gmail.com) \\>wrote: > > > Hello, > > > > I'm trying to make search highlighting work without storing sources in > > Elasticsearch. > > > > In other to do this, I intend to get search hits position from Lucene's \*TermFreqVector > > \*class. I gather from an older thread that I need to write a plugin for > > Elasticsearch. > > > > I have two questions : > > > > 1. Is it feasible? Is there a way to extend that part of > > Elasticsearch? > > 2. If so, where should I hook my plugin? > > > > Cheers, > > Greg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [February 18, 2013, 6:51pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/4 "2013-02-18T18:51:11Z")
it might be a useful feature to add returning a stream of offset tuples rather than highlighted strings. Even if we expose TermVectors they might be too expensive to transfer while a "snipet descriptor" might be what some people need. simon On Monday, February 18, 2013 4:55:39 PM UTC+1, gelmajjouti wrote: > Thanks for your reply. > > Most of my data is searchable fulltext. I can't afford to store it in ES. > > Greg > > On Monday, February 18, 2013 4:45:12 PM UTC+1, Martijn v Groningen wrote: > > > Before you start and create a plugin / fork ES, you can just store a > > specific field instead of the whole source and > > highlight on that instead. Is that sufficient in your case? > > > > Martijn > > > > On 18 February 2013 14:10, gelmajjouti [gelma...@gmail.com](mailto:gelma...@gmail.com) wrote: > > > > > Hello, > > > > > > I'm trying to make search highlighting work without storing sources in > > > Elasticsearch. > > > > > > In other to do this, I intend to get search hits position from Lucene's > > > \*TermFreqVector \*class. I gather from an older thread that I need to > > > write a plugin for Elasticsearch. > > > > > > I have two questions : > > > > > > 1. Is it feasible? Is there a way to extend that part of > > > Elasticsearch? > > > 2. If so, where should I hook my plugin? > > > > > > Cheers, > > > Greg > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![roytmana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roytmana/32/44855_2.png) [@roytmana](https://discuss.elastic.co/u/roytmana)\ **Post date:** [February 18, 2013, 10:43pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/5 "2013-02-18T22:43:35Z")
+100 To me getting data about highlighting is more useful than highlighted text. In case of searching \_all (or any other non stored multi\_field) I would like to get say a list of matched tokens (ideally including any "fuzzy" tokens) and I will do client side highlighting of my text based on that. For cases when we search against set of stored/source fields, name of field(s) with array of offset tuples (or matched tokens) would be great!. We often end up highlighting data from our business objects (JPA/JDO/Hibernate) and not elastic stored data so for this case, most useful would be a highlight consisting of list of matched fields with array of matched tokens from the source (case does not matter) On Monday, February 18, 2013 1:51:11 PM UTC-5, simonw wrote: > it might be a useful feature to add returning a stream of offset tuples > rather than highlighted strings. Even if we expose TermVectors they might > be too expensive to transfer while a "snipet descriptor" might be what some > people need. > > simon > > On Monday, February 18, 2013 4:55:39 PM UTC+1, gelmajjouti wrote: > > > Thanks for your reply. > > > > Most of my data is searchable fulltext. I can't afford to store it in ES. > > > > Greg > > > > On Monday, February 18, 2013 4:45:12 PM UTC+1, Martijn v Groningen wrote: > > > > > Before you start and create a plugin / fork ES, you can just store a > > > specific field instead of the whole source and > > > highlight on that instead. Is that sufficient in your case? > > > > > > Martijn > > > > > > On 18 February 2013 14:10, gelmajjouti [gelma...@gmail.com](mailto:gelma...@gmail.com) wrote: > > > > > > > Hello, > > > > > > > > I'm trying to make search highlighting work without storing sources in > > > > Elasticsearch. > > > > > > > > In other to do this, I intend to get search hits position from Lucene's > > > > \*TermFreqVector \*class. I gather from an older thread that I need to > > > > write a plugin for Elasticsearch. > > > > > > > > I have two questions : > > > > > > > > 1. Is it feasible? Is there a way to extend that part of > > > > Elasticsearch? > > > > 2. If so, where should I hook my plugin? > > > > > > > > Cheers, > > > > Greg > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > Met vriendelijke groet, > > > > > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 18, 2013, 10:56pm UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/6 "2013-02-18T22:56:50Z")
Not the cleanest solution, but you can extract some of this information from the explain object. -- Ivan On Mon, Feb 18, 2013 at 2:43 PM, AlexR [roytmana@gmail.com](mailto:roytmana@gmail.com) wrote: > +100 To me getting data about highlighting is more useful than highlighted > text. > In case of searching \_all (or any other non stored multi\_field) I would > like to get say a list of matched tokens (ideally including any "fuzzy" > tokens) and I will do client side highlighting of my text based on that. > For cases when we search against set of stored/source fields, name of > field(s) with array of offset tuples (or matched tokens) would be great!. > We often end up highlighting data from our business objects > (JPA/JDO/Hibernate) and not elastic stored data so for this case, most > useful would be a highlight consisting of list of matched fields with array > of matched tokens from the source (case does not matter) > > On Monday, February 18, 2013 1:51:11 PM UTC-5, simonw wrote: > > > it might be a useful feature to add returning a stream of offset tuples > > rather than highlighted strings. Even if we expose TermVectors they might > > be too expensive to transfer while a "snipet descriptor" might be what some > > people need. > > > > simon > > > > On Monday, February 18, 2013 4:55:39 PM UTC+1, gelmajjouti wrote: > > > > > Thanks for your reply. > > > > > > Most of my data is searchable fulltext. I can't afford to store it in ES. > > > > > > Greg > > > > > > On Monday, February 18, 2013 4:45:12 PM UTC+1, Martijn v Groningen wrote: > > > > > > > Before you start and create a plugin / fork ES, you can just store a > > > > specific field instead of the whole source and > > > > highlight on that instead. Is that sufficient in your case? > > > > > > > > Martijn > > > > > > > > On 18 February 2013 14:10, gelmajjouti [gelma...@gmail.com](mailto:gelma...@gmail.com) wrote: > > > > > > > > > Hello, > > > > > > > > > > I'm trying to make search highlighting work without storing sources in > > > > > Elasticsearch. > > > > > > > > > > In other to do this, I intend to get search hits position from > > > > > Lucene's \*TermFreqVector \*class. I gather from an older thread that I > > > > > need to write a plugin for Elasticsearch. > > > > > > > > > > I have two questions : > > > > > > > > > > 1. Is it feasible? Is there a way to extend that part of > > > > > Elasticsearch? > > > > > 2. If so, where should I hook my plugin? > > > > > > > > > > Cheers, > > > > > Greg > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@googlegroups.\*\*com. > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > Met vriendelijke groet, > > > > > > > > Martijn van Groningen > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ghoumard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghoumard/32/44886_2.png) [@ghoumard](https://discuss.elastic.co/u/ghoumard)\ **Post date:** [February 19, 2013, 2:04am UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/7 "2013-02-19T02:04:38Z")
+1 exactly a solution I might need. An array of matched field with position instead of highlighted text would be great ! On Monday, February 18, 2013 5:43:35 PM UTC-5, AlexR wrote: > +100 To me getting data about highlighting is more useful than highlighted > text. > In case of searching \_all (or any other non stored multi\_field) I would > like to get say a list of matched tokens (ideally including any "fuzzy" > tokens) and I will do client side highlighting of my text based on that. > For cases when we search against set of stored/source fields, name of > field(s) with array of offset tuples (or matched tokens) would be great!. > We often end up highlighting data from our business objects > (JPA/JDO/Hibernate) and not elastic stored data so for this case, most > useful would be a highlight consisting of list of matched fields with array > of matched tokens from the source (case does not matter) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![roytmana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roytmana/32/44855_2.png) [@roytmana](https://discuss.elastic.co/u/roytmana)\ **Post date:** [February 19, 2013, 3:20am UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/8 "2013-02-19T03:20:24Z")
i do not know if it is possible, but if it would be fantastic if it were possible to store start/end of individual fields in \_all (or similar composite fields) and translate highlight positions in \_all to which source fields got matched and positions in those fields in the source. That would take care of virtually all highlighting needs! On Mon, Feb 18, 2013 at 9:04 PM, Gildas Houmard [ghoumard@gmail.com](mailto:ghoumard@gmail.com) wrote: > +1 exactly a solution I might need. An array of matched field with > position instead of highlighted text would be great ! > > On Monday, February 18, 2013 5:43:35 PM UTC-5, AlexR wrote: > > > +100 To me getting data about highlighting is more useful than > > highlighted text. > > In case of searching \_all (or any other non stored multi\_field) I would > > like to get say a list of matched tokens (ideally including any "fuzzy" > > tokens) and I will do client side highlighting of my text based on that. > > For cases when we search against set of stored/source fields, name of > > field(s) with array of offset tuples (or matched tokens) would be great!. > > We often end up highlighting data from our business objects > > (JPA/JDO/Hibernate) and not elastic stored data so for this case, most > > useful would be a highlight consisting of list of matched fields with array > > of matched tokens from the source (case does not matter) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:50am UTC](https://discuss.elastic.co/t/sourceless-highlighting/10775/9 "2017-07-06T02:50:49Z")