# Can't Get Highlighting Working **URL:** \ **Category:** Elasticsearch\ **Created:** [January 3, 2011, 5:11pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731 "2011-01-03T17:11:02Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 3, 2011, 5:11pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/1 "2011-01-03T17:11:02Z")
I am trying to get highlighting working without luck. I'm still pretty much an ES noob. curl -XGET [http://10.0.1.2:9200/tscott/\_search](http://10.0.1.2:9200/tscott/_search) -d' { "query":{ "field":{ "Summary":{ "query":""The quick brown fox"" } } }, "highlight":{ "fields":{ "\_all":{ } } } }' Which returns the one hit but with no highlighting: "highlight" : { "\_all" : null } When I check the mapping for the type, I see that the Summary field is mapped thus: "Summary" : { "term\_vector" : "with\_positions\_offsets", "type" : "string" } I'm sure I'm missing something simple. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [January 3, 2011, 6:16pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/2 "2011-01-03T18:16:02Z")
Hi, you are highlighting on \_all field (not on Summary field) so you need to check mapping of \_all field (is it stored?). Also it would be good if you can post version of ES being used because there have been some enhancements/new features implemented for highlighting in recently released version 0.14 (and in master as well). Regards, Lukas On Mon, Jan 3, 2011 at 6:11 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I am trying to get highlighting working without luck. I'm still pretty > much > an ES noob. > > curl -XGET [http://10.0.1.2:9200/tscott/\_search](http://10.0.1.2:9200/tscott/_search) -d' > { > "query":{ > "field":{ > "Summary":{ > "query":""The quick brown fox"" > } > } > }, > "highlight":{ > "fields":{ > "\_all":{ } > } > } > }' > > Which returns the one hit but with no highlighting: > > "highlight" : { "\_all" : null } > > When I check the mapping for the type, I see that the Summary field is > mapped thus: > > "Summary" : { > "term\_vector" : "with\_positions\_offsets", > "type" : "string" > } > > ## I'm sure I'm missing something simple. > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2186155.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2186155.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 3, 2011, 9:57pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/3 "2011-01-03T21:57:59Z")
Thanks for the reply. I am using version 0.14.1 (the latest binary download). I guess I don't understand what "\_all" fields means. I thought it was shorthand for listing every field individually. In any case when I put "Summary" instead of "\_all" I get highlight results as expected. That should do it for me. Thanks again! On Jan 3, 12:16 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > Hi, > > you are highlighting on \_all field (not on Summary field) so you need to > check mapping of \_all field (is it stored?). > Also it would be good if you can post version of ES being used because there > have been some enhancements/new features implemented for highlighting in > recently released version 0.14 (and in master as well). > > Regards, > Lukas > > On Mon, Jan 3, 2011 at 6:11 PM, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com)wrote: > > > I am trying to get highlighting working without luck. I'm still pretty > > much > > an ES noob. > > > curl -XGEThttp://10.0.1.2:9200/tscott/\_search-d' > > { > > "query":{ > > "field":{ > > "Summary":{ > > "query":""The quick brown fox"" > > } > > } > > }, > > "highlight":{ > > "fields":{ > > "\_all":{ } > > } > > } > > }' > > > Which returns the one hit but with no highlighting: > > > "highlight" : { "\_all" : null } > > > When I check the mapping for the type, I see that the Summary field is > > mapped thus: > > > "Summary" : { > > "term\_vector" : "with\_positions\_offsets", > > "type" : "string" > > } > > > ## I'm sure I'm missing something simple. > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin)... > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [January 3, 2011, 10:30pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/4 "2011-01-03T22:30:33Z")
Hi, On Mon, Jan 3, 2011 at 10:57 PM, Tim Scott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > Thanks for the reply. I am using version 0.14.1 (the latest binary > download). I guess I don't understand what "\_all" fields means. I > thought it was shorthand for listing every field individually. Most of the time it can be helpful for search but it is not a good candidate for highlighting IMHO. When you search in Lucene (which Elasticsearch is built on top of) you need to specify which document field(s) to query against. At Lucene low level it is necessary to specify document field name for queries. \_all field concept is used to help in situations when the field name is not known in advance at query time (it gives you ability to search for documents that have given term in any of its filed). You can read more about \_all field here: [http://www.elasticsearch.com/docs/elasticsearch/mapping/all\_field/](http://www.elasticsearch.com/docs/elasticsearch/mapping/all_field/) > In any case when I put "Summary" instead of "\_all" I get highlight > results as expected. That should do it for me. Just note that it was required for the field to be stored to allow highlighting but as of 0.14 if the field is not set as stored its content is extracted directly from the \_source field for the highlighting. But \_all field is not part of the \_source (and if I am not mistaken it is not stored by default as well) so you can not use it for highlighting without explicitly setting it stored. > Thanks again! > > On Jan 3, 12:16 pm, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > Hi, > > > > you are highlighting on \_all field (not on Summary field) so you need to > > check mapping of \_all field (is it stored?). > > Also it would be good if you can post version of ES being used because > > there > > have been some enhancements/new features implemented for highlighting in > > recently released version 0.14 (and in master as well). > > > > Regards, > > Lukas > > > > On Mon, Jan 3, 2011 at 6:11 PM, timscott \<[tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) > > wrote: > > > > > I am trying to get highlighting working without luck. I'm still pretty > > > much > > > an ES noob. > > > > > curl -XGEThttp://10.0.1.2:9200/tscott/\_search-d' > > > { > > > "query":{ > > > "field":{ > > > "Summary":{ > > > "query":""The quick brown fox"" > > > } > > > } > > > }, > > > "highlight":{ > > > "fields":{ > > > "\_all":{ } > > > } > > > } > > > }' > > > > > Which returns the one hit but with no highlighting: > > > > > "highlight" : { "\_all" : null } > > > > > When I check the mapping for the type, I see that the Summary field is > > > mapped thus: > > > > > "Summary" : { > > > "term\_vector" : "with\_positions\_offsets", > > > "type" : "string" > > > } > > > > > ## I'm sure I'm missing something simple. > > > > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > .. > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 4, 2011, 12:42am UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/5 "2011-01-04T00:42:42Z")
I got highlighting to work...well sorta. The highlighting is frequently shifted. For example, when I search for "balance" many of the highlights are like this: ... Average led_ger bal_ance this period ... As you can see the highlighting is shifted to the left by 4 characters. The shift seems to vary from 0 to the length of the term. Did something go bad at indexing time? Ideas? ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 4, 2011, 9:18pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/6 "2011-01-04T21:18:11Z")
I re-indexed everything, and the shifting problem went away. Now I have a much worse problem. In my query: "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} Because I am requesting highlighting for every field that is queried, I expect some highlighting for every hit. However, only a minority of hits have any. Most hits come back like this: "highlight" : {"Content" : , "Summary" : null} But it gets much weirder. For those hits that do have highlighting, some are okay, but others contain fragments from the wrong document. That is, the highlighting for the hit on document A contains fragments found nowhere in document A but instead in document B. Document B is also a hit, which may or may not have any highlighting, and if it does it may or may not be the correct highlighting. I re-mapped and re-indexed everything several times with the same results each time. The relevant part of the mapping is: "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} I can't see what could be wrong, but obviously something is bad wrong. Ideas? On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > I got highlighting to work...well sorta. The highlighting is frequently > shifted. For example, when I search for "balance" many of the highlights > are like this: > > ... Average led_ger bal_ance this period ... > > As you can see the highlighting is shifted to the left by 4 characters. The > shift seems to vary from 0 to the length of the term. > > ## Did something go bad at indexing time? Ideas? > > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [January 4, 2011, 9:42pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/7 "2011-01-04T21:42:39Z")
Hi, do you think you are able to recreate this with curl examples? Also can you try set store to yes in mappings for highlighted fields not to get the content from the \_source and see if this makes any change? Regards, Lukas On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I re-indexed everything, and the shifting problem went away. Now I > have a much worse problem. In my query: > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > Because I am requesting highlighting for every field that is queried, > I expect some highlighting for every hit. However, only a minority of > hits have any. Most hits come back like this: > > "highlight" : {"Content" : , "Summary" : null} > > But it gets much weirder. For those hits that do have highlighting, > some are okay, but others contain fragments from the wrong document. > That is, the highlighting for the hit on document A contains fragments > found nowhere in document A but instead in document B. Document B is > also a hit, which may or may not have any highlighting, and if it does > it may or may not be the correct highlighting. > > I re-mapped and re-indexed everything several times with the same > results each time. The relevant part of the mapping is: > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > I can't see what could be wrong, but obviously something is bad > wrong. Ideas? > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > I got highlighting to work...well sorta. The highlighting is frequently > > shifted. For example, when I search for "balance" many of the highlights > > are like this: > > > > ... Average led_ger bal_ance this period ... > > > > As you can see the highlighting is shifted to the left by 4 characters. > > The > > shift seems to vary from 0 to the length of the term. > > > > ## Did something go bad at indexing time? Ideas? > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin)... > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 4, 2011, 9:51pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/8 "2011-01-04T21:51:25Z")
I'm just ahead of you. I tried setting Content and Summary fields to store="yes". It made no difference. It will take a good bit of effort to replicate with a series of curl statements, but I guess that's the only next step. I'll try to do that in the next day or two. Thanks for your help Lukáš. Hi, do you think you are able to recreate this with curl examples? Also can you try set store to yes in mappings for highlighted fields not to get the content from the \_source and see if this makes any change? Regards, Lukas On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I re-indexed everything, and the shifting problem went away. Now I > have a much worse problem. In my query: > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > Because I am requesting highlighting for every field that is queried, > I expect some highlighting for every hit. However, only a minority of > hits have any. Most hits come back like this: > > "highlight" : {"Content" : , "Summary" : null} > > But it gets much weirder. For those hits that do have highlighting, > some are okay, but others contain fragments from the wrong document. > That is, the highlighting for the hit on document A contains fragments > found nowhere in document A but instead in document B. Document B is > also a hit, which may or may not have any highlighting, and if it does > it may or may not be the correct highlighting. > > I re-mapped and re-indexed everything several times with the same > results each time. The relevant part of the mapping is: > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > I can't see what could be wrong, but obviously something is bad > wrong. Ideas? > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > I got highlighting to work...well sorta. The highlighting is frequently > > shifted. For example, when I search for "balance" many of the highlights > > are like this: > > > > ... Average led_ger bal_ance this period ... > > > > As you can see the highlighting is shifted to the left by 4 characters. > > The > > shift seems to vary from 0 to the length of the term. > > > > ## Did something go bad at indexing time? Ideas? > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin)... > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [January 4, 2011, 10:00pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/9 "2011-01-04T22:00:01Z")
If curl replication sounds like a big undertake then you can try some earlier ES releases ([http://www.elasticsearch.com/download/](http://www.elasticsearch.com/download/)) and see if the issue starts with some particular version. Regards, Lukas On Tue, Jan 4, 2011 at 10:51 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I'm just ahead of you. I tried setting Content and Summary fields to > store="yes". It made no difference. > > It will take a good bit of effort to replicate with a series of curl > statements, but I guess that's the only next step. I'll try to do that in > the next day or two. Thanks for your help Lukáš. > > Lukáš Vlček wrote: > > > Hi, > > > > do you think you are able to recreate this with curl examples? > > Also can you try set store to yes in mappings for highlighted fields not > > to > > get the content from the \_source and see if this makes any change? > > > > Regards, > > Lukas > > > > On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > I re-indexed everything, and the shifting problem went away. Now I > > > have a much worse problem. In my query: > > > > > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > > > > > Because I am requesting highlighting for every field that is queried, > > > I expect some highlighting for every hit. However, only a minority of > > > hits have any. Most hits come back like this: > > > > > > "highlight" : {"Content" : , "Summary" : null} > > > > > > But it gets much weirder. For those hits that do have highlighting, > > > some are okay, but others contain fragments from the wrong document. > > > That is, the highlighting for the hit on document A contains fragments > > > found nowhere in document A but instead in document B. Document B is > > > also a hit, which may or may not have any highlighting, and if it does > > > it may or may not be the correct highlighting. > > > > > > I re-mapped and re-indexed everything several times with the same > > > results each time. The relevant part of the mapping is: > > > > > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > > > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > > > > > I can't see what could be wrong, but obviously something is bad > > > wrong. Ideas? > > > > > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > > > > > I got highlighting to work...well sorta. The highlighting is > > > > frequently > > > > shifted. For example, when I search for "balance" many of the > > > > highlights > > > > are like this: > > > > > > > > ... Average led_ger bal_ance this period ... > > > > > > > > As you can see the highlighting is shifted to the left by 4 > > > > characters. > > > > The > > > > shift seems to vary from 0 to the length of the term. > > > > > > > > ## Did something go bad at indexing time? Ideas? > > > > > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > > .. > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![mbro](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@mbro](https://discuss.elastic.co/u/mbro)\ **Post date:** [January 4, 2011, 11:07pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/10 "2011-01-04T23:07:53Z")
Hi, I recently discovered the same behavior that Tim is reporting, but only when I was using a query\_string search that contains a wildcard (e.g., "\*someword"). In those cases, the same query issued multiple times would always return the same list of hits, but the highlighting results wouldn't be consistent. Some hits wouldn't have any highlights returned. And it was always a different set of hits missing the highlighting. I'm a new ES user just getting started with 0.14 so I can't say when this started happening. I'll see if I can recreate it with some curls. Mike On Tue, Jan 4, 2011 at 5:00 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > If curl replication sounds like a big undertake then you can try some > earlier ES releases ([http://www.elasticsearch.com/download/](http://www.elasticsearch.com/download/)) and see if > the issue starts with some particular version. > > Regards, > Lukas > > On Tue, Jan 4, 2011 at 10:51 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > I'm just ahead of you. I tried setting Content and Summary fields to > > store="yes". It made no difference. > > > > It will take a good bit of effort to replicate with a series of curl > > statements, but I guess that's the only next step. I'll try to do that in > > the next day or two. Thanks for your help Lukáš. > > > > Lukáš Vlček wrote: > > > > > Hi, > > > > > > do you think you are able to recreate this with curl examples? > > > Also can you try set store to yes in mappings for highlighted fields not > > > to > > > get the content from the \_source and see if this makes any change? > > > > > > Regards, > > > Lukas > > > > > > On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott > > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > > > I re-indexed everything, and the shifting problem went away. Now I > > > > have a much worse problem. In my query: > > > > > > > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > > > > > > > Because I am requesting highlighting for every field that is queried, > > > > I expect some highlighting for every hit. However, only a minority of > > > > hits have any. Most hits come back like this: > > > > > > > > "highlight" : {"Content" : , "Summary" : null} > > > > > > > > But it gets much weirder. For those hits that do have highlighting, > > > > some are okay, but others contain fragments from the wrong document. > > > > That is, the highlighting for the hit on document A contains fragments > > > > found nowhere in document A but instead in document B. Document B is > > > > also a hit, which may or may not have any highlighting, and if it does > > > > it may or may not be the correct highlighting. > > > > > > > > I re-mapped and re-indexed everything several times with the same > > > > results each time. The relevant part of the mapping is: > > > > > > > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > > > > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > > > > > > > I can't see what could be wrong, but obviously something is bad > > > > wrong. Ideas? > > > > > > > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > > > > > > > I got highlighting to work...well sorta. The highlighting is > > > > > frequently > > > > > shifted. For example, when I search for "balance" many of the > > > > > highlights > > > > > are like this: > > > > > > > > > > ... Average led_ger bal_ance this period ... > > > > > > > > > > As you can see the highlighting is shifted to the left by 4 > > > > > characters. > > > > > The > > > > > shift seems to vary from 0 to the length of the term. > > > > > > > > > > ## Did something go bad at indexing time? Ideas? > > > > > > > > > > View this message in context: > > > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > > > .. > > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [January 4, 2011, 11:42pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/11 "2011-01-04T23:42:49Z")
I tried to replicate via curl with naive data. No luck. My real data is sensitive so I spent some time to redact a few samples. However, the contents are quite large and have many different non-alphanumeric characters throughout. These characters give bash fits. I spent an hour or so trying to clean it (adding escapes), but I felt like I was in quicksand. I'm giving up for now. Highlighting is an important feature for my app, but I have far exceeded a reasonable time-box to make it work, so I'm moving on for now. I'll keep checking back as new versions come out. There's one thing that I wonder. Could the problem be related to all the non-alphanumeric characters in my docs? Somehow these muck up the indexing? If curl replication sounds like a big undertake then you can try some earlier ES releases (http://www.elasticsearch.com/download/) and see if the issue starts with some particular version. Regards, Lukas On Tue, Jan 4, 2011 at 10:51 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I'm just ahead of you. I tried setting Content and Summary fields to > store="yes". It made no difference. > > It will take a good bit of effort to replicate with a series of curl > statements, but I guess that's the only next step. I'll try to do that in > the next day or two. Thanks for your help Lukáš. > > Lukáš Vlček wrote: > > > Hi, > > > > do you think you are able to recreate this with curl examples? > > Also can you try set store to yes in mappings for highlighted fields not > > to > > get the content from the \_source and see if this makes any change? > > > > Regards, > > Lukas > > > > On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > I re-indexed everything, and the shifting problem went away. Now I > > > have a much worse problem. In my query: > > > > > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > > > > > Because I am requesting highlighting for every field that is queried, > > > I expect some highlighting for every hit. However, only a minority of > > > hits have any. Most hits come back like this: > > > > > > "highlight" : {"Content" : , "Summary" : null} > > > > > > But it gets much weirder. For those hits that do have highlighting, > > > some are okay, but others contain fragments from the wrong document. > > > That is, the highlighting for the hit on document A contains fragments > > > found nowhere in document A but instead in document B. Document B is > > > also a hit, which may or may not have any highlighting, and if it does > > > it may or may not be the correct highlighting. > > > > > > I re-mapped and re-indexed everything several times with the same > > > results each time. The relevant part of the mapping is: > > > > > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > > > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > > > > > I can't see what could be wrong, but obviously something is bad > > > wrong. Ideas? > > > > > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > > > > > I got highlighting to work...well sorta. The highlighting is > > > > frequently > > > > shifted. For example, when I search for "balance" many of the > > > > highlights > > > > are like this: > > > > > > > > ... Average led_ger bal_ance this period ... > > > > > > > > As you can see the highlighting is shifted to the left by 4 > > > > characters. > > > > The > > > > shift seems to vary from 0 to the length of the term. > > > > > > > > ## Did something go bad at indexing time? Ideas? > > > > > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > > .. > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![mbro](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@mbro](https://discuss.elastic.co/u/mbro)\ **Post date:** [January 5, 2011, 5:34am UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/12 "2011-01-05T05:34:37Z")
I was able to put together a small script and Java program (see attached zip) that demonstrates what I'm seeing. Run the load-highlight-data bash script to put documents into the index. Then run test-highlight-rest to do REST API queries via curl. There are some comments in the file regarding query\_string patterns that I used. There is also a small Java project included that you can run. Just drop the elasticsearch-0.14.0.jar into the lib directory and it should compile/run. The REST API consistently returns the same results, although they are wrong for some queries. The Java API returns inconsistent results. Both APIs alway return the correct list of hits...but sometimes the highlight information returned with the hits is incorrect. Hope this helps... Mike On Tue, Jan 4, 2011 at 6:42 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > I tried to replicate via curl with naive data. No luck. My real data is > sensitive so I spent some time to redact a few samples. However, the > contents are quite large and have many different non-alphanumeric > characters > throughout. These characters give bash fits. I spent an hour or so trying > to clean it (adding escapes), but I felt like I was in quicksand. I'm > giving up for now. Highlighting is an important feature for my app, but I > have far exceeded a reasonable time-box to make it work, so I'm moving on > for now. I'll keep checking back as new versions come out. > > There's one thing that I wonder. Could the problem be related to all the > non-alphanumeric characters in my docs? Somehow these muck up the > indexing? > > Lukáš Vlček wrote: > > > If curl replication sounds like a big undertake then you can try some > > earlier ES releases ([http://www.elasticsearch.com/download/](http://www.elasticsearch.com/download/)) and see if > > the > > issue starts with some particular version. > > > > Regards, > > Lukas > > > > On Tue, Jan 4, 2011 at 10:51 PM, timscott > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > I'm just ahead of you. I tried setting Content and Summary fields to > > > store="yes". It made no difference. > > > > > > It will take a good bit of effort to replicate with a series of curl > > > statements, but I guess that's the only next step. I'll try to do that > > > in > > > the next day or two. Thanks for your help Lukáš. > > > > > > Lukáš Vlček wrote: > > > > > > > Hi, > > > > > > > > do you think you are able to recreate this with curl examples? > > > > Also can you try set store to yes in mappings for highlighted fields > > > > not > > > > to > > > > get the content from the \_source and see if this makes any change? > > > > > > > > Regards, > > > > Lukas > > > > > > > > On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott > > > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > > > > > I re-indexed everything, and the shifting problem went away. Now I > > > > > have a much worse problem. In my query: > > > > > > > > > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > > > > > > > > > Because I am requesting highlighting for every field that is queried, > > > > > I expect some highlighting for every hit. However, only a minority > > > > > of > > > > > hits have any. Most hits come back like this: > > > > > > > > > > "highlight" : {"Content" : , "Summary" : null} > > > > > > > > > > But it gets much weirder. For those hits that do have highlighting, > > > > > some are okay, but others contain fragments from the wrong document. > > > > > That is, the highlighting for the hit on document A contains > > > > > fragments > > > > > found nowhere in document A but instead in document B. Document B is > > > > > also a hit, which may or may not have any highlighting, and if it > > > > > does > > > > > it may or may not be the correct highlighting. > > > > > > > > > > I re-mapped and re-indexed everything several times with the same > > > > > results each time. The relevant part of the mapping is: > > > > > > > > > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > > > > > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > > > > > > > > > I can't see what could be wrong, but obviously something is bad > > > > > wrong. Ideas? > > > > > > > > > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > > > > > > > > > I got highlighting to work...well sorta. The highlighting is > > > > > > frequently > > > > > > shifted. For example, when I search for "balance" many of the > > > > > > highlights > > > > > > are like this: > > > > > > > > > > > > ... Average led_ger bal_ance this period ... > > > > > > > > > > > > As you can see the highlighting is shifted to the left by 4 > > > > > > characters. > > > > > > The > > > > > > shift seems to vary from 0 to the length of the term. > > > > > > > > > > > > ## Did something go bad at indexing time? Ideas? > > > > > > > > > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > > .. > > > > > > > > > Sent from the Elasticsearch Users mailing list archive at > > > > > > [Nabble.com](http://Nabble.com). > > > > > > -- > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html) > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2195175.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2195175.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 5, 2011, 2:19pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/13 "2011-01-05T14:19:54Z")
I've found the problem with highlighting the wrong content (from another doc) opened an issue for it and pushed a fix: [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/closed#issue/600). Will release 0.14.2 soonish to include this fix and other bug fixes. On Wed, Jan 5, 2011 at 7:34 AM, Mike Brocious [mike.brocious@gmail.com](mailto:mike.brocious@gmail.com)wrote: > I was able to put together a small script and Java program (see attached > zip) that demonstrates what I'm seeing. > > Run the load-highlight-data bash script to put documents into the index. > Then run test-highlight-rest to do REST API queries via curl. There are > some comments in the file regarding query\_string patterns that I used. > There is also a small Java project included that you can run. Just drop > the elasticsearch-0.14.0.jar into the lib directory and it should > compile/run. > > The REST API consistently returns the same results, although they are wrong > for some queries. > The Java API returns inconsistent results. > > Both APIs alway return the correct list of hits...but sometimes the > highlight information returned with the hits is incorrect. > > Hope this helps... > > Mike > > On Tue, Jan 4, 2011 at 6:42 PM, timscott [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > I tried to replicate via curl with naive data. No luck. My real data is > > sensitive so I spent some time to redact a few samples. However, the > > contents are quite large and have many different non-alphanumeric > > characters > > throughout. These characters give bash fits. I spent an hour or so > > trying > > to clean it (adding escapes), but I felt like I was in quicksand. I'm > > giving up for now. Highlighting is an important feature for my app, but I > > have far exceeded a reasonable time-box to make it work, so I'm moving on > > for now. I'll keep checking back as new versions come out. > > > > There's one thing that I wonder. Could the problem be related to all the > > non-alphanumeric characters in my docs? Somehow these muck up the > > indexing? > > > > Lukáš Vlček wrote: > > > > > If curl replication sounds like a big undertake then you can try some > > > earlier ES releases ([http://www.elasticsearch.com/download/](http://www.elasticsearch.com/download/)) and see if > > > the > > > issue starts with some particular version. > > > > > > Regards, > > > Lukas > > > > > > On Tue, Jan 4, 2011 at 10:51 PM, timscott > > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > > > I'm just ahead of you. I tried setting Content and Summary fields to > > > > store="yes". It made no difference. > > > > > > > > It will take a good bit of effort to replicate with a series of curl > > > > statements, but I guess that's the only next step. I'll try to do that > > > > in > > > > the next day or two. Thanks for your help Lukáš. > > > > > > > > Lukáš Vlček wrote: > > > > > > > > > Hi, > > > > > > > > > > do you think you are able to recreate this with curl examples? > > > > > Also can you try set store to yes in mappings for highlighted fields > > > > > not > > > > > to > > > > > get the content from the \_source and see if this makes any change? > > > > > > > > > > Regards, > > > > > Lukas > > > > > > > > > > On Tue, Jan 4, 2011 at 10:18 PM, Tim Scott > > > > > [tscott@lunaversesoftware.com](mailto:tscott@lunaversesoftware.com)wrote: > > > > > > > > > > > I re-indexed everything, and the shifting problem went away. Now I > > > > > > have a much worse problem. In my query: > > > > > > > > > > > > "highlight":{"fields":{"Summary":{},"Content":{"order":"score"}}} > > > > > > > > > > > > Because I am requesting highlighting for every field that is > > > > > > queried, > > > > > > I expect some highlighting for every hit. However, only a minority > > > > > > of > > > > > > hits have any. Most hits come back like this: > > > > > > > > > > > > "highlight" : {"Content" : , "Summary" : null} > > > > > > > > > > > > But it gets much weirder. For those hits that do have highlighting, > > > > > > some are okay, but others contain fragments from the wrong document. > > > > > > That is, the highlighting for the hit on document A contains > > > > > > fragments > > > > > > found nowhere in document A but instead in document B. Document B > > > > > > is > > > > > > also a hit, which may or may not have any highlighting, and if it > > > > > > does > > > > > > it may or may not be the correct highlighting. > > > > > > > > > > > > I re-mapped and re-indexed everything several times with the same > > > > > > results each time. The relevant part of the mapping is: > > > > > > > > > > > > "Content":{"term\_vector":"with\_positions\_offsets", "type":"string"}, > > > > > > "Summary":{"term\_vector":"with\_positions\_offsets","type":"string"} > > > > > > > > > > > > I can't see what could be wrong, but obviously something is bad > > > > > > wrong. Ideas? > > > > > > > > > > > > On Jan 3, 6:42 pm, timscott [tsc...@lunaversesoftware.com](mailto:tsc...@lunaversesoftware.com) wrote: > > > > > > > > > > > > > I got highlighting to work...well sorta. The highlighting is > > > > > > > frequently > > > > > > > shifted. For example, when I search for "balance" many of the > > > > > > > highlights > > > > > > > are like this: > > > > > > > > > > > > > > ... Average led_ger bal_ance this period ... > > > > > > > > > > > > > > As you can see the highlighting is shifted to the left by 4 > > > > > > > characters. > > > > > > > The > > > > > > > shift seems to vary from 0 to the length of the term. > > > > > > > > > > > > > > ## Did something go bad at indexing time? Ideas? > > > > > > > > > > > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlightin). > > > > > > .. > > > > > > > > > > > Sent from the Elasticsearch Users mailing list archive at > > > > > > > [Nabble.com](http://Nabble.com). > > > > > > > > -- > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2194542.html) > > > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2195175.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2195175.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![mbro](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@mbro](https://discuss.elastic.co/u/mbro)\ **Post date:** [January 6, 2011, 4:09pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/14 "2011-01-06T16:09:23Z")
Shay, I just downloaded and tried 0.14.2, but it does not fix the inconsistent highlighting results that I'm seeing. The attachment in my earlier post in this thread can be used to demonstrate the problem. Just run the load-highlight-data script to post data to ES, then run the Java program a couple of times and you should see different highlight results. I'll be happy to open an issue for this if you'd like. Thanks for your efforts on ES...it's awesome. Regards, Mike ---
**Author:** ![mbro](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@mbro](https://discuss.elastic.co/u/mbro)\ **Post date:** [January 7, 2011, 1:30pm UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/15 "2011-01-07T13:30:31Z")
Shay, I just downloaded and tried 0.14.2, but it does not fix the inconsistent highlighting results that I'm seeing. The attachment in my earlier post in this thread can be used to demonstrate the problem. Just run the load-highlight-data script to post data to ES, then run the Java program a couple of times and you should see different highlight results. I'll be happy to open an issue for this if you'd like. Thanks for your efforts on ES...it's awesome. Regards, Mike ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 8, 2011, 1:21am UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/16 "2011-01-08T01:21:44Z")
Well, that one was a nasty one to track down... . Its not the Java transport client compared to the REST one, its just that its a game of statistics (I was getting it once out of about 10000 runs). Pushed the fix to master and the 0.14 branch if you want to test. On Thu, Jan 6, 2011 at 6:09 PM, mbrocious [mike.brocious@gmail.com](mailto:mike.brocious@gmail.com) wrote: > Shay, > > I just downloaded and tried 0.14.2, but it does not fix the inconsistent > highlighting results that I'm seeing. The attachment in my earlier post in > this thread can be used to demonstrate the problem. Just run the > load-highlight-data script to post data to ES, then run the Java program a > couple of times and you should see different highlight results. > > I'll be happy to open an issue for this if you'd like. > > Thanks for your efforts on ES...it's awesome. > > Regards, > Mike > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2206391.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2206391.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![mbro](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@mbro](https://discuss.elastic.co/u/mbro)\ **Post date:** [January 8, 2011, 4:26am UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/17 "2011-01-08T04:26:43Z")
Yup, that fixed it! Thanks. Mike BTW, sorry for the double post of my last message....the first one was 'pending' for about a day and I hadn't seen that happen before so I posted again. On Fri, Jan 7, 2011 at 8:21 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Well, that one was a nasty one to track down... . Its not the Java > transport client compared to the REST one, its just that its a game of > statistics (I was getting it once out of about 10000 runs). Pushed the fix > to master and the 0.14 branch if you want to test. > > On Thu, Jan 6, 2011 at 6:09 PM, mbrocious [mike.brocious@gmail.com](mailto:mike.brocious@gmail.com) wrote: > > > Shay, > > > > I just downloaded and tried 0.14.2, but it does not fix the inconsistent > > highlighting results that I'm seeing. The attachment in my earlier post > > in > > this thread can be used to demonstrate the problem. Just run the > > load-highlight-data script to post data to ES, then run the Java program a > > couple of times and you should see different highlight results. > > > > I'll be happy to open an issue for this if you'd like. > > > > Thanks for your efforts on ES...it's awesome. > > > > Regards, > > Mike > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2206391.html](http://elasticsearch-users.115913.n3.nabble.com/Can-t-Get-Highlighting-Working-tp2186155p2206391.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:14am UTC](https://discuss.elastic.co/t/cant-get-highlighting-working/3731/18 "2017-07-06T04:14:07Z")