# Highlight stopped working for me **URL:** \ **Category:** Elasticsearch\ **Created:** [July 23, 2012, 1:41am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484 "2012-07-23T01:41:46Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 1:41am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/1 "2012-07-23T01:41:46Z")
We were previously getting perfectly good highlight results back, changed some queries and now it's not working anymore. here is the query data. Am I missing something? { "query": { "filtered": { "query": { "query\_string": { "query": "a Ruby on Rail Engineer" } }, "filter": { "and": [ { "not": { "term": { "id": 3453 } } } ] } } }, "highlight": { "fields": { "\_all": {}, "jobs.industry": {}, "jobs.employer": {}, "jobs.position": {}, "educations.school": {}, "educations.major": {}, "skills": {}, "linkedin\_summary": {}, "linkedin\_groups": {}, "linkedin\_certifications": {}, "headline": {} }, "pre\_tags": [ "\" ], "post\_tags": [ "" ] }, "size": 10 } ---
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 2:59am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/2 "2012-07-23T02:59:41Z")
As a follow up, the only changes made were to how the data was indexed. I'm using the Tire Ruby gem and am attempting to include some associations on my model. Before, I wasn't properly include them so the objects :educations and :jobs weren't actually being indexed, and this is when the highlighting worked. However, when i properly include the associations, the highlighting stopped working. Here is my mapping: { - state: open - settings: { - index.number\_of\_shards: 5 - index.number\_of\_replicas: 1 - index.version.created: 190399 } - mappings: { - user: { - properties: { - skills: { - type: string } - headline: { - type: string } - jobs: { - properties: { - position: { - type: string } - employer: { - type: string } - state: { - type: string } - city: { - type: string } - country: { - type: string } - id: { - type: long } - end\_date: { - format: dateOptionalTime - type: date } - updated\_at: { - format: dateOptionalTime - type: date } - description: { - type: string } - created\_at: { - format: dateOptionalTime - type: date } - user\_id: { - type: long } - industry: { - type: string } - start\_date: { - format: dateOptionalTime - type: date } } } - linkedin\_connections: { - include\_in\_all: false - index: not\_analyzed - type: string } - is\_active: { - type: boolean } - fb\_connections: { - include\_in\_all: false - index: not\_analyzed - type: string } - fb\_user\_id: { - include\_in\_all: false - index: not\_analyzed - type: string } - linkedin\_certifications: { - type: string } - is\_email\_confirmed: { - type: boolean } - id: { - index: not\_analyzed - type: string } - profile\_pic\_url: { - type: string } - first\_name: { - type: string } - is\_reachable\_on\_fb: { - type: boolean } - updated\_at: { - format: dateOptionalTime - type: date } - linkedin\_id: { - include\_in\_all: false - index: not\_analyzed - type: string } - is\_work\_updated: { - type: boolean } - last\_name: { - type: string } - linkedin\_public\_profile\_url: { - type: string } - linkedin\_groups: { - type: string } - created\_at: { - format: dateOptionalTime - type: date } - slug: { - type: string } - educations: { - properties: { - school\_type: { - type: string } - id: { - type: long } - updated\_at: { - format: dateOptionalTime - type: date } - school: { - type: string } - grad\_year: { - format: dateOptionalTime - type: date } - degree: { - type: string } - created\_at: { - format: dateOptionalTime - type: date } - user\_id: { - type: long } - major: { - type: string } } } - linkedin\_summary: { - type: string } - login\_hash: { - type: string } } } } - aliases: } Any reason highlighting wouldn't work with the above setup? On Sunday, July 22, 2012 9:41:46 PM UTC-4, Brandon Hilkert wrote: > We were previously getting perfectly good highlight results back, changed > some queries and now it's not working anymore. here is the query data. Am I > missing something? > > { > "query": { > "filtered": { > "query": { > "query\_string": { > "query": "a Ruby on Rail Engineer" > } > }, > "filter": { > "and": [ > { > "not": { > "term": { > "id": 3453 > } > } > } > ] > } > } > }, > "highlight": { > "fields": { > "\_all": {}, > "jobs.industry": {}, > "jobs.employer": {}, > "jobs.position": {}, > "educations.school": {}, > "educations.major": {}, > "skills": {}, > "linkedin\_summary": {}, > "linkedin\_groups": {}, > "linkedin\_certifications": {}, > "headline": {} > }, > "pre\_tags": [ > "\" > ], > "post\_tags": [ > "" > ] > }, > "size": 10 > } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 23, 2012, 3:07am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/3 "2012-07-23T03:07:53Z")
Did you try to prefix with your type name? ``` "user.educations.school": {}, ``` Not sure it will help... David -- Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > ``` > "educations.school": {}, > > ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 23, 2012, 3:11am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/4 "2012-07-23T03:11:39Z")
Also, are you doing this on a clean index? Does it remain some old mapping? David -- Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > Did you try to prefix with your type name? > > ``` > "user.educations.school": {}, > > ``` > > Not sure it will help... > > David > > -- > > Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > > > ``` > > "educations.school": {}, > > > > ``` ---
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 1:23pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/5 "2012-07-23T13:23:05Z")
Hey David, The only difference from when it previously worked is the change in data i'm sending with those objects. I make sure I delete the index entirely before every test. I was going to mess around with the mapping today. Any other ideas? Brandon On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote: > Also, are you doing this on a clean index? > Does it remain some old mapping? > > David > > -- > > Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > > Did you try to prefix with your type name? > > ``` > "user.educations.school": {}, > > ``` > > Not sure it will help... > > David > > -- > > Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > > ``` > "educations.school": {}, > > ``` ---
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 1:46pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/6 "2012-07-23T13:46:35Z")
I've confirmed that if I include both associations in my JSON representation when indexing the data, the highlighting breaks for whatever reason. So with this in my Ruby code: def to\_indexed\_json to\_json(include: [:educations, :jobs]) end _Highlighting doesn't work._ However, without that and just sending the root data, _highlighting works fine._ Any thoughts on what might cause this? Thanks, Brandon On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote: > Hey David, > > The only difference from when it previously worked is the change in data > i'm sending with those objects. I make sure I delete the index entirely > before every test. > > I was going to mess around with the mapping today. Any other ideas? > > Brandon > > On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote: > > > Also, are you doing this on a clean index? > > Does it remain some old mapping? > > > > David > > > > -- > > > > Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > > > > Did you try to prefix with your type name? > > > > ``` > > "user.educations.school": {}, > > > > ``` > > > > Not sure it will help... > > > > David > > > > -- > > > > Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > > > > ``` > > "educations.school": {}, > > > > ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 23, 2012, 1:56pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/7 "2012-07-23T13:56:20Z")
Can you gist a full curl recreation ? I will try to test it. David 😉 Twitter : @dadoonet / @elasticsearchfr Le 23 juil. 2012 à 15:46, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > I've confirmed that if I include both associations in my JSON representation when indexing the data, the highlighting breaks for whatever reason. > > So with this in my Ruby code: > > def to\_indexed\_json > to\_json(include: [:educations, :jobs]) > end > > Highlighting doesn't work. > > However, without that and just sending the root data, highlighting works fine. > > Any thoughts on what might cause this? > > Thanks, > Brandon > > On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote: > Hey David, > > The only difference from when it previously worked is the change in data i'm sending with those objects. I make sure I delete the index entirely before every test. > > I was going to mess around with the mapping today. Any other ideas? > > Brandon > > On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote: > Also, are you doing this on a clean index? > Does it remain some old mapping? > > David > > -- > > Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > > > Did you try to prefix with your type name? > > > > ``` > > "user.educations.school": {}, > > > > ``` > > > > Not sure it will help... > > > > David > > > > -- > > > > Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > > > > > ``` > > > "educations.school": {}, > > > > > > ``` ---
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 2:29pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/8 "2012-07-23T14:29:56Z")
[https://gist.github.com/3163918](https://gist.github.com/3163918) On Monday, July 23, 2012 9:56:20 AM UTC-4, David Pilato wrote: > Can you gist a full curl recreation ? > > I will try to test it. > > David 😉 > Twitter : @dadoonet / @elasticsearchfr > > Le 23 juil. 2012 à 15:46, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : > > I've confirmed that if I include both associations in my JSON > representation when indexing the data, the highlighting breaks for whatever > reason. > > So with this in my Ruby code: > > def to\_indexed\_json > to\_json(include: [:educations, :jobs]) > end > > _Highlighting doesn't work._ > > However, without that and just sending the root data, _highlighting works > fine._ > > Any thoughts on what might cause this? > > Thanks, > Brandon > > On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote: > > > Hey David, > > > > The only difference from when it previously worked is the change in data > > i'm sending with those objects. I make sure I delete the index entirely > > before every test. > > > > I was going to mess around with the mapping today. Any other ideas? > > > > Brandon > > > > On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote: > > > > > Also, are you doing this on a clean index? > > > Does it remain some old mapping? > > > > > > David > > > > > > -- > > > > > > Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : > > > > > > Did you try to prefix with your type name? > > > > > > ``` > > > "user.educations.school": {}, > > > > > > ``` > > > > > > Not sure it will help... > > > > > > David > > > > > > -- > > > > > > Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit > > > : > > > > > > ``` > > > "educations.school": {}, > > > > > > ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 23, 2012, 8:20pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/9 "2012-07-23T20:20:25Z")
Hummm. Strange. It looks like you cannot use the type name as a property: Note to Shay: do you think it’s an issue we should track? If you send: curl -XPOST '[http://localhost:9200/spokely\_development\_users/user/3457](http://localhost:9200/spokely_development_users/user/3457)' -d ' { "myuser":{ ``` "headline":"Ruby on Rails Engineer" } ``` } ' curl -XGET '[http://localhost:9200/spokely\_development\_users/user/\_search?size=10&pretty=true](http://localhost:9200/spokely_development_users/user/_search?size=10&pretty=true)' -d ' { "query":{ ``` "filtered":{ "query":{ "query_string":{ "query":"a Ruby on Rail Engineer", "fields":[ "myuser.headline^2" ] } } } ``` }, "highlight":{ ``` "fields":{ "fields":{ "myuser.headline":{ } } } ``` }, "size":10 } ' It will work. But with: curl -XPOST '[http://localhost:9200/spokely\_development\_users/user/3457](http://localhost:9200/spokely_development_users/user/3457)' -d ' { "user":{ ``` "headline":"Ruby on Rails Engineer" } ``` } ' curl -XGET '[http://localhost:9200/spokely\_development\_users/user/\_search?size=10&pretty=true](http://localhost:9200/spokely_development_users/user/_search?size=10&pretty=true)' -d ' { "query":{ ``` "filtered":{ "query":{ "query_string":{ "query":"a Ruby on Rail Engineer", "fields":[ "user.headline^2" ] } } } ``` }, "highlight":{ ``` "fields":{ "fields":{ "user.headline":{ } } } ``` }, "size":10 } ' It’s broken. Hope this will help you De : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] De la part de Brandon Hilkert Envoyé : lundi 23 juillet 2012 16:30 À : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Objet : Re: Highlight stopped working for me [https://gist.github.com/3163918](https://gist.github.com/3163918) On Monday, July 23, 2012 9:56:20 AM UTC-4, David Pilato wrote: Can you gist a full curl recreation ? I will try to test it. David 😉 Twitter : @dadoonet / @elasticsearchfr Le 23 juil. 2012 à 15:46, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : I've confirmed that if I include both associations in my JSON representation when indexing the data, the highlighting breaks for whatever reason. So with this in my Ruby code: def to\_indexed\_json ``` to_json(include: [:educations, :jobs]) ``` end Highlighting doesn't work. However, without that and just sending the root data, highlighting works fine. Any thoughts on what might cause this? Thanks, Brandon On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote: Hey David, The only difference from when it previously worked is the change in data i'm sending with those objects. I make sure I delete the index entirely before every test. I was going to mess around with the mapping today. Any other ideas? Brandon On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote: Also, are you doing this on a clean index? Does it remain some old mapping? David -- Le 23 juil. 2012 à 05:07, David Pilato [david@pilato.fr](mailto:david@pilato.fr) a écrit : Did you try to prefix with your type name? ``` "user.educations.school": {}, ``` Not sure it will help... David -- Le 23 juil. 2012 à 04:59, Brandon Hilkert [brandon@meeteor.com](mailto:brandon@meeteor.com) a écrit : ``` "educations.school": {}, ``` ---
**Author:** ![Brandon\_Hilkert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/brandon_hilkert/32/2182_2.png) [@Brandon\_Hilkert](https://discuss.elastic.co/u/Brandon_Hilkert)\ **Post date:** [July 23, 2012, 8:44pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/10 "2012-07-23T20:44:54Z")
Ah....nice catch. I turned off printing the Root JSON element in my app and all is well. Thanks for tracking this down. Brandon On Sunday, July 22, 2012 9:41:46 PM UTC-4, Brandon Hilkert wrote: > We were previously getting perfectly good highlight results back, changed > some queries and now it's not working anymore. here is the query data. Am I > missing something? > > { > "query": { > "filtered": { > "query": { > "query\_string": { > "query": "a Ruby on Rail Engineer" > } > }, > "filter": { > "and": [ > { > "not": { > "term": { > "id": 3453 > } > } > } > ] > } > } > }, > "highlight": { > "fields": { > "\_all": {}, > "jobs.industry": {}, > "jobs.employer": {}, > "jobs.position": {}, > "educations.school": {}, > "educations.major": {}, > "skills": {}, > "linkedin\_summary": {}, > "linkedin\_groups": {}, > "linkedin\_certifications": {}, > "headline": {} > }, > "pre\_tags": [ > "\" > ], > "post\_tags": [ > "" > ] > }, > "size": 10 > } ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 23, 2012, 9:38pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/11 "2012-07-23T21:38:57Z")
Hi David, On Monday, July 23, 2012 10:20:25 PM UTC+2, David Pilato wrote: > Hummm. Strange. > > It looks like you cannot use the type name as a property: > > _Note to Shay_: do you think it’s an issue we should track? I stumbled upon this a while ago, it's commented in the source by Shay: DocumentMapper.java in package org.elasticsearch.index.mapper, around Line 473: if (type.equals(parser.currentName())) { // first field is the same as the type, this might be because the type is provided, and the object exists within it // or because there is a valid field that by chance is named as the type ``` // Note, in this case, we only handle plain value types, an ``` object type will be analyzed as if it was the type itself // and other same level fields will be ignored token = parser.nextToken(); countDownTokens++; It's a kind of a "safety bag" mechanism, for the case a user puts the index type into the JSON, then it is respected as the type name. As a consequence, you can't index a JSON object on root level with same name as the index type, only a plain JSON value. Best regards, Jörg ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 23, 2012, 9:45pm UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/12 "2012-07-23T21:45:26Z")
Thanks Jörg for the update. Cheers De : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] De la part de Jörg Prante Envoyé : lundi 23 juillet 2012 23:39 À : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Objet : Re: Highlight stopped working for me Hi David, On Monday, July 23, 2012 10:20:25 PM UTC+2, David Pilato wrote: Hummm. Strange. It looks like you cannot use the type name as a property: Note to Shay: do you think it’s an issue we should track? I stumbled upon this a while ago, it's commented in the source by Shay: DocumentMapper.java in package org.elasticsearch.index.mapper, around Line 473: ``` if (type.equals(parser.currentName())) { // first field is the same as the type, this might be because the type is provided, and the object exists within it // or because there is a valid field that by chance is named as the type // Note, in this case, we only handle plain value types, an object type will be analyzed as if it was the type itself // and other same level fields will be ignored token = parser.nextToken(); countDownTokens++; ``` It's a kind of a "safety bag" mechanism, for the case a user puts the index type into the JSON, then it is respected as the type name. As a consequence, you can't index a JSON object on root level with same name as the index type, only a plain JSON value. Best regards, Jörg ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/highlight-stopped-working-for-me/8484/13 "2017-07-06T03:19:22Z")