Highlight stopped working for me


(Brandon Hilkert) #1

We were previously getting perfectly good highlight results back, changed
some queries and now it's not working anymore. here is the query data. Am I
missing something?

{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "a Ruby on Rail Engineer"
}
},
"filter": {
"and": [
{
"not": {
"term": {
"id": 3453
}
}
}
]
}
}
},
"highlight": {
"fields": {
"_all": {},
"jobs.industry": {},
"jobs.employer": {},
"jobs.position": {},
"educations.school": {},
"educations.major": {},
"skills": {},
"linkedin_summary": {},
"linkedin_groups": {},
"linkedin_certifications": {},
"headline": {}
},
"pre_tags": [
"<span class="highlight">"
],
"post_tags": [
""
]
},
"size": 10
}


(Brandon Hilkert) #2

As a follow up, the only changes made were to how the data was indexed. I'm
using the Tire Ruby gem and am attempting to include some associations on
my model. Before, I wasn't properly include them so the objects :educations
and :jobs weren't actually being indexed, and this is when the highlighting
worked.

However, when i properly include the associations, the highlighting stopped
working. Here is my mapping:

{

  • state: open
  • settings: {
    • index.number_of_shards: 5
    • index.number_of_replicas: 1
    • index.version.created: 190399
      }
  • mappings: {
    • user: {
      • properties: {
        • skills: {
          • type: string
            }
        • headline: {
          • type: string
            }
        • jobs: {
          • properties: {
            • position: {
              • type: string
                }
            • employer: {
              • type: string
                }
            • state: {
              • type: string
                }
            • city: {
              • type: string
                }
            • country: {
              • type: string
                }
            • id: {
              • type: long
                }
            • end_date: {
              • format: dateOptionalTime
              • type: date
                }
            • updated_at: {
              • format: dateOptionalTime
              • type: date
                }
            • description: {
              • type: string
                }
            • created_at: {
              • format: dateOptionalTime
              • type: date
                }
            • user_id: {
              • type: long
                }
            • industry: {
              • type: string
                }
            • start_date: {
              • format: dateOptionalTime
              • type: date
                }
                }
                }
        • linkedin_connections: {
          • include_in_all: false
          • index: not_analyzed
          • type: string
            }
        • is_active: {
          • type: boolean
            }
        • fb_connections: {
          • include_in_all: false
          • index: not_analyzed
          • type: string
            }
        • fb_user_id: {
          • include_in_all: false
          • index: not_analyzed
          • type: string
            }
        • linkedin_certifications: {
          • type: string
            }
        • is_email_confirmed: {
          • type: boolean
            }
        • id: {
          • index: not_analyzed
          • type: string
            }
        • profile_pic_url: {
          • type: string
            }
        • first_name: {
          • type: string
            }
        • is_reachable_on_fb: {
          • type: boolean
            }
        • updated_at: {
          • format: dateOptionalTime
          • type: date
            }
        • linkedin_id: {
          • include_in_all: false
          • index: not_analyzed
          • type: string
            }
        • is_work_updated: {
          • type: boolean
            }
        • last_name: {
          • type: string
            }
        • linkedin_public_profile_url: {
          • type: string
            }
        • linkedin_groups: {
          • type: string
            }
        • created_at: {
          • format: dateOptionalTime
          • type: date
            }
        • slug: {
          • type: string
            }
        • educations: {
          • properties: {
            • school_type: {
              • type: string
                }
            • id: {
              • type: long
                }
            • updated_at: {
              • format: dateOptionalTime
              • type: date
                }
            • school: {
              • type: string
                }
            • grad_year: {
              • format: dateOptionalTime
              • type: date
                }
            • degree: {
              • type: string
                }
            • created_at: {
              • format: dateOptionalTime
              • type: date
                }
            • user_id: {
              • type: long
                }
            • major: {
              • type: string
                }
                }
                }
        • linkedin_summary: {
          • type: string
            }
        • login_hash: {
          • type: string
            }
            }
            }
            }
  • aliases: [ ]

}

Any reason highlighting wouldn't work with the above setup?

On Sunday, July 22, 2012 9:41:46 PM UTC-4, Brandon Hilkert wrote:

We were previously getting perfectly good highlight results back, changed
some queries and now it's not working anymore. here is the query data. Am I
missing something?

{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "a Ruby on Rail Engineer"
}
},
"filter": {
"and": [
{
"not": {
"term": {
"id": 3453
}
}
}
]
}
}
},
"highlight": {
"fields": {
"_all": {},
"jobs.industry": {},
"jobs.employer": {},
"jobs.position": {},
"educations.school": {},
"educations.major": {},
"skills": {},
"linkedin_summary": {},
"linkedin_groups": {},
"linkedin_certifications": {},
"headline": {}
},
"pre_tags": [
"<span class="highlight">"
],
"post_tags": [
""
]
},
"size": 10
}


(David Pilato) #3

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(David Pilato) #4

Also, are you doing this on a clean index?
Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(Brandon Hilkert) #5

Hey David,

The only difference from when it previously worked is the change in data
i'm sending with those objects. I make sure I delete the index entirely
before every test.

I was going to mess around with the mapping today. Any other ideas?

Brandon

On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote:

Also, are you doing this on a clean index?
Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(Brandon Hilkert) #6

I've confirmed that if I include both associations in my JSON
representation when indexing the data, the highlighting breaks for whatever
reason.

So with this in my Ruby code:

def to_indexed_json
to_json(include: [:educations, :jobs])
end

Highlighting doesn't work.

However, without that and just sending the root data, highlighting works
fine.

Any thoughts on what might cause this?

Thanks,
Brandon

On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote:

Hey David,

The only difference from when it previously worked is the change in data
i'm sending with those objects. I make sure I delete the index entirely
before every test.

I was going to mess around with the mapping today. Any other ideas?

Brandon

On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote:

Also, are you doing this on a clean index?
Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(David Pilato) #7

Can you gist a full curl recreation ?

I will try to test it.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 15:46, Brandon Hilkert brandon@meeteor.com a écrit :

I've confirmed that if I include both associations in my JSON representation when indexing the data, the highlighting breaks for whatever reason.

So with this in my Ruby code:

def to_indexed_json
to_json(include: [:educations, :jobs])
end

Highlighting doesn't work.

However, without that and just sending the root data, highlighting works fine.

Any thoughts on what might cause this?

Thanks,
Brandon

On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote:
Hey David,

The only difference from when it previously worked is the change in data i'm sending with those objects. I make sure I delete the index entirely before every test.

I was going to mess around with the mapping today. Any other ideas?

Brandon

On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote:
Also, are you doing this on a clean index?
Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(Brandon Hilkert) #8

https://gist.github.com/3163918

On Monday, July 23, 2012 9:56:20 AM UTC-4, David Pilato wrote:

Can you gist a full curl recreation ?

I will try to test it.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 15:46, Brandon Hilkert brandon@meeteor.com a écrit :

I've confirmed that if I include both associations in my JSON
representation when indexing the data, the highlighting breaks for whatever
reason.

So with this in my Ruby code:

def to_indexed_json
to_json(include: [:educations, :jobs])
end

Highlighting doesn't work.

However, without that and just sending the root data, highlighting works
fine.

Any thoughts on what might cause this?

Thanks,
Brandon

On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote:

Hey David,

The only difference from when it previously worked is the change in data
i'm sending with those objects. I make sure I delete the index entirely
before every test.

I was going to mess around with the mapping today. Any other ideas?

Brandon

On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote:

Also, are you doing this on a clean index?
Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit
:

 "educations.school": {},

(David Pilato) #9

Hummm. Strange.

It looks like you cannot use the type name as a property:

Note to Shay: do you think it’s an issue we should track?

If you send:

curl -XPOST 'http://localhost:9200/spokely_development_users/user/3457' -d '

{

"myuser":{

  "headline":"Ruby on Rails Engineer"

}

}

'

curl -XGET 'http://localhost:9200/spokely_development_users/user/_search?size=10&pretty=true' -d '

{

"query":{

  "filtered":{

     "query":{

        "query_string":{

           "query":"a Ruby on Rail Engineer",

           "fields":[

              "myuser.headline^2"

           ]

        }

     }

  }

},

"highlight":{

  "fields":{

     "fields":{

        "myuser.headline":{   }

     }

  }

},

"size":10

}

'

It will work.

But with:

curl -XPOST 'http://localhost:9200/spokely_development_users/user/3457' -d '

{

"user":{

  "headline":"Ruby on Rails Engineer"

}

}

'

curl -XGET 'http://localhost:9200/spokely_development_users/user/_search?size=10&pretty=true' -d '

{

"query":{

  "filtered":{

     "query":{

        "query_string":{

           "query":"a Ruby on Rail Engineer",

           "fields":[

              "user.headline^2"

           ]

        }

     }

  }

},

"highlight":{

  "fields":{

     "fields":{

        "user.headline":{   }

     }

  }

},

"size":10

}

'

It’s broken.

Hope this will help you

De : elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] De la part de Brandon Hilkert
Envoyé : lundi 23 juillet 2012 16:30
À : elasticsearch@googlegroups.com
Objet : Re: Highlight stopped working for me

https://gist.github.com/3163918

On Monday, July 23, 2012 9:56:20 AM UTC-4, David Pilato wrote:

Can you gist a full curl recreation ?

I will try to test it.

David :wink:

Twitter : @dadoonet / @elasticsearchfr

Le 23 juil. 2012 à 15:46, Brandon Hilkert brandon@meeteor.com a écrit :

I've confirmed that if I include both associations in my JSON representation when indexing the data, the highlighting breaks for whatever reason.

So with this in my Ruby code:

def to_indexed_json

to_json(include: [:educations, :jobs])

end

Highlighting doesn't work.

However, without that and just sending the root data, highlighting works fine.

Any thoughts on what might cause this?

Thanks,

Brandon

On Monday, July 23, 2012 9:23:05 AM UTC-4, Brandon Hilkert wrote:

Hey David,

The only difference from when it previously worked is the change in data i'm sending with those objects. I make sure I delete the index entirely before every test.

I was going to mess around with the mapping today. Any other ideas?

Brandon

On Sunday, July 22, 2012 11:11:39 PM UTC-4, David Pilato wrote:

Also, are you doing this on a clean index?

Does it remain some old mapping?

David

--

Le 23 juil. 2012 à 05:07, David Pilato david@pilato.fr a écrit :

Did you try to prefix with your type name?

 "user.educations.school": {},

Not sure it will help...

David

--

Le 23 juil. 2012 à 04:59, Brandon Hilkert brandon@meeteor.com a écrit :

 "educations.school": {},

(Brandon Hilkert) #10

Ah....nice catch. I turned off printing the Root JSON element in my app and
all is well.

Thanks for tracking this down.

Brandon

On Sunday, July 22, 2012 9:41:46 PM UTC-4, Brandon Hilkert wrote:

We were previously getting perfectly good highlight results back, changed
some queries and now it's not working anymore. here is the query data. Am I
missing something?

{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "a Ruby on Rail Engineer"
}
},
"filter": {
"and": [
{
"not": {
"term": {
"id": 3453
}
}
}
]
}
}
},
"highlight": {
"fields": {
"_all": {},
"jobs.industry": {},
"jobs.employer": {},
"jobs.position": {},
"educations.school": {},
"educations.major": {},
"skills": {},
"linkedin_summary": {},
"linkedin_groups": {},
"linkedin_certifications": {},
"headline": {}
},
"pre_tags": [
"<span class="highlight">"
],
"post_tags": [
""
]
},
"size": 10
}


(Jörg Prante) #11

Hi David,

On Monday, July 23, 2012 10:20:25 PM UTC+2, David Pilato wrote:

Hummm. Strange.

It looks like you cannot use the type name as a property:

Note to Shay: do you think it’s an issue we should track?

I stumbled upon this a while ago, it's commented in the source by Shay:

DocumentMapper.java in package org.elasticsearch.index.mapper, around Line
473:

if (type.equals(parser.currentName())) {
// first field is the same as the type, this might be
because the type is provided, and the object exists within it
// or because there is a valid field that by chance is
named as the type

            // Note, in this case, we only handle plain value types, an 

object type will be analyzed as if it was the type itself
// and other same level fields will be ignored
token = parser.nextToken();
countDownTokens++;

It's a kind of a "safety bag" mechanism, for the case a user puts the index
type into the JSON, then it is respected as the type name. As a
consequence, you can't index a JSON object on root level with same name as
the index type, only a plain JSON value.

Best regards,

Jörg


(David Pilato) #12

Thanks Jörg for the update.

Cheers

De : elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] De la part de Jörg Prante
Envoyé : lundi 23 juillet 2012 23:39
À : elasticsearch@googlegroups.com
Objet : Re: Highlight stopped working for me

Hi David,

On Monday, July 23, 2012 10:20:25 PM UTC+2, David Pilato wrote:

Hummm. Strange.

It looks like you cannot use the type name as a property:

Note to Shay: do you think it’s an issue we should track?

I stumbled upon this a while ago, it's commented in the source by Shay:

DocumentMapper.java in package org.elasticsearch.index.mapper, around Line 473:

 if (type.equals(parser.currentName())) {

            // first field is the same as the type, this might be because the type is provided, and the object exists within it

            // or because there is a valid field that by chance is named as the type



            // Note, in this case, we only handle plain value types, an object type will be analyzed as if it was the type itself

            // and other same level fields will be ignored

            token = parser.nextToken();

            countDownTokens++;

It's a kind of a "safety bag" mechanism, for the case a user puts the index type into the JSON, then it is respected as the type name. As a consequence, you can't index a JSON object on root level with same name as the index type, only a plain JSON value.

Best regards,

Jörg


(system) #13