Postings highlighter throws exception on some queries

I'm using Elasticsearch 1.4.1 (lucene_version 4.10.2) and have a search
that works well with most search strings, but fails spectacularly on
others. Experimenting, it appears to fail when at least one word in the
query doesn't match in any documents (like the second word in the made up
search phrase below). It fails with the error:

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[w3zfoix_Qi-xwpVGbCbQWw][ia_test][0]:
ElasticsearchIllegalArgumentException[the field [content] should be indexed
with positions and offsets in the postings list to be used with postings
highlighter]}]",
"status": 400
}

The simplest search which gives this error is shown below:

POST /myindex/_search
{
"from" : 0,
"size" : 25,
"query": {
"filtered" : {
"query" : {
"multi_match" : {
"type" : "most_fields",
"fields": ["title", "content", "content.english"],
"query": "Box Fexye"
}
}
}
},
"highlight" : {
"fields" : {
"content" : {
"type" : "postings"
}
}
}
}

The index definition for these fields looks like:

{
"properties" : {
"title" : {
"type" : "string",
"fields": {
"shingles": {
"type": "string",
"analyzer": "my_shingle_analyzer"
}
}
},
"content" : {
"type" : "string",
"analyzer" : "standard",
"fields": {
"english": {
"type": "string",
"analyzer": "my_english"
},
"shingles": {
"type": "string",
"analyzer": "my_shingle_analyzer"
}
},
"index_options" : "offsets",
"term_vector" : "with_positions_offsets"
}
}
}

Is user error involved here, or is there something ES can't handle?

Thanks,
Peter

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30081940-1d4e-4116-af61-8953302beeb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This looks like a bug: clearly from your mappings, field "content" was
indexed with offsets, yet the error message (incorrectly) claims otherwise.

Does the bug still happen on the last 1.4.x release (1.4.2)?

If you search only on the content/content.english field does the error
still happen? (i.e., remove title from the search since it doesn't have
offsets)

Mike McCandless

http://blog.mikemccandless.com

On Tue, Jan 13, 2015 at 10:09 AM, Peter Bowyer peter@mapledesign.co.uk
wrote:

I'm using Elasticsearch 1.4.1 (lucene_version 4.10.2) and have a search
that works well with most search strings, but fails spectacularly on
others. Experimenting, it appears to fail when at least one word in the
query doesn't match in any documents (like the second word in the made up
search phrase below). It fails with the error:

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query_fetch], all shards failed; shardFailures
{[w3zfoix_Qi-xwpVGbCbQWw][ia_test][0]:
ElasticsearchIllegalArgumentException[the field [content] should be indexed
with positions and offsets in the postings list to be used with postings
highlighter]}]",
"status": 400
}

The simplest search which gives this error is shown below:

POST /myindex/_search
{
"from" : 0,
"size" : 25,
"query": {
"filtered" : {
"query" : {
"multi_match" : {
"type" : "most_fields",
"fields": ["title", "content", "content.english"],
"query": "Box Fexye"
}
}
}
},
"highlight" : {
"fields" : {
"content" : {
"type" : "postings"
}
}
}
}

The index definition for these fields looks like:

{
"properties" : {
"title" : {
"type" : "string",
"fields": {
"shingles": {
"type": "string",
"analyzer": "my_shingle_analyzer"
}
}
},
"content" : {
"type" : "string",
"analyzer" : "standard",
"fields": {
"english": {
"type": "string",
"analyzer": "my_english"
},
"shingles": {
"type": "string",
"analyzer": "my_shingle_analyzer"
}
},
"index_options" : "offsets",
"term_vector" : "with_positions_offsets"
}
}
}

Is user error involved here, or is there something ES can't handle?

Thanks,
Peter

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/30081940-1d4e-4116-af61-8953302beeb1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/30081940-1d4e-4116-af61-8953302beeb1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRetw_nU1_G71xXTwPwGN%3D4ZLEgAR-uodJOCPyd9u0hu4w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Michael,

Thanks for your response - it turned out to be user error. I'd set up the
mappings correctly, but a few records in my bulk import file were created
with the wrong 'type' - and so had no mapping for offsets. Once I corrected
this, it worked.

Peter

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJvSBVMQZd1ZJiwq33pFghpg9WZzHkBtdQXAbmnkdKMjM%2B1n9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Super, thanks for brining closure.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jan 14, 2015 at 9:59 AM, Peter Bowyer peter@mapledesign.co.uk
wrote:

Hi Michael,

Thanks for your response - it turned out to be user error. I'd set up the
mappings correctly, but a few records in my bulk import file were created
with the wrong 'type' - and so had no mapping for offsets. Once I corrected
this, it worked.

Peter

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJvSBVMQZd1ZJiwq33pFghpg9WZzHkBtdQXAbmnkdKMjM%2B1n9g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJvSBVMQZd1ZJiwq33pFghpg9WZzHkBtdQXAbmnkdKMjM%2B1n9g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRdBft81f_hN%3D8%2BcAS7w1DvqhqVu4v734Rk6T6dz9iacMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.