Highlight problem

locojay · December 22, 2010, 7:29pm

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

Lukas_Vlcek1 · December 22, 2010, 7:36pm

Hi, which version of ES are you using?
Regards,
Lukas
Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

locojay · December 22, 2010, 7:39pm

Hi i m using 0.13.1

On Dec 22, 2010, at 2:36 PM, Lukáš Vlček wrote:

Hi, which version of ES are you using?
Regards,
Lukas

Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

Lukas_Vlcek1 · December 22, 2010, 7:41pm

Can you try master? There has been fixed the following bug

github.com/elastic/elasticsearch

Highlighting bug where fragments are broken when number_of_fragments is 0

opened 12:42AM - 05 Nov 10 UTC

closed 06:53PM - 18 Nov 10 UTC

jessiehernandez

>bug v0.14.0

This is happening on version 0.12.1. When highlighting is specified, some of the… highlight fields come complete, and some are broken. When those documents are access directly by providing the direct index as "from" and setting "size" as 1, it comes fine: Reproduce code is below (bash script): ``` curl -XPUT 'http://server:9200/indexname/highbug/_mapping' -d ' { "highbug": { "properties": { "body": { "store": "yes", "term_vector": "with_positions_offsets", "type": "string" }, "title": { "store": "yes", "term_vector": "with_positions_offsets", "type": "string" } } } } '; for (( i = 0; i < 20; ++i )) do curl -XPUT "http://server:9200/indexname/highbug/$i" -d " { \"body\": \"Test body\", \"date\": \"$(date +%s)\", \"title\": \"This is a test on the highlighting bug present in elasticsearch\" }"; done ``` The call below returns the full highlighted fragment for the 1st and 5th results, and the 2nd through 4th results return "hting <em>bug</em> present in elasticsearch" instead of "This is a test on the highlighting <em>bug</em> present in elasticsearch". ``` curl -XGET 'http://server:9200/indexname/highbug/_search?pretty=true' -d ' {"query":{"query_string":{"query":"bug"}},"from":0,"size":5,"highlight":{"fields":{"title":{"number_of_fragments":0}}}} '; ```

Lukáš
Dne 22.12.2010 20:39 "Jassin Meknassi" jassin.meknassi@googlemail.com
napsal(a):

Hi i m using 0.13.1

On Dec 22, 2010, at 2:36 PM, Lukáš Vlček wrote:

Hi, which version of ES are you using?
Regards,
Lukas

Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com
napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

Jassin_MEKNASSI · December 22, 2010, 8:44pm

took longer to build the master. elasticsearch-0.14.0-SNAPSHOT

instead of having a smaller size most have no entry:

result['hits'][i]['highlight']['fulltext'] is an empty list

On Dec 22, 2010, at 2:41 PM, Lukáš Vlček wrote:

Can you try master? There has been fixed the following bug Highlighting bug where fragments are broken when number_of_fragments is 0 · Issue #479 · elastic/elasticsearch · GitHub

Lukáš

Dne 22.12.2010 20:39 "Jassin Meknassi" jassin.meknassi@googlemail.com napsal(a):

Hi i m using 0.13.1

On Dec 22, 2010, at 2:36 PM, Lukáš Vlček wrote:

Hi, which version of ES are you using?
Regards,
Lukas

Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

Lukas_Vlcek1 · December 22, 2010, 8:49pm

Can you please gist some curl recreation?

On Wed, Dec 22, 2010 at 9:44 PM, Jassin MEKNASSI
jassin.meknassi@gmail.comwrote:

took longer to build the master. elasticsearch-0.14.0-SNAPSHOT

instead of having a smaller size most have no entry:

result['hits'][i]['highlight']['fulltext'] is an empty list

On Dec 22, 2010, at 2:41 PM, Lukáš Vlček wrote:

Can you try master? There has been fixed the following bug
Highlighting bug where fragments are broken when number_of_fragments is 0 · Issue #479 · elastic/elasticsearch · GitHub

Lukáš
Dne 22.12.2010 20:39 "Jassin Meknassi" jassin.meknassi@googlemail.com
napsal(a):

Hi i m using 0.13.1

On Dec 22, 2010, at 2:36 PM, Lukáš Vlček wrote:

Hi, which version of ES are you using?
Regards,
Lukas

Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com
napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?

Jassin_MEKNASSI · December 23, 2010, 12:10am

unfortunately its confidential data.
reindexing on the failing docs using master works.
I will rebuild my index overnight and see if it works

Thanks for the help

On Dec 22, 2010, at 3:49 PM, Lukáš Vlček wrote:

Can you please gist some curl recreation?

On Wed, Dec 22, 2010 at 9:44 PM, Jassin MEKNASSI jassin.meknassi@gmail.com wrote:
took longer to build the master. elasticsearch-0.14.0-SNAPSHOT

instead of having a smaller size most have no entry:

result['hits'][i]['highlight']['fulltext'] is an empty list

On Dec 22, 2010, at 2:41 PM, Lukáš Vlček wrote:

Can you try master? There has been fixed the following bug Highlighting bug where fragments are broken when number_of_fragments is 0 · Issue #479 · elastic/elasticsearch · GitHub

Lukáš

Dne 22.12.2010 20:39 "Jassin Meknassi" jassin.meknassi@googlemail.com napsal(a):

Hi i m using 0.13.1

On Dec 22, 2010, at 2:36 PM, Lukáš Vlček wrote:

Hi, which version of ES are you using?
Regards,
Lukas

Dne 22.12.2010 20:29 "locojay" jassin.meknassi@googlemail.com napsal(a):

Hi

my mapping is

"mongodocid":{"boost":1.0,
"store": "yes",
"type": "string"},
"fulltext": {"boost": 1.0,
"index": "analyzed",
"store": "yes",
"type": "string",
"term_vector" : "with_positions_offsets"}}
to highlight the complete fulltext i am setting number_of_framgments
to 0.

if i do the following lucene like string query:

'{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test"}}, "size": 100}'

for some documents in the resulset the lengh of the highlighted
fulltext is smaller than ten fulltext itself. Since i am setting
number_of_fragments to 0 and pre_tags/post_tags are added this should
not happen.

Now comes the strange behaviour. if i only search for one of this
failing elements by doing this

{"highlight": {"pre_tags": "", "fields": {"fulltext":
{"number_of_fragments": 0}}, "post_tags": ""}, "query":
{"query_string": {"query": "fulltext:test AND mongodocid:
4d0a861c2ebef6032c00b1ec"}}, "size": 100}

then all works fine.

Any idea?