Suggestion problem after upgrading to elasticsearch 1.0.0

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the search
will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donmisme@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]]; nested: IllegalStateException[TokenStream contract violation: close() call missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}
If I remove the last two lines for pre_filter and post_filter, the search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"
Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FEC3FEBF-B8E5-43A5-8654-393B200A3785%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

What is the definition of analyzer "body" ?

Jörg

On Sun, Feb 16, 2014 at 2:27 PM, Zhihai Xian donmisme@gmail.com wrote:

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the search
will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFr7u-OAaGX7tQ7FGsVJq-iTrOUOQC7NOqW6WqfvS28NA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

I did not use any plugins

Thanks

On Sunday, February 16, 2014 11:52:01 PM UTC+8, Jörg Prante wrote:

What is the definition of analyzer "body" ?

Jörg

On Sun, Feb 16, 2014 at 2:27 PM, Zhihai Xian <donm...@gmail.com<javascript:>

wrote:

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {

"text" : "Xor the Got-Jewel",

"simple_phrase" : {

  "phrase" : {

    "analyzer" : "body",

    "field" : "bigram",

    "size" : 4,

    "real_word_error_likelihood" : 0.95,

    "confidence" : 2.0,

    "gram_size" : 2,

    "direct_generator" : [ {

      "field" : "body",

      "suggest_mode" : "always",

      "min_word_length" : 1

    }, {
      "field" : "reverse",

      "suggest_mode" : "always",

      "min_word_length" : 1,

      "pre_filter" : "reverse",

      "post_filter" : "reverse"

    } ]
  }

}

}

If I remove the last two lines for pre_filter and post_filter, the search
will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",

"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7918233-dbaf-4492-b66d-9124443c06ae%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:

I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian <donm...@gmail.com <javascript:>> a
écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the search
will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian donmisme@gmail.com a écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:
I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]]; nested: IllegalStateException[TokenStream contract violation: close() call missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}
If I remove the last two lines for pre_filter and post_filter, the search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"
Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/608ABF3F-6978-44E3-AC78-769D14DC7536%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David

I think I solve the problem by separating the full search query into match
query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.

my original full search string is shown below

http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en",
"byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf",
"photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}

and now I moved the below phrase suggestion query out of the full query
search and do a single phrase suggest search as below

http://searchnode:9200/es_search/
POST _suggest
{

"text" : "raise the capabilities of Singaporeans as well as to 

calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}

But I am not sure if this is a correct approach for my issue.

Thanks

On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:

Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian <donm...@gmail.com <javascript:>> a
écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:

I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the search
will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5d95a4b-e59a-48e1-b3e2-08028d6da802%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

It could be really interesting to have a full script on gist.github.com which reproduce your initial error.
I think we should not fail with an error such as: TokenStream contract violation: close() call missing.

So, if you can create that script, that would be awesome.

Thanks!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 février 2014 à 08:03:32, Zhihai Xian (donmisme@gmail.com) a écrit:

Hi David

I think I solve the problem by separating the full search query into match query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.

my original full search string is shown below

http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en", "byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf", "photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}

and now I moved the below phrase suggestion query out of the full query search and do a single phrase suggest search as below

http://searchnode:9200/es_search/
POST _suggest
{

"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {      
  "phrase" : {
    "analyzer" : "suggestion_analyzer",
    "field" : "suggest_phrase_en",
    "size" : 2,
    "shard_size" : 2,
    "real_word_error_likelihood" : 0.05,
    "confidence" : 0.1,
    "max_errors" : 0.9,
    "gram_size" : 5,
    "direct_generator" : [ {
      "field" : "suggest_phrase_en",
      "suggest_mode" : "always",
      "size" : 3,
      "max_edits" : 2,
      "min_word_length" : 3
    }, {
      "field" : "reverse_phrase",
      "suggest_mode" : "always",
      "min_word_length" : 3,
      "pre_filter": "reverse_analyzer",
      "post_filter" : "reverse_analyzer"
    } ]
  }
}  

}

But I am not sure if this is a correct approach for my issue.

Thanks

On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:
Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian donm...@gmail.com a écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:
I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]]; nested: IllegalStateException[TokenStream contract violation: close() call missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}
If I remove the last two lines for pre_filter and post_filter, the search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"
Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5d95a4b-e59a-48e1-b3e2-08028d6da802%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5301ca4a.3006c83e.f2%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David

As requested, I created a gist on gist.github.com* . This is my first time
to create somthing on gist. I am not sure if it is correct format. The
following is the url :*

https://gist.github.com/donmis/9065851 http://gist.github.com/

Regards

On Mon, Feb 17, 2014 at 4:37 PM, David Pilato david@pilato.fr wrote:

It could be really interesting to have a full script on gist.github.com which
reproduce your initial error.
I think we should not fail with an error such as: TokenStream contract violation: close() call missing.

So, if you can create that script, that would be awesome.

Thanks!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 février 2014 à 08:03:32, Zhihai Xian (donmisme@gmail.com) a écrit:

Hi David

I think I solve the problem by separating the full search query into match
query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.

my original full search string is shown below

http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en",
"byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf",
"photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}

and now I moved the below phrase suggestion query out of the full query
search and do a single phrase suggest search as below

http://searchnode:9200/es_search/
POST _suggest
{

"text" : "raise the capabilities of Singaporeans as well as to

calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}

But I am not sure if this is a correct approach for my issue.

Thanks

On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:

Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian donm...@gmail.com a écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:

I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close()
call missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the
search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f5d95a4b-e59a-48e1-b3e2-08028d6da802%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/UmLSl9_cTU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.5301ca4a.3006c83e.f2%40MacBook-Air-de-David.local
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGGrwQFXEb_WJ7%3DvoQk7QqKvzjB4Lv5Q%3DqJ%2BbV-Zm2Q4E7R00Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks Don for this.

A full curl recreation means that we can just download your script and replay it with a single command line.

Somehow, it's a shell script which looks like this: https://gist.github.com/dadoonet/2965777#file-test-sh

You need to delete old test index, create the tes index, add the mapping if any, put some documents, refresh the index and then run your queries.

Could you update your script so I can replay it?

Thanks for your help so far!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 18 février 2014 à 08:12:53, Don (donmisme@gmail.com) a écrit:

Hi David

As requested, I created a gist on gist.github.com . This is my first time to create somthing on gist. I am not sure if it is correct format. The following is the url :

Regards

On Mon, Feb 17, 2014 at 4:37 PM, David Pilato david@pilato.fr wrote:
It could be really interesting to have a full script on gist.github.com which reproduce your initial error.
I think we should not fail with an error such as: TokenStream contract violation: close() call missing.

So, if you can create that script, that would be awesome.

Thanks!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 février 2014 à 08:03:32, Zhihai Xian (donmisme@gmail.com) a écrit:

Hi David

I think I solve the problem by separating the full search query into match query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.

my original full search string is shown below

http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en", "byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf", "photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}

and now I moved the below phrase suggestion query out of the full query search and do a single phrase suggest search as below

http://searchnode:9200/es_search/
POST _suggest
{

"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {      
  "phrase" : {
    "analyzer" : "suggestion_analyzer",
    "field" : "suggest_phrase_en",
    "size" : 2,
    "shard_size" : 2,
    "real_word_error_likelihood" : 0.05,
    "confidence" : 0.1,
    "max_errors" : 0.9,
    "gram_size" : 5,
    "direct_generator" : [ {
      "field" : "suggest_phrase_en",
      "suggest_mode" : "always",
      "size" : 3,
      "max_edits" : 2,
      "min_word_length" : 3
    }, {
      "field" : "reverse_phrase",
      "suggest_mode" : "always",
      "min_word_length" : 3,
      "pre_filter": "reverse_analyzer",
      "post_filter" : "reverse_analyzer"
    } ]
  }
}  

}

But I am not sure if this is a correct approach for my issue.

Thanks

On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:
Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian donm...@gmail.com a écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:
I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]]; nested: IllegalStateException[TokenStream contract violation: close() call missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"
Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5d95a4b-e59a-48e1-b3e2-08028d6da802%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/UmLSl9_cTU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5301ca4a.3006c83e.f2%40MacBook-Air-de-David.local.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGGrwQFXEb_WJ7%3DvoQk7QqKvzjB4Lv5Q%3DqJ%2BbV-Zm2Q4E7R00Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53046b6a.109cf92e.2753%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David

I am quite busy at work, I will try to reform a full script at weekend,

Regards

On Wed, Feb 19, 2014 at 4:29 PM, David Pilato david@pilato.fr wrote:

Thanks Don for this.

A full curl recreation means that we can just download your script and
replay it with a single command line.

Somehow, it's a shell script which looks like this:
Test case for multifield fields · GitHub

You need to delete old test index, create the tes index, add the mapping
if any, put some documents, refresh the index and then run your queries.

Could you update your script so I can replay it?

Thanks for your help so far!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 18 février 2014 à 08:12:53, Don (donmisme@gmail.com) a écrit:

Hi David

As requested, I created a gist on gist.github.com* . This is my first
time to create somthing on gist. I am not sure if it is correct format. The
following is the url :*

https://gist.github.com/donmis/9065851 http://gist.github.com/

Regards

On Mon, Feb 17, 2014 at 4:37 PM, David Pilato david@pilato.fr wrote:

It could be really interesting to have a full script on gist.github.com which
reproduce your initial error.
I think we should not fail with an error such as: TokenStream contract violation: close() call missing.

So, if you can create that script, that would be awesome.

Thanks!

 --

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 février 2014 à 08:03:32, Zhihai Xian (donmisme@gmail.com) a écrit:

Hi David

I think I solve the problem by separating the full search query into
match query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.

my original full search string is shown below

http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en",
"byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf",
"photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to
calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}

and now I moved the below phrase suggestion query out of the full query
search and do a single phrase suggest search as below

http://searchnode:9200/es_search/
POST _suggest
{

"text" : "raise the capabilities of Singaporeans as well as to

calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}

But I am not sure if this is a correct approach for my issue.

Thanks

On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:

Could you gist a full curl recreation to reproduce your issue?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 févr. 2014 à 02:21, Zhihai Xian donm...@gmail.com a écrit :

Hi David,

I did not use any plugins

The following is my actual suggestion search query :

"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}

Setting in Analyzer:

"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"

    ]
  },

"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},

Thanks

On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:

I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :

Hi

After upgrading to elasticsearch 1.0.0, I got a suggestion search error:

RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]];
nested: IllegalStateException[TokenStream contract violation: close() call
missing]; }]

I found that the problem come with the suggestion query:

"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}

If I remove the last two lines for pre_filter and post_filter, the
search will be fine, but the suggestion result will not be expected.

"pre_filter" : "reverse",
"post_filter" : "reverse"

Is there any solution on the above issue?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a6ab296e-f49e-4fcc-a7bc-a1ca32f0988e%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52eec1ed-cf39-42b6-a308-084770d4823d%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f5d95a4b-e59a-48e1-b3e2-08028d6da802%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/UmLSl9_cTU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.5301ca4a.3006c83e.f2%40MacBook-Air-de-David.local.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGGrwQFXEb_WJ7%3DvoQk7QqKvzjB4Lv5Q%3DqJ%2BbV-Zm2Q4E7R00Q%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/UmLSl9_cTU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.53046b6a.109cf92e.2753%40MacBook-Air-de-David.local
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGGrwQF6eOncTEu%3Dn2UAi1r5-yzksCFdpbdWkkTdmwMi2JSXCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.