Thanks Don for this.
A full curl recreation means that we can just download your script and replay it with a single command line.
Somehow, it's a shell script which looks like this: https://gist.github.com/dadoonet/2965777#file-test-sh
You need to delete old test index, create the tes index, add the mapping if any, put some documents, refresh the index and then run your queries.
Could you update your script so I can replay it?
Thanks for your help so far!
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
Le 18 février 2014 à 08:12:53, Don (donmisme@gmail.com) a écrit:
Hi David
As requested, I created a gist on gist.github.com . This is my first time to create somthing on gist. I am not sure if it is correct format. The following is the url :
Regards
On Mon, Feb 17, 2014 at 4:37 PM, David Pilato david@pilato.fr wrote:
It could be really interesting to have a full script on gist.github.com which reproduce your initial error.
I think we should not fail with an error such as: TokenStream contract violation: close() call missing.
So, if you can create that script, that would be awesome.
Thanks!
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
Le 17 février 2014 à 08:03:32, Zhihai Xian (donmisme@gmail.com) a écrit:
Hi David
I think I solve the problem by separating the full search query into match query (_search) and suggest query (_suggest),
The search result and suggest result both looks fine so far.
my original full search string is shown below
http://searchnode:9200/es_search/article,fullpagepdf,graphics,photographics
POST _search
{
"from" : 0,
"size" : 10,
"query" : {
"filtered" : {
"query" : {
"multi_match" : {
"query" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign work",
"fields" : [ "headline_en^4", "headline_cn^4", "byline_en", "byline_cn", "caption_en^2", "caption_cn^2", "body_en^3", "body_cn^3" ] ,
"analyzer": "phrase_analyzer"
}
},
"filter" : {
"and" : {
"filters" : [ {
"term" : {
"wires" : "1"
}
}, {
"term" : {
"language" : "en"
}
}, {
"or" : {
"filters" : [ {
"terms" : {
"hideinasiaone" : [ "no", "NO", "No" ]
}
}, {
"and" : {
"filters" : [ {
"term" : {
"language" : "zh"
}
}, {
"range" : {
"publicationdate" : {
"from" : null,
"to" : "2008-12-31",
"include_lower" : true,
"include_upper" : true
}
}
} ]
}
}, {
"terms" : {
"sourcetype" : [ "ad_metadata", "fullpagepdf", "photographics"]
}
} ]
}
} ]
}
}
}
},
"sort" : [ {
"_score" : {
"order" : "desc"
}
} ],
"facets" : {
"Company" : {
"terms" : {
"field" : "facet_company",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Section" : {
"terms" : {
"field" : "facet_pagename",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Publication" : {
"terms" : {
"field" : "publication",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
},
"Advertiser" : {
"terms" : {
"field" : "facet_advertiser",
"size" : 20,
"exclude" : [ "" ],
"order" : "count"
}
}
},
"highlight" : {
"pre_tags" : [ "<span class="highlighting_text">" ],
"post_tags" : [ "" ],
"fields" : {
"headline_en" : {
"fragment_size" : 200,
"number_of_fragments" : 1
},
"body_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
},
"pdfbody_en" : {
"fragment_size" : 350,
"number_of_fragments" : 1
}
}
},
"suggest" : {
"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 3,
"shard_size" : 3,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter" : "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
}
and now I moved the below phrase suggestion query out of the full query search and do a single phrase suggest search as below
http://searchnode:9200/es_search/
POST _suggest
{
"text" : "raise the capabilities of Singaporeans as well as to calibrate the existing work pass framework to ensure that foreign worker",
"search_suggest" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
}
But I am not sure if this is a correct approach for my issue.
Thanks
On Monday, February 17, 2014 1:18:21 PM UTC+8, David Pilato wrote:
Could you gist a full curl recreation to reproduce your issue?
--
David 
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 17 févr. 2014 à 02:21, Zhihai Xian donm...@gmail.com a écrit :
Hi David,
I did not use any plugins
The following is my actual suggestion search query :
"suggest" : {
"text" : "ationel stadiem",
"simple_phrase" : {
"phrase" : {
"analyzer" : "suggestion_analyzer",
"field" : "suggest_phrase_en",
"size" : 2,
"shard_size" : 2,
"real_word_error_likelihood" : 0.05,
"confidence" : 0.1,
"max_errors" : 0.9,
"gram_size" : 5,
"direct_generator" : [ {
"field" : "suggest_phrase_en",
"suggest_mode" : "always",
"size" : 3,
"max_edits" : 2,
"min_word_length" : 3
}, {
"field" : "reverse_phrase",
"suggest_mode" : "always",
"min_word_length" : 3,
"pre_filter": "reverse_analyzer",
"post_filter" : "reverse_analyzer"
} ]
}
}
Setting in Analyzer:
"suggestion_analyzer": {
"type" : "custom",
"tokenizer": "standard",
"filter":[
"standard",
"asciifolding",
"word_delimiter",
"text_shingle",
"lowercase",
"stop",
"en_GB"
]
},
"reverse_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"reverse"
]
},
Thanks
On Sunday, February 16, 2014 9:40:06 PM UTC+8, David Pilato wrote:
I have seen that in past but with analysis plugins.
Any chance you are using a plugin and did not update it ?
--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 16 févr. 2014 à 14:27, Zhihai Xian donm...@gmail.com a écrit :
Hi
After upgrading to elasticsearch 1.0.0, I got a suggestion search error:
RemoteTransportException[[data98][inet[/172.28.96.98:9300]][search/phase/query]]; nested: IllegalStateException[TokenStream contract violation: close() call missing]; }]
I found that the problem come with the suggestion query:
"suggest" : {
"text" : "Xor the Got-Jewel",
"simple_phrase" : {
"phrase" : {
"analyzer" : "body",
"field" : "bigram",
"size" : 4,
"real_word_error_likelihood" : 0.95,
"confidence" : 2.0,
"gram_size" : 2,
"direct_generator" : [ {
"field" : "body",
"suggest_mode" : "always",
"min_word_length" : 1
}, {
"field" : "reverse",
"suggest_mode" : "always",
"min_word_length" : 1,
"pre_filter" : "reverse",
"post_filter" : "reverse"
} ]
}
}
}
If I remove the last two lines for pre_filter and post_filter, the search will be fine, but the suggestion result will not be expected.
"pre_filter" : "reverse",
"post_filter" : "reverse"
Is there any solution on the above issue?
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/UmLSl9_cTU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5301ca4a.3006c83e.f2%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGGrwQFXEb_WJ7%3DvoQk7QqKvzjB4Lv5Q%3DqJ%2BbV-Zm2Q4E7R00Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53046b6a.109cf92e.2753%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.
