MoreLikeThis change between ES 0.16 and ES 0.17

ruflin_2 · August 5, 2011, 11:32am

Hi

@juneym just made a pull request for Elastica with the MoreLikeThis query:

Now it seems like the query is running under 0.16 but does not work under
0.17. The query looks the following:

```
       [more_like_this] => Array*
```
```
           (*
```
```
               [fields] => Array*
```
```
                   (*
```
```
                       [0] => email*
```
```
                       [1] => content*
```
```
                   )*
```
```
               [boost] => 1*
```

               [like_text] => fake gmail sample*

```
               [max_query_terms] => 1*
```

               [percent_terms_to_match] => 0.3*

```
               [min_term_freq] => 1*
```
```
               [min_doc_freq] => 1*
```
```
               [max_doc_freq] => *
```
```
               [min_word_len] => 0*
```
```
               [max_word_len] => 0*
```
```
               [boost_terms] => 1*
```
```
           )*
```
```
   )*
```

And elasticsearch throws the following error

[2011-08-05 12:26:05,427][DEBUG][action.search.type ] [Lorna Dane]
[test][0], node[XC2VHqNjQsmuGCUIfVOPVA], [P], s[STARTED]: Failed to execute
[org.elasticsearch.action.search.SearchRequest@58c3d9ac]
org.elasticsearch.search.query.QueryPhaseExecutionException: [test][0]:
query[like:fake gmail sample],from[0],size[10]: Query Failed [Failed to
execute main query]

at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:221)*
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:295)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:224)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:191)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)*
Caused by: java.io.IOException: Stream closed

Any idea what is wrong with this query? I tried to figure out what changed
between ES 0.16 and 0.17, but couldn't figure out what it is.

Vladimir_Shkurin · August 5, 2011, 3:09pm

I also meet with this. The problem in "fields" parameter.

Here's a sample query and it don't work:

gist.github.com

https://gist.github.com/kimchy/553660

gistfile1.txt

{
     "query" : {
         "more_like_this" : {
             "fields" : ["content", "title"],
             "like_text" : "jetblue blew tire",
             "min_term_freq" : 1
         }
     },
     "from" : 2000
}

But if you remove the "fields", the query works

Offtop *

Shay, I have another question about relevance. MLT function in
ElasticSearch working not as good as in Solr, the problem in fields
weight. I have set "boost" for the fields in the mapping, but they are
ignored. Could you make that request could look like for the
query_string { "fields" : ["title^7", "tags^3", "categories^2",
"description"] } ?

Many thanks in advance!

yark · August 5, 2011, 3:13pm

There is a typo in docs. fields must be a mlt_fields

5 серп. 2011, в 18:09, Vladimir Shkurin написал(а):

I also meet with this. The problem in "fields" parameter.

Here's a sample query and it don't work:

gist:553660 · GitHub

But if you remove the "fields", the query works

Offtop *

Shay, I have another question about relevance. MLT function in
Elasticsearch working not as good as in Solr, the problem in fields
weight. I have set "boost" for the fields in the mapping, but they are
ignored. Could you make that request could look like for the
query_string { "fields" : ["title^7", "tags^3", "categories^2",
"description"] } ?

Many thanks in advance!

Vladimir_Shkurin · August 5, 2011, 3:30pm

Yaroslav, "mlt_fields" or "mlt_fieldsafaf" is work, but it has no
effect Because the parameter "mlt_fields" for More Like This API,
not for More Like This Query

kimchy · August 5, 2011, 8:34pm

This looks like this bug in Lucene:
[LUCENE-3326] MoreLikeThis reuses a reader after it has already closed it - ASF JIRA. It will be fixed in 3.4,
I will see if I can workaround it for now.

On Fri, Aug 5, 2011 at 6:30 PM, Vladimir Shkurin vshkurin@gmail.com wrote:

Yaroslav, "mlt_fields" or "mlt_fieldsafaf" is work, but it has no
effect Because the parameter "mlt_fields" for More Like This API,
not for More Like This Query

ruflin_2 · August 6, 2011, 10:40am

It works when I rename fields to mlt_fields. But if I understand you right,
the fields can be completely skipped as they don't have any affect? This
means, they probably should be removed from the docu?
http://www.elasticsearch.org/guide/reference/query-dsl/mlt-query.html

Vladimir_Shkurin · August 6, 2011, 11:10am

No need to delete anything, this section of the documentation is
correct. We just have to wait when will the new release of Lucene, or
maybe Shay can fix it earlier

ruflin_2 · August 7, 2011, 9:51am

Ok, good to know.